Docker Hardened Images Migration: A Healthy Database That Kept Nothing

The image is documented, the compose file is not#

Docker’s guide to the hardened PostgreSQL image says it plainly: the data directory is versioned by the major release, and a new install should mount its volume at the parent directory so the image can create the versioned one itself. The official image, through version 17, keeps its data in a fixed directory that every compose file written for it mounts. Neither image is wrong. The trap is in the file that sits between them.

On the official path, the hardened image finds no cluster where it looks for one, creates a new cluster in its versioned directory inside the container’s own writable layer, and starts. The template’s health check, the PostgreSQL readiness probe, answers; connections work, writes succeed. The named volume, mounted one directory away, receives nothing.

The template I moved runs the database, like every service but Keycloak, with a read-only root filesystem, and that is the only reason the problem showed up at once: the hardened PostgreSQL tried to create its cluster on a read-only filesystem and stopped with an error. A stack without that setting gets no error at all. The repository’s trap test shows the quiet version. It starts the hardened image on both paths without a read-only root, writes a row, counts the files on the volume, recreates the container and reads the row again:

1
# Verify, run 40742373013, the trap test's output, condensed
2
=== the hardened PostgreSQL on two mount paths ===
3
image: dhi.io/postgres:16@sha256:e3b4059f05cecc8c4b452ed9c20cd189b6ae9b64f6181aacf9940eb95a11049f
4
  PASS: on the official path the volume stays empty and the row is gone after a recreate, while the database looked healthy throughout
5
  PASS: on the hardened path the data is on the volume and outlives the recreate
6
passed: 2   failed: 0

Until the recreate, the two databases answered the same queries the same way. Only one of them had anything on its volume.

The official image shows what a guard for this looks like. It moved to a versioned data directory in version 18, and its entrypoint refuses a first start, with an error that names the old path, when it finds a volume mounted there, even an empty one. The hardened image for version 16 has the versioned layout without that check.

The fix is one mount. The override that goes next to the template’s own compose file, which stays untouched, replaces the database volume:

1
# stacks/keycloak/dhi.override.yml in docker-hardened-images-migration at
2
# v1.0.1, the database, shortened
3
# An existing install moves its database volume first, once, with
4
# scripts/move-postgres-volume.sh. `!override` needs Docker Compose 2.24.4 or
5
# newer.
6
services:
7
  postgres:
8
    image: dhi.io/postgres:16@sha256:e3b4059f05cecc8c4b452ed9c20cd189b6ae9b64f6181aacf9940eb95a11049f
9
    # DHI keeps the cluster in /var/lib/postgresql/16/data, not in the
10
    # official image's /var/lib/postgresql/data, and runs as uid 70, not 999.
11
    # Left on the template's mount, it initialises a new cluster beside the
12
    # volume: here that fails on the read-only root, and in a stack without
13
    # read_only the data lives in the container and is gone at the next
14
    # recreate (tests/e2e-trap.sh shows both). Docker's guide suggests the
15
    # parent, /var/lib/postgresql, for a new install; the versioned path keeps
16
    # an install moved from the official image working, its files being at
17
    # the root of the volume.
18
    volumes: !override
19
      - keycloak-postgres:/var/lib/postgresql/16/data

It mounts the versioned directory rather than the parent the guide recommends, and the comment says why: an install moved from the official image has its cluster at the root of the volume, and the versioned mount reads it where it is. A new install with no history can follow the guide.

Four more things a straight swap breaks#

With the database fixed, the template on hardened images still did not come up. Each of the other four changes in the override exists because something broke without it, and each has a sabotage test in CI that takes that one change out and requires that exact failure.

Traefik never turned healthy, so nothing that waits for it started. Its health check called wget, and the image has no wget. Anyone who has moved to minimal images knows this one. The answer here is Traefik’s own health-check command against its ping entry point, a check that needs no tool from outside the binary.

Keycloak went into a restart loop with an access-denied exception. The template pins the official image’s user ID, 1000, while the hardened image’s files belong to user 65532, and on its first start Keycloak rebuilds itself into those files. Run as 1000, it cannot. The change is to run it as 65532.

Keycloak’s health check then failed with grep not found. This image has bash and little else: no grep, no cat, no tee. The check now opens a connection to the management port with a bash redirection, sends the request by hand and matches the status line in a case statement, builtins only:

1
# stacks/keycloak/dhi.override.yml in docker-hardened-images-migration at
2
# v1.0.1, Keycloak and Traefik, shortened
3
  keycloak:
4
    image: dhi.io/keycloak:26@sha256:85c0863e9c0534c15e096e01db61c516112acd2ba626b427f11d1afc8ed002fe
5
    # The template pins the official image's uid 1000; this image's files
6
    # belong to 65532, and the first start rebuilds Keycloak into them.
7
    user: "65532:65532"
8
    healthcheck:
9
      # The image has bash but no grep, cat or tee: builtins only.
10
      test:
11
        - "CMD-SHELL"
12
        - |
13
          exec 3<>/dev/tcp/localhost/9000 &&
14
          printf 'GET /health/ready HTTP/1.1\r\nHost: localhost\r\nConnection: close\r\n\r\n' >&3 &&
15
          read -r status <&3 &&
16
          case "$$status" in *" 200 "*) exit 0 ;; *) exit 1 ;; esac
17
  traefik:
18
    image: dhi.io/traefik:3@sha256:6bda6cb02db098057796a26856b1afb45b875dc1e9ac671ff080b0bf21b3fa3a
19
    healthcheck:
20
      # No wget in the image: Traefik checks its own ping entry point.
21
      test: ["CMD", "/usr/bin/traefik", "healthcheck", "--ping", "--ping.entrypoint=ping", "--entrypoints.ping.address=:8082"]

The backup sidecar went into a restart loop of its own. Its loop is a shell script, and the runtime PostgreSQL image has no shell, so the sidecar needs the image’s dev variant, which has one. That variant’s entrypoint always drops root before it runs anything, and dropping root needs the set-user-ID capability the template’s hardening took away. The official image’s entrypoint ran the loop as it was, so the override empties this one to do the same.

Each change was then taken out on its own, in its own CI job, and each job required its own failure:

1
# Verify, run 40742373013, the five sabotage jobs, the result line of each
2
  PASS: postgres fails as the comment says: Read-only file system
3
  PASS: traefik fails as the comment says: "wget": executable file not found
4
  PASS: keycloak fails as the comment says: AccessDeniedException
5
  PASS: keycloak fails as the comment says: grep: command not found
6
  PASS: backups fails as the comment says: failed switching to 'postgres'

A line in a migration override that no test has seen fail is a guess about why it is there. These five are not guesses.

Moving an install that already holds data#

A new install is the easy half. An install that has run on the official images has a database volume full of files owned by user 999, and the hardened PostgreSQL runs as user 70. Started on that volume with the override in place, it stops with permission denied and starts nothing. That is the right failure: loud, and before a single write.

The move is three steps. Stop the stack; its volumes stay. Hand the database volume to user 70, once, with the repository’s script, which refuses while anything uses the volume, checks that the volume holds a PostgreSQL data directory, and changes only the owner of its files. Start with the override.

Docker’s guide offers two other routes. A logical dump and restore costs a copy of the data. The other copies the old files into the versioned directory of a new volume, and as the guide words it today, it has no step that creates that directory and none that hands the files to the new user. The repository’s guide-copy test runs that route three ways:

1
# Verify, run 40750042068, the guide-copy test's output, condensed
2
=== the guide's direct data copy ===
3
  PASS: as written: the copy fails, and the next step starts the hardened image on an empty cluster
4
  PASS: with the directory made: the copy keeps uid 999, and the hardened image cannot read it (Permission denied)
5
  PASS: made and handed to uid 70: the hardened image starts on the copied data
6
passed: 3   failed: 0

As written, the copy fails and the guide’s next command starts the hardened image on an empty cluster, the same quiet loss as the trap. I opened a pull request against Docker’s guide that adds both steps on 28 October; on 12 November it is still open, with no review yet. Changing the owner in place costs one command and a stopped stack, which is why the override mounts the versioned path.

The upgrade test runs the whole sequence on a fresh runner: the template on its official images, a realm created in Keycloak, the refusal without the move, the move, and the same realm on the hardened images. All six checks pass:

1
# Verify, run 40742373013, the upgrade test's output, condensed
2
=== an official install moves to Docker Hardened Images ===
3
  PASS: the template comes up on its official images
4
  PASS: a realm is created on the official images
5
  PASS: without the move, the hardened PostgreSQL refuses the old volume (Permission denied)
6
  PASS: the volume is handed to uid 70
7
  PASS: the same install is up and healthy on DHI
8
  PASS: the realm made on the official images is there on DHI
9
passed: 6   failed: 0

Two scanners, two answers#

The reason to move is in the numbers, and the numbers depend on who counts. The repository measures every image with two scanners, on the pinned digests, every Monday. From the scan of 9 November, on AMD64, counted once per CVE and package:

By Docker Scout, which applies the vulnerability statements Docker publishes with these images, the hardened PostgreSQL goes from 4 critical and 35 high findings to no critical and 2 high, and Keycloak from 1 critical and 14 high to none and 1. By Trivy, which applies no vendor statements, PostgreSQL goes from 2 critical and 93 high to 1 and 16, and Keycloak from 7 high to 4. Traefik has no critical or high finding left by Trivy; by Scout it still has 1 critical and 4 high, against 2 and 5 on the official image.

The counts also move while the images stay put. Every image here is pinned by digest, both scanners are pinned too, and the scans of 28 October and 9 November read the same images. In those twelve days the official PostgreSQL went from 86 to 93 high findings by Trivy while the hardened one stayed at 16. The official Keycloak went from 3 to 7, so the one place where Trivy had counted the hardened image worse, 4 against 3 on 28 October, is now a place where it counts it better. And by Scout, Traefik went from no critical findings to 2 on the official image and 1 on the hardened one; on 28 October several of its findings still had no severity assigned. A pinned digest freezes the image, not what is known about it. The fix for a finding in a pinned image is a newer digest, and in this repository those pins move by hand, each move proven by the tests before it lands.

The gap between the two hardened columns is those statements: a “not affected” for each CVE, with a reason. Whether a gate trusts them is a policy decision, and it belongs before the move. A pipeline that gates on Trivy will not see the near-zero a pipeline gating on Scout sees, and the first week on hardened images is the wrong time to learn which one the release process reads. When I hardened one of my published images earlier this year, the scanner was the measure of progress. Here it is a choice that has to be written down.

The dev variant the backup sidecar needs costs more than the runtime image, 44 high by Trivy against 16, because it carries a shell and a package manager. Only the sidecar runs it, and the override keeps it that way.

What the move costs#

Three costs came with the move, and none of them shows up in a scan.

• A login for every pull. Docker’s hardened registry refuses anonymous pulls, the free images included: its token endpoint answers 401 without credentials, which I checked again on 12 November. Every host and CI runner that pulls needs a Docker account, and an air-gapped install mirrors the images with one. So does every bot. The repository’s first Dependabot pull request, on 11 November, failed at the registry login in every job that pulls a hardened image: a Dependabot run reads its own set of secrets, and the registry credentials were not among them. Adding the same two credentials as Dependabot secrets fixed it: on 12 November the pull request passed all 11 jobs and merged on its own.
• Health checks rewritten. Anything that ran a shell tool inside a container, a wget, a grep, a shell loop, is gone or has to move to a dev variant. The pain is well known for minimal images, and it applies to every service here.
• A user ID change per service, with a one-time ownership change for every volume that already holds data.

What I check before a stack moves to hardened images#

Every time a stack moves to images it did not grow up with, I want these settled before the first hardened digest is pinned:

• A write, a recreate and a read for every stateful service on the new image, with a count of the files on its volume. A healthy database proves nothing about where it writes.
• A read-only root filesystem on every service that can run with one. In this stack it turned a silent data loss into an error at the first start.
• The user each image runs as, checked against every pinned user in the compose file and the owner of every volume that already holds data.
• Every health check’s command, run inside the new image before the move. If the tool is not in the image, the check is already broken.
• The scanner and the vendor statements the release gate trusts, decided and written down before the first hardened image ships.
• Pull credentials on every host, every runner, every mirror and every bot that opens pull requests.
• Who moves the pinned digests, and how often. A pin nobody moves keeps last month’s image under this month’s findings.

The hardened image took away what an attacker could use. It also took away what the compose file had quietly relied on, and only the first of those shows up in a scan.

Sources#

• heyvaldemar, docker-hardened-images-migration (the override, the volume move, the tests and the CVE counts), 28 October 2026.
• GitHub Actions, the run with every test and both scanners, 28 October 2026.
• GitHub Actions, the weekly scan with the counts in the table, 9 November 2026, on the same pinned digests.
• heyvaldemar, the trap test, at the first patch release.
• heyvaldemar, the Keycloak template the migration starts from.
• heyvaldemar, the pull request that adds the missing steps to Docker’s PostgreSQL guide, opened 28 October 2026.
• GitHub Actions, the run with the guide-copy test, 28 October 2026.
• Docker, the guide to the hardened PostgreSQL image (the versioned data directory, the parent mount, the two migration routes), checked 12 November 2026.
• docker-library, the change that gave the official PostgreSQL image a versioned data directory from version 18, merged 6 June 2025, and the entrypoint check for a volume on the old path, read 12 November 2026.
• GitHub Actions, the first Dependabot run, 11 November 2026: the registry login fails in every job that pulls a hardened image. The same run, run again on 12 November with the credentials as Dependabot secrets: all 11 jobs pass.
• Docker, Docker makes hardened images free, open and transparent for everyone, 17 December 2025.
• Docker, Docker Hardened Images documentation, accessed October 2026.
• Docker, the Compose merge reference (the override tag), accessed October 2026.

Discussion#

If you have moved a stack to hardened or distroless images and found something the image swap alone broke, or you settled the scanner question another way, drop a comment below. Counterarguments are welcome and the comment thread is where I respond first. For longer back-and-forth with senior practitioners, join the discussion on Discord.