Docker Health Checks That Cannot Fail — Seventeen Hours Green

What a Docker health check that cannot fail looks like#

The check was ordinary. It asked whether the game server process was running.

1
# Dockerfile: the check that was green over a core dump
2
HEALTHCHECK --interval=30s --timeout=5s --retries=3 \
3
  CMD pgrep -f "ShooterGameServer" || exit 1

Run that against a pattern that cannot possibly exist, and watch it pass:

1
$ docker run --rm alpine:3.21 sh -c 'pgrep -f missing; echo "exit: $?"'
2
1
3
exit: 0

The PID it prints is the shell running the check itself. Exit zero, for a process that has never existed on any machine. Every container using that pattern had a health check with no red state at all. The fix is one character class. It breaks the self-match and still matches the real process:

1
# Dockerfile: the pattern no longer matches the shell that runs it
2
HEALTHCHECK --interval=30s --timeout=5s --retries=3 \
3
  CMD pgrep -f "[S]hooterGameServer" || exit 1

There is a second trap behind the first, and it bites the other way. The earliest version of that check matched the supervising script, not the binary. The supervisor starts immediately. So the container reported healthy while SteamCMD was still pulling twenty-five gigabytes and no game server existed yet. A check that passes before the thing it checks exists is worse than no check. It is believed.

Why this keeps happening#

Monitoring is written to detect failure, and it is tested by watching it not fire. That is the whole defect in one sentence. A check that has only ever been silent has not been shown to work; it has been shown to be quiet, and those are different facts that look identical on a dashboard.

Docker does not close the loop either. An unhealthy container keeps running, because the engine has no restart-on-unhealthy behaviour outside Swarm. Watch it happen:

1
$ docker run -d --name probe --health-cmd='exit 1' \
2
> --health-interval=2s --health-retries=1 alpine:3.21 sleep 60
3
$ docker inspect probe \
4
> --format '{{.State.Health.Status}} restarts={{.RestartCount}}'
5
unhealthy restarts=0
6
$ docker ps --filter name=probe --format '{{.Status}}'
7
Up 6 seconds (unhealthy)

A health check is a label, not a control. If nothing reads the label and acts, a correct red state changes nothing, and an incorrect green state costs you the same as having no check at all while feeling like diligence.

A silenced probe fails the same way, one layer down. A host reported its backups current every night for weeks. The probe used find -newermt to ask what had changed, the date expression failed to parse, and its stderr went to /dev/null. The command printed nothing, and nothing was read as “no changes found” rather than “the question was never asked”. Redirecting stderr had converted a broken probe into a clean answer. Not a wrong one. A clean one.

If a command can fail, the failure has to be a different value from “found nothing”.

The same shape has a nastier variant: a guard that its own failure switches off. A chat bridge dropped bot lines under one rule, the roster was read and nobody human was on the server. A failed roster read blanked the cache and cleared the flag the rule depended on, so one unlucky read switched the guard off rather than weakening it. The branch doing its job looked exactly like a quiet server, and the only evidence it had ever stopped working was a bot line arriving in a private channel. A retry does not fix this. What fixes it is deciding what “unread” means: the last good reading is evidence while it is fresh, and the flag that says “we know who is here” gets set on purpose, never inherited from whether the latest attempt happened to work.

The pattern generalises past containers. A freshness job across the fleet compared a pinned version against the upstream release with a plain not-equal test, and went red on repositories that were already current. GitHub’s own documentation is explicit about why: its latest release is the most recent non-prerelease, non-draft release, ordered by creation date. Created, not highest. A backport published after a newer line makes every up-to-date repository look behind.

Equality answers “is it different”. The question was always “is it behind”.

Two rules shipped in the same commit failed in opposite directions, which is the clearest picture of the problem I have. One compared a vendored config against its upstream copy and reported every single file as changed. It captured file contents through command substitution, which strips trailing newlines, so every file differed from a byte-identical copy of itself. Loud, constantly, about nothing. The other read a pin of v3.1.0 and a README saying 3.1.0 as two different versions, and therefore skipped every v-prefixed pin in the fleet. Silent, for months, about everything.

The noisy one got fixed in a day, because someone had to look at it. The silent one survived because silence is indistinguishable from success.

That distinction is about to matter more, because the volume of checks is about to stop coming from humans. A generator asked for a Dockerfile does not test whether its health check can go red. It reproduces the shape of the checks it has seen, and if that shape is the self-matching one, it ships at whatever volume you asked for. A person copies a broken pattern into ten images over a year. A generator copies it into ten thousand over a weekend, and every one of them is green on the first run. The failure mode of generated infrastructure is not code that breaks. It is code that cannot.

Risk and blast radius#

Direct exposure is the incident you do not get told about. Seventeen hours is a full working day and a night, during which an operator would have sworn the service was up. The tool they trust said so.

Systemic exposure is worse, because these defects arrive by copy-paste and then sit still. A health check pattern that works is pasted into the next image, and the fact that it cannot fail travels with it. The not-equal comparison ran across every repository at once. Two backup bugs found the same way needed patch releases across eighteen and thirty-one repositories, and a single misplaced permission line sat in eighty-three of ninety-four.

None of those were discovered by the monitoring. They were discovered by going to look. One reporting layer on a host had been green for months. Five scheduled timers were firing on a schedule nobody had written down, and nothing noticed. The report read the table of cron jobs. Nothing read systemd. A timer whose service unit was deleted is the purest case. It fires, systemd finds nothing to start, and the job does not run. It never appears in systemctl --failed, because it never failed.

Waiting for a failure cannot detect a job that has lost the ability to fail.

One more class belongs here, because it is the reason a fleet number can be wrong in a way no check catches. A conformance script was patched twice, re-run, and reported every repository meeting the standard. That verdict was passed on. The next day a fresh clone showed neither patch had ever been pushed: the clean number had been produced by a local copy that no longer existed anywhere. A fleet-wide verdict is worth exactly as much as the reachability of the code that produced it.

Assurance exposure is the one that reaches the boardroom, and it arrives quietly. SOC 2 CC7.2 asks whether the entity monitors system components for anomalies. ISO 27001 control 8.16 asks whether networks, systems and applications are monitored for anomalous behaviour and action taken. Both are satisfied by a health check that exists, runs on schedule, and reports. Neither asks whether it can go red. A check that cannot fail produces flawless operating-effectiveness evidence: every sample an auditor pulls shows it ran and passed. It will pass the audit. It will pass every security questionnaire that asks whether you monitor container health. And each of those answers will be false in the only way that matters.

That is the exposure a CEO owns. Not the outage. The statement.

So the question to ask of any monitoring, yours or a vendor’s, is not whether it exists. Ask to watch it go red. A team that can show you that in under a minute has monitoring. A team that cannot has a dashboard.

The checks that lie, by failure mode#

The last row is the one that hides the other six.

Framework: prove the red case, cross-check, detect silence#

1
#!/usr/bin/env bash
2
# tests/e2e-healthcheck.sh: the check must go red, or the suite fails
3
set -euo pipefail
4

5
cid=$(docker run -d --health-interval=2s --health-retries=1 "$IMAGE")
6
trap 'docker rm -f "$cid" >/dev/null' EXIT
7

8
status() { docker inspect -f '{{.State.Health.Status}}' "$cid"; }
9
# settle <state> <tries>: poll every two seconds until the state is seen
10
settle() {
11
  for _ in $(seq "$2"); do
12
    [ "$(status)" = "$1" ] && return 0
13
    sleep 2
14
  done
15
  return 1
16
}
17

18
# green path: wait for the real process to come up
19
settle healthy 45 || { echo "never went green"; exit 1; }
20

21
# red path: kill what the check watches, require the status to follow
22
docker exec "$cid" pkill -f "[S]hooterGameServer"
23
settle unhealthy 30 \
24
  || { echo "check never went red: it cannot fail"; exit 1; }

1
#!/usr/bin/env bash
2
# deadman-switch, the "agree" check kind: two answers to one question
3
set -euo pipefail
4

5
from_cron=$(crontab -l 2>/dev/null | grep -cE '^[^#]' || true)
6
from_systemd=$(systemctl list-timers --all --no-legend --no-pager \
7
  | grep -c . || true)
8
declared=$(jq '.schedules | length' /etc/fleet/schedules.json)
9

10
if [ "$((from_cron + from_systemd))" -ne "$declared" ]; then
11
  echo "schedules disagree:" \
12
    "cron=$from_cron systemd=$from_systemd declared=$declared"
13
  exit 1
14
fi
15

16
# both sides answering zero is not agreement
17
[ "$declared" -gt 0 ] \
18
  || { echo "nothing declared: that is not agreement"; exit 1; }

1
#!/usr/bin/env bash
2
# fleet-ops heartbeat: did it FIRE, not did it pass
3
set -euo pipefail
4

5
timers=$(systemctl list-units --type=timer --all --no-legend --plain \
6
  | awk '{print $1}')
7
for unit in $timers; do
8
  svc="${unit%.timer}.service"
9
  systemctl cat "$svc" >/dev/null 2>&1 \
10
    || { echo "orphan timer: $unit fires, $svc is missing"; exit 1; }
11

12
  last=$(systemctl show "$svc" --value \
13
    -p ExecMainStartTimestampMonotonic)
14
  [ "$last" != "0" ] \
15
    || { echo "$svc has never run since boot"; exit 1; }
16
done

Tradeoffs#

The cost is test time and a small amount of discipline at review. Proving the red case on a container that pulls a large image adds minutes to a pipeline, and the agreement check needs a declared inventory that somebody has to keep current. Neither is free.

What you buy is the difference between a dashboard and a claim. Seventeen hours of false green cost two players, a crashed round, and an afternoon spent looking for a bug in the wrong layer. The test that would have caught it runs in ninety seconds. Across a fleet the arithmetic only gets easier, because the same seven failure modes repeat and one rig covers them all.

Ninety seconds against a working day.

The closing argument#

Treat every check as untested until you have watched it go red. That audit is uncomfortable to run against a fleet you built yourself. I would rather run it on a Tuesday than have an incident run it for me. One test per check. The test is shorter than the check.

If you cannot produce the failing case on demand, you do not have monitoring. You have a green light wired to nothing.

Sources#

• AICPA Trust Services Criteria via ISMS.online, SOC 2 CC7.2, monitoring of system components for anomalies (the control asks that monitoring exist and detect, not that it be shown to fail), accessed September 2026
• ISO/IEC 27001, 2022 edition, via ISMS.online, Annex A 8.16, monitoring activities, accessed September 2026
• Docker Docs, Dockerfile reference: HEALTHCHECK (health status is reported, not acted on), accessed September 2026
• Docker Docs, docker inspect (State.Health.Status and RestartCount used in the reproduction above), accessed September 2026
• GitHub REST API, Get the latest release (latest is ordered by creation date, not by version), accessed September 2026
• procps-ng, pgrep manual page (-f matches the full command line), accessed September 2026
• systemd, systemd.timer manual page (a timer activates a unit of the same name by default), accessed September 2026

Discussion#

If you have found a check in your own fleet that turned out to have no red state, or you keep a different discipline for proving monitoring works, drop a comment below. Counterarguments are welcome and the comment thread is where I respond first. For longer back-and-forth with senior practitioners, join the discussion on Discord.