07 from Docker Health Checks That Cannot Fail — Seventeen Hours Green
The red-case audit: prove every health check can fail
Monitoring is written to detect failure and tested by watching it not fire. That is the whole defect.
A dashboard of green checks cannot tell a working check from one that has no red state, so the audit asks each check to fail, on purpose, once.
- 1
Run every check against a case it must fail
Inside the container,
pgrep -f missingshould exit 1; exit 0 means the check matches its own shell. Then stop the real process and watch the check go red, within the interval and retries it is given. - 2
Break the self-match, and name what serves
Bracket one character,
pgrep -f "[n]ame", and match the binary that serves, not the supervisor that starts first. Better still, ask the service itself: its own query or its health endpoint. - 3
Keep an error apart from an empty answer
Never send a probe's stderr to
/dev/null. A command that failed must give a different value from “found nothing”. - 4
Compare versions as versions
sort -V, never!=: GitHub's latest release is the newest created, not the highest. - 5
Give the label a consumer
Docker leaves an unhealthy container running, a restart policy included. Something has to read the state and act, and that path needs its own red test.
- 6
Find the checks that never fired
Break a conforming case on purpose for every rule that has been silent since it shipped, and compare the timers against their units: a timer whose service was deleted never shows up in
systemctl --failed.
The checks that lie, by failure mode
| Failure mode | What it looks like | What settles it |
|---|---|---|
| Self-matching pattern | Green over a dead process | Run the check against a pattern that cannot exist |
| Passes too early | Green before the binary is on disk | Name the binary, not the supervisor |
| Silenced stderr | “Nothing changed” from a command that errored | Make failure a different value from “found nothing” |
| Wrong comparison | Current repositories reported behind | sort -V, never != |
| Nothing reads it | Correct red state, no action taken | Give the label a consumer |
| Guard disabled by its own failure | A rule stops applying and looks like the quiet case it exists for | Keep the last good reading while it is fresh, and set “do we know” explicitly |
| Never red, ever | Rule silent since the day it shipped | Break a conforming case on purpose |
What shows it works
A public commit, a decision, and two measurements anyone can repeat.
- Commit b47fadd6, 5 September 2026: the self-matching check found green on an empty container, and the bracket fix measured against the same base image.
- Decision 6, 7 September 2026: every check in the fleet ships with the test that plants its violation.
- Measured on 13 November 2026 in alpine:3.21, each command in its own container:
pgrep -f missingexits 0, andpgrep -f "[m]issing"exits 1. - Measured on 13 November 2026 on Docker 29.7.2: a container whose health command always fails, started with
--restart unless-stopped, reports unhealthy with restarts=0. - The ledger, checks that lie: every finding of this kind, with the rule it produced.
Test each check alone. Inside one sh -c that also contains the word, even the bracketed pattern matches, because the shell's own command line now carries it. A red test proves the check can fail today; renaming the binary or the image can take that away, which is why the test runs on every push.
Green is the state nobody investigates.
The long version is the post Docker Health Checks That Cannot Fail. GitHub Docs, the latest release (the most recent non-prerelease, non-draft release, by creation date); procps, pgrep(1); ISMS.online on SOC 2 CC7.2 and ISO 27001 8.16.