What the fleet taught, in the order it hurt

pgrep -f "ZZZ_no_such_process" exits 0 inside that container. pgrep -f searches full command lines, and the shell running the check has the pattern in its own. Seventeen hours green.

The container reported healthy while SteamCMD was still fetching 25 GB and no game server process existed at all.

GitHub's releases/latest is the most recently published release, not the highest version. A backport published after a newer line makes every up-to-date repository look behind.

find -newermt failed to parse its own date expression. Its stderr went to /dev/null, the output was empty, and empty meant "nothing changed".

The next day a fresh clone showed neither patch had ever been pushed. The clean number had been produced by a local copy that no longer existed anywhere.

Asking both the same question, "how many timers are scheduled", differed by five. Five timers were firing on a schedule nobody had written down.

The timer fires, systemd finds no unit to start, and the job does not run. Five such timers were left behind by renamed scripts on one host.

The diff captured file contents with $(...), which strips trailing newlines, so every file differed from its own byte-identical copy. The version rule read a pin of v3.1.0 and a README saying 3.1.0 as different versions, leaving every v-prefixed pin unchecked.

The sweep now says what it did: 105 of 105 repositories read, 0 open pull requests seen, while a pull request 45 hours old sat on one of them. Reading worked. Seeing did not.

GitHub answers 404, not 403, when a credential lacks the Pull requests permission, the same shape as "no such repository". It also answers 404 to anyone when the repository has Issues turned off, because a pull request is an issue. Across all 105 repositories on the account: issues on, the endpoint answered 103 of 103; issues off, it refused 2 of 2. No exceptions either way.

Every one of those runs was true and none of them was new. The finding was filed, the fix belonged to a person, and the red it produced was indistinguishable from the red that means something stopped overnight.

In the run that condemned it: docker compose up and the HTTPS smoke test passed, all three Trivy scans passed, the linter passed. The only red job was the freshness check, which goes red the moment any pin in that repository lags upstream, and says nothing about whether the digest just written boots.

The date-filtered query for that workflow's scheduled runs came back with nothing newer than the 18th while runs from the 19th, 20th, 21st and 22nd sat in the same repository, the last of them finishing two hours before the run that asked. A date filter had already been added for this exact failure a week earlier. It made it rarer, not impossible.

The credential could not write there: the API refused with Resource not accessible by personal access token. The create ran unchecked, so the message after it was printed on the way past. Two reports lost, zero signals raised, by the one piece of machinery whose entire job is raising them.

Three of the four were false. A script with the same filename as a published tool carried none of that tool's settings, zero occurrences of the variable the finding was about. Two others were each machine's own version of a job, already internally consistent: the directories they copy with no exclusions are verified with no exclusions.

The roster of machines was derived from the rows. Zero rows, zero visits: the report went silent in exactly the state where it had the most to correct, and silence on an open thread reads as a standing complaint.

The expected count was 0, and 0 >= 0 passes on the first pass. The glob matched nothing because a failed cd was not fatal: the script sets -uo pipefail without -e, so the error printed and everything below it ran in whatever directory the script had been started from.

Half of that repository is not the working tree. A snapshot script runs before every commit and copies the live files over it, live to repository and never back, and deletes anything the sources do not have. A merge into that half survives until the next run, at most an hour, and the revert is committed under the name config drift, which does not mention what it undid. The repository’s own README records the mechanism doing exactly that three times in one day, the third time to the warning about it that had just been written there.

The variable naming the apex was referenced by the router and was not in the example environment file everyone is told to copy, so the rule rendered as Host(``). The proxy accepts that configuration, creates the load balancer, creates the server, attaches its middlewares, and only then discards the router: error while checking rule Host: empty args for matcher Host, [], one line, at debug level. Run against the same file, the compose tool says it out loud first: variable is not set. Defaulting to a blank string.

The job wrote its own environment file, eight lines of it. The documented path is to copy the example file and edit it. Those are two different deployments, and they differed by exactly the variable that was broken: the job had it, the example file did not. Checked across the fleet, 64 stacks ship an example file and two required a variable it never mentioned.

The page announced every four hours through a change to twice a day and then to once a day. The same words sat in the profile summary, in the watcher’s own docstring and in two workflow comments. Nothing compared any of them to the cron.

The workflow behind that badge also runs the freshness check, which is a designed alarm: the pin is behind upstream and the fleet’s triage moves it within the day. Over fourteen days: 281 failed runs on the default branch, 254 of them with no failing job except that alarm, and 27 real. At the moment of writing, 64 badges green and none red.

One pin sat a patch release behind and was reported by hand every day, because its check says is behind its line: pinned 7.0.30, latest in 7.0 is 7.0.31 while the triage matched a different phrasing. Tested across the fleet by rendering every alarm’s text and running it against that matcher: 51 alarms it could act on, 51 it could not, including every proxy pin in the fleet, whose sentence says “pinned minor 3.6” and where the word minor was the whole problem.

It was wrong. The suffix had nothing to do with it; the alarm’s wording did, which the finding above measured an hour later. By then the sentence was in two changelogs, two published releases and a closed report.

The rewritten concurrency group read pin-freshness-${{ github.ref }} github.ref }}: the pattern had stopped at the first space and left the tail of the old value behind. The workflow linter passed it, because a garbled string is still a valid string. It was caught by reading the output, not by any gate.

The audit skipped every value containing a $, and the newer generation of scripts writes paths as ${VAR:-default}, so it passed that generation unread. It also recognised a one-word variable prefix only, and where it could not read a stack’s paths it compared nothing and said nothing. Taught both, it reported eight and named the two repositories it could not compare instead of passing them.

v2.0.4 was out. A release had been created late for an old version that was announced and never tagged, and the host’s “latest” flag goes to the most recently created release, so the older version took it. The page read the flag.

One test cleared the data directory before unpacking, with a comment explaining that tar does not delete. The other put its marker in a collection the archive already held, with a comment explaining that mongorestore --drop replaces only those. Both tests knew the limitation and worked around it. Neither shipped script did. The green run proved the test's restore.

The loop runs inside bash -c '...', and a new comment said Garage's. The apostrophe closed the quote; bash stopped at unexpected end of file and no backup ran. It was caught because CI waits for a Data backup OK line rather than for the container to be up.

The success condition accepted exit 1 because GNU tar returns it for "a file changed while reading". BusyBox also returns it when the output cannot be opened at all. Blocking the destination on purpose produced two unreadable files named like backups.

Reproduced on a bench: killed mid-dump, the old loop leaves a file that fails gzip -t. Five templates then failed CI because their success condition still tested the final name, which the fix had moved away from.

Every one of those files was on the disk and identical. The verification ran at 13:00 and the backup at 12:00; the report was an hour of ordinary writes. With the last-success stamp missing, the same code classified every difference as corruption instead.

restic restores to <target>/<path in the snapshot>, so the documented command produced data/data/world/level.dat. The drill looked for level.dat anywhere under the restore directory and found it.

Ten is a number nobody revisits. It passes for a dump that restored a fifth of the schema, and it keeps passing as the application grows away from it.

That archive is a copy of a live SQLite file with its -wal and -shm beside it, taken by an ordinary archiver. Fine for a daily copy. Not for the one backup that matters, on the day it is the only copy.

Nobody performs that maintenance. The floor passes for a dump that restored a fifth of the schema and keeps passing as the application grows away from it, the same defect this fleet had already found and fixed in an application, still living in the published tool.

Against the pinned image rather than reasoned about: mysqldump exits 0, and the dump it writes is byte-for-byte what --no-tablespaces produces, same sha256, 1819 bytes each. The application user has no PROCESS privilege and needs none; the dump was asking for tablespace information nothing wants. The four MariaDB templates were measured too and print nothing at all.

Of ninety-four public templates it was the only one carrying a database and no backup at all. The game offers no export; the data exists in one volume and is written continuously.

One library is copied under a different name on the backup disk, films excluded and structure and metadata kept, and that pair was in no list. On a stand-in tree with one metadata file corrupted on the backup side and the source untouched since the last backup, the report was green: every byte matches. With the pair added, the same disk produced one unexplained difference and exit 1.

Not one of those tests runs the shipped scripts: each restores with its own copy of the commands. Checked against the stacks they belong to, eight scripts disagree with them. One listed a backup directory the stack does not write to, so on the day it was needed it would offer nothing to restore. Several still target the paths of an image family the templates left months ago; one of those refuses to run unless that old path exists, and refuses any override of it. The tests stayed green throughout, because the copy was right and the script was not.

Since the stack replaced MinIO with Garage in 2.0.0, the loop had archived the old minio-data volume, which nothing writes to. The attachments were in no backup at all. The restore script looked for a service called minio and stopped before touching anything.

The tests restored with their own copies of the commands. The scripts a person would run on the day they need one: four written for Bitnami images on stacks that run official ones, one of which refused to run at all; one that signed in with the wrong account and would have dropped a database that does not exist; restores that merged instead of replacing, through rm -rf dir/*, which keeps dotfiles, tar over live data, and mongorestore --drop, which keeps collections the archive does not hold; paths read from the operator's shell instead of the stack; and a database image with no client of the name the script called, found only when CI first ran it.

The server publishes by renaming: write Saves.next, move Saves aside, rename the staged copy into place. rename(2) on a mount point returns EBUSY: mv: cannot move '/app/Saves': Resource busy. The fallback empties the host directory file by file and the rename in still fails. The only complete world lived in the container's writable layer.

On a bench against an origin that produces its response over four seconds: without the middleware the first byte arrives at 0.03s; with the all-zero one, at 4.15s. Zero means no size ceiling, not disabled.

cap_drop: ALL removes DAC_READ_SEARCH. Root without it cannot traverse a directory it does not own.

Its entrypoint renders the routing table into conf.d at start. Mounted read-only, that write fails and is not fatal: nginx comes up listening on an empty configuration.

Compose requires $ doubled in a label. In an unquoted heredoc, $$ is the shell's PID, so the doubled hash was written to disk as a five-digit number.

Docker Desktop virtualises bind-mount ownership, so a container copying files into a mounted directory succeeds locally and fails on Linux. Four consecutive CI failures came from that gap alone.

Its cron sidecar had no restart policy, so after a reboot every background job - file scans, cleanup, notifications - stopped with no error anywhere. The same gap was in another template’s main application, and a template generated from that one today inherited it before it was caught.

Across nineteen runs: 78 waits, 2 reverts. Twenty scheduled runs in a cycle, 462 minutes, 23 minutes a run; one run took three hours. Ninety-seven per cent of the waiting confirmed what was already true.

generating optimized images ✓ Completed in 18m 17s out of an 18m 48s build, over 2,991 images. The deploy platform is the build of record and did the same work minutes later. 27 runs a fortnight.

Token-Permissions 0/10, caused by one file in 83 of 94 public repositories: dependabot-automerge.yml granted contents: write at the top of the file, where it applies to every job including ones nobody has written yet.

One offered 11.0.0 against a main already on 11.0.1. The other offered Nextcloud 32.0.15 against a main on 34.0.3: opening it would have proposed a two-major downgrade of a database that only migrates forward.

GitHub disables a scheduled workflow after 60 days without repository activity. That is a state on the workflow, not a failed run: no red badge, no email, and the profile keeps showing the numbers it last wrote.

last fired 51 hours ago and the same line a day later hash differently, so an unfixed problem would have commented and mailed twice a day for as long as it stayed unfixed, worst on the day a schedule actually stopped, which is the one alarm that must not arrive buried.

Issues are off on both, and a pull request is an issue, so Dependabot pushed a branch and then failed to open the pull request that would deliver it, quietly, on a schedule. The branches were from 2026-09-06 and held checkout, scorecard-action and codeql-action digest bumps. Fourteen days, one of them the profile repository.

The job triggers on workflow_run when the verification completes. That verification completed at 07:35:14; the auto-merge workflow reached the default branch at 23:58:59 the same day. Sixteen hours too late to ever hear about it, and nothing re-fires, so the pull request was stranded permanently.

It re-ranks one catalogue and the scores barely move, so the shortlist is stable by construction. Nothing recorded that an answer had been given, so the same question would have arrived every Monday for as long as the job existed.

Measured across every scheduled job in the repository, each lands five to six and a half hours after the minute it names. Declared 05:00 arrives 10:22; 06:45 arrives 11:47; 07:15 arrives 12:20; 08:00 arrives 14:55. GitHub runs scheduled workflows at low priority and the delay grows with load.

Of twelve pieces of work it proposed, five were already in the target, one of them for eight days. The job was shown only each target's latest release note, so a rule that shipped six releases ago, or lives in the code with no release of its own, was invisible to it.

It selected by the end of a repository's name. Of 87 repositories carrying a changelog, 21 were never checked at all: every infrastructure pipeline and every published tool that is not a container. One of them had announced three releases in its changelog and carried none of those tags, in public, for four months, with a rule for exactly that sitting unreached inside the checker.

The wave that made every verification daily ran on the morning of 2026-09-02; three templates received their CI that afternoon from a template that still said weekly, and the wave’s own log would have read no drift job, skipped. They ran weekly for three weeks under a security policy saying daily and a profile saying every template boots in CI daily. The wave that stopped hiding failed security scans walked the compose templates, and the one pipeline that publishes an image was not a template, so it kept the flag on the scan of what it ships.

Judged by the fleet’s standard as it stands, it failed five rules and lacked a supply-chain workflow every public repository carries; its version check compared with !=, which the rest of the fleet had fixed. The other three predated the standard entirely: every image reference came from an environment file whose example left them all empty, so the documented deploy could not start, and one mounted its library from a Windows path. Every wave and every daily check walks public repositories only.

Traefik's acme.json carries its ACME account private key as base64 inside JSON. It passed every content pattern in the same commit where the scanner correctly rejected an ordinary DB_PASSWORD= line.

On a host that deploys by git pull, that commit removes the file from the working tree. The stack fails on its next docker compose up, and the secrets it needed are gone with it.

GitHub secret scanning is free on public repositories only; private ones need an enterprise plan. Eleven private repositories had no scanning at all.

in=44175 out=4000, stop_reason max_tokens, and not one text block in the response. Extended thinking is on by default and produces no text until it finishes; raising the ceiling to 16,000 bought more thinking. Two sibling jobs had the same defect and got away with it on smaller inputs.

The split that holds: an edge container with no socket, no credentials, a read-only filesystem and an internal: true network, whose only verb is writing a file into a shared volume; a worker with the socket and no inbound path at all, not even on the edge's network. Each button carries a one-time nonce; using any button burns every nonce of that alert.

MinIO's community server was archived in April 2026, and the newest image is the one already pinned: green for ever. MongoDB 8.x refuses to start on kernel 6.19 and later (TCMalloc rseq), while GitHub runners are on 6.17, so CI would pass a bump a modern host rejects. And a database pinned at a major on purpose never "lags", so a major reaching end of support sits in a template with every check green.

Every image the fleet pins was resolved the way the reviewer resolves it and the answer asked of GitHub: 34 of 100 named a repository that is not there. Each produced a verdict about the lookup wearing the clothes of a verdict about the upgrade. Found through beszel, whose agent ships from the hub's repository.

The release notes carry the commit title and nothing else. The change does explain itself: a directory rename plus two lines of Dockerfile: base image steamrt/sniper/platform to steamrt/steamrt4/sdk, and the RCON forwarder simpleproxy to socat. Both inside the image; ports, variables and the compose interface unchanged.

The plugins it pins had no build for that version yet. Nothing in the stack could have known: the answer lives in the plugin registry's API, not in any container. Asked directly, the registry answers exactly: 11, 7 and 1 builds for the version that works, 0, 0 and 0 for one that does not.

The watcher's query filters by loader and not by game version, so the build it offers can target a release none of the servers run. The registry answers exactly: one version of the same plugin lists 26.2 and not 26.3, the next lists both. A pin the running version cannot load does not fail where it is written. It fails at container start, in ten backends that have already been recreated.