What the fleet taught, in the order it hurt

pgrep -f "ZZZ_no_such_process" exits 0 inside that container. pgrep -f searches full command lines, and the shell running the check has the pattern in its own. Seventeen hours green.

The container reported healthy while SteamCMD was still fetching 25 GB and no game server process existed at all.

GitHub's releases/latest is the most recently published release, not the highest version. A backport published after a newer line makes every up-to-date repository look behind.

find -newermt failed to parse its own date expression. Its stderr went to /dev/null, the output was empty, and empty meant "nothing changed".

The next day a fresh clone showed neither patch had ever been pushed. The clean number had been produced by a local copy that no longer existed anywhere.

Asking both the same question, "how many timers are scheduled", differed by five. Five timers were firing on a schedule nobody had written down.

The timer fires, systemd finds no unit to start, and the job does not run. Five such timers were left behind by renamed scripts on one host.

The diff captured file contents with $(...), which strips trailing newlines, so every file differed from its own byte-identical copy. The version rule read a pin of v3.1.0 and a README saying 3.1.0 as different versions, leaving every v-prefixed pin unchecked.

The sweep now says what it did: 105 of 105 repositories read, 0 open pull requests seen, while a pull request 45 hours old sat on one of them. Reading worked. Seeing did not.

GitHub answers 404, not 403, when a credential lacks the Pull requests permission, the same shape as "no such repository". It also answers 404 to anyone when the repository has Issues turned off, because a pull request is an issue. Across all 105 repositories on the account: issues on, the endpoint answered 103 of 103; issues off, it refused 2 of 2. No exceptions either way.

Every one of those runs was true and none of them was new. The finding was filed, the fix belonged to a person, and the red it produced was indistinguishable from the red that means something stopped overnight.

In the run that condemned it: docker compose up and the HTTPS smoke test passed, all three Trivy scans passed, the linter passed. The only red job was the freshness check, which goes red the moment any pin in that repository lags upstream, and says nothing about whether the digest just written boots.

The date-filtered query for that workflow's scheduled runs came back with nothing newer than the 18th while runs from the 19th, 20th, 21st and 22nd sat in the same repository, the last of them finishing two hours before the run that asked. A date filter had already been added for this exact failure a week earlier. It made it rarer, not impossible.

The credential could not write there: the API refused with Resource not accessible by personal access token. The create ran unchecked, so the message after it was printed on the way past. Two reports lost, zero signals raised, by the one piece of machinery whose entire job is raising them.

Three of the four were false. A script with the same filename as a published tool carried none of that tool's settings, zero occurrences of the variable the finding was about. Two others were each machine's own version of a job, already internally consistent: the directories they copy with no exclusions are verified with no exclusions.

The roster of machines was derived from the rows. Zero rows, zero visits: the report went silent in exactly the state where it had the most to correct, and silence on an open thread reads as a standing complaint.

The expected count was 0, and 0 >= 0 passes on the first pass. The glob matched nothing because a failed cd was not fatal: the script sets -uo pipefail without -e, so the error printed and everything below it ran in whatever directory the script had been started from.

Half of that repository is not the working tree. A snapshot script runs before every commit and copies the live files over it, live to repository and never back, and deletes anything the sources do not have. A merge into that half survives until the next run, at most an hour, and the revert is committed under the name config drift, which does not mention what it undid. The repository’s own README records the mechanism doing exactly that three times in one day, the third time to the warning about it that had just been written there.

The variable naming the apex was referenced by the router and was not in the example environment file everyone is told to copy, so the rule rendered as Host(``). The proxy accepts that configuration, creates the load balancer, creates the server, attaches its middlewares, and only then discards the router: error while checking rule Host: empty args for matcher Host, [], one line, at debug level. Run against the same file, the compose tool says it out loud first: variable is not set. Defaulting to a blank string.

The job wrote its own environment file, eight lines of it. The documented path is to copy the example file and edit it. Those are two different deployments, and they differed by exactly the variable that was broken: the job had it, the example file did not. Checked across the fleet, 64 stacks ship an example file and two required a variable it never mentioned.

The page announced every four hours through a change to twice a day and then to once a day. The same words sat in the profile summary, in the watcher’s own docstring and in two workflow comments. Nothing compared any of them to the cron.

The workflow behind that badge also runs the freshness check, which is a designed alarm: the pin is behind upstream and the fleet’s triage moves it within the day. Over fourteen days: 281 failed runs on the default branch, 254 of them with no failing job except that alarm, and 27 real. At the moment of writing, 64 badges green and none red.

One pin sat a patch release behind and was reported by hand every day, because its check says is behind its line: pinned 7.0.30, latest in 7.0 is 7.0.31 while the triage matched a different phrasing. Tested across the fleet by rendering every alarm’s text and running it against that matcher: 51 alarms it could act on, 51 it could not, including every proxy pin in the fleet, whose sentence says “pinned minor 3.6” and where the word minor was the whole problem.

It was wrong. The suffix had nothing to do with it; the alarm’s wording did, which the finding above measured an hour later. By then the sentence was in two changelogs, two published releases and a closed report.

The rewritten concurrency group read pin-freshness-${{ github.ref }} github.ref }}: the pattern had stopped at the first space and left the tail of the old value behind. The workflow linter passed it, because a garbled string is still a valid string. It was caught by reading the output, not by any gate.

The audit skipped every value containing a $, and the newer generation of scripts writes paths as ${VAR:-default}, so it passed that generation unread. It also recognised a one-word variable prefix only, and where it could not read a stack’s paths it compared nothing and said nothing. Taught both, it reported eight and named the two repositories it could not compare instead of passing them.

v2.0.4 was out. A release had been created late for an old version that was announced and never tagged, and the host’s “latest” flag goes to the most recently created release, so the older version took it. The page read the flag.

One test cleared the data directory before unpacking, with a comment explaining that tar does not delete. The other put its marker in a collection the archive already held, with a comment explaining that mongorestore --drop replaces only those. Both tests knew the limitation and worked around it. Neither shipped script did. The green run proved the test's restore.

The loop runs inside bash -c '...', and a new comment said Garage's. The apostrophe closed the quote; bash stopped at unexpected end of file and no backup ran. It was caught because CI waits for a Data backup OK line rather than for the container to be up.

Both sides ran the same stack. The drill took "the release before main" by tag order, and main was one commit past v1.9.1 with nothing in the stack changed, so the "previous release" was the current one. A version-crossing test that did not cross passed exactly as a real one would.

The live counter written from the same test said 72 of 73. The 73rd, in restore-drill, runs in its test as "$ROOT/restore-drill.sh", and the rule only recognised ./restore-drill.sh. The rule had not been wrong about any repository it judged; it would have been wrong about the next one written that way.

The real gh() never raises: it calls sys.exit. In CI the fleet token cannot read eleven private repositories, so the first of them would have ended the whole run at 07:15 the next morning, with every unit test green. Found by reading the function the fake stood in for, an hour after pushing, before the schedule fired.

Their last run was the weekly one, two days earlier. Against the new 24-hour period that read as 62 hours late, before the new daily time had come round once. A cron edited is a clock reset, and the check measured from the old clock.

Its dump names carry the database, so the drill's pattern was -dr_drill-…\.bak$, and grep -E "$pat" read the leading dash as a flag: no error to the caller, no match, a loop that saw nothing. Every other template's pattern began with a bracket or a caret and never met the bug.

On the dying host, running the previous release, the drill wrote the marker into $OUTLINE_DATA_PATH/data; that release kept its data under /data, not /var/lib/garage, so the directory did not exist and the shell said cannot create … Directory nonexistent. The drill's write ended in || true, so the log line was there and the run went on to fail twenty minutes later on the read, blaming the restore.

The sentence was written before the page, the page was never made, and nothing checked that a URL in a policy answers: the profile-claims check reads the profile, the conformance check reads the templates, and neither opens a link. It surfaced on 25 September while answering the OpenSSF Best Practices question that asks for exactly that URL. There was no key to publish either.

The OpenSSF Best Practices questionnaire asks seven questions about an automated test suite. Answered honestly for 89 repositories on 25 September: 74 have one, eight Terraform pipelines, six tools and the one published image have none. Lint is a check on the text; a test is a check on the behaviour, and green on the first was being read as the second.

Every scheduled job was green through those fourteen hours, and the heartbeat, which reads scheduled runs only, said the machinery was fine. Verify runs on a push, not a cron. With mail off, a red push run in a private repository reached nobody until a person opened the Actions tab.

Two days earlier the freshness check had moved out of the verification workflow into its own, which runs on a schedule and skips on a push by design. The refresh pass recorded that workflow for the judgement. The answer it read was skipped, and the branch that handled every answer but green, running and cancelled was the revert. One code path, twenty-three reverts, each followed by "needs a human".

The test slept the backup interval plus sixty seconds: 120 s in CI. A backup cycle is a database dump, a data backup that takes as long as the data does, the prune, then the interval. With 35.0.1 the data backup took 98 s; the cycle ran 160 s; the test asked at 120 s, before the prune had run. find and touch in the old and new images behave identically, checked locally. Thirty-six templates carried the same three lines.

The public copy's catalog run met GitHub's hourly API limit half way through reading the fleet, during the hour the fleet's engine moved to its public repository and every job ran twice; the private copy's run in the same minute read everything. The public one read 37 of 97 repositories and published what it had. The evidence line even said that 60 could not be read, which made it honest and still wrong: every visitor's browser replaced the page's numbers with the smaller ones, and the site's copy job, which refuses a fleet that shrank by a fifth, was not due for three hours.

Both surfaced on 26 September, on the first run of test suites written for the fifteen repositories decision 18 left short of the OpenSSF badge. New-Item and Export-Csv inherited the dry-run preference along with the account changes. Every suite came with planted violations, 153 in all, and the plants found four tests that could not fail: a -WhatIf case that Remove-Item satisfied on its own, a Mattermost plant no test could reach, a tree-size check that needed to measure one row, and an SSH host-key guard sitting behind a second lock, so removing it changed nothing a test could see. That one became a man-in-the-middle test: after the key scan the target's network name moves to an impostor with fresh host keys and the same account, and the job has to refuse to log in. Writing the Terraform tests moved three load balancers to AWS's TLS 1.3 policy and put what the defaults do not deliver into each README: SSH open to the internet on four EC2 pipelines, Redis unencrypted in two, a public EKS endpoint.

Docker Hub keeps sha-*, version and kube-v* tags immutable on this repository. The release was tagged on the commit main had published minutes earlier, so the tag run rebuilt it and tried to move sha-83be903 to the new digest. The refusal came last, after the version tags were already out. Immutability did its job; the workflow assumed a tag run always builds a commit nobody has built. kube-v1.37.1 now points at the unsigned image for good, because an immutable tag cannot be moved to the fixed one either.

The heartbeat read schedules, tokens, Dependabot pull requests and heads with no check on them, once a day across 94 public repositories. No job read what a person had written. Answering the issue meant looking at the rest of the fleet: 40 of the 50 Traefik stacks gave an operator no way to set the entry point's timeouts, nine had them under nine different names, and one had only the read timeout.

The success condition accepted exit 1 because GNU tar returns it for "a file changed while reading". BusyBox also returns it when the output cannot be opened at all. Blocking the destination on purpose produced two unreadable files named like backups.

Reproduced on a bench: killed mid-dump, the old loop leaves a file that fails gzip -t. Five templates then failed CI because their success condition still tested the final name, which the fix had moved away from.

Every one of those files was on the disk and identical. The verification ran at 13:00 and the backup at 12:00; the report was an hour of ordinary writes. With the last-success stamp missing, the same code classified every difference as corruption instead.

restic restores to <target>/<path in the snapshot>, so the documented command produced data/data/world/level.dat. The drill looked for level.dat anywhere under the restore directory and found it.

Ten is a number nobody revisits. It passes for a dump that restored a fifth of the schema, and it keeps passing as the application grows away from it.

That archive is a copy of a live SQLite file with its -wal and -shm beside it, taken by an ordinary archiver. Fine for a daily copy. Not for the one backup that matters, on the day it is the only copy.

Nobody performs that maintenance. The floor passes for a dump that restored a fifth of the schema and keeps passing as the application grows away from it, the same defect this fleet had already found and fixed in an application, still living in the published tool.

Against the pinned image rather than reasoned about: mysqldump exits 0, and the dump it writes is byte-for-byte what --no-tablespaces produces, same sha256, 1819 bytes each. The application user has no PROCESS privilege and needs none; the dump was asking for tablespace information nothing wants. The four MariaDB templates were measured too and print nothing at all.

Of ninety-four public templates it was the only one carrying a database and no backup at all. The game offers no export; the data exists in one volume and is written continuously.

One library is copied under a different name on the backup disk, films excluded and structure and metadata kept, and that pair was in no list. On a stand-in tree with one metadata file corrupted on the backup side and the source untouched since the last backup, the report was green: every byte matches. With the pair added, the same disk produced one unexplained difference and exit 1.

Not one of those tests runs the shipped scripts: each restores with its own copy of the commands. Checked against the stacks they belong to, eight scripts disagree with them. One listed a backup directory the stack does not write to, so on the day it was needed it would offer nothing to restore. Several still target the paths of an image family the templates left months ago; one of those refuses to run unless that old path exists, and refuses any override of it. The tests stayed green throughout, because the copy was right and the script was not.

Since the stack replaced MinIO with Garage in 2.0.0, the loop had archived the old minio-data volume, which nothing writes to. The attachments were in no backup at all. The restore script looked for a service called minio and stopped before touching anything.

The tests restored with their own copies of the commands. The scripts a person would run on the day they need one: four written for Bitnami images on stacks that run official ones, one of which refused to run at all; one that signed in with the wrong account and would have dropped a database that does not exist; restores that merged instead of replacing, through rm -rf dir/*, which keeps dotfiles, tar over live data, and mongorestore --drop, which keeps collections the archive does not hold; paths read from the operator's shell instead of the stack; and a database image with no client of the name the script called, found only when CI first ran it.

The backup loop starts with the stack. On the new machine its first cycle wrote an empty backup into the same directory before the restore ran, so the newest file was the new host's own. Nextcloud got the new, empty database next to the old files and answered 500; Gitea came back without its files. Backup names carry the minute, not the second, so that first cycle can also overwrite an imported backup of the same name.

Restored onto a machine that had never run the stack, it answered 500 on every request: password authentication failed for user "oc_admin". Nextcloud's installer creates its own database account with a random password and writes it into config.php; pg_dump saves the data and none of the server's accounts. The new host's installer had created that account again, with a different password. Every same-host restore passed, because there the account already had the right one.

Each of the three prepares something before the stack: a directory of secret files, a bcrypt hash made by a documented command, a bind directory chowned to the container's uid. CI does it in steps the drill had not copied, so the clean machine was cleaner than any real machine would be. Authelia's secrets are also the one thing that must come from the dead host, not be made anew: the encryption key in them is what the restored database was written with.

On this machine, main cold-started and asked directly, front answered /admin/ with 302 to /sso/login every time, so front was fine. The difference in CI was Traefik in front of it. The compose file gave webmail its own Traefik router with the same Host() rule as front's and no priority, and Traefik chose between two equal routers at every start; when it chose webmail, every path on the mail host went to Roundcube. One start in three, and every deployment of the template had the same odds. Read side by side, the two routers were the whole finding; the drill only made it repeatable.

Surveyed on 25 September after the Nextcloud prune test failed: thirty-six templates polled for the prune with a wait chosen when the data was small, nine polled with a ceiling of interval plus ninety seconds, one slept a fixed forty-five, two checked it their own way, and one, Zammad, did not check at all. A file dated 2020 placed beside Zammad's real backups was gone after the next cycle and six recent archives stayed, which is the first time that sentence could be written.

The server publishes by renaming: write Saves.next, move Saves aside, rename the staged copy into place. rename(2) on a mount point returns EBUSY; BusyBox's mv says can't rename '/app/Saves': Resource busy. The fallback empties the host directory file by file and the rename in still fails. The only complete world lived in the container's writable layer.

On a bench against an origin that produces its response over four seconds: without the middleware the first byte arrives at 0.03s; with the all-zero one, at 4.15s. Zero means no size ceiling, not disabled.

cap_drop: ALL removes DAC_READ_SEARCH. Root without it cannot traverse a directory it does not own.

Its entrypoint renders the routing table into conf.d at start. Mounted read-only, that write fails and is not fatal: nginx comes up listening on an empty configuration.

Compose requires $ doubled in a label. In an unquoted heredoc, $$ is the shell's PID, so the doubled hash was written to disk as a five-digit number.

Docker Desktop virtualises bind-mount ownership, so a container copying files into a mounted directory succeeds locally and fails on Linux. Four consecutive CI failures came from that gap alone.

Its cron sidecar had no restart policy, so after a reboot every background job - file scans, cleanup, notifications - stopped with no error anywhere. The same gap was in another template’s main application, and a template generated from that one today inherited it before it was caught.

Across the fifteen scheduled runs that still waited, 7 to 14 September: 62 waits and 2 decisions to revert, one of which reverted a good change and one of which failed at git revert. 439 minutes, 29 a run, the longest 104. Counting every run, scheduled and started by hand: 88 waits in 586 minutes. Ninety-seven per cent of the waiting confirmed what was already true. A first count said 78, because every note prints twice in a run's log, once as it happens and once in the report.

generating optimized images ✓ Completed in 18m 17s out of an 18m 48s build, over 2,991 images. The deploy platform is the build of record and did the same work minutes later. 27 runs a fortnight.

Token-Permissions 0/10, caused by one file in 83 of 94 public repositories: dependabot-automerge.yml granted contents: write at the top of the file, where it applies to every job including ones nobody has written yet.

One offered 11.0.0 against a main already on 11.0.1. The other offered Nextcloud 32.0.15 against a main on 34.0.3: opening it would have proposed a two-major downgrade of a database that only migrates forward.

GitHub disables a scheduled workflow after 60 days without repository activity. That is a state on the workflow, not a failed run: no red badge, no email, and the profile keeps showing the numbers it last wrote.

last fired 51 hours ago and the same line a day later hash differently, so an unfixed problem would have commented and mailed twice a day for as long as it stayed unfixed, worst on the day a schedule actually stopped, which is the one alarm that must not arrive buried.

Issues are off on both, and a pull request is an issue, so Dependabot pushed a branch and then failed to open the pull request that would deliver it, quietly, on a schedule. The branches were from 2026-09-06 and held checkout, scorecard-action and codeql-action digest bumps. Fourteen days, one of them the profile repository.

The job triggers on workflow_run when the verification completes. That verification completed at 07:35:14; the auto-merge workflow reached the default branch at 23:58:59 the same day. Sixteen hours too late to ever hear about it, and nothing re-fires, so the pull request was stranded permanently.

It re-ranks one catalogue and the scores barely move, so the shortlist is stable by construction. Nothing recorded that an answer had been given, so the same question would have arrived every Monday for as long as the job existed.

Measured on 21 September across every scheduled job in the repository, each landed five to six and a half hours after the minute it names: declared 05:00 arrived 10:22; 06:45 arrived 11:47; 07:15 arrived 12:20; 08:00 arrived 14:55. Over the whole month the delay ranges from two to eight hours, by hour and by day. GitHub runs scheduled workflows at low priority and the delay grows with load.

Of twelve pieces of work it proposed, five were already in the target, one of them for eight days. The job was shown only each target's latest release note, so a rule that shipped six releases ago, or lives in the code with no release of its own, was invisible to it.

It selected by the end of a repository's name. Of 87 repositories carrying a changelog, 21 were never checked at all: every infrastructure pipeline and every published tool that is not a container. One of them had announced three releases in its changelog and carried none of those tags, in public, for four months, with a rule for exactly that sitting unreached inside the checker.

The wave that made every verification daily ran on the morning of 2026-09-02; three templates received their CI that afternoon from a template that still said weekly, and the wave’s own log would have read no drift job, skipped. They ran weekly for three weeks under a security policy saying daily and a profile saying every template boots in CI daily. The wave that stopped hiding failed security scans walked the compose templates, and the one pipeline that publishes an image was not a template, so it kept the flag on the scan of what it ships.

Judged by the fleet’s standard as it stands, it failed five rules and lacked a supply-chain workflow every public repository carries; its version check compared with !=, which the rest of the fleet had fixed. The other three predated the standard entirely: every image reference came from an environment file whose example left them all empty, so the documented deploy could not start, and one mounted its library from a Windows path. Every wave and every daily check walks public repositories only.

The fleet's heartbeat watches that schedule for stopping, not for failing, and a run that refuses a bad file is a run that fired. So the one failure the job exists to catch would have kept yesterday's numbers in the HTML, correctly, and told nobody. Planted on a throwaway branch: the first failing run opened an issue, the second commented on it instead of opening another, and the next passing run closed it.

The clone had lost the files along with its config, and git add -A staged the absences as deletions. Sixteen hours: the weekly jobs had not yet missed a run, so the heartbeat, which reads this repository's schedules from the checkout, saw a shorter list and no gap; the daily triage reported No module named 'anthropic_federation' on every upstream review and proceeded on the CI gate alone, as it is written to. The triage issue was the only place it showed, and I read it because the user asked whether everything was closed.

Traefik's acme.json carries its ACME account private key as base64 inside JSON. It passed every content pattern in the same commit where the scanner correctly rejected an ordinary DB_PASSWORD= line.

On a host that deploys by git pull, that commit removes the file from the working tree. The stack fails on its next docker compose up, and the secrets it needed are gone with it.

GitHub secret scanning is free on public repositories only; private ones need an enterprise plan. Eleven private repositories had no scanning at all.

in=44175 out=4000, stop_reason max_tokens, and not one text block in the response. Extended thinking is on by default and produces no text until it finishes; raising the ceiling to 16,000 bought more thinking. Two sibling jobs had the same defect and got away with it on smaller inputs.

The split that holds: an edge container with no socket, no credentials, a read-only filesystem and an internal: true network, whose only verb is writing a file into a shared volume; a worker with the socket and no inbound path at all, not even on the edge's network. Each button carries a one-time nonce; using any button burns every nonce of that alert.

4.7.5 declares User=node and 4.7.7 declares User=1000:1000, and both resolve to uid 1000, gid 1000. In the same report ghost 6.64.0 to 6.65.0 differed in GHOST_VERSION and nothing else, and mc-backup 2026.9.1 to 2026.9.2 was identical in every field that changes behaviour.

MinIO's community server was archived in April 2026, and the newest image is the one already pinned: green for ever. MongoDB 8.0 before 8.0.30 refuses to start on kernels 6.19 through 7.0.13 (TCMalloc rseq), while GitHub runners are on 6.17, so CI would pass a bump a modern host rejects. And a database pinned at a major on purpose never "lags", so a major reaching end of support sits in a template with every check green.

Every image the fleet pins was resolved the way the reviewer resolves it and the answer asked of GitHub: 34 of 100 named a repository that is not there. Each produced a verdict about the lookup wearing the clothes of a verdict about the upgrade. Found through beszel, whose agent ships from the hub's repository.

The release notes carry the commit title and nothing else. The change does explain itself: a directory rename plus two lines of Dockerfile: base image steamrt/sniper/platform to steamrt/steamrt4/sdk, and the RCON forwarder simpleproxy to socat. Both inside the image; ports, variables and the compose interface unchanged.

The plugins it pins had no build for that version yet. Nothing in the stack could have known: the answer lives in the plugin registry's API, not in any container. Asked directly, the registry answers exactly: 11, 7 and 1 builds for the version that works, 0, 0 and 0 for one that does not.

The watcher's query filters by loader and not by game version, so the build it offers can target a release none of the servers run. The registry answers exactly: one version of the same plugin lists 26.2 and not 26.3, the next lists both. A pin the running version cannot load does not fail where it is written. It fails at container start, in ten backends that have already been recreated.

The SLSA generator attaches its provenance by editing the release, and GitHub answers that edit with Resource not accessible by integration for the workflow token when the release's target_commitish is a commit rather than a branch. Every release here cut from a verified commit targets that commit, on purpose. The step that only adds an asset had never been refused.

Checked from outside CI: a fresh anonymous token from quay.io gets 401 on every manifest of minio/minio, and hub.docker.com lists zero tags. The migration script starts a throwaway MinIO to read the old volume, so a user on a fresh machine had no way to run it either. Bitnami's frozen archive, bitnamilegacy/minio, still carries the last public build; tested here against a volume before it was pinned.