What the green checks hid

Case one · a backup of nothing

12 days, 8 releases, and not one attachment in any backup

Data backup OK
CI: data archive opens    ✓

An archive that opens proves the archive. CI now proves the backup the way a user would notice it missing: it stores a file through S3, takes a backup, stores a second file, restores, and requires the first to read back and the second to be gone. The restore uses Garage's own consistent metadata snapshot rather than a copy of a live database.

• The fix, and what the loop had been archiving
• v2.0.0, 11 September
• v2.1.0, 23 September

Case two · the restore nobody ran

72 restore scripts, some never run by CI since May 2021

test_restore_roundtrip   PASS
(every push, 47 templates)

A decade of production infrastructure taught me one pattern above the rest: a backup that has never been restored is a belief. The tests now run the shipped script, and a fleet rule fails any repository whose restore script no test invokes. A comment or an echo naming the script does not count. Its list of exceptions can only shrink, and it has been empty since the day it was written.

• The WordPress script before and after

Case three · the check that could not fail

17 hours healthy over a crashed server

docker inspect → healthy

The rule it produced: a check whose pattern can match the check itself has never failed, so prove the red case against a real container before believing the green one. The fix shipped to every game-server template with that test, and every fleet rule since has been tested against a planted violation.

• The test that kills the game and requires the check to go red

The question a CISO asks first

The host is gone. How long until it works again?

Every week, for each template below, one runner starts the previous release, writes a marker row into the database and a marker file into the data, and takes a backup. Only what an operator keeps off the host leaves it: the backup files and the environment file. A second runner that has never run the stack then brings the current release back from those files alone, with the scripts the template ships, and the clock stops when the application answers over HTTPS with both markers back. So the number below is an older backup restored into a newer release, on an empty machine, image downloads included.

Fastest 17 s · slowest 6 min 12 s

A drill that fails is listed as failing and carries no time. A restore time from a restore that did not work is not a time.

In the words of a vendor questionnaire. The recovery time objective is the measured column, not a target somebody wrote down. The recovery point is the backup interval each template ships with, read from the file an operator copies; shortening it is one variable. The date of the last recovery test is the last column, and while the drill stays green it is never more than a week old.

Scored by someone else

OpenSSF Scorecard, across the whole fleet, low marks included

Scorecard runs its own checks against every public repository here and publishes the result. The fleet reads that result each morning and repeats it, rather than awarding itself a number. Median 7.3 of 10 across 100 repositories; the lowest is 6.5, fleet-ops. Below, every check that is under ten somewhere, with how many repositories and what that means here. A check Scorecard marks as not applicable is not counted.

Read from Scorecard's own API, results dated 2026-10-13. The decision to leave Code-Review at zero, and why, is decision 16.

Answered by me, published by them

OpenSSF Best Practices, with every answer public and the short ones named

The badge is a questionnaire, not a scan: sixty-seven criteria, each answered Met, Unmet or N/A with a sentence saying why, and OpenSSF publishes the answers beside the badge. A badge is only as honest as its answers, so each one here names the workflow or file that makes it true, and a repository with no test suite says Unmet where it is unmet. Passing on 90 of 90 registered repositories.

Read from the badge site's own API, answers last changed 2026-09-27; 0 could not be read at the last count and are in neither number. The 10 public repositories that hold no code to rate are not registered. Why a criterion that is unmet is answered Unmet rather than Met is decision 18.

Threat model

What an attacker, or an ordinary Tuesday, would try, and what answers

A control that is not tied to a threat is a feature. Each row names the failure first, then the control, then where the control is proven to work, which for a public template is a file a reader can open.

How the AI is run

The agent is fast, and it is wrong on a schedule. The operating model assumes both.

An AI agent does most of the typing across this fleet, and nothing it says about its own work counts as evidence. The typing is the part anyone can buy now. What cannot be bought is the set of rules that decide what the agent's work is allowed to become.

Nothing ships on a claim

A release is cut only after the release commit itself passes a full deploy, backup and restore in CI. The agent's report that something works is not an input.

A check that cannot fail is not a check

Every fleet rule is exercised against a planted violation. A rule that stays quiet over its own violation fails the build.

Exceptions only shrink

Anything excused from a rule sits on a written list. An item that has since been fixed but is still listed is itself a failure, so the list cannot quietly become permanent.

The agent does not hold the keys

It never reads decrypted secrets and never changes a production server. A server is changed only by the agent that lives on it, from a written brief, and it reports back what it did.

Its mistakes are published

The public record of findings includes the agent's own, among them a cause it published before measuring it, and the correction.

Caught before release on 23 September 2026. Five mistakes by the agent in one day of work across 47 repositories. None of them reached a release.