Production-tested. Partner links.
🛸 Proton VPN – Encrypted tunnel
🔐 Proton Pass – Zero-knowledge vault
🦑 GitKraken Pro – Visual Git client
🐧 Linux Foundation – Code: VALDEMAR
Evidence
What the green checks hid
For twelve days a team wiki's backup reported success while it saved none of the files the team had uploaded. Every check was green. This page is three of those moments, what was actually true, a restore measured on a machine that had never run the stack, and the system that catches the difference, including the mistakes of the AI agent that writes most of this code.
- repositories under one standard
- 88
- restore scripts, all run by CI
- 73of 73
- clean-machine restores measured
- 1
- releases tagged
- 830
Counted by fleet-ops, which rewrites the numbers when one changes. Last changed 2026-09-23 22:03 UTC.
Case one · a backup of nothing
12 days, 8 releases, and not one attachment in any backup
Data backup OK CI: data archive opens ✓
Release 2.0.0 moved Outline's file storage from MinIO to Garage. The backup kept archiving the old MinIO volume, which nothing writes to any more. The restore script looked for a service that no longer existed and would have stopped before doing anything.
A team restoring after a lost disk gets its documents back with every image and attachment missing. Every log line in those twelve days would have said OK.
An archive that opens proves the archive. CI now proves the backup the way a user would notice it missing: it stores a file through S3, takes a backup, stores a second file, restores, and requires the first to read back and the second to be gone. The restore uses Garage's own consistent metadata snapshot rather than a copy of a live database.
Case two · the restore nobody ran
72 restore scripts, some never run by CI since May 2021
test_restore_roundtrip PASS (every push, 47 templates)
Each test restored with its own copy of the commands. The scripts a person would run were never executed. Four were written for a different vendor's image, and one of those refused to run at all. One signed in with the wrong account. Several merged the backup into live data instead of replacing it.
The day you need a restore would have been the first day the script ran. In 16 of the 47 templates that script was more than three years old.
Twenty years of production infrastructure taught me one pattern above the rest: a backup that has never been restored is a belief. The tests now run the shipped script, and a fleet rule fails any repository whose restore script no test invokes. A comment or an echo naming the script does not count. Its list of exceptions can only shrink, and it has been empty since the day it was written.
Case three · the check that could not fail
17 hours healthy over a crashed server
docker inspect → healthy
A game server had crashed and dropped two players mid-round. The health check searched for the process with pgrep -f, which matches its own command line. It found itself, every time.
Time to detect was never. Docker does not restart an unhealthy container on its own, and this one was never reported unhealthy.
The rule it produced: a check whose pattern can match the check itself has never failed, so prove the red case against a real container before believing the green one. The fix shipped to every game-server template with that test, and every fleet rule since has been tested against a planted violation.
The question a CISO asks first
The host is gone. How long until it works again?
Every week, for each template below, one runner starts the previous release, writes a marker row into the database and a marker file into the data, and takes a backup. Only what an operator keeps off the host leaves it: the backup files and the environment file. A second runner that has never run the stack then brings the current release back from those files alone, with the scripts the template ships, and the clock stops when the application answers over HTTPS with both markers back. So the number below is an older backup restored into a newer release, on an empty machine, image downloads included.
Fastest 1 min 3 s · slowest 1 min 3 s
| Template | Backup taken on → restored into | Empty machine to answering | Of that, restoring | Last run |
|---|---|---|---|---|
| WordPress | v1.9.1 → v1.9.1-1-gb612bd5 | 1 min 3 s | 20 s | 2026-09-23 |
A drill that fails is listed as failing and carries no time. A restore time from a restore that did not work is not a time.
How the AI is run
The agent is fast, and it is wrong on a schedule. The operating model assumes both.
An AI agent does most of the typing across this fleet. That is the part anyone can buy now. What cannot be bought is the set of rules that decide what the agent's work is allowed to become.
- Nothing ships on a claim
- A release is cut only after the release commit itself passes a full deploy, backup and restore in CI. The agent's report that something works is not an input.
- A check that cannot fail is not a check
- Every fleet rule is exercised against a planted violation. A rule that stays quiet over its own violation fails the build.
- Exceptions only shrink
- Anything excused from a rule sits on a written list. An item that has since been fixed but is still listed is itself a failure, so the list cannot quietly become permanent.
- The agent does not hold the keys
- It never reads decrypted secrets and never changes a production server. A server is changed only by the agent that lives on it, from a written brief, and it reports back what it did.
- Its mistakes are published
- The public record of findings includes the agent's own, among them a cause it published before measuring it, and the correction.
Caught before release on 23 September 2026. Six mistakes by the agent in one day of work across 47 repositories. None of them reached a release.
| What the agent got wrong | What caught it | Proof |
|---|---|---|
| An apostrophe in a comment closed a shell quote, so Outline's backup loop never started. | CI waits for a "Data backup OK" line, not for a running container | fix |
| A bulk rename also renamed an unrelated variable in five test suites. | strict mode stopped every one of those runs | fix |
| A new restore script was committed without its executable bit, and the same test exported the environment file, so the stack rendered without one. | the test runs the shipped file, and the check that the stack must refuse to render | fix |
| A generated restore called a database client the image does not ship. | the first CI run of the shipped script | fix |
| Tests passed locally against a stale copy of the reference repository. | CI, which checks against the current one | private repository |
Green is a claim. A restore is evidence.