Architecture decisions
Decisions, and what each one cost
An architecture is the sum of its decisions, and a decision without its cost is a slogan. These are the ones behind the fleet, dated from the commit that made each, with the alternative refused and the price accepted. Where the evidence is public it is linked. Where it lives in a private repository, the entry says so instead of pointing at a description.
17 decisions. The findings that forced most of them are on the ledger; the numbers they produced are on the evidence page, read from the fleet daily.
-
One host, Compose, Traefik in front, a backup sidecar beside every database
ContextThe reader of these templates has one machine, no platform team, and a service to stand up this afternoon. Kubernetes answers a question they do not have and adds a control plane they would have to run. The upstream project's own compose file usually answers a different question again: it gets the developer running, with a floating tag and no certificate.
DecisionEvery template is one compose file: the application, its database, Traefik terminating TLS from Let's Encrypt, and a sidecar that takes scheduled backups of the database and the data and prunes them. Five or six moving parts, and the same shape from template to template, so an operator who has run one has run them all.
What it costNo horizontal scaling and no rolling upgrade. A template that outgrows one host outgrows this fleet, and says so in its README rather than pretending otherwise.
-
Upstream images are pinned by digest, and a daily job goes red when upstream moves
ContextA tag is a promise made by someone else.
stablemoved under three templates in one week, and a deployment that worked on Monday pulled something different on Thursday with the same file. The alternative, floating tags with a note to check upstream, is what every other example repository ships.DecisionEvery upstream image is written as
tag@sha256:digest, as an interpolation default in the compose file, so a plaingit pulldelivers the exact combination CI tested. Dependabot moves the digest through a pull request. A separate daily job re-resolves each pin against the registry and fails when one has drifted.What it costA red run for every upstream release, every day until the pin moves. Fourteen days measured across the fleet: nine red runs in ten were this alarm, which is what forced the decision of 23 September below.
-
GitHub Actions are pinned by commit SHA, and Dependabot's updates arrive as one pull request per repository per week
ContextAn action referenced by tag can be replaced under that tag by whoever holds the upstream repository, and it runs with this repository's token. Pinning by SHA closes that and opens another problem: a pinned SHA never moves on its own, so it goes stale in silence. Ungrouped, a week in which four actions publish is four pull requests per repository, and across ninety repositories that is a queue nobody reads to the end.
DecisionEvery
uses:carries a full commit SHA with the version in a comment. Dependabot groups all action updates into one pull request per repository per week.What it costA queue that still has to be read. That is the decision of 6 September.
-
A backup is proven by a restore, in CI, on every push
ContextEvery template had taken backups for years. Whether the archive contained what an operator would need on the day was a belief:
tarexits 0 over a directory that is missing half its files, and a database dump that opens is not a database dump that restores.DecisionThe end-to-end test writes a marker into the database and a marker file into the data, waits for a real backup cycle, destroys the live state, restores, and requires both markers back over HTTPS. It runs on every push and once a day on a schedule.
What it costDeployment Verification grew from a boot check to a twenty-minute job, and the backup sidecar had to learn to tolerate files changing under it while it archived. Both were worth it on 11 September, when a backup that reported OK for twelve days turned out to contain none of the team's attachments.
-
Dependabot merges itself only after the repository's own verification is green on that commit, and not through GitHub's auto-merge
Contextgh pr merge --autohands the decision to GitHub, which waits for the checks a branch rule marks as required. Where no rule marks any, nothing is required, and auto-merge merges immediately, before the verification it was meant to wait for. That is exactly the state of a fleet that refuses required checks on purpose.DecisionA workflow triggered by the completion of the repository's own verification merges the Dependabot pull request if, and only if, that run concluded success for that commit. Its write permissions sit on the one job that merges, not at the top of the file.
What it costA pull request Dependabot opens while verification is broken stays open, by design, and the heartbeat reports it after 24 hours. Two such requests sat for four and seven days in a private repository before the heartbeat learned to look there.
-
Every check ships with the test that plants its violation
ContextA game server crashed and Docker reported it healthy for seventeen hours. The health check searched for the process with
pgrep -f, which matches its own command line, so it had never failed and never could. A check that cannot go red is a green light wired to the wall.DecisionA check is not done until a test has made it fail against a real container or a planted violation and seen it go red. Every fleet rule since carries such a test, and a rule that stays quiet over its own violation fails the build.
What it costEvery check is two pieces of work, and the second is the one that gets skipped under time pressure. It is the reason the ledger records the fleet's own tests being wrong: on 24 September a fake in a test failed the way the real function never does, and 116 green tests hid a crash.
-
The agent's model credentials come from workload identity federation; no API key exists anywhere
ContextThree jobs in the operations repository ask a model for judgement: what upstream changed between two versions, what is worth adding to the fleet, and what the home servers learned this week. An API key in a repository secret is a key that any workflow in that repository, present or future, can spend, and that a leaked log can spend from anywhere.
DecisionThe runner mints a GitHub OIDC token for the job and the model provider exchanges it under a federation rule pinned to this repository and branch. The only thing that can spend these tokens is a workflow in that repository on that branch. The ids live in one module that all three jobs import, so two copies cannot drift.
What it costThe jobs cannot be run from a laptop. Reviewing upstream by hand means dispatching the workflow and reading its output, which is slower and is the point.
- fleet-ops, private; the workflow logs are quoted in the ledger where a run mattered
-
The scout may propose a template. It may not create one
ContextReading a catalogue of self-hosted software and generating a compose file for each entry is a week's work for an agent, and the result would be ninety repositories nobody had run. The value of a template here is what running it taught, and that cannot be generated.
DecisionA weekly job reads the catalogue, drops what the fleet already covers and what cannot be pinned or maintained, asks the model to assess what is left against how these templates are actually built, and opens one issue. A repository is created only after a decision, by a person, and the scaffold comes after that.
What it costGrowth is bounded by reading. The fleet gets slower to grow than an agent could make it, and every repository in it has been run.
- fleet-ops, private; its issues are the shortlist
-
The watcher measures whether a schedule fired, not whether it passed, and is itself watched from outside GitHub
ContextGitHub disables a scheduled workflow after sixty days without a push. No run fails, no badge turns red, and the last green run stays on the README. A daily job that fails daily looks, to a check that counts successes, exactly like a job that stopped.
DecisionA daily heartbeat reads every workflow's own cron, asks GitHub when it last fired, and reports one that has not fired inside its own interval or that GitHub has switched off. Whether it passed is a different question with a different report. The heartbeat's own run pings a switch outside GitHub that alarms on a missing ping, so GitHub not running the heartbeat is noticed by something that is not GitHub.
What it costTwo false alarms in ten days, each of which became a rule: the order GitHub returns runs in is not a contract, and a cron changed since its last run has not missed anything yet.
-
Findings go to one issue, rewritten in place. Nobody is mailed for an age that grew by itself
ContextActions mail is switched off across the fleet: at ninety repositories it is a hundred messages a week that say the same thing, and a hundred messages nobody reads are the same as none. A report that opens a new issue every day is a report that gets muted the same way.
DecisionEach watcher keeps exactly one open issue, updates its body when the findings change, comments once when they change, and closes it when they clear. A finding that is the same as yesterday's, with a larger number of hours in it, is not a change and sends nothing. A finding that is true, filed, and not going to change today is listed as standing, and does not turn the run red.
What it costThe issue has to be read. A list of accepted gaps sits beside it and may only shrink: a repository that no longer needs accepting and is still listed is itself a finding, so the list cannot become permanent by neglect.
- fleet-ops, private; the issue bodies are what the watchers write
-
A second, read-only credential rather than a wider one
ContextThe heartbeat could not read eleven private repositories, and in one of them two Dependabot pull requests had sat four and seven days, one of them green and mergeable the whole time. The quick fix was to add those repositories to the token the fleet already held.
DecisionA new fine-grained token with one permission, pull requests read-only, on all repositories, used by the one function that needs it. The existing token was not widened.
What it costOne more secret to rotate. Against it: adding a repository to a token grants that repository every permission the token already holds, and the existing one can write.
- fleet-ops, private; the reasoning is the header of its accepted-gaps list, which has been empty since
-
CI runs the restore script the operator would run, not a copy of its commands
ContextThe end-to-end test restored with its own commands, and passed. The scripts a person would run on the day were never executed by anything. Four had been written for a different vendor's image, one refused to run, one signed in with the wrong account, and several merged the backup into live data instead of replacing it. In sixteen templates the script was more than three years old.
DecisionThe test invokes the shipped script. A fleet rule fails any repository with a restore script that no test invokes, and a comment or an echo naming it does not count. Its list of exceptions may only shrink and has been empty since the day it was written.
What it costForty-seven templates' scripts rewritten in one day, with the mistakes that come with a day like that, six of them caught before release and listed on the evidence page.
-
The freshness alarm moves out of the workflow whose badge is on the README
ContextPin Freshness ran inside Deployment Verification. It is a designed alarm and goes red when a pin is one version behind, which triage fixes within the day. Measured over fourteen days across the fleet, nine red runs in ten were this alarm and nothing else, and a visitor cannot tell one version behind from does not boot.
DecisionThe job is unchanged and moves to its own workflow, with its own name. The badge on the README now means one thing: whether the stack boots, deploys, backs up and restores.
What it costOne more workflow per template, and a rollout across the fleet by script. The alternative, muting the alarm, was refused.
-
Recovery time is measured weekly on a machine that has never run the stack, from the previous release into the current one
ContextA restore test that runs on the host that took the backup answers an easier question than the one a security questionnaire asks. The questionnaire asks how long until it works again after the host is gone, and whether the backup taken under last month's release restores into this month's.
DecisionOne runner starts the previous release, writes markers, takes a backup, and exports only what an operator keeps off the host: the backup files and the environment file. A second runner that has never run the stack brings the current release back from those files alone, using the scripts the template ships. The clock stops when the application answers over HTTPS with both markers back. A drill that fails is listed as failing and carries no time.
What it costAbout thirty minutes of runner time per template per week, and a drill that took five attempts to make honest: it measured the wrong release pair, raced its own backup loop, and once accepted a backup taken before its markers. Each attempt is in the ledger.
-
No number on this site or the profile is typed. They are read from the fleet, daily, and the copy's failure opens an issue
ContextThe profile said 72 restore scripts across 47 templates on the day that was true, and would have said it for a year. A number a person typed is a number that is wrong from the next release on.
DecisionThe fleet recounts itself every morning and publishes one JSON file. The profile's evidence line, this site's evidence and ledger pages, and the strip on the front page read that file; the site's build reads it, and the visitor's browser reads it again. A daily job copies it into the site so the HTML carries today's numbers too, refuses a file that is not whole, and opens an issue when it refuses, because a red run in a private repository with mail off reaches nobody.
What it costA page that cannot show a number the fleet has not measured, which is the point.
-
Branch rules block force-push and deletion, and nothing else
ContextOpenSSF Scorecard, read across all 97 public repositories, scored Branch-Protection 0 on 91 of them: main could be rewritten or deleted by anyone holding the token, and nothing would have noticed until a clone came back different. The full answer, required reviews and required checks, assumes a second reviewer this fleet does not have, and a rule that would be bypassed on day one is worse than none.
DecisionOne ruleset on every default branch: no force-push, no deletion. The heartbeat asks every repository for both rules from now on, and for a SECURITY.md, which nine repositories around the templates were missing.
What it costScorecard's Code-Review check stays at zero, and the evidence page says so beside the score rather than hiding the check. The review here is CI against planted violations, not a second person, and a reader deciding whether to trust the fleet is owed that fact.
-
A release is an archive, a keyless signature and SLSA provenance, not a tag and its notes
ContextA tag is a name, and a name can be moved. Across 833 tags nothing a person deployed could be checked against anything: no archive, no signature, no statement of how it was built. Scorecard marked Signed-Releases as not applicable for the whole fleet, which is the polite form of zero. The keyed alternative, a GPG key in a repository secret, is a key every workflow in that repository can use and a leaked log can use from anywhere.
DecisionOn every published release a workflow makes a
git archiveof exactly the tree the tag points at, signs it with Sigstore keyless signing under the workflow's own short-lived identity, verifies that signature on the runner before uploading anything, and has the SLSA generator produce build provenance as.intoto.jsonl. The README says how to verify all three with nothing from the repository trusted. The same workflow, started by hand with a tag, gave the last five releases of every repository the same files; their provenance names the branch the run started from, and the README says so.What it costThree more assets per release, one more workflow per repository, and one reusable workflow referenced by tag, because the generator halts when referenced by commit: Scorecard's Pinned-Dependencies dropped from 10 to 8 on every repository that carries it, the same afternoon, and stays there. The median still rose, 6.8 to 7.2, because Signed-Releases went from not applicable to 10. The heartbeat asks every repository that has a release for the workflow from now on.
A decision you cannot date is a preference.