A Partial Read Is Not a Smaller Fleet: A GitHub API Rate Limit and the Numbers That Lied for an Hour

For one hour in the early morning of 26 September, UTC, my GitHub profile said 24 restore scripts across 15 repositories, and the evidence page on this site showed no clean-machine restores at all. The fleet had 73 scripts across 48 repositories and 47 restores measured that week. Every number came out of one partial read of the fleet by a job, was published by a job, was refreshed in the reader’s browser by a script, and every one of them was wrong in the same direction. I had designed each of those three steps so that no number on the page would ever be typed by a person.

A partial API read published as the whole is what happens when a job that reads ninety-seven repositories gets through thirty-seven, is refused on the rest, and writes what it has. The file it wrote was honest: it listed the sixty it could not read. It was still wrong, because every consumer downstream took the numbers and left the list. The same design decision that made the numbers trustworthy, that they are never typed, made them capable of shrinking by themselves.

What a partial read published as the whole looks like#

The catalog job reads every repository’s changelog, tests, workflows and drill results, and writes one JSON file the profile and the website read. The step that had the hole, before:

1
# scripts/fleet-catalog.py in fleet-ops, before the fix (e276c50^): read what can be read, write what was read.
2
# ev["unread"] lists the repositories the API refused, and nothing between the read and the write looks at it
3
ev = collect_evidence(names)
4
# ... the drills, the Scorecard results and the badges are read the same way
5
if a.json:
6
    print("json:", write_json(a.json, fleet_json(rows, ev, drills, sc, bp)))

The file carried an unread list with sixty names in it, which is the honest part. The profile line gave 24 restore scripts across 15 repositories, because that is what the thirty-seven it had read added up to, and then it even said why: 60 repositories could not be read today and are not counted.

The job, after, reads everything before it writes anything and refuses to write a file that is not whole:

1
# scripts/fleet-catalog.py in fleet-ops, after the fix (e276c50): a partial read is not a smaller fleet
2
UNREAD_LIMIT = 3
3

4
def refuse_partial(ev, drills, previous):
5
    """The reasons this run must not publish, or an empty list."""
6
    why = []
7
    if len(ev["unread"]) > UNREAD_LIMIT:
8
        why.append("%d repositories could not be read (%s%s); more than %d is a partial read, not a smaller fleet"
9
                   % (len(ev["unread"]), ", ".join(ev["unread"][:4]), " …" if len(ev["unread"]) > 4 else "", UNREAD_LIMIT))
10
    before = len((previous or {}).get("clean_machine_restores") or [])
11
    if before and len(drills) < before / 2:
12
        why.append("the clean-machine restores measured fell from %d to %d; a drill does not vanish overnight, a read does"
13
                   % (before, len(drills)))
14
    return why

A refusal is a red run; the heartbeat calls it a finding once the catalog has gone two days without a successful run, and yesterday’s whole numbers stay up until a run can replace them. Three repositories unread is the threshold because a single refused request happens on a normal morning; sixty does not.

Why this keeps happening#

The rate limit is documented and shared. GitHub’s REST API allows an authenticated user 5,000 requests per hour, and that limit is combined across every personal access token the same user holds. On the night of 26 September the fleet’s engine was moving from a private repository to a public one, and during the cutover hour both copies ran the same jobs, under tokens that all belonged to one account. Two catalog runs started within a minute of each other, a little before four in the morning UTC, one from each copy. The first read everything. The second, the public copy’s, met the limit after thirty-seven repositories and published what it had six minutes later.

The second reason is that “honest” and “correct” are different properties of a file, and only one of them survives a consumer. Ours said sixty were unread. The profile’s evidence line was rendered from the file and repeated that sentence. Then the same line gave the counts, and the counts were the sum of what was read. A reader would have had to hold both sentences in mind to understand that the second was not a measurement of the fleet. Every program downstream held only the second.

The third reason is that the consumers were built to trust the file, because trusting the file was the design. Our build reads it. Each visitor’s browser fetches it again on every page view and replaces the built numbers with fresher ones. A daily job copies it into the site’s repository so the HTML carries the day’s numbers, and that job refuses a file whose counts fall too far from the last copy’s: the repositories by more than a fifth, the release tags below ninety-five percent. The partial file still said 88 repositories, so the first rule would not have fired; the second would have, since the tags had fallen from 865 to 291. All three consumed the partial file. The browser replaced the whole numbers with the smaller ones for every visitor for an hour; the copy job, the one with the guard, was not due for three hours.

The guard was in the one place the partial file had not reached yet.

Risk and blast radius#

Direct exposure is the claim on the page. The evidence page exists so that a reader deciding whether to trust the fleet can check every number against a run. For an hour it showed no clean-machine restores, on a site whose thesis is that restores are measured weekly on a clean machine. A reader in that hour would have concluded, correctly from what they saw, that the claim was false.

Systemic exposure is every pipeline that reads a paginated or rate-limited source and publishes an aggregate. Cost dashboards that sum accounts, security dashboards that count findings across projects, inventory pages that count assets across regions: each one reads a list of things, and each one has a branch for “could not read this one” that logs and continues. The aggregate at the end of that branch is smaller than the truth by exactly the number of failures, and it carries no sign of it. A dashboard that dips overnight and recovers by morning is read as a blip. It was a partial read, and the blip is the only trace it leaves.

The same assumption is cheaper to make and more expensive to be wrong about when the number decides an action. In October an update roll on a host with live sessions counted the users on each server before recreating it, and skipped any server that had someone on it (a game host; the public Project Zomboid template carries the same fix). Three of the stacks did not answer the count. The roll read them as empty, and one of the three was a relay carrying ten worlds on a full evening; it was caught and stopped by hand on 15 October. The public template had the same branch in its weekly refresh: a server that did not answer the player query was read as empty, and recreated under whoever was on it. Now an unanswered count is a question mark, skipped like an occupied server, and a test plays the silent server that the old script would have recreated.

Unknown is not empty.

That hour is also told from the other side in the reviews that read nothing, where the same fleet’s model reviews were found judging upgrades whose release notes they had not read. Two ways for a pipeline to publish a confident document about nothing, in one month, on one fleet.

The third exposure is the cutover itself. A migration that runs the old and new copies side by side doubles every read for the duration, and the rate limit is per account, not per repository. The rehearsal had proven that the new copy produced the same reports as the old. It had not proven what happened when both copies asked at once.

The read side has a fix of its own. A GitHub App’s installation token has its own limit, 5,000 requests an hour and more as the repositories grow, and does not share a person’s budget; this fleet reads with personal tokens, which do. An App would have kept the two copies from starving each other. It would not have made a partial read whole on the day the App’s own limit runs out, which is why the guard sits at the writer and not at the token.

What it took to restore the number#

The numbers came back about an hour after that, the same night, from a whole run. Guards followed the same day, in three layers, because the first one to meet a partial file was the one nobody had thought to guard.

The catalog refuses to write, as above, with seven planted cases in its test suite, among them sixty unread and three, restores at none and at nearly all of the previous file, and a whole read passing. In the browser, the page keeps the numbers it was built with when the file it fetched is partial:

1
// FleetLive.astro on this site: a file that could not read more than a handful leaves the built numbers alone
2
if ((f.unread || []).length > 3) return;
3
if (!(f.clean_machine_restores || []).length) return;

And the daily copy refuses one, in the same terms, so a partial file that gets past the catalog cannot get into the site’s repository; its check is the third layer below.

Three layers for one file is not redundancy for its own sake. Each consumer had its own way of reading the file, so each had its own way of believing it.

Consumer	What it did with the partial file	What it does now
The catalog job	Wrote it	Refuses to write; a red run, and a heartbeat finding if it lasts two days
The visitor’s browser	Replaced the page’s numbers with it	Keeps the built numbers when the file is partial
The daily copy into the site	Was not due for three hours	Refuses it and opens an issue

The file is the contract. Each side of the contract has to enforce it.

Framework: a file that says what it could not read, refused at the writer, checked at every reader#

The three layers are a contract for the file, validation where it is written, and validation where it is read.

Layer 1: the file names what it could not read, and when it was measured (week 1)#

A file that is only an aggregate cannot be checked. The fleet’s file carries the list of repositories the run could not read, the time it was measured, and one row per drill with the run it came from, so a consumer can decide for itself whether the file is whole:

1
{
2
  "measured_at": "2026-09-26 15:11 UTC",
3
  "repositories": 88,
4
  "restore_scripts": 73,
5
  "restore_scripts_run": 73,
6
  "restore_repositories": 48,
7
  "unread": [],
8
  "clean_machine_restores": [
9
    {
10
      "repository": "gitea-traefik-letsencrypt-docker-compose",
11
      "from": "v1.8.3",
12
      "to": "v1.8.5",
13
      "ok": true,
14
      "seconds_total": 48,
15
      "run": "https://github.com/heyvaldemar/gitea-traefik-letsencrypt-docker-compose/actions/runs/36088411990"
16
    }
17
  ]
18
}

An unread list that is missing is worse than one that is long: it means the writer never asked.

Owner: the maintainer of the job that writes the file.

Layer 2: read everything before writing anything, and refuse a partial (week 2)#

The refusal function above, called after every read and before any write, so a refusal leaves every file exactly as the last whole run left it. Both thresholds are numbers with reasons: three unread because one refused request is a normal morning, half the restores because a drill does not vanish overnight and a read does. Data teams call this a volume check, the row count compared with the last load. The planted cases sit well inside and well outside the limits rather than on them, so a change that loosens a threshold a little would still pass; the next cases to plant are four unread, and twenty-three restores of forty-seven.

The refusal is only as good as the step that carries it, and the first version of this step lost it. The generator exited with an error and printed why, but its output ran through the tee command into a log, and a shell pipeline reports the exit code of its last command. On the afternoon of 15 October the API quota ran out partway through a read. The guard refused, printed twice that nothing was published, and the run went green. Nothing watching the fleet could tell the catalog had stopped. Sixteen minutes later the step set pipefail, and its own comment keeps the reason:

1
# .github/workflows/fleet-catalog.yml in fleet-ops, since 1d0e0a7, shortened: a refusal is the step's exit code, not tee's
2
- name: Rewrite both blocks
3
  id: gen
4
  env:
5
    GH_TOKEN: ${{ secrets.FLEET_PAT }}
6
  # pipefail: the generator exits 1 when it refuses to publish a partial
7
  # read, and without it tee's 0 was the step's, so the refusal of
8
  # 2026-10-15 16:11 (an exhausted API quota) ran green and nothing
9
  # watching the fleet could tell the catalog had stopped.
10
  run: |
11
    set -o pipefail
12
    python3 scripts/fleet-catalog.py --catalog catalog/README.md --summary profile/README.md --evidence profile/README.md --json catalog/fleet.json | tee /tmp/gen.txt
13
    # ... four lines that read which files changed from /tmp/gen.txt

A refusal nobody sees is a log line.

Owner: platform engineer.

Layer 3: every reader checks the contract, and the last reader opens an issue (week 3 onward)#

The browser guard above is one reader, and a coarser one: it keeps the built numbers when more than three repositories are unread or no restores came back at all. Last comes the copy job, and its check is the writer’s contract in the writer’s words:

1
# fleet-numbers.yml on this site: the copy refuses a file that does not look whole, and says why
2
need(len(new.get("unread") or []) <= 3, "%d repositories unread: a partial read" % len(new.get("unread") or []))
3
need(len(new["clean_machine_restores"]) >= len(old.get("clean_machine_restores") or []) / 2,
4
     "the clean-machine restores fell below half of the last copy")
5
if new.get("unread"):
6
    print("::warning::fleet-ops could not read %s; their numbers are missing from this file" % ", ".join(new["unread"]))

Its failure is the finding, so it opens one issue on the first refusal, comments on repeats, and closes it on recovery; all three paths were planted on a throwaway branch before the job was merged, because a red run in a private repository with mail off reaches nobody by default.

Owner: SRE, who reads that issue.

Tradeoffs#

The guards cost a morning when the file is refused: the page keeps yesterday’s numbers, the run goes red, and a person decides whether the refusal was a rate limit or a real shrink. They also cost the ability to publish a smaller fleet on purpose. If twenty repositories are archived in one afternoon, the copy job refuses the file that says so, and the threshold has to be raised by hand for that day. That is the right cost. A fleet shrinking by a fifth in a day is a thing a person should have to confirm.

Not guarding costs an hour, or a night, of numbers that are arithmetically correct about a fleet that does not exist, on the page whose purpose is to be believed.

A day of yesterday’s numbers, against an hour of wrong ones. Yesterday’s numbers were at least true.

The closing argument#

A partial read is not a smaller fleet. A file that says what it could not read is honest and still wrong, because every consumer takes the numbers and leaves the list. Read everything before writing anything, refuse to publish a file that is not whole, and make every reader check the same contract, because the first consumer to meet a partial file will be the one you did not guard. The numbers on this site are never typed. That is exactly why they need a rule for the morning the source cannot be read.

Sources#

GitHub, Rate limits for the REST API (5,000 requests per hour, combined across a user’s personal access tokens), accessed September 2026
heyvaldemar.com, Decisions (decision 15: no number on the site or the profile is typed) and Ledger (an honest partial count, published as the fleet)
heyvaldemar, fleet-ops (the catalog, the refuse-partial guard and its seven planted cases), the refusal that ran green and the fix that made it red, 15 October 2026
heyvaldemar, catalog (the published file the site reads)
heyvaldemar, the Project Zomboid server template, the release that stops recreating a server it cannot count and its verification run, 18 October 2026

Discussion#

If a rate limit has ever shrunk a number on one of your dashboards, or you guard aggregates over partial reads some other way, drop a comment below. Counterarguments are welcome and the comment thread is where I respond first. For longer back-and-forth with senior practitioners, join the discussion on Discord.

Vladimir Mikhalev

Docker Captain · CNCF Ambassador · IBM Champion

The Verdict — production-tested analysis on YouTube.

YouTube GitHub LinkedIn