AI Reviews That Read Nothing — Thirty-Four Verdicts in a Hundred

What an AI review of nothing looks like#

Before any version bump moves, a model reads the upstream release notes against the compose file that pins the image and answers on one line: SAFE TO APPLY, NEEDS ATTENTION, or DO NOT APPLY UNATTENDED. To read the notes it needs a repository, so something has to turn an image name into one. That something was a short table of exceptions, and past the table, convention.

1
# review-source.sh: the convention past the table, image name in, repository out
2
image="${1#docker.io/}"
3
case "$image" in
4
  ghcr.io/*|quay.io/*|lscr.io/*) echo "none" ;;   # these registries name no GitHub owner
5
  */*) echo "$image" ;;   # henrygd/beszel-agent becomes henrygd/beszel-agent
6
  *) echo "none" ;;       # an official image has no owner to guess
7
esac

Convention is right often enough to look right, and it fails in the direction that produces an answer rather than an error. Renovate and Dependabot do not guess: they read the address the publisher wrote into the image, the standard source label in its configuration, and keep a list of exceptions behind it. That is the right order, and it is not an answer either. I read that label from the registry for the images in this fleet’s table. Thirteen named the right repository. Thirty carried no label at all. Five pointed at the repository that builds the image rather than the one that writes the software, which is the same wrong project as a redirect, written down by the publisher. A label is a claim. Something still has to check it.

The beszel agent is published under an image name that looks like a repository of its own, and no repository has that name. The agent ships from the hub’s own repository. Dify publishes four images from one repository and moves them together, so on 10 September a single Dify release produced four reviews, four not-found answers, and four verdicts of NEEDS ATTENTION that carried no information about the upgrade at all.

The fix is to stop trusting any name, guessed or declared, and prove it with the one fact the fleet already holds: the version it pins.

1
#!/usr/bin/env bash
2
# verify-mapping.sh: a mapping is accepted only when the repository exists
3
# under exactly that name and carries the version the fleet pins, as a tag.
4
# Exit 0 proven, 1 disproven, 2 GitHub did not answer, which is no verdict.
5
set -euo pipefail
6
repo="$1"
7
pin="$2"
8
api="https://api.github.com/repos/$repo"
9
auth=(-H "Authorization: Bearer ${GITHUB_TOKEN:?}")
10
lower() { tr '[:upper:]' '[:lower:]' <<<"$1"; }
11
body=$(mktemp)
12
trap 'rm -f "$body"' EXIT
13

14
code=$(curl -sSL -o "$body" -w '%{http_code}' "${auth[@]}" "$api" || true)   # a transport failure prints 000
15
case "$code" in
16
  200) ;;
17
  404) echo "no repository at $repo"; exit 1 ;;
18
  *) echo "GitHub answered $code for $repo: no verdict"; exit 2 ;;
19
esac
20
name=$(jq -r '.full_name' "$body")
21
if [ "$(lower "$name")" != "$(lower "$repo")" ]; then
22
  echo "$repo redirects to $name: map to $name only if that is the project"
23
  exit 1
24
fi
25

26
for tag in "$pin" "v$pin"; do
27
  code=$(curl -sS -o /dev/null -w '%{http_code}' "${auth[@]}" "$api/git/ref/tags/$tag" || true)
28
  case "$code" in
29
    200) echo "proven: $repo carries $tag"; exit 0 ;;
30
    404) ;;
31
    *) echo "GitHub answered $code for $repo tag $tag: no verdict"; exit 2 ;;
32
  esac
33
done
34
echo "$repo exists but has no tag for $pin: mapping unproven"
35
exit 1

A repository that answers and carries your version is evidence. A repository that merely answers is a coincidence, and coincidences scale. A GitHub that does not say yes or no, on a rate limit, a dead token or no network at all, has said nothing, so the script exits 2 rather than report an absence it never saw. Ask GitHub for a repository called mysql under the mysql account and it answers, through a redirect, with one called mysql-docker: a real project, the packaging and not the database. It was one of four plausible names that failed the tag test and never went in. A mapping that points at the wrong project is worse than none. It produces confident notes about somebody else’s software.

I have been writing glue like this since it resolved hostnames instead of source trees, and this failure has never once announced itself where anyone was looking.

The table has a history, and the history is the finding. The first fix, on 10 September, was twenty-six mappings written by hand, and the commit that added them said every image whose notes live elsewhere now had an entry. Ten days later, with eleven more images in the fleet, the sweep found thirty-four of a hundred pointing at nothing, so at least twenty-three of them had been pointing at nothing on the day the list was called complete. The thirty-four came apart into three kinds. Ten were missing mappings, each accepted only after the pinned version turned up as a tag. Seven were publishers that do not use GitHub at all, moved to a table of their own with the reason beside each. The rest were official images and registries that name no owner, for which convention had never produced an address in the first place. The first fix was a list. The second came with a test, and with a fourth answer: not found is a finding about the lookup, and the record now says so in a field of its own, beside a top line that still has only three values to choose from. That top line is the next thing to change.

Why this keeps happening#

Four mechanisms stack, and each one turns a missing thing into an ordinary-looking thing.

A missing address answers like an empty one. GitHub returns 404 for a repository that does not exist and, by its own documentation, the same 404 for a private one your credential cannot see, so that it never confirms a private repository exists. The reviewer’s fetch failed, the prompt said the notes were not available, and the model did exactly what it was told. It said so, and listed what to check by hand. A publisher that does not use GitHub at all is a third answer. Atlassian, Microsoft’s SQL Server, GitLab Runner and Forgejo publish their notes elsewhere, and the fleet now says so from a table instead of reporting a failure it was always going to have. A rate limit is a fourth, and it is the one that publishes a smaller world. On 26 September the fleet’s own catalog was refused a third of the way through and published what it had. For an hour that was a fleet with a third of its restore scripts.

Nobody opens the envelope. A weekly job asks a model which of the week’s commits carry a rule worth keeping, and opens an issue with the answer. One week the issue arrived with its table of commits and nothing said about any of them. The response had come back over HTTP 200 with a stop reason of max tokens, 44,175 tokens in, 4,000 out, and not one text block. On the model it uses, Claude Sonnet 5, thinking is on by default, and Anthropic’s documentation is explicit that the token ceiling covers thinking and answer combined. The whole budget went into reasoning, and the answer never started. Joining an empty list of text blocks yields an empty string, which prints, evaluates as false, and writes to a file without complaint. The stop details field, the natural place to look, is filled in only for a refusal; for a truncated answer it is null. The fix was not a bigger ceiling. The job turns thinking off, raises when no text comes back, and says in the report itself when an answer was cut short.

The reviewer reads a description of the software, not the software. On 22 September dashy’s release notes said the default user was changing from node to 1000, and the review called it NEEDS ATTENTION, correctly from the notes alone: a changed container user over a bind-mounted file is how an upgrade breaks quietly. The registry disagreed. Version 4.7.5 declares the user called node, 4.7.7 declares 1000 for user and group, and both resolve to the same user and group. A rename, with nothing to change anywhere. A release note is a sentence somebody wrote. The image is the thing that runs. Since that night the review receives both image configurations from the registry, and where they disagree with the notes about the user, ports, volumes, entrypoint or health check, the configurations win. Silence is a description too: a game server’s release arrived marked breaking with no notes at all, and its diff was two lines of Dockerfile plus a new variable that decides whether the spectator relay forwards player voice. The marker said breaking. The diff said privacy.

An automation reads a summary instead of the jobs. The job that refreshes pinned image digests pushed a change, read its CI, and reverted it. The change was good. The stack came up, the HTTPS smoke test passed, all three Trivy scans passed. The only red job was the freshness check, a designed alarm that goes red whenever any pin in the repository lags upstream and says nothing about whether the digest just written boots. The automation read the run’s overall conclusion, and a conclusion cannot tell a designed alarm from a failure; over fourteen days, 254 of 281 red runs across the fleet had no failing job except that alarm. A model in the triage seat would have made the revert eloquent. It would not have made it right.

Four failures, one shape: a lookup, an envelope, a description and a summary, each standing in for the thing itself. Published research has the same shape from the model’s side. Google’s work on retrieval found that frontier models, handed insufficient context, answer wrong rather than abstain. Atlassian measured its production reviewer and found 22 percent of comments not grounded in what the reviewer was given. In August a team at Ionix watched their reviewer lose read access to the repository and keep producing confident, well-formatted reviews of code it had never read a byte of. None of those counted what this fleet counted, a verdict whose source was never fetched, and every one of them points the same way.

A pipeline that cannot tell an empty answer from no answer will eventually publish the second one.

Risk and blast radius#

Direct exposure is the verdict somebody acts on. A third of the reviews told a person to look hard at an upgrade nobody had read, which is how a human was called in for beszel on 20 September. Raise the alarm on a third of all bumps and everyone downstream learns that the alarm means nothing, which is the lesson a review exists to prevent.

Systemic exposure is the gate. The review runs in front of every version bump of every image the fleet pins, a hundred of them that week, and one resolver fed every one of those reviews. A defect in a gate is the same wrong decision applied to everything that passes through it, at machine speed, with a paper trail that looks like diligence. This fleet is one maintainer’s. Put the same resolver in front of a thousand engineers and the failure keeps its shape. What grows is the number of people who learn to ignore it.

The code in this fleet comes from the same kind of machine, and the rule holds for it too: nothing an AI agent says about its own work counts as evidence. A fleet conformance checker was patched, re-run, and reported every repository meeting the standard, and the next day a fresh clone showed the patch had never been pushed. The clean number came from code that existed nowhere. The evidence page keeps the agent’s mistakes beside what caught each one.

Assurance exposure is the one that reaches the board, and it is quiet. SOC 2’s change management criterion asks whether changes are documented, assessed and approved through a structured process with version control. ISO 27001’s change management control asks whether changes to information systems are planned, assessed, authorized, tested and documented. Both are satisfied by an assessment that happened. Neither asks what the assessment was holding, and no change record I have audited has had a required field for the inputs of the assessment, only for its outcome. An AI verdict pasted into a change record satisfies every word of both while describing an upgrade it never read. OWASP’s Top 10 for LLM applications files this under misinformation, its ninth entry, and names the cure as grounding and verification, which is a standard a CISO can cite and a vendor cannot wave away. Documents rot the same way. Every security policy in this fleet told a reporter to fetch an encryption key from a page that did not exist, in ninety-seven repositories, from April until 25 September, and none of the checks that read the fleet ever opened the link.

Since 24 September every release in this fleet carries a git archive, a keyless signature and SLSA provenance, a statement of what was built from what, which anyone can check without trusting the repository. The review of the version bump inside that same release is a paragraph in its notes. We learned to ask a binary where it came from. Ask the verdict what it read.

The shape of the answer already exists. Provenance has a subject, the thing attested; a predicate, what was done to it and from what; and a builder. A verdict has the same three parts: the image at a version, the documents the reviewer read with a hash of each, and the model, its version, its token counts and how it stopped. Write the record in that shape and a verdict becomes something a second system can check, not a paragraph a person has to believe. Nothing on the market publishes it yet. Dependency bots attach the release notes they found; no AI reviewer I have evaluated attaches what it read.

The same question travels to the review of a merge request without changing a word. Which files were in the context window, and was the diff cut to fit it? A reviewer that cannot say has reviewed the part of the change it happened to see.

So the question to put to any automated review, yours or a vendor’s, stops being how good the model is. The questions about confidentiality, what leaves your perimeter and for how long, you ask already. These five are the ones nobody asks, and a team that buys a reviewer instead of building one has nothing else to hold it with:

• Which document did it read for this verdict, and at which version?
• Show me the last ten verdicts whose source was missing, and their top line.
• In the record, not the logs: can it tell a refused answer, a cut-off answer and an empty one apart?
• What can the verdict change by itself, with nothing but its own word behind it?
• Whose credentials does it spend, for how long are they valid, and could anything outside its pipeline spend them?

A system that can answer all five in one click has a reviewer. A system that cannot has a writer.

The failure modes, by what the reviewer was holding#

Sorted by what the reviewer was holding when it ruled, the failures above come to nine: nothing, the wrong project, a blank, a description, silence, a third of the fleet, a summary, another workflow’s answer, and a local copy of code nobody had pushed. Each one is a dated finding on the ledger, with the rule it produced.

Framework: prove the source, open the envelope, bound the verdict#

1
# tests/test-review-sources.sh: a lookup that cannot say no proves nothing
2
rc=0
3
./verify-mapping.sh henrygd/beszel-agent 0.20.0 || rc=$?
4
if [ "$rc" -ne 1 ]; then
5
  echo "FAIL: a missing repository came back as exit $rc, not as absent"
6
  exit 1
7
fi
8
./verify-mapping.sh henrygd/beszel 0.20.0

1
# review.py: the envelope decides whether there is an answer at all
2
import anthropic
3

4
client = anthropic.Anthropic()
5

6

7
def answer(prompt: str) -> str:
8
    msg = client.messages.create(
9
        model="claude-sonnet-5",
10
        max_tokens=6000,
11
        thinking={"type": "disabled"},  # three verdicts need no reasoning pass
12
        messages=[{"role": "user", "content": prompt}],
13
    )
14
    if msg.stop_reason == "refusal":
15
        raise RuntimeError(f"refused: {msg.stop_details}")
16
    text = "".join(b.text for b in msg.content if b.type == "text").strip()
17
    if not text:
18
        raise RuntimeError(
19
            f"no answer: stop_reason={msg.stop_reason}, "
20
            f"in={msg.usage.input_tokens}, out={msg.usage.output_tokens}"
21
        )
22
    if msg.stop_reason == "max_tokens":
23
        text += "\n\n(Cut off at the token ceiling. Nothing below this was assessed.)"
24
    return text

1
# registry-user.sh: what the image declares, read from the registry, nothing pulled
2
for v in 4.7.5 4.7.7; do
3
  printf '%s: ' "$v"
4
  docker buildx imagetools inspect "lissy93/dashy:$v" \
5
    --format '{{ (index .Image "linux/amd64").Config.User }}'
6
  echo
7
done

1
# chatops-privilege-wall: the half that listens cannot act
2
services:
3
  edge:
4
    image: python:3.13-alpine
5
    read_only: true
6
    cap_drop: [ALL]
7
    networks: [edge-network]   # Traefik joins this network; nothing else does
8
    volumes:
9
      - wall-queue:/queue      # its one verb: write a request file
10

11
  worker:
12
    image: python:3.13-alpine
13
    read_only: true
14
    cap_drop: [ALL]
15
    networks: [worker-network] # no ports, no router, not on the edge's network
16
    volumes:
17
      - wall-queue:/queue
18
      - /var/run/docker.sock:/var/run/docker.sock:ro
19

20
networks:
21
  edge-network:
22
    internal: true
23
  worker-network: {}
24

25
volumes:
26
  wall-queue: {}

Tradeoffs#

The cost is two or three GitHub calls per pinned image per run, six to eight registry requests per reviewed image, a seven-line table somebody keeps honest, a dozen lines of envelope checking, and a second container in a stack that had one. None of it is free, and the table is the part that rots. Ours rotted in ten days, which is why it now ships with a test.

What you buy is a verdict that names its source. For a hundred images that is a few hundred calls a day, inside one personal token’s hourly allowance of five thousand. The limit this fleet did meet was the catalog’s, on a night every job ran twice, and the rule that came out of it is the one the script above already keeps: a read that did not finish is not an answer. Reading one wrong verdict carefully enough to catch it takes longer than all of those calls together.

A few hundred requests a day. Thirty-four verdicts about nothing in a hundred. Take the trade.

The closing argument#

The model was the most reliable component in that pipeline. It read what it was handed, said plainly what it could not read, and answered the question it was asked. Everything that failed happened before the prompt and after the response: in the lookup that chose the document, in the envelope nobody opened, in the summary read instead of the jobs. That is ordinary software, and none of it was tested until it failed, because a model was standing next to it.

Ask any automated reviewer to show you what it read, with an address, a version and a hash on it. If it cannot answer, you do not have a reviewer. You have a very fluent guess.

Sources#

• Anthropic, Thinking (on by default on Claude Sonnet 5, and how to turn it off), accessed September 2026
• Anthropic, Steering thinking (the token ceiling covers thinking and answer combined), accessed September 2026
• Anthropic, Messages API reference (stop reasons, and stop details filled in only for a refusal), accessed September 2026
• GitHub Docs, Troubleshooting the REST API (404 instead of 403 for private repositories), accessed September 2026
• GitHub Docs, Git references (used above to prove a tag exists), accessed September 2026
• SLSA, Provenance, version 1.0, accessed September 2026
• OCI Image Format Specification, Annotations (the standard source label), accessed October 2026
• Renovate, Docker datasource (the source URL comes from the image’s label), accessed October 2026
• OWASP, Top 10 for LLM Applications 2025, Misinformation, accessed October 2026
• Tantithamthavorn et al., HalluJudge: reference-free hallucination detection for context misalignment in code review automation, 27 January 2026
• Joren et al., Sufficient context: a new lens on retrieval augmented generation systems, ICLR 2025
• Ionix, We built an AI PR reviewer, 13 August 2026
• AICPA Trust Services Criteria via ISMS.online, SOC 2 CC8.1, change management, accessed September 2026
• ISO 27001, 2022 edition, via ISMS.online, Annex A 8.32, change management, accessed September 2026

Discussion#

If you run a model inside a pipeline and can produce, on demand, the exact document it read for its last verdict, drop a comment below with how. If you cannot, that is the more interesting comment. Counterarguments are welcome and the comment thread is where I respond first. For longer back-and-forth with senior practitioners, join the discussion on Discord.