Backups Verified by Restore — Twelve Days of OK Over an Empty Archive

What a backup verified by nothing looks like#

Every template in this fleet runs a sidecar beside its database that dumps the database and archives the data on a schedule, prunes old archives, and logs a line. For years the loop wrote straight to the final file name, and its success condition read the exit code of the archiver:

1
# the backup loop, before, as a sketch of the shape the templates shared
2
# (each compose file has its own variables): the final name from the first
3
# byte, and an exit code as the verdict
4
if tar -czf "$BACKUPS/data-$(date +%F_%H-%M).tar.gz" -C /data . \
5
    || [ $? -eq 1 ]; then
6
  echo "Data backup OK"
7
fi

The clause that accepts an exit code of one is there because GNU tar returns 1 when a file changed while it was being read, which is harmless on a live data directory. BusyBox tar, on the alpine images two in five of the fleet’s backup sidecars run, also returns 1 when it cannot open the output file at all. Blocking the destination on purpose produced two unreadable files named like backups, and the log said OK over both. Interrupted mid-dump, the same loop left a truncated file under exactly the name a restore selects.

The loop as the fleet’s rule now requires it writes to a partial name, reads the archive back, and only then gives it the name a restore would pick:

1
# the backup loop, after, as a sketch of the shape, not one template's text:
2
# nothing carries the final name until it has been read back
3
f="$BACKUPS/data-$(date +%F_%H-%M).tar.gz"
4
tar -czf "$f.partial" -C /data . 2>/dev/null; rc=$?
5
if { [ "$rc" -eq 0 ] || [ "$rc" -eq 1 ]; } \
6
    && tar -tzf "$f.partial" >/dev/null 2>&1; then
7
  mv "$f.partial" "$f" && echo "Data backup OK: $f"
8
else
9
  mv "$f.partial" "$f.failed" 2>/dev/null
10
  echo "Data backup FAILED: tar exited $rc, or the archive does not open" >&2
11
fi

An exit code shared by a harmless case and a fatal one is not a result. Reading the archive back is.

I have carried this rule since a telecom environment where a nightly database dump had been “succeeding” for months against a disk that was full. The file was there. It was four kilobytes.

Why this keeps happening#

The success line is written by the thing that took the backup, and it describes the thing it did rather than the thing that is needed. tar reports that it archived what it was pointed at. It cannot report that it was pointed at the wrong directory. The Outline template replaced MinIO with Garage in its version 2 release, and the backup loop kept archiving the minio-data volume, which nothing wrote to from that day on. Every cycle logged OK. CI checked that the archive it named opened, which it did, because an empty volume archives beautifully. The attachments were in no backup at all, and the restore script for them looked for a service called minio and stopped before touching anything.

The second reason is that the restore is tested with a copy. Every template’s end-to-end test restored a backup on every push and went green, and every one of those tests restored with its own commands. The scripts a person would run on the day they need one had been run by nobody. Checked against their stacks on 23 September: four written for Bitnami images on stacks that run official ones, one of which refused to run at all; one that signed in with the wrong account and would have dropped a database that does not exist; restores that merged instead of replacing, through a recursive delete that keeps dotfiles, an archive extracted over live data, which deletes nothing, and a Mongo restore with the drop option, which keeps collections the archive does not hold; paths read from the operator’s shell instead of the stack; and a database image with no client of the name the script called. In sixteen templates the script was more than three years old.

The tests stayed green throughout, because the copy was right and the script was not.

Two tests were more honest than the scripts they stood beside. One cleared the data directory before extracting, with a comment explaining that tar does not delete. The other put its marker in a collection the archive already held, with a comment explaining that the drop option replaces only those. Both tests knew the limitation and worked around it. Neither shipped script did. When a test has to work around the thing under test, the workaround is the finding.

Risk and blast radius#

Direct exposure is the day the backup is the only copy. ISO 27001 control 8.13 asks for backups to be tested regularly, and SOC 2’s availability criterion A1.3 asks for the recovery plan procedures to be tested. A green run that restores with its own commands can satisfy either auditor and still fail the operator. The Outline case is the exact shape: twelve days in which any disk failure would have lost every attachment the team had uploaded since the migration, under a log that said OK once a day, twelve days running.

Systemic exposure is the copy count. These templates are deployed by people who take the backup loop and the restore script as a pair that has been proven together. Until 23 September the pair had never been proven together, and the script not at all. A defect in a shipped restore script is not one incident. It is one incident per deployment, on the day each deployment needs it, with no way to know in advance which of them will. A restore procedure handed to a customer in a runbook is the same kind of shipped script, and the same test applies: run it as written before the customer has to.

The prune is the third exposure and it points the other way. Every sidecar prunes archives older than a number of days, one find command that deletes by age. Thirteen templates shipped that line with no test of it, and one, Zammad, had carried a retention setting since the day it was written without anything ever asking whether it deleted anything, or whether it stopped at the recent archives. A prune that is only intended fills the disk. One that takes everything is a backup that is not there on the day.

What running the shipped scripts found#

On 23 September the fleet’s tests were changed to invoke the shipped restore scripts, non-interactively, against a marker written after the backup they restore. Forty-seven templates’ scripts were rewritten in one day, and six of the mistakes that come with a day like that were caught before release and listed on the evidence page. The rule that closes the class lives in the fleet’s conformance checker rather than in forty-seven workflows:

1
# scripts/fleet-conformance.py in fleet-ops at c06df73, reflowed to fit:
2
# a restore script CI does not run is a guess
3
scripts, unrun, looked = unrun_restore_scripts(repo, names)
4
if scripts:
5
    if unrun is None:
6
        bad.append(f"ships {', '.join(scripts)}, and nothing under tests/ "
7
                   f"or .github/workflows/ could be read to see whether CI "
8
                   f"runs them (looked in: "
9
                   f"{', '.join(looked) or 'nothing found'})")
10
    elif repo.name in RESTORE_NOT_YET_RUN:
11
        if unrun:
12
            exempt.append(f"restore scripts CI does not run yet, listed in "
13
                          f"RESTORE_NOT_YET_RUN: {', '.join(unrun)}")
14
        else:
15
            bad.append("CI runs every restore script now: take it off "
16
                       "RESTORE_NOT_YET_RUN, so a regression here fails "
17
                       "instead of being excused")
18
    elif unrun:
19
        bad.append(f"CI never runs {', '.join(unrun)}: a restore script "
20
                   f"that no test invokes can be wrong for its stack while "
21
                   f"every run is green")

A comment or an echo that names the script does not count. The rule’s list of exceptions can only shrink, and it has been empty since the day it was written.

The Outline test now proves the backup the way a user would notice it missing. It stores an object through the application’s S3 interface, waits for an archive written after it, stores a second object, restores, and requires the first to read back and the second to be gone:

1
# tests/e2e-backup-restore.sh in the Outline template at v2.2.8, shortened
2
# and reflowed to fit: prove the backup the way a user would miss it
3
tag="e2e-$(date +%s)"
4
if ! s3obj put "$tag/before.txt" "stored before the backup"; then
5
  bad "could not store an object through S3"
6
else
7
  # ... Garage is asked for its metadata snapshot rather than waited for
8
  ol_sh "touch $OUTLINE_DATA_BACKUPS_PATH/.e2e-stamp"
9
  archive=$(newest_after_marker "$OL_BK" "$OUTLINE_DATA_BACKUPS_PATH" \
10
    "$OUTLINE_DATA_PREFIX" ".tar.gz")
11
  if [ -z "$archive" ]; then
12
    bad "no attachment archive taken after the object within ${CYCLE_WAIT}s"
13
  else
14
    note "archive: $(basename "$archive")"
15
    s3obj put "$tag/after.txt" "stored after the backup" \
16
      || note "could not store the second object"
17
    if ! COMPOSE_PROJECT_NAME="$PROJECT" \
18
        ./outline-restore-application-data.sh "$(basename "$archive")"; then
19
      bad "./outline-restore-application-data.sh failed"
20
    else
21
      waited=0
22
      until got=$(s3obj get "$tag/before.txt" 2>/dev/null) \
23
          || [ "$waited" -ge 120 ]; do
24
        sleep 3; waited=$((waited + 3))
25
      done
26
      if [ "$got" = "stored before the backup" ]; then
27
        ok "the object stored before the backup reads back through S3 …"
28
      else
29
        bad "the object stored before the backup is missing after the restore …"
30
      fi
31
      if s3obj absent "$tag/after.txt"; then
32
        ok "the object stored after the backup is gone, so the restore …"
33
      else
34
        bad "the object stored after the backup survived the restore"
35
      fi
36
    fi
37
  fi
38
fi

The second object is the half people skip. A restore that leaves newer files in place is a merge, and a merge passes every test that only asks for the old marker back.

The prune got the same treatment two days later, after a good Nextcloud upgrade was reverted twice by a test that slept for a number. The test waited the backup interval plus sixty seconds and then asked whether a file dated 2020 was gone. With the new release the data backup took 98 seconds, the cycle ran 160, and the test asked at 120, before the prune had run. Thirty-six templates carried the same three lines. The test now waits for the event it is testing, with a ceiling of two cycles, and prints how long the prune took:

1
# tests/e2e-backup-restore.sh in the Keycloak template at v1.8.8, shortened
2
# and reflowed to fit: wait for the prune itself, not for a number chosen
3
# when the data was small
4
local ceiling=$(( secs * 2 + 300 )) waited=0
5
echo "  waiting up to ${ceiling}s for a prune cycle to remove it..."
6
while backups_sh "ls $fake_old 2>/dev/null" > /dev/null 2>&1; do
7
  if [ "$waited" -ge "$ceiling" ]; then
8
    fail "fake old file survived ${ceiling}s, longer than two backup cycles"
9
    return 1
10
  fi
11
  sleep 5; waited=$(( waited + 5 ))
12
done
13
echo "  pruned after ${waited}s"

A file dated 2020 placed beside Zammad’s real backups was gone after the next cycle and six recent archives stayed. It was the first time that sentence could be written about that template.

A restore can put back every byte and still lose the service, because the bytes land in the wrong place or belong to the wrong user. October produced one of each, and both were caught by a restore that checked the service, not the exit code.

On a VEIN game server I run, the world is backed up with restic, which recreates a snapshot’s original path under the restore target. A plain restore of the latest snapshot would have put the world one directory too deep, where the server does not look, and the server would have started a new world over it. A test restore caught it before anyone needed the backup. Restic’s own form for restoring the contents of a path, the snapshot, a colon and the path, put the world file back byte for byte at the path the server reads. The public VEIN template archives with tar rather than restic, and its test guards the same class of mistake: it asserts places and bytes, not exit codes, including the contrast case, in which the same archive extracted at the volume’s root must leave no world where the server looks.

The second never reached a user. The fleet’s shared restore pattern sets aside what the archive leaves out on purpose, clears the volume, unpacks the archive, and puts the set-aside paths back. For months it did the last step in one line:

1
# the shared restore pattern, before, as it runs with the data root at /data:
2
# the set-aside paths put back in one copy
3
cp -a /tmp/restore-keep/. /data/

Copying a directory’s dot entry in archive mode copies the directory itself. Its owner and mode land on the target. The volume’s root came back owned by root, and the restored database was given that owner. AnythingLLM runs as an unprivileged user. After the restore it passed every read. Every write failed: attempt to write a readonly database. The archives record the moment: sets taken before the first restore show the volume’s root owned by user 1000, sets taken after it by root. It surfaced on 19 October, in the first local runs of the template’s restore test, before its first release, and the test has asserted a write after every restore since. The fix puts the set-aside paths back one by one and gives the volume’s root the owner of the files the application created:

1
# anythingllm-restore-data.sh in the AnythingLLM template at v1.0.0,
2
# lines 98 to 104, without the escaping of the sh -c string they run in:
3
for p in $KEEP; do
4
  [ -e "/tmp/restore-keep/$p" ] || continue
5
  rm -rf "$p"; mkdir -p "$(dirname "$p")"; mv "/tmp/restore-keep/$p" "$p"
6
done
7
rm -rf /tmp/restore-keep
8
owner=$(stat -c %u:%g /data/comkey 2>/dev/null || stat -c %u:%g /data)
9
chown "$owner" /data

Since the next release, the set-aside paths wait in a hidden directory on the volume itself rather than in the container’s temporary directory, so each move is a rename and nothing is copied into the container’s layer.

The same line was in seven other templates, and none of them broke as shipped, because their applications run as root, take ownership of their data on every start, or restore into a directory root already owns. A line that is harmless seven times and fatal the eighth is invisible to a test that only reads.

Framework: prove the archive, prove the script, prove the prune#

Three things have to be shown to work, and each is a separate test with a separate owner. A stack that has one of them has one third of a backup.

1
# restore-drill.sh in restore-drill at v1.1.0, shortened and reflowed to fit:
2
# the live database is the reference, not a number in the test
3
# ... table_list <container>: one table name per line, sorted,
4
# empty on any failure
5
if [ -n "$DRILL_LIVE_CONTAINER" ]; then
6
  live="$(table_list "$DRILL_LIVE_CONTAINER")"
7
  # An unreachable live database answers the same as one with no tables, and
8
  # reading that as "nothing is missing" would turn this into a check that
9
  # cannot fail — the exact shape this whole tool exists to refuse.
10
  [ -n "$live" ] \
11
    || die "could not read the table list from $DRILL_LIVE_CONTAINER — …"
12
  restored="$(table_list "$CONTAINER")"
13
  missing="$(comm -23 <(printf '%s\n' "$live" | sort -u) \
14
    <(printf '%s\n' "$restored" | sort -u))"
15
  if [ -n "$missing" ]; then
16
    count="$(printf '%s\n' "$missing" | wc -l | tr -d ' ')"
17
    die "$count table(s) the live database has did not come back: $(
18
      printf '%s' "$missing" | tr '\n' ' ' | cut -c1-300)"
19
  fi
20
  # ... the line that reports the drill, and the table floor used only
21
  # when no live database is named

1
# tests/e2e-backup-restore.sh in the Keycloak template at v1.8.8, reflowed
2
# to fit: the shipped file, its two prompts answered on stdin, the file name
3
# and then DESTROY
4
if ! printf '%s\nDESTROY\n' "$(basename "$baseline")" \
5
    | COMPOSE_PROJECT_NAME="$COMPOSE_PROJECT_NAME" \
6
      ./keycloak-restore-database.sh; then
7
  fail "the shipped restore script failed"
8
  return 1
9
fi

1
# tests/e2e-backup-restore.sh in the AnythingLLM template at v1.0.0, reflowed
2
# to fit: a restore that reads is not yet a restore that works
3
wrote="$(api /api/v1/workspace/new -H "Content-Type: application/json" \
4
  -d '{"name":"after-restore"}' \
5
  | python3 -c 'import json,sys
6
print((json.load(sys.stdin).get("workspace") or {}).get("slug") or "")')"
7
[[ -n "$wrote" ]] || {
8
  fail "AnythingLLM cannot write after the restore (a new workspace was refused)"
9
  return 1
10
}

1
# scripts/fleet-conformance.py in fleet-ops at c06df73, reflowed to fit:
2
# a prune that is only intended is not a prune
3
prunes = [n for n in composes
4
          if re.search(r"PRUNE_DAYS|HOLD_DAYS|-mtime", repo.read(n) or "")]
5
if prunes:
6
    tests = [f"tests/{n}" for n in repo.names_in("tests") if n.endswith(".sh")]
7
    exercised = any(re.search(r"(?im)^(?!\s*#).*prun", repo.read(t) or "")
8
                    for t in tests)
9
    if tests and not exercised:
10
        bad.append(f"{', '.join(prunes)} prunes old backups and no test "
11
                   f"under tests/ exercises the prune: a prune that is only "
12
                   f"intended fills the disk, and one that takes everything "
13
                   f"is a backup that is not there on the day")

Tradeoffs#

Deployment verification grew from a boot check to a job of about seven minutes on every push, ten for the heaviest stack, and the sidecar had to learn to tolerate files changing under it while it archived. The restore rewrite cost a day for 47 templates and produced six mistakes of its own before release, each caught by the tests the rewrite had just made honest.

Skipping it costs what the Outline case cost: a window, twelve days here, in which the backup exists, reports OK, and holds nothing you would want back. Nobody finds that window until the day they need the backup, which is the one day it cannot be fixed.

Seven minutes a push, against an archive of a volume nothing writes to. That is not a close call.

The closing argument#

An archive that opens proves the archive. A restore that runs with its own commands proves the copy. A prune that is only intended is not a prune. The backup that exists is the one a test has restored, with the shipped script, on a machine that did not have the data, then found the marker through the application and written something new. Everything short of that is a log line that says OK, and the log line is written by the thing that took the backup, not by the thing that would need it.

Sources#

• Debian, the tar manual page (return value 1: some files differ), accessed September 2026
• ISO 27001, 2022 edition, via ISMS.online, Annex A 8.13, information backup, accessed September 2026
• AICPA, 2017 Trust Services Criteria, with revised points of focus (2022) (criterion A1.3, recovery plan procedures tested), accessed October 2026
• MongoDB, mongorestore and its drop option (drops only the collections in the dump), accessed September 2026
• restic, Restoring from backup (a snapshot’s original path recreated under the target, and the form that restores a subfolder’s contents), accessed October 2026
• heyvaldemar, vein-server-docker-compose (the restore asserted by path and by byte), 18 October 2026
• heyvaldemar, the AnythingLLM template, version 1.0 and the run that restores and then writes, 19 October 2026
• heyvaldemar, restore-drill version 1.1 (the drill that compares the restored tables with the live database’s, and refuses a live database it cannot read)
• heyvaldemar, the conformance rule that refuses a directory copied over the data root, 19 October 2026
• heyvaldemar.com, Ledger (the backup findings, from BusyBox tar to the Zammad prune) and Decisions (decisions 4 and 12)

Discussion#

If you have found a backup that reported OK over the wrong directory, or your restore test runs the shipped script rather than a copy of it, drop a comment below. Counterarguments are welcome and the comment thread is where I respond first. For longer back-and-forth with senior practitioners, join the discussion on Discord.