Restore Drills on a Clean Machine — Measured RTO for 47 Stacks, 19 Seconds to 7 Minutes

What a restore drill that proves the archive looks like#

The drill every template carried until 24 September was a real test, and it was wrong in one word. It restored on the same runner, minutes after the backup, into the same containers. Its shape, shortened to the lines that matter:

1
# deployment-verification.yml, before: backup and restore on one runner
2
- name: Backup and restore end-to-end
3
  run: |
4
    ./tests/e2e-backup-restore.sh   # marker in, backup, destroy, restore, marker back

The marker came back every time. So did the database account, the encryption keys in the environment, the files an installer had written on first boot, and every other thing the host still had because the host was still there. The test proved the archive. It did not prove the procedure an operator would follow at three in the morning with a new machine and a directory of backup files.

The drill that replaced it runs on two runners that never meet, and the second one has never seen the stack:

1
# dr-drill.yml, shortened: the host that dies, then a machine that has never run the stack
2
jobs:
3
  before:
4
    name: The host that dies runs the previous release and is backed up
5
    runs-on: ubuntu-latest
6
    steps:
7
      - run: ./tests/dr-drill.sh before
8
      - name: Keep only what an operator keeps off the host
9
        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
10
        with:
11
          name: dr-offsite
12
          path: dr-out
13
  after:
14
    name: A clean machine brings the current release back from those files
15
    needs: before
16
    runs-on: ubuntu-latest
17
    steps:
18
      - uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
19
        with:
20
          name: dr-offsite
21
          path: dr-out
22
      - run: ./tests/dr-drill.sh after

The first job starts the previous release and writes a marker row into the database and a marker file into the data. It waits for a backup whose cycle began after both, then exports only what an operator keeps off a host: the backup files and the environment file. The second job starts the current release empty on a fresh virtual machine, waits until it answers, copies the backup files in, runs the shipped restore scripts, and requires the application to answer over HTTPS with both markers back. The time from the first command to that answer is the number.

In a Series D environment I have watched a quarterly restore drill pass for a year on the host that took the backups, and fail the first time the host was gone. The archive was fine. The runbook was not.

Why this keeps happening#

NIST SP 800-34 defines the recovery time objective as the maximum time a system can be down before the outage compromises the mission. Its test section names the case directly: system recovery on an alternate platform from backup media, in as close to an operating environment as possible. Most drills stop short of it. A test on the host that took the backup is easier to schedule, and it answers a different question. That host still holds everything the backup left out: server accounts, keys the installer generated, files written on first boot, and a running application that was never actually gone.

Nextcloud is the clean example. Its installer creates a database account with a random password and writes it into its configuration file. A dump saves the data and none of the server’s accounts, which is what a dump is for; the textbook answer is a second dump, of the server’s roles, kept beside it. On the same host the account exists with the right password, so every restore passes. On a new host the installer creates the account again with a different password, the restored configuration carries the old one, and every request answers 500 with password authentication failed for the application’s database user. Both restore scripts here take the other road and bring the account in line with that file, as the installer would, and the clean machine is the only test that can tell whether that line is still there.

The second reason is that the new host is not idle. Its backup loop starts with the stack, and with CI’s fifteen-second warm-up the first cycle wrote an empty backup into the same directory before the restore ran. A restore that picks “the newest file” picks that one. Nextcloud got an empty database next to the old files; Gitea came back without its files. File names carry the minute, not the second, so the new host’s first cycle can even overwrite an imported backup of the same name.

A restore onto a rebuilt host restores by name, from the files you brought, never by newest.

Risk and blast radius#

Direct exposure is the questionnaire. Every enterprise security review I have filled in asks for the recovery time objective and how it was measured, and ISO 27001 control 8.13 asks for backups to be tested regularly. A same-host restore answers both with a number that is true on the day nobody needs it. The number that matters is measured from an empty machine, and until it has been, the field on the form is a belief. The question for whoever signs the questionnaire is on which machine the RTO was measured.

Systemic exposure is the fleet. These 47 templates are copied into other people’s hosts, and each carries its own restore scripts. Until 23 September no test had run those scripts; each test used its own copy of the commands, so eight of them disagreed with the stack they belonged to and four had been written for a different vendor’s image. A restore script CI has never run is a guess with a shebang line, and a guess copied into a hundred deployments is a hundred guesses.

The drill’s own history is the third exposure. It took five attempts to make honest. The first crossed from version 1.9.1 to version 1.9.1, because “the release before main” by tag order was the current one when main was one commit past the tag. One raced its own backup loop. One accepted a backup that finished after the markers but had started before them, so it was newer than the stamp and held no marker. One waited fifteen minutes for a SQL Server dump that landed every sixty seconds, because its file pattern began with a dash and grep read the pattern as an option. One wrote its marker file into a directory the previous release did not have, and the write was allowed to fail without a word, so the drill failed twenty minutes later blaming the restore.

A drill that cannot fail on its own mistakes is a drill that reports the number you hoped for.

What 47 drills found in their first week#

The first full week measured 47 of 47 templates restoring, from the previous release into the current one, on a machine that had never run the stack. The evidence page reads the result of every drill from the fleet’s own file; a failing drill is listed as failing and carries no time. The figures in the title are that first week’s, as the catalog recorded them on 26 September; the page shows the latest week, and its numbers move.

The number is not the interesting part. What the drill found on the way to it is.

Three templates failed because the clean machine was cleaner than any real machine. Authelia’s Postgres refused to initialise, Dozzle could not write a password hash, Audiobookshelf’s permission job exited 1. Each of them prepares something before the stack starts: a directory of secret files, a bcrypt hash made by a documented command, a bind directory handed over to the container’s user. CI did those in steps the drill had not copied. The drill generator now borrows every step between the network creation and the start-up from the deployment workflow, and a per-template keep list names what has to travel off the dying host besides the backups and the environment file. For Authelia that is its secrets directory, because the encryption key in them is what the restored database was written with.

Mailu was red on one cold start in three, and I published a cause the same hour that the next run disproved. The fault was there before any restore: two Traefik routers with the same host rule and no priority, so Traefik chose between them at every start, and when it chose the webmail router every path on the mail host went to Roundcube. The drill now proves the empty stack answers before it puts anything back, so a start-up fault and a restore fault are two different log lines. The webmail router is gone; front serves that path itself.

OTRS stayed unhealthy on two attempts in six after the restore. Each restore script stops and starts the application, and two stop-starts seconds apart left Apache on a stale pid file. An operator waits for the first restore to come up before starting the second, and the drill now waits with them.

Framework: inventory, two runners, a published number#

The drill is one script and one workflow, generated for each template from its own compose file. The three layers below are what any stack needs before the number it produces means anything.

1
# tests/dr-drill.sh, before: export only what an operator keeps off the host
2
say "exporting what an operator keeps off the host: the backup files and .env"
3
cp .env "$OUT/env"
4
for v in $(dirs); do
5
  dir="$(env_of "$v")"
6
  mkdir -p "$OUT/$v"
7
  docker cp "$(cid "$BACKUPS_SVC"):$dir/." "$OUT/$v/"
8
done
9
for k in ${DR_KEEP:-}; do
10
  mkdir -p "$OUT/keep/$(dirname "$k")"
11
  cp -a "$k" "$OUT/keep/$k"
12
done
13
printf '%s\n' "$MARK" > "$OUT/marker"

1
# tests/dr-drill.sh: a backup that finished after the markers is not one that contains them
2
wait_backup_started_after() {  # directory variable, file regex, marker minute, log word
3
  local dir pat f waited=0
4
  dir="$(env_of "$1")"; pat="$(expand "$2")"
5
  while [ "$waited" -lt 900 ]; do
6
    for f in $(bk "ls -1 '$dir'" | grep -E -e "$pat" | sort -r); do
7
      [ "$(printf '%s' "$f" | cycle_of)" \> "$3" ] || continue
8
      if docker logs "$(cid "$BACKUPS_SVC")" 2>&1 | grep -qiF "backup OK: $dir/$f"; then
9
        say "backup started after the markers: $f"; return 0
10
      fi
11
    done
12
    sleep 5; waited=$((waited + 5))
13
  done
14
  echo "no $4 backup started after $3 within 900s in $dir (matching $pat)" >&2
15
  return 1
16
}

1
# tests/dr-drill.sh, after: the file you brought, never "the newest in the directory"
2
newest() {  # the newest exported file matching a pattern, by name: the names carry the time
3
  local pat; pat="$(expand "$2")"
4
  find "$OUT/$1" -maxdepth 1 -type f -printf '%f\n' | grep -E -e "$pat" | sort | tail -n 1
5
}
6
dbf="$(newest "$DB_DIR_ENV" "$DB_FILE_MATCH")"
7
F="$dbf" S="$(printf '%s' "$dbf" | cycle_of)" bash -c "$DB_RESTORE"

1
{
2
  "repository": "nextcloud-traefik-letsencrypt-docker-compose",
3
  "from": "v2.0.4",
4
  "to": "v2.0.7",
5
  "ok": true,
6
  "seconds_total": 154,
7
  "seconds_restore": 87,
8
  "backup_interval": "24h",
9
  "rpo_hours": 24,
10
  "run": "https://github.com/heyvaldemar/nextcloud-traefik-letsencrypt-docker-compose/actions/runs/36088409515",
11
  "finished_at": "2026-09-25T03:06:37Z"
12
}

Tradeoffs#

The drill costs four to five minutes of runner time per template per week, and thirteen for the slowest, Jira. Most of it is waiting for a real backup cycle on the dying host and for the application to come up twice on the clean one. On public repositories that is free. On private ones the 47 templates would come to roughly 230 minutes a week, about a thousand a month, half of a Free plan’s 2,000 included minutes. A larger private fleet would drill a rotating subset rather than everything at once.

A single host with no CI can run the smallest version of the same idea. restore-drill restores the newest database dump into a throwaway container on a timer and counts what came back, which catches a dump that will not load or loads empty, though not an application that cannot log in to what came back.

The alternative is the number you already have, measured on a machine that still had everything the backup left out. Nextcloud’s would have been correct on every host except a new one.

Five minutes a template a week, against a 500 on the day the host is gone. The minutes are cheaper.

The closing argument#

A backup is proven on a machine that has never seen the stack, from the previous release into the current one, by the scripts the operator would run. Anything the application needs from outside what the dump saves, server accounts, keys, first-boot files, is either in the export or recreated by the restore, and only the clean machine tells you which one you forgot. The same-host drill is not wrong. It is a test of the archive, and the archive was never the thing that failed.

Sources#

• NIST, SP 800-34 Revision 1, Contingency Planning Guide for Federal Information Systems, May 2010, updated November 2010 (recovery time objective and plan testing)
• ISO 27001, 2022 edition, via ISMS.online, Annex A 8.13, information backup, accessed September 2026
• GitHub, About GitHub-hosted runners (each runner is a new virtual machine), accessed September 2026
• GitHub, About billing for GitHub Actions (free for public repositories; 2,000 included minutes on the Free plan), accessed September 2026
• Nextcloud, Backup, in the administration manual (what a backup has to carry besides the database), accessed September 2026
• heyvaldemar.com, Evidence (this week’s 47 drill results, read from the fleet’s own file) and Decisions (decision 14, the weekly clean-machine drill)

Discussion#

If you measure recovery time on a machine that has never run your stack, or you have a reason to trust the same-host number, drop a comment below. Counterarguments are welcome and the comment thread is where I respond first. For longer back-and-forth with senior practitioners, join the discussion on Discord.