Mutation Testing for Infrastructure Code — 153 Planted Bugs, Four Tests That Could Not Fail

Terraform: testing a plan with no AWS account#

The eight Terraform pipelines are tested with terraform test, which since Terraform 1.7, released on 17 January 2024, can replace a provider with a mock. Each test plans the configuration with its default variables, against mocked providers, and asserts what would be built. No account, no credentials, nothing created:

1
# tests/posture.tftest.hcl in the RDS pipeline, shortened: mocked providers, no AWS account
2
mock_provider "aws" {}
3

4
run "the_defaults_are_the_secure_ones" {
5
  command = plan
6

7
  assert {
8
    condition     = aws_db_instance.db_instance_1.storage_encrypted == true
9
    error_message = "db_instance_1 stores data unencrypted"
10
  }
11

12
  assert {
13
    condition     = aws_db_instance.db_instance_1.publicly_accessible == false
14
    error_message = "db_instance_1 is reachable from the internet"
15
  }
16
}

Between them the eight files make 164 assertions: KMS key rotation, S3 public access blocks, encryption and versioning, the state lock’s encryption and point-in-time recovery, IMDSv2 on instances, invalid-header dropping on load balancers, TLS policy on listeners, flow logs, EKS secrets encryption. On the first run 109 plants broke the promises behind them, and every one was caught.

The assertions were generated from each configuration’s own resources, which is how the more useful output appeared. Generated that way, they could only assert what the defaults deliver, so every place those defaults fall short became a line in the README instead of a silent gap. SSH is open to the whole internet in four EC2 pipelines. Redis is not encrypted at rest or in transit in two. The EKS API endpoint is public. None of these is hidden any more, and none is claimed.

The load balancers show why a list of plants is never finished. Version 1.2.0 moved three of them to AWS’s TLS 1.3 policy from 2021 and called it the policy AWS recommends. It was not. AWS now recommends its hybrid post-quantum policies, and version 1.3.0 moved the three to one of them. The assertion now requires both TLS 1.3 and a post-quantum key exchange in the policy’s name, and two new plants in each pipeline try the old policy and a TLS 1.2 one. Six more plants, 159 in all, and a changelog entry that says the previous release was wrong.

1
# 17-alb.tf in the GitLab pipeline, inside the HTTPS listener
2
ssl_policy = "ELBSecurityPolicy-TLS13-1-2-Res-PQ-2025-09"

A test that asserts only the good news is a brochure.

The dry run that reported a file it never wrote#

The Active Directory script disables accounts that have not signed in for a set number of days, stamps them, and moves them to a holding OU. It supports WhatIf properly, through SupportsShouldProcess. Under WhatIf it printed how many dormant accounts it had written to its report.

It wrote no report.

WhatIf sets a preference variable for the whole script scope, and every cmdlet that supports ShouldProcess inherits it, including New-Item and Export-Csv. The account changes were correctly skipped. So was the report, the one thing a dry run exists to produce. The line after it announced a file that did not exist. A two-parameter fix exempts those two cmdlets from the dry run:

1
# disable-inactive-users-active-directory.ps1: the report is written even under -WhatIf
2
New-Item -Path $LogFolder -ItemType Directory -Force -WhatIf:$false | Out-Null
3
$report | Export-Csv -LiteralPath $logFile -NoTypeInformation -Encoding UTF8 -WhatIf:$false

The Pester suite that found it runs the script against a stand-in ActiveDirectory module that records every call instead of making it. That is what makes an AD script testable on a CI runner with no domain. Its tests assert the sequence of changes the script would make, in order, and that WhatIf makes none of them. For three weeks, since the script learned WhatIf on 3 September, the dry run had been a sentence with no file behind it.

The other defect was older. In the terminal tree, the Ctrl-C handler called tput reset, which wipes the scrollback and, on a terminal that does not answer it, stalls long enough for the script to be killed before the cursor comes back. It had been there since the first commit, in December 2021, and a headless test that runs the tree under util-linux script found it on its first run. The handler now restores the colours and the cursor, clears the screen, and exits with the signal’s code:

1
# terminal-christmas-tree.sh: give the terminal back on Ctrl-C
2
restore() { tput sgr0; tput cnorm; clear; }
3
trap 'restore; exit 130' INT
4
trap 'restore; exit 143' TERM

Both had shipped. Both surfaced the first time a test asked what the script does, not what it prints.

Four tests that could not fail#

The plants did their work on the tests, not the code. Five of them missed on the first runs, from four tests, and each of those tests looked right and proved nothing.

The first is the WhatIf test above. In the Mattermost retention test, one plant broke a code path the test never reached. It was replaced by a plant the test could see, and by a 20-day-old post that a 30-day retention has to keep, because a retention job that deletes too much is the expensive failure, not the one that deletes too little. The terminal tree’s size check read the drawn output as a whole, and a plant that shrank the tree passed it. It now measures the widest row drawn between cursor moves.

The fourth is worth the whole exercise.

The host key check that sat behind a second lock#

Two example jobs ship with the OS update pipeline, one for GitHub Actions and one for GitLab CI, and both SSH into a host and run apt-get upgrade. Each scans the host key, refuses to continue when the scan comes back empty, and then connects with strict host key checking on. The first versions, years ago, connected with it off.

1
# .github/workflows/00-os-update.yml.example, shortened: the scan, the guard, the lock
2
ssh-keyscan -H "$EC2_HOST" > ~/.ssh/known_hosts 2>/dev/null
3
test -s ~/.ssh/known_hosts || { echo "::error::no host key from $EC2_HOST"; exit 1; }
4
ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=yes -o BatchMode=yes \
5
  "$SSH_USER@$EC2_HOST" 'sudo apt-get update && sudo DEBIAN_FRONTEND=noninteractive apt-get upgrade -y'

Two plants removed the empty-scan guard, one in each example. The test stayed green.

With the guard gone, strict host key checking refuses the unknown host anyway, so the job still stopped and nothing a test could see had changed. The guard was a second lock behind the first. Testing it proved nothing, because the thing that mattered was the first lock, and nothing tested that.

Strict host key checking exists for one case. The host that answers is not the host whose key you scanned. So the test now builds that case. The runner scans the real target’s key, then the target’s network name moves to an impostor container with freshly generated host keys and the same account, and only then does the job’s SSH step run:

1
# The target's name moves to the impostor between the key scan and the login.
2
swap_to_impostor() {
3
  docker network disconnect "$RUN" "$RUN-target" >/dev/null
4
  docker network connect --alias target "$RUN" "$RUN-impostor" >/dev/null
5
}

The job has to fail, and nothing may run on the impostor. Now the plant that matters turns strict host key checking off. With that plant in place the job logs straight into the impostor and upgrades it. The test goes red, and the run reports the plant as caught. That is a man-in-the-middle test running on a free GitHub runner inside that job, for a pipeline that is copied into other people’s CI.

A security control is tested when an attacker’s move is shown to fail against it. Removing the control’s neighbour is not that.

Layer 1: write the promises and their plants (week 1)#

Write down what the repository promises, one sentence per promise, in the terms a reader would use. “The database is not reachable from the internet”, not the name of the setting that makes it so. For each promise, write the smallest edit that breaks it. That edit is a plant. In a large organisation, start with the promises whose failure would be an audit finding rather than a bug: who can reach the data, what gets deleted, what a dry run records, and which host a job will log into. Every tested repository here keeps them in one tab-separated file beside its tests. Each line names a file, a piece of text in it, what to put there instead, and a sentence saying what the test must notice. This line from the RDS pipeline, shortened here to the lines that change, flips the default for public accessibility:

1
# tests/plants.tsv: file, old text, new text, what the tests must notice
2
"00-variables.tf"  "type        = bool\n  default     = false"  "type        = bool\n  default     = true"  "publicly_accessible flipped in the default of rds_publicly_accessible"

The fields are JSON strings separated by tabs, so a plant can span lines and carry quotes without an escaping scheme of its own.

Owner: the repository’s maintainer.

Layer 2: break each promise on its own copy (week 2)#

The runner that reads the plants is about eighty lines of Python. Script repositories share one, and the Terraform pipelines use a variant that runs terraform test in a pinned container. Its core runs the suite on an untouched copy, then applies each plant to a fresh copy and counts what stayed green:

1
# tests/plant-violations.py: the untouched copy first, then one plant per copy
2
work = copy()
3
r = run(cmd, work)
4
if r.returncode != 0:
5
    sys.exit("the untouched code does not pass its own tests")
6

7
missed = stale = 0
8
for fname, old, new, what in plants:
9
    work = copy()
10
    path = os.path.join(work, fname)
11
    text = open(path, encoding="utf-8").read()
12
    if old not in text:
13
        print("  STALE   %s: the text this plant breaks is no longer in %s" % (what, fname))
14
        stale += 1
15
    else:
16
        open(path, "w", encoding="utf-8").write(text.replace(old, new, 1))
17
        if run(cmd, work).returncode == 0:
18
            print("  MISSED  %s" % what)
19
            missed += 1

The untouched copy has to pass. Otherwise every plant “fails” for a reason that has nothing to do with the plant, and the run reports a perfect score over a broken suite. And every plant runs on its own copy, because two violations at once can mask each other.

Owner: platform engineer.

Layer 3: run the plants on every push, and fail on a stale one (week 3 onward)#

A plant whose text is gone is stale, and stale fails the run. Code changes. A plant that silently stops applying is a test that silently stops being tested, which is the exact failure this whole exercise exists to catch. So the plants run in CI on every push, beside the tests:

1
# .github/workflows/verification.yml in the log cleaner
2
# A test that cannot fail is not a test: each promise is broken on a
3
# copy of the script, and the run fails if the tests stay green.
4
- name: The tests, shown each violation they exist for
5
  run: python3 tests/plant-violations.py -- pwsh -NoProfile -NonInteractive -File tests/run.ps1

When a plant survives, fix the test first. The survivor is information, not noise.

Owner: platform engineer, with security owning the plants that guard access: public endpoints, TLS policy and SSH host keys.