06 from Mutation Testing for Infrastructure Code — 153 Planted Bugs, Four Tests That Could Not Fail
Mutation testing for infrastructure code, one planted violation per promise
Mutation tools exist for Java, JavaScript and Python, but none knows what a Terraform default or a PowerShell guard promises. Written by hand, one plant per promise, the same idea fits infrastructure code.
Its job is to test the tests: a check that has never been seen to fail has not been shown to work.
- 1
Write the promises
One sentence each, in the terms a reader would use: “the database is not reachable from the internet”, not the name of the setting. Start with the promises whose failure is an audit finding: who can reach the data, what gets deleted, what a dry run records, which host a job logs into.
- 2
Plant one violation per promise
The smallest edit that breaks it, one line in a tab-separated
plants.tsvbeside the tests: the file, the text to replace, its replacement, and what the plant breaks. The fields are JSON strings, so a plant can span lines and carry quotes. - 3
Run the untouched copy first
The suite has to pass on a clean copy. Otherwise every plant fails for a reason that has nothing to do with the plant, and the run reports a perfect score over a broken suite.
- 4
One plant, one fresh copy
Apply each plant to its own copy and run the suite; it counts as caught only if the suite goes red. Two violations at once can mask each other.
- 5
Fail on a stale plant
A plant whose text is gone fails the run. A plant that silently stops applying is a test that silently stops being tested.
- 6
Run the plants on every push, and fix the test first
A surviving plant is information about the test. For a security control, plant the attacker's move, not the removal of the control's neighbour.
One line per promise
| Promise | The plant | What catches it | Owner |
|---|---|---|---|
| A dry run deletes nothing and says so | The script's own ShouldProcess check always answers yes | The count the script reports, “Deleted 0 file(s)” | Maintainer |
| The job logs in only to the host whose key it scanned | Strict host key checking turned off | An impostor with fresh host keys under the target's name | Security |
| The database is not reachable from the internet | The default of publicly_accessible flipped to true | terraform test with a mock provider, no cloud account | Platform engineer |
The neighbour Remove the guard that refuses an empty key scan. Strict checking refuses the unknown host anyway, the test stays green, and it proves nothing.
The attacker's move Turn strict checking off and put an impostor under the target's name. The job logs in and upgrades it, and the test goes red.
What shows it works
Each is a public commit, run or file.
- Commit fd18b15c, released in v1.1.0 on 26 September 2026: the impostor test and the plant runner, run on every push, and the host-key plants that first proved nothing.
- Verification run 42893910994, 10 November 2026: 5 plants, 5 caught, 0 missed, 0 stale.
- The plant runner, 78 lines of Python: the clean copy, one copy per plant, stale fails.
- The first pass, in the post on 27 September 2026: 153 plants across fifteen repositories, and four tests that could not fail.
Plants are not free. A repository with eighteen plants runs its suite nineteen times, and in September the slowest here spent about three minutes a push on its test and plants together. A private monorepo would run them on a schedule, or when the tested files change.
A security control is tested when an attacker's move is shown to fail against it.
The long version is the post Mutation Testing for Infrastructure Code. Inozemtseva and Holmes, Coverage is not strongly correlated with test suite effectiveness, ICSE 2014; OpenSSH, ssh_config (StrictHostKeyChecking); PIT, Stryker and mutmut.