09 from EKS Validation on a Real AWS Account: Seven Failures No Mock Saw
The bounded cloud run: prove the platform on a real account, with the bill capped twice and read from the invoice
By Vladimir Mikhalev Docker Captain · CNCF Ambassador · IBM Champion
A mocked plan proves the configuration parses. Only a real account proves the platform comes up, and only the bill proves what that cost.
The first passing run on AWS was the eighth (run 38695231460, 15 October 2026). The seven before it found what no mock could, and each finding is now a check. The guards below are why those seven cost about a dollar together, and the one that left something behind was clean within the hour.
- 1
Give the run an account of its own
An account that exists for these runs and nothing else: standalone, or a member account of its own in the organization. The roles a run needs can create and delete, and in a shared account those rights reach everything else.
- 2
Two identities, by OIDC
A creation role for the apply and the test, and a delete-only role for the sweeper, each trusted for one repository and one environment. No long-lived keys.
- 3
Bound every run from inside
A hard job timeout, every resource tagged with the run's ID, teardown in its own job with
if: always(), and after it a residue search by tag plus the types the tagging API misses: load balancers, network interfaces, volumes. Anything found fails the run. - 4
Bound every run from outside
A daily sweeper on the delete-only role destroys anything tagged with a run ID older than three hours. Budget alerts at $10 and $20 a month, and at $30 a budget action that attaches deny-all to the creation role. The sweeper does not depend on that role, so the kill-switch never stops the cleanup.
- 5
Schedule it weekly, behind a switch
Wednesday at 01:17 UTC, two days after the week's dependency updates, and only while a repository variable is
true. A push or a pull request never starts a run. - 6
Report the bill, not the estimate
The run's summary carries its estimate from resource-hours and list prices. The settled number is read from the account's 30-day cost table some days later; the next day's reading is a floor.
Estimate against the bill, a passing run
| Line | 28 October 2026 | 4 November 2026 |
|---|---|---|
| The run's own estimate | $0.16 | $0.16 |
| The bill for the day, before credits, read 13 November | $0.49 | $0.52 |
| EC2 - Other, of which | $0.36 | $0.39 |
| NAT gateway data processing (not in the estimate) | $0.16 | $0.18 |
| Regional data transfer (not in the estimate) | $0.09 | $0.11 |
| NAT gateway hours (in the estimate) | $0.09 | $0.09 |
| Volumes (in the estimate) | $0.01 | $0.01 |
A cent of each day's bill is the sweeper's own Cost Explorer requests.
What shows it works
A decision record, the first passing run, and the bill.
- ADR 10, AWS validation runs: guards and cost reporting: every guard above, when it was built and when it was approved.
- Run 38695231460, 15 October 2026: 101 resources applied, 19 Applications Synced and Healthy, everything destroyed, nothing left.
- The seven failures before it, each now a check: GitHub's immutable OIDC subject, an AWS name limit, a guardrail that denied reads, the tagging API listing deleted resources, an anonymous registry limit, EKS 1.37's default metrics-server build breaking the OpenAPI document Argo CD reads, and a teardown that lost the load balancer controller before the load balancer.
- Sweeper run 43447026743, 13 November 2026: the 30-day table the bill figures come from.
- Sweeper run 41482155146, 2 November 2026: the whole account's 1 November read at $0.56 the next day, against $2.80 settled.
The estimate is what the run believes. The invoice is what it cost.
The long version is the post EKS Validation on a Real AWS Account. ADR 10 in aws-eks-reference-platform, and its workflows eks-validation.yml and eks-sweeper.yml.