EKS Validation on a Real AWS Account: Twenty Cents a Run, Seven Failures No Mock Saw

What a mocked provider cannot see#

The platform under test is a public reference platform for Amazon EKS: Terraform for the network, EKS and its add-ons, Argo CD for everything inside the cluster, Kyverno for admission, Karpenter for nodes, Velero for backups. Its Terraform tests run on every change against a mocked provider, and the whole platform runs on kind on every change and every day. The night of 15 October was the first time any of it met an AWS account. Each run reports its own length and estimates its cost from resource-hours at list prices, before data transfer, logs and load balancer capacity units.

The seven failures cost a dollar and nine cents by the same estimates, more than five times the run that passed. What they printed is what an engineer pastes into a search box, so the post quotes it verbatim.

1
# the seven failed validation runs, 15 October 2026, verbatim, account ID removed
2
Not authorized to perform sts:AssumeRoleWithWebIdentity
3
"name" cannot be longer than 64 characters: "reference-platform-38666856654-karpenter-capacity-reservation-interruption"
4
is not authorized to perform: iam:GetRole on resource: role platform-validation-create with an explicit deny in an identity-based policy
5
response status code 429: toomanyrequests: Data limit exceeded
6
error getting openapi resources: the server is currently unable to handle the request
7
DependencyViolation: The subnet 'subnet-0b2546018e28dfddd' has dependencies and cannot be deleted.

Three of the seven came from outside the repository, and nothing in it had changed. GitHub moved repositories created, renamed or transferred after 15 July 2026 to an immutable OIDC subject that carries the owner’s and the repository’s IDs, so a trust policy written for the old owner-and-name format refused every run before it started. ECR Public gives an anonymous client 500 gigabytes a month and one pull a second, and a hosted runner fetching Karpenter’s chart without logging in was told the data limit was exceeded. And the default metrics-server add-on for EKS 1.37, in October 2026, is built with Go 1.27, where the new JSON implementation is the default.

That last one took two runs. The cluster’s OpenAPI document still answered as JSON, which is how a person checks it, and failed as protobuf, which is how Argo CD reads it. Argo CD could not build its cluster cache, and the root Application sat in Unknown behind an error about a server that could not handle the request. The upstream issue has been open since 1 September, and on 15 October I added the EKS builds and the pin that works to it. For three days the platform held the add-on on its last Go 1.26 build. On 16 October the add-on’s thirteenth EKS build was reported fixed on the same issue: the same Go 1.27, with the new JSON implementation switched off. The platform moved to it on 18 October, and two checks keep the move honest. Before it creates anything, the weekly run asks EKS whether that build is offered for the cluster’s Kubernetes version. Before it installs anything, the bootstrap asks for the document the way Argo CD does.

1
# bootstrap/bootstrap.sh in aws-eks-reference-platform at v1.3.0, shortened: read the cluster's OpenAPI document the way Argo CD reads it, as protobuf
2
proxy_dir="$(mktemp -d)"
3
kubectl proxy --unix-socket="$proxy_dir/api.sock" >/dev/null 2>&1 &
4
proxy=$!
5
trap 'kill "$proxy" 2>/dev/null || true; rm -rf "$proxy_dir"' EXIT
6
code=000
7
for _ in $(seq 1 30); do
8
  kill -0 "$proxy" 2>/dev/null || { echo "kubectl proxy exited; cannot ask for the OpenAPI document" >&2; exit 1; }
9
  if [ -S "$proxy_dir/api.sock" ]; then
10
    code="$(curl -s --max-time 30 --unix-socket "$proxy_dir/api.sock" -o /dev/null -w '%{http_code}' \
11
      -H 'Accept: application/[email protected]+protobuf' http://localhost/openapi/v2 || true)"
12
    [ "$code" = 200 ] && break
13
  fi
14
  sleep 10
15
done
16
# ... the proxy is stopped and its directory removed
17
if [ "$code" != 200 ]; then
18
  echo "/openapi/v2 as protobuf answers HTTP $code; Argo CD cannot build its cluster cache on this cluster." >&2
19
  # ... the upstream issue is named and the aggregated APIs are listed
20
  exit 1
21
fi

The pin carries its own history, so the next person to lift it knows what it is guarding against.

1
# terraform/addons.tf in aws-eks-reference-platform at v1.3.0, shortened: the pin and the reason for it
2
# metrics-server is pinned to v0.9.0-eksbuild.13. eksbuild.12, the default
3
# for Kubernetes 1.37 in October 2026, is built with Go 1.27, whose default
4
# jsonv2 experiment makes kube-openapi leak field names into the schema it
5
# publishes; the API server then answers /openapi/v2 as protobuf with 503,
6
# and Argo CD cannot build its cluster cache ("error getting openapi
7
# resources: the server is currently unable to handle the request"). The
8
# first EKS validation runs found it, and the platform held eksbuild.11, the
9
# last build on Go 1.26. eksbuild.13 is Go 1.27 built with
10
# GOEXPERIMENT=nojsonv2, the fix AWS reported on
11
# https://github.com/kubernetes-sigs/metrics-server/issues/1868. Two checks
12
# keep it honest: the weekly run asks EKS before applying anything whether
13
# this build is offered for the cluster's Kubernetes version, and bootstrap
14
# reads /openapi/v2 as protobuf, so a build that brings the 503 back fails
15
# there.
16
resource "aws_eks_addon" "after_nodes" {
17
  for_each = {
18
    coredns            = { service_account = null, role_arn = null, version = null }
19
    metrics-server     = { service_account = null, role_arn = null, version = "v0.9.0-eksbuild.13" }
20
    aws-ebs-csi-driver = { service_account = "ebs-csi-controller-sa", role_arn = aws_iam_role.ebs_csi.arn, version = null }
21
  }
22
  # ...

The first scheduled run, on 21 October, put both checks to work on a real cluster. EKS offered the thirteenth build for Kubernetes 1.37, the document answered as protobuf, and every Application came up Synced and Healthy.

The other four were the platform’s own, and only the real API could judge them. An EventBridge rule name reached 74 characters once the cluster name carried the run’s ID, and AWS stops at 64. The guardrail that keeps workflow roles away from the bootstrap’s own IAM denied reads as well as writes, so Terraform could not look up the role it was running as. A residue search failed a teardown that had left nothing, because the tagging API kept listing a deleted NAT gateway, subnet, security group and flow log, and three keys that were only scheduled for deletion. And deleting Argo CD’s root Application cascaded through the load balancer controller before the Gateway’s Service was gone, so a network load balancer outlived the cluster and held the network open.

In the production environment I run, the first apply of anything new has always been where the account’s own rules showed up, never the plan.

Why this keeps happening#

A mocked provider answers with what the test’s author believed about AWS on the day the test was written. It has no quotas, no name limits, no identity formats and no default add-on builds, and it never changes on its own. Everything around it does. Three of the seven failures were changes and limits outside the repository. A fourth trap sits in AWS’s own defaults: Budgets counts credits unless told otherwise, so in an account running on promotional credits a budget alert waits for the credits to run out.

The kind suite has the opposite blind spot. It runs the Kubernetes the platform deploys, every day, and catches what admission, GitOps and backups do inside a cluster. It has no cloud controllers, no load balancers made by AWS, no IAM and no add-ons that someone else builds. Each layer proved what it could reach. None of them reached the account.

A long-lived staging cluster looks like the answer and is a different test. It proves the platform keeps running. It does not prove the platform can be built from nothing, which is the case that matters on the day a region, an account or a customer’s environment has to be created. It is the gap behind the restore drill on a clean machine, one layer up, and its bill runs whether anyone reads it or not.

Proof that never meets the account is a forecast.

Risk and blast radius#

Direct exposure is the first real apply happening when someone needs it: a rebuild after an incident, a new region, a customer’s account during onboarding. Every failure on this list would have arrived then, one at a time, with the clock running. For a solutions architect or a forward deployed engineer, that first apply happens in a customer’s account, in front of the customer. The night of 15 October found all seven in under five hours, for about a dollar.

Systemic exposure is the copy. A reference platform exists to be copied into other accounts, and every copy inherits the trust policy that no longer matches GitHub’s subject, the add-on build that blinds Argo CD and the teardown that leaves a load balancer billing. For a CISO, two of the seven sat in identity: a security improvement at GitHub that refuses every trust policy written the old way, and a guardrail strict enough to deny the role it protects. For a CTO or a finance lead, the credits default means a company living on credits can have budget alerts configured and still never receive one.

Three of the seven, and the credits default, are four questions any account can answer today, before anything is built.

1
# four questions for an account of your own. The first, the first half of the second and the fourth are adapted from
2
# aws-eks-reference-platform at v1.3.0 (terraform/bootstrap-aws/variables.tf and .github/workflows/eks-validation.yml).
3
# The third reads the setting that repository's budget turns off (terraform/bootstrap-aws/main.tf, include_credit = false).
4

5
# 1. The subject prefix GitHub writes into this repository's OIDC token, and what the role trusts. The role's sub must begin with it.
6
gh api repos/OWNER/NAME/actions/oidc/customization/sub --jq .sub_claim_prefix
7
aws iam get-role --role-name ROLE --query 'Role.AssumeRolePolicyDocument.Statement[].Condition' --output json
8

9
# 2. The metrics-server builds EKS offers for a Kubernetes version (here 1.37), and the build a cluster runs.
10
aws eks describe-addon-versions --kubernetes-version 1.37 --addon-name metrics-server \
11
  --query 'addons[0].addonVersions[].addonVersion' --output text
12
aws eks describe-addon --cluster-name CLUSTER --addon-name metrics-server --query addon.addonVersion --output text
13

14
# 3. Whether each budget counts credits. True means the budget waits for the credits to run out.
15
aws budgets describe-budgets --account-id "$(aws sts get-caller-identity --query Account --output text)" \
16
  --query 'Budgets[].[BudgetName,CostTypes.IncludeCredit]' --output text
17

18
# 4. Log in to ECR Public before pulling a chart from it, so the pull is the account's and not an anonymous one.
19
aws ecr-public get-login-password --region us-east-1 | helm registry login --username AWS --password-stdin public.ecr.aws

Ways to prove an EKS platform, compared#

The table is not a ranking. The first two run on every change because they are free and fast, and they stay. The third answers the one question the first two cannot, for less than a dollar a month.

None of the four is new: ephemeral environments and Terratest have applied real infrastructure from CI for years. What this night adds is a price and a list.

Framework: a ceiling, a clean teardown, a weekly run#

1
# kill-switch.tf: a standalone version for an account of your own, adapted from terraform/bootstrap-aws/main.tf in
2
# aws-eks-reference-platform, where the limits are variables ($10, $20 and $30 by default) and the role is the module's own.
3
# Alerts on usage before credits, then deny-all at the ceiling.
4
variable "creation_role_name" {
5
  type = string
6
}
7

8
variable "alert_email" {
9
  type = string
10
}
11

12
data "aws_caller_identity" "current" {}
13

14
resource "aws_budgets_budget" "validation" {
15
  name         = "platform-validation-monthly"
16
  budget_type  = "COST"
17
  limit_amount = "30.00"
18
  limit_unit   = "USD"
19
  time_unit    = "MONTHLY"
20

21
  cost_types {
22
    include_credit = false # Budgets counts credits by default: on promotional credits nothing would ever fire.
23
  }
24

25
  dynamic "notification" {
26
    for_each = [10, 20]
27
    content {
28
      comparison_operator        = "GREATER_THAN"
29
      threshold                  = notification.value
30
      threshold_type             = "ABSOLUTE_VALUE"
31
      notification_type          = "ACTUAL"
32
      subscriber_email_addresses = [var.alert_email]
33
    }
34
  }
35
}
36

37
resource "aws_iam_policy" "deny_all" {
38
  name   = "platform-validation-deny-all"
39
  policy = jsonencode({ Version = "2012-10-17", Statement = [{ Effect = "Deny", Action = "*", Resource = "*" }] })
40
}
41

42
resource "aws_iam_role" "budget_action" {
43
  name = "platform-validation-budget-action"
44
  assume_role_policy = jsonencode({
45
    Version = "2012-10-17"
46
    Statement = [{
47
      Effect    = "Allow"
48
      Action    = "sts:AssumeRole"
49
      Principal = { Service = "budgets.amazonaws.com" }
50
      Condition = { StringEquals = { "aws:SourceAccount" = data.aws_caller_identity.current.account_id } }
51
    }]
52
  })
53
}
54

55
resource "aws_iam_role_policy" "budget_action" {
56
  name = "attach-deny-all-to-the-creation-role"
57
  role = aws_iam_role.budget_action.id
58
  policy = jsonencode({
59
    Version = "2012-10-17"
60
    Statement = [{
61
      Effect    = "Allow"
62
      Action    = ["iam:AttachRolePolicy", "iam:DetachRolePolicy"]
63
      Resource  = "arn:aws:iam::${data.aws_caller_identity.current.account_id}:role/${var.creation_role_name}"
64
      Condition = { ArnEquals = { "iam:PolicyARN" = aws_iam_policy.deny_all.arn } }
65
    }]
66
  })
67
}
68

69
# At the ceiling, Budgets attaches deny-all to the creation role. The sweeper's role is not touched.
70
resource "aws_budgets_budget_action" "kill_switch" {
71
  budget_name        = aws_budgets_budget.validation.name
72
  action_type        = "APPLY_IAM_POLICY"
73
  approval_model     = "AUTOMATIC"
74
  notification_type  = "ACTUAL"
75
  execution_role_arn = aws_iam_role.budget_action.arn
76

77
  action_threshold {
78
    action_threshold_type  = "ABSOLUTE_VALUE"
79
    action_threshold_value = 30
80
  }
81

82
  definition {
83
    iam_action_definition {
84
      policy_arn = aws_iam_policy.deny_all.arn
85
      roles      = [var.creation_role_name]
86
    }
87
  }
88

89
  subscriber {
90
    address           = var.alert_email
91
    subscription_type = "EMAIL"
92
  }
93
}

1
# residue.sh: a standalone version for one run ID, adapted from scripts/aws-residue.sh in aws-eks-reference-platform,
2
# not that file's text. The tagging API lists what was just deleted, so ask each ARN
3
set -uo pipefail
4
run_id="${1:?run-id}"
5
left=0
6

7
exists() { # exists <arn>: 0 while the resource is there and not on its way out
8
  local arn="$1" id="${1##*/}"
9
  case "$arn" in
10
    *:natgateway/*) aws ec2 describe-nat-gateways --nat-gateway-ids "$id" \
11
      --query 'NatGateways[].State' --output text 2>/dev/null | grep -qvE '^(deleted)?$' ;;
12
    *:subnet/*) aws ec2 describe-subnets --subnet-ids "$id" >/dev/null 2>&1 ;;
13
    *:security-group/*) aws ec2 describe-security-groups --group-ids "$id" >/dev/null 2>&1 ;;
14
    *:vpc-flow-log/*) aws ec2 describe-flow-logs --flow-log-ids "$id" \
15
      --query 'FlowLogs[].FlowLogId' --output text 2>/dev/null | grep -q . ;;
16
    *:volume/*) aws ec2 describe-volumes --volume-ids "$id" >/dev/null 2>&1 ;;
17
    *:key/*) aws kms describe-key --key-id "$id" --query 'KeyMetadata.KeyState' \
18
      --output text 2>/dev/null | grep -qvE '^(PendingDeletion|PendingReplicaDeletion)?$' ;;
19
    *) return 0 ;; # a type this script cannot ask about counts as left, never as gone
20
  esac
21
}
22

23
for arn in $(aws resourcegroupstaggingapi get-resources --tag-filters "Key=run-id,Values=${run_id}" \
24
  --query 'ResourceTagMappingList[].ResourceARN' --output text); do
25
  if exists "$arn"; then
26
    echo "left behind: $arn"
27
    left=1
28
  fi
29
done
30

31
[ "$left" -eq 0 ] && echo "nothing left from run ${run_id}"
32
exit "$left"

1
# .github/workflows/eks-validation.yml in aws-eks-reference-platform at v1.3.0, shortened: weekly, by hand only with a typed word, stoppable without a commit
2
on:
3
  schedule:
4
    # Wednesday: after Monday's Dependabot updates have merged, so the week's
5
    # pins reach AWS in days, not a week later.
6
    - cron: "17 1 * * 3"
7
  workflow_dispatch:
8
    inputs:
9
      confirm:
10
        description: "Type SPEND to create billed AWS resources (about $0.20 a passing run, up to about $0.50)"
11
        required: true
12
        type: string
13
# ...
14
jobs:
15
  up:
16
    name: Apply, install, check
17
    if: github.ref == 'refs/heads/main' && (inputs.confirm == 'SPEND' || (github.event_name == 'schedule' && vars.AWS_VALIDATION_ENABLED == 'true'))
18
    runs-on: ubuntu-latest
19
    environment: aws-validation
20
    timeout-minutes: 100
21
    # ... apply, install and check; a second job destroys, searches for residue and estimates the cost

1
# .github/workflows/eks-sweeper.yml in aws-eks-reference-platform at v1.3.0, shortened: what the account used yesterday, by service, before credits and refunds
2
start="$(date -u -d yesterday +%F)"; end="$(date -u +%F)"
3
# ... a table header for the run's summary
4
aws ce get-cost-and-usage --time-period "Start=$start,End=$end" --granularity DAILY \
5
  --metrics UnblendedCost --group-by Type=DIMENSION,Key=SERVICE \
6
  --filter '{"Not":{"Dimensions":{"Key":"RECORD_TYPE","Values":["Credit","Refund"]}}}' \
7
  --query 'ResultsByTime[0].Groups[].[Keys[0],Metrics.UnblendedCost.Amount]' --output text |
8
  awk -F'\t' '$2 > 0.005 {printf "| %s | $%.2f |\n", $1, $2; t += $2} END {printf "| **Total** | **$%.2f** |\n", t}'

Tradeoffs#

A passing run takes under an hour of runner time across its two jobs and costs between fourteen and twenty cents in AWS by its own list-price estimate, so a weekly run costs under a dollar a month. Failures cost more than success: the seven failed runs cost more than five times the one that passed, and the most expensive of them was the one that got furthest. The account has to exist inside an AWS Organization and hold nothing else. And the run proves a build from nothing with two system nodes, demo tenants and a Let’s Encrypt staging certificate. It says nothing about scale or load, and the platform’s README says so in its known limitations.

Skipping it costs nothing on every day before the first real apply. On that day the seven failures arrive in someone else’s account.

The closing argument#

Mocked tests and a kind suite are the right defaults, and they keep running on every change. They prove the platform agrees with itself. Only an apply proves it agrees with AWS, and GitHub and EKS moved the ground under this one twice since July without a commit in the repository. Twenty cents a week buys the answer before anyone needs it.

Sources#

• heyvaldemar, the EKS reference platform, the October release (its changelog carries every finding) and the passing validation run, 15 October 2026
• GitHub Docs, OpenID Connect reference, immutable subject claims (repositories created, renamed or transferred after 15 July 2026), accessed November 2026
• Kubernetes SIGs, metrics-server issue on a broken OpenAPI document, opened 1 September 2026
• heyvaldemar, my comment there, with the EKS builds and the working pin, 15 October 2026, and the fixed build reported, 16 October 2026
• heyvaldemar, the move to the fixed build, 18 October 2026
• Go, Go 1.27 release notes (the new JSON implementation is the default), accessed November 2026
• AWS, Budgets cost types (credits included by default) and budget actions, accessed November 2026
• AWS, EventBridge PutRule reference (a rule name of at most 64 characters), accessed November 2026
• AWS, Amazon ECR Public service quotas (500 gigabytes a month and one pull a second without authentication), accessed November 2026
• AWS, Amazon EKS pricing (the control plane per hour, standard and extended support), accessed November 2026
• AWS Builder Center, Testing Terraform for AWS With No AWS Account, 3 October 2026

Discussion#

If you prove infrastructure on a real account on a schedule, or you have a reason to trust the plan alone, drop a comment below. Counterarguments are welcome and the comment thread is where I respond first. For longer back-and-forth with senior practitioners, join the discussion on Discord.