Breaking up a 9,000-line Terraform root
One state file, 25-minute plans, and a lock everyone fought over. Split into 40 small stacks with a clear dependency order.
- Date
- 2024-07-03
- Type
- build
- Duration
- 5 weeks
- Impact
- Median plan time from 25 minutes to under 90 seconds. State lock contention gone.
PLACEHOLDER POST — seed data. Replace with your real writeup.
Every infrastructure change, from a DNS record to a VPC, went through the same root module and the same state file. A plan refreshed 3,100 resources. Two engineers could not work at the same time.
How we split it
Stacks were cut along blast radius and rate of change, not along AWS service lines:
| Layer | Changes | Examples |
|---|---|---|
| foundation | yearly | accounts, VPCs, transit gateway |
| platform | monthly | EKS, RDS, shared IAM |
| service | daily | queues, buckets, per-service IAM |
Lower layers publish outputs to SSM parameters. Higher layers read them, so no stack reads another stack’s remote state directly.
Moving resources without recreating them
terraform state mv -state-out=../platform/eks.tfstate \
module.eks module.eksWe scripted every move and ran it against a copy of state first. The rule was that each move had to end in an empty plan on both sides before we did the next one.
Result
- 40 stacks, each planning in seconds
- CI runs plans only for the stacks a PR touches
- Applying the foundation layer now needs two approvals; the service layer needs one
← all writeups