aman
← all writeups

depends_on on a module replaced my NAT

A module-level depends_on made Terraform plan to replace the only NAT instance, plus OIDC and IAM, every time anything in the VPC module changed.

Date
2026-06-08
Type
note
Impact
Applying the plan would have replaced the cluster's only NAT instance, and every private subnet would have lost egress while it was rebuilt. Caught at plan time. Nothing was applied.

I made a small change to the S3 gateway endpoint in the VPC module. The plan came back wanting to replace the NAT instance. It also churned the EKS OIDC provider and some IAM. None of those resources had been touched. At the time that NAT was the only egress for every private subnet, so this plan was an outage.

What actually happens

The root module had this:

module "eks" {
  private_subnet_ids     = module.vpc.private_subnet_ids
  vpc_id                 = module.vpc.vpc_id
  private_route_table_id = module.vpc.private_route_table_id
  depends_on             = [module.vpc]
  # ...
}

depends_on on a module doesn’t mean “wait for the VPC”. It means that every data source inside module.eks depends on everything in module.vpc. When anything in module.vpc has a pending change, Terraform won’t read those data sources at plan time. It defers them to apply.

Why

The NAT instance’s subnet_id came from a data "aws_subnet" lookup by tag, inside module.eks. Once that read was deferred, the subnet ID became (known after apply). subnet_id forces a new instance, so Terraform had to assume it might change and planned a replacement. The OIDC and IAM churn came from the same deferred reads.

The depends_on added nothing. The module already used three module.vpc outputs, and those references are real dependencies at the attribute level.

What to do instead

  • Delete module-level depends_on when the module already uses the other module’s outputs. The implicit graph is correct and it’s precise.
  • If you really need ordering, put the output reference on the single resource or data source that needs it, not on the whole module.
  • Read the reason on every must be replaced. If it’s known after apply on an input you never changed, a data source got deferred.

My first move was dumb. I reverted the endpoint change to make the plan go quiet. Thirteen minutes later the real fix landed: delete one line in the root module. The endpoint went back in four days later with a clean plan.

Module-level depends_on doesn’t make ordering safe. It hides the dependency graph from the plan.


← all writeups