aman®
← all writeups

A Terraform plan that looked like a no-op

One renamed resource and one missing moved block. The plan said 1 to destroy, and nobody read that line.

Date
2024-10-29
Type
postmortem
Severity
SEV2
Duration
2h 30m
Impact
Production DB parameter group recreated; 11 minutes of connection failures during the forced reboot.

PLACEHOLDER POST — seed data. Replace with your real writeup.

The PR was a tidy-up that renamed a few resources to match a naming convention. The plan output was 400 lines long. The line that mattered was in the middle of it: Plan: 1 to add, 0 to change, 1 to destroy.

Timeline

Time Event
16:10 PR approved, pipeline runs terraform apply
16:12 Parameter group destroyed and recreated with defaults
16:14 RDS instance rebooted to apply the new group
16:25 Connections recover, but max_connections is now the default
17:30 Pool exhaustion under evening load; correct values restored
18:40 Verified settings, incident closed

The actual cause

Renaming a resource address without a moved block tells Terraform to delete the old resource and create a new one. Code review looked at the diff, not the plan. The pipeline applied automatically after approval.

moved {
  from = aws_db_parameter_group.pg
  to   = aws_db_parameter_group.orders_primary
}

What changed

  • A policy check fails the pipeline on any delete against a protected resource type
  • The plan summary (add/change/destroy) is posted to the PR as a comment
  • lifecycle { prevent_destroy = true } on stateful resources
← all writeups