← All posts

Terraform state: lessons from three disasters

Terraform state is the most dangerous file in your infrastructure, and most teams treat it more casually than their application code.

Disaster one: two engineers ran apply concurrently against local state. The fix is table stakes now — remote state with locking from day one — but the deeper lesson was that convenience defaults are production risks.

Disaster two: a CI misconfiguration pointed a destroy at the wrong workspace. Lesson: production applies go through a plan that a human reads, environments live in separate state files with separate credentials, and destroy requires a flag nobody sets by accident.

Disaster three: a stuck lock, a hasty force-unlock, and a state file that no longer matched reality. Lesson: state surgery is an incident. Declare it, snapshot the state first, and do imports and moves one resource at a time with a second pair of eyes.

Boring process is what makes infrastructure as code safe. The code is the easy part.

// more like this

Blameless postmortems that actually change things

Set SLOs before you build dashboards

Comments

Comments are moderated before appearing.