← All posts
Devesh SainiDevesh SainiDevesh Saini — SRE · Cloud & DevOps Engineer

Terraform state: lessons from three disasters

// listen

Listen to this article

Neural voiceDevesh Saini

Terraform state is the most dangerous file in your infrastructure, and most teams treat it more casually than their application code.

Disaster one: two engineers ran apply concurrently against local state. The fix is table stakes now — remote state with locking from day one — but the deeper lesson was that convenience defaults are production risks.

Disaster two: a CI misconfiguration pointed a destroy at the wrong workspace. Lesson: production applies go through a plan that a human reads, environments live in separate state files with separate credentials, and destroy requires a flag nobody sets by accident.

Disaster three: a stuck lock, a hasty force-unlock, and a state file that no longer matched reality. Lesson: state surgery is an incident. Declare it, snapshot the state first, and do imports and moves one resource at a time with a second pair of eyes.

Boring process is what makes infrastructure as code safe. The code is the easy part.

// more like this

AWS Cloud Explained: Services, Benefits & How to Get Started

Blameless postmortems that actually change things

Kubernetes Requests and Limits: A Practical Sizing Guide

Comments

Comments are moderated before appearing. Leave your email to be notified when yours is approved or replied to.