Terraform Drift Detection and Recovery
Bringing a large application portfolio back in line with its infrastructure code
Overview
Over time, manual changes made outside Terraform had left the infrastructure of more than 30 applications out of step with the code meant to describe it. Nobody noticed until the drift caused an incident. I recovered every affected application and set up scheduled drift detection with alerts, so new drift is flagged the day it appears.
Problems
Manual changes made outside Terraform had pulled deployed infrastructure away from the code across the portfolio
More than 30 applications had drifted, several with multiple discrepancies, so the Terraform code was no longer a reliable record of what was running
Nothing compared deployed state with approved configuration, so unreviewed changes went unseen
With no detection in place, drift built up until something broke
Goals
Restore an accurate picture of what infrastructure actually exists
Bring each drifted application back into line without causing new outages
Turn a one-off cleanup into a standing process so drift does not creep back
Alert the team when drift happens instead of when something fails
What Was Done
Recovered more than 30 applications containing multiple instances of drift
Documented a remediation process the team now uses as its standard
Added automated alerting so new drift is caught early
Results and Impact
More than 30 applications back in line with their Terraform code
Every application's live state is now checked against its approved configuration
Infrastructure across the portfolio is consistent, with the code as the single source of truth
New drift is caught by alerts, and the team has a repeatable process for handling it
Stack
