← Back
Terraform Drift Detection and Recovery
Energy

Terraform Drift Detection and Recovery

Bringing a large application portfolio back in line with its infrastructure code

Overview

Over time, manual changes made outside Terraform had left the infrastructure of more than 30 applications out of step with the code meant to describe it. Nobody noticed until the drift caused an incident. I recovered every affected application and set up scheduled drift detection with alerts, so new drift is flagged the day it appears.

Problems

Manual changes made outside Terraform had pulled deployed infrastructure away from the code across the portfolio

More than 30 applications had drifted, several with multiple discrepancies, so the Terraform code was no longer a reliable record of what was running

Nothing compared deployed state with approved configuration, so unreviewed changes went unseen

With no detection in place, drift built up until something broke

Goals

Restore an accurate picture of what infrastructure actually exists

Bring each drifted application back into line without causing new outages

Turn a one-off cleanup into a standing process so drift does not creep back

Alert the team when drift happens instead of when something fails

What Was Done

01

Recovered more than 30 applications containing multiple instances of drift

02

Documented a remediation process the team now uses as its standard

03

Added automated alerting so new drift is caught early

Results and Impact

More than 30 applications back in line with their Terraform code

Every application's live state is now checked against its approved configuration

Infrastructure across the portfolio is consistent, with the code as the single source of truth

New drift is caught by alerts, and the team has a repeatable process for handling it

Stack

AWSTerraform