Disaster recovery
The first full recovery drill in years, run by AI
A B2B SaaS company's recovery plan had not been seen to work since 2022. In 2026 the drill ran end to end as a one-click pipeline, verified 100 of 100 tables and was signed off as audit evidence.
Where it stood
No disaster recovery drill since 2022 had completed a database restore. The previous drill's data step had failed while the dashboard still showed green, so the record said the company could recover and nobody had watched it happen.
The recovery environment had been deployed once, ever, and sat four releases behind production. One recovery web server had stopped replicating more than a year earlier. The automation that was supposed to bring it all back lived on one engineer's laptop.
What I built
A CI/CD pipeline that runs the drill, start to finish, from one click: cloud failover, disk resizing, database configuration, backup restore, a 31-script data deploy with a row-count gate, table-by-table replication checks, traffic cutover and automated browser tests against the recovered app. When the drill is done it tears the recovery environment down, so a test never leaves a second copy of production running up the cloud bill.
I scoped the work, set the design and moved the laptop automation into a pipeline anyone on the team can run. An AI coding agent working under my direction wrote much of the pipeline code; I reviewed every change and ran production. The agent added one guardrail on its own: the pipeline refuses to run if any production name appears in its inputs. Failed runs tear themselves down too.
We ran it 13 times in a week. The drill found four blockers that would have stopped a live recovery, and each one was fixed by change request before the next run.
The numbers
| Measure | Before | After |
|---|---|---|
| Last drill with a completed database restore | 2022 | 2026 |
| Replication check after restore | Showed green, had failed | 100 of 100 tables verified |
| Cloud failover step | No timed run | About 11 minutes |
| Failover to a verified app | Not reached since 2022 | 6h 14m on the first full run |
| A clean drill, end to end | No complete run | Near 5h 20m by the rewritten runbook |
| Who can run it | One engineer's laptop | Anyone on the team, one click |
Step times and the table count come from the pipeline logs and the drill record. Nobody timed the earlier manual drills, so the before column says so.
Why it matters
A recovery plan nobody has seen finish is a guess, and the one test that ran had failed while showing green. Insurers, auditors and large customers' security questionnaires increasingly ask for proof the business comes back after a bad day, and a filed drill record is that proof. The drill is also the only way to find the four things that would have stopped a live recovery before the day you need them gone.
The first full run came back in 6h 14m against a 24-hour recovery target. A drill that cheap to run gets run, and a recovery environment that gets run stops drifting.
For a client I do the same job: read what the recovery plan says, run it end to end, fix what stops it, and leave behind a drill the team can run with one click.
We spin up, we test, we tear down. A drill cheap enough to run often, or DR rots.
