Several support incidents taught me the same hard lesson: when data is at risk, the first priority is preserving the data, not troubleshooting.
Recovery process
- Stop making changes.
- Create backups before touching anything.
- Preserve evidence (logs, exact state).
- Verify storage health.
- Verify database integrity.
- Recover first, optimize later.
Why the order is strict
Every well-intentioned "quick fix" applied before a backup is a chance to turn a recoverable incident into permanent data loss. Discipline beats cleverness here.
An incident, anonymized
A production database holding years of traceability data stopped accepting writes. The line kept running - results were queuing in memory - so the pressure was enormous and rising by the minute.
The almost-fatal moment: someone suggested restarting the server "to see if it clears". The disk was failing. A restart would very likely have been the last time that disk ever spun up. Instead:
- Writes were redirected to a spare machine so the line could keep running - removing the time pressure was the single most valuable move.
- A raw block-level copy of the failing disk was taken first, before any repair tool ran. Repair tools write to the disk they are repairing; on dying hardware they can destroy what they are trying to save.
- All recovery experiments ran against copies of that image, never the original. Failed attempts cost nothing but time.
- The database was rebuilt from the image plus transaction logs, then verified against production records before anyone declared success.
Total data loss: a few minutes around the initial failure. Without step 2, it would have been years.
What makes this hard in practice
It is not technical knowledge - every step above is standard. It is executing them under pressure, with a line manager asking for an ETA every ten minutes and a "quick fix" always available that probably works and possibly destroys everything. The team that has written the procedure down beforehand, and decided in advance who can authorize risky steps, recovers in hours. The team improvising recovers in weeks, or never.
The unglamorous conclusion: the best recovery story is a tested backup that made the recovery boring. Backups that have never been restored from are hopes, not backups.