Running a Restore Drill: Scenario, Roles and Pass Criteria

Most organizations have backups. Very few know for certain that they can restore. The gap between the two is not closed by a dashboard full of green ticks. It is closed by periodically sitting down, assuming a specific failure, and actually recovering the system while somebody holds a stopwatch.

This article describes how to run a restore drill so that it produces something usable: a scenario specific enough to act on, a clear role assignment, pass criteria agreed before you start, timing that matches the RTO you committed to, verification that the data is genuinely correct rather than merely present, and a write-up that makes the next drill better.

Why a successful backup job is not evidence

A backup job reporting success proves only that a write process finished without raising an error. It says nothing about the things that actually decide the outcome of an incident:

• Whether the backup can be expanded on the software versions you run today.
• Whether it contains everything the system needs to run — configuration, secrets, schema, reference data — and not just the primary tables.
• Whether the person on call at three in the morning knows where it is and what permissions they need to fetch it.
• Whether the real expansion time fits inside a window the business can tolerate.

There is exactly one way to answer those four questions, and that is to try.

Step 1 — design the scenario

A good scenario is specific enough that participants do not have to ask clarifying questions. It states what broke, when it broke, who noticed, and what is still working. That last point matters more than it looks. If you do not say what survived, everyone assumes everything survived, and the drill becomes far too easy.

Three scenarios worth keeping in the catalogue

• Loss of a primary database. The classic: the primary instance is unreachable and the replica state is ambiguous. Tests both restore capability and the failover procedure.
• Data encrypted across a shared file server. Here the interesting part is not the restore itself but identifying the last known-good point: which backup generation is unaffected. It forces the team to work with backup version history rather than reflexively grabbing the newest copy.
• A deletion noticed late. A job or a manual action removed a dataset, and nobody reported it for several days. This is the hardest scenario, because it requires restoring part of the data into a live, running system rather than restoring everything into an empty one.

Run one scenario per drill. Cramming all three into a single session guarantees none of them is finished properly.

Pick a realism level

You do not have to break production to run a serious drill. Three levels are commonly used: restore into an isolated environment while production keeps running; restore and let a group of internal users work on it for real; or actually shift traffic to the restored environment. Start at level one and only move up once the previous level has passed several times in a row.

Step 2 — assign roles

Drills usually fail for lack of clarity about who decides what, not for lack of technical skill. Assign roles to named individuals before you start, and state the boundary of each role explicitly.

Role	Responsible for	Must not
Drill coordinator	Announcing the scenario, keeping the timeline, calling a stop if needed	Perform the restore steps personally
Restorer	Executing the restore, working only from the existing runbook	Use undocumented tribal knowledge without noting it
Decision maker	Choosing the recovery point, accepting the data loss, declaring completion	Interfere in technical execution
Verifier	Running the integrity checks and business acceptance tests	Take part in the restore, so independence is preserved
Scribe	Recording every timestamp, every blocker, every open question	Offer hints to the restorer
Observer	Representing the business or a dependent team, asking questions afterwards	Direct the work while it is running

Two details are worth calling out. First, the restorer should be the least familiar qualified person on the team, not the expert who built the system. The expert will finish quickly from memory, and you will never discover that the runbook is missing three steps. Second, verification has to be independent. If the person who ran the restore also declares the data correct, you are marking your own homework.

Step 3 — agree the pass criteria before you start

Pass criteria must be written down and agreed before the scenario is announced. Defined after the fact, they always end up shaped to fit whatever happened. A workable set usually covers:

1. Time. The service returns to a serving state within the committed RTO.
2. Data loss. Loss falls inside the committed RPO, expressed as a measured number rather than an opinion.
3. Integrity. Every integrity check defined below passes.
4. Documentation only. The restore completes without calling someone who is not part of the assumed on-call rotation.
5. No side effects. No system outside the scenario scope is affected.

State explicitly that a failed drill is still an organizational success. The point is to find problems while nothing is at stake, and if the team believes a failure will be held against them, they will unconsciously pick easier scenarios.

Step 4 — time it honestly

Timing numbers only mean something if everyone agrees on the start and end markers. The most commonly disputed method is starting the clock when the engineer types the first command, which conveniently skips the slowest part of a real incident.

Set t0 at detection, not at the first command, and record the following:

• t0 — the failure is detected.
• t1 — the decision maker confirms a restore is required.
• t2 — the backup and recovery point are selected.
• t3 — the data has finished expanding.
• t4 — the application starts and connects successfully.
• t5 — the verifier confirms the checks pass.
• t6 — the service is declared open to real users.

RTO is measured as t6 minus t0. The intermediate markers tell you where the time went, and in most first drills the stretch from t0 to t2 is far longer than anyone predicted — finding documentation, finding someone with the right permissions, and arguing about which copy to use.

Step 5 — verify integrity, not just presence

This is the step most often done superficially. Files appearing and a service starting does not mean the data is right. Prepare the check set before the drill and hand it to the verifier:

• Record counts. Compare row counts on key tables against the expected value at the recovery point, not against current production.
• Checksums. For files and blobs, compare checksums rather than sizes.
• Referential integrity. Run queries that look for orphaned child records, foreign keys pointing nowhere, or broken state sequences.
• Landmark records. Pick specific records from several points in time in advance, and check them field by field after the restore.
• Business reconciliation. Re-run a familiar reconciliation report and compare results — a balance total, daily order totals, or record counts by status.
• Real user journeys. Perform a few end-to-end business actions on the restored system: sign in, look something up, create, edit, then reload and confirm the change persisted.

If your system has components restored independently, add a cross-component consistency check. A frequent failure is a database restored to one point in time and an object store restored to another, leaving records that reference files which do not exist. That defect rarely shows up in a smoke test but surfaces immediately when a real user opens the affected record.

Step 6 — write it up

The drill report should be short and always structured the same way, so that runs can be compared. One page is usually enough:

• Scenario run, date and time, participants and their roles.
• A table of t0 through t6 and total elapsed time against RTO.
• Measured data loss against RPO.
• Result of each integrity check: pass or fail, with detail on any failure.
• Every point where the team got stuck, with the cause.
• Corrective actions, each with a named owner and a due date.

That last item determines whether the drill was worth running at all. A drill that surfaces nine problems and fixes none of them will surface the same nine next time. Put the corrective actions into whatever work tracking system the team actually uses daily, rather than leaving them inside the report file.

Cadence and common mistakes

A workable rhythm for most organizations is a small drill per quarter for a given system, and one larger annual exercise that involves the business. Add an extra run after any major change to the storage architecture or after switching backup tooling.

Recurring mistakes: announcing the drill in so much detail that everyone pre-stages the work; always using the same expert as restorer; only ever drilling a full restore and never a partial one; skipping configuration and secret recovery entirely; and forgetting to check that the backup system itself can be recovered. The last one is more serious than it sounds. If the backup catalogue lives alongside the system you just lost, you have the data but no way to locate it.

How HiTechCloud can help

HiTechCloud is an AWS partner and a Google Premier Partner, and operates in-country infrastructure. On restore drills we can help design scenarios that fit your systems, act as coordinator or independent verifier during the exercise, build an automated integrity check suite you can re-run each time, and revise the recovery runbook based on where the team actually got stuck.