Ask most firm administrators whether their backups work and they will say yes, because the backup software shows a green checkmark. But a green checkmark means the job ran, not that the data can be restored, quickly, completely and in a usable state. The only way to know is to restore something and see.
Restore testing is cheap insurance and also useful evidence for clients, auditors and cyber insurers. Here is a practical program.
Why backups fail quietly
Common causes of a failed recovery that nobody noticed:
- A new server or database was never added to the backup job
- Credentials changed and the job has been failing partially
- Files were backed up but the application database was not, so the application will not start
- The backup is corrupted, or the media is damaged
- The restore takes far longer than anyone assumed
- Backup encryption keys or passwords are lost
- The backup was deleted or encrypted by an attacker who had access to it
Three levels of testing
Level 1: File and mailbox restores (monthly)
Pick a few random files, from different folders and dates, and restore them to a temporary location. Open them to confirm that they are intact. Restore one mailbox item or a folder from Microsoft 365 backup. Record how long the process took and who did it.
This check is quick and catches broken jobs and permission problems.
Level 2: Full system or application restore (quarterly or semiannually)
Restore a full server or critical application, such as your document management database or practice management server, into an isolated test environment. Confirm that:
- The system boots and the application launches
- Recent data is present
- Users can log in and perform a sample task
- The time taken is consistent with your recovery time objective
Testing in isolation matters, so that you do not disturb production or create duplicates.
Level 3: Full scenario exercise (annually)
Simulate a major event, such as loss of the primary office or ransomware, and walk through the recovery with partners, staff and your IT provider. Include the human questions: who declares a disaster, how are clients contacted, how do attorneys work meanwhile, and who handles deadlines.
What to measure
- Recovery time: how long from start to a usable system?
- Recovery point: how recent was the data you recovered compared to the failure moment?
- Completeness: was anything missing, including permissions, metadata or integrations?
- Dependencies: did you need something unexpected, like a license key, a domain controller or a specific person?
- Bandwidth: if restoring from the cloud, how long did the data take to download?
What to record
Keep a simple log that includes:
- Date of the test and who performed it
- System, data or mailbox tested
- Backup date and location used
- Steps taken and any problems encountered
- Time to restore
- Result, pass or fail, and corrective actions
- Sign-off by the administrator or a partner
Store the log somewhere that survives an outage. Insurers often ask whether you test restores, and a dated record is a convincing answer.
Protecting the backups themselves
Testing should also confirm that your backup is protected from compromise:
- Is there an immutable or offline copy that an administrator account cannot delete?
- Are backup credentials separate from your regular administrator accounts and protected with MFA?
- Is the backup encrypted, and are the keys stored safely and separately?
- Is someone alerted when a job fails, and does someone act on it?
Do not forget cloud data
Test restores for Microsoft 365 and other cloud applications, too. Deleted items, accidental overwrites and compromised accounts all require recovery, and the vendor's own retention is not a substitute for your own backup.
Fix what you find
A failed test is a success for the program, since you found the problem on a quiet Tuesday rather than in a crisis. Assign each issue an owner and date, and retest after the fix.
Making it routine
Put tests on the calendar for the year, assign them to named people and review results at a quarterly partner or administrator meeting. When tests become routine, they stop being a project.
How Counsel Cyber can help
We run and document scheduled restore tests for the firms we support, and can set up a testing program for your existing backups. Ask us to run a first test and report the findings.