Why Most Backup Tools Fail the Restore Test

Restore testing is the practice of verifying that a backup can be fully recovered into a working state, not just confirming the backup was created. For git repositories, this means extracting the backup to a clean environment and confirming that the object database passes git fsck, all branches and tags are present, the full commit history is intact, and any uncommitted work (staged changes, stashes, untracked files) was captured. A backup that has never been restore-tested is an assumption, not a guarantee.

Every developer has backups. Almost none of them have tested a restore.

This is the backup/restore asymmetry: creating a backup is easy, fast, and gives you a green checkmark. Restoring from that backup is hard, slow, and most people never try it until the day they actually need it. By then, it is too late to find out the backup was incomplete, corrupted, or locked behind a vendor you can no longer reach.

The backup succeeds silently. The restore fails loudly. And it fails on the worst possible day, when your machine is dead, your disk is gone, or your repository is corrupted beyond repair.

A backup you have never restored is not a backup. It is a hope.

Most backup tools are optimized for the backup side of this equation. They report success when the upload completes. They count bytes transferred. They log timestamps. What they do not do is prove that the data on the other end can be turned back into a working repository, with every file, every commit, every branch, and every piece of uncommitted work intact.

Backup tools fail restores in predictable ways. These are the categories we see most often.

Filesystem-level backup misses git internals

The most common mistake is treating a git repository like a folder of files. Tools that copy the working directory get the files you can see, but miss the data that makes git work: the object database, the pack files, the refs, the reflog, the hooks, the submodule configurations. A filesystem copy of a repository is not a repository. It is a snapshot of one moment's checkout.

Try restoring a filesystem copy and running git log. If the object database was not captured correctly, you will get errors. If the pack files were copied mid-write, you will get corruption. If the refs were copied out of order, branches will point to commits that do not exist. The files are there. The history is gone.

Incremental backup chains break

Incremental backups save space by only storing what changed since the last backup. This is efficient on the backup side. On the restore side, it means every restore depends on an unbroken chain of increments going back to the last full backup.

If any link in that chain is corrupted, missing, or inaccessible, the restore fails. Not partially. Completely. One bad increment in a chain of 200 means all 200 are useless. The more increments you have, the more fragile the restore becomes. The backup tool shows "199 successful backups." The restore shows "chain broken at increment 47."

Encrypted backups with lost keys

Encryption is necessary. But encryption without key management is a time bomb. If the key that encrypted your backup is stored on the same machine the backup is protecting, you lose both at the same time. If the key is stored in a password manager that requires a working machine to access, same problem.

The worst version of this: the backup tool manages the key for you, stored on the vendor's servers. Your backup is encrypted. Your key is held by someone else. If that vendor goes down, gets acquired, or changes their terms, your encrypted backup becomes an encrypted brick.

Vendor-dependent restore processes

Some backup tools require their own software, their own servers, or their own authentication to restore. This creates a dependency chain: to get your data back, you need the vendor to be online, authenticated, and running the same version of their software that created the backup.

This is the worst possible architecture for an emergency. Emergencies happen when things are broken. If your restore process requires a functioning internet connection, a running vendor service, and a valid subscription, you have added three new failure modes to the moment when you can least afford them.

Missing uncommitted work

Most backup tools that are git-aware will capture committed data. But the work developers lose most painfully is the work they had not committed yet: the staging area, the working directory changes, the stashed experiments, the in-progress merge or rebase.

A backup that captures only committed history is a backup of last week. The hours of work sitting in your working directory, not yet committed, are gone. And those are the hours that hurt the most, because they are the hours you remember doing.

There is a fundamental difference between two things that most backup tools conflate:

• "Upload succeeded" means bytes were transferred to a destination and the destination acknowledged receipt.
• "Restore will work" means those bytes, when retrieved and reassembled, will produce a complete, functional, verified copy of the original data.

Most backup tools only check the first one. They verify that the upload completed, that the file size matches, maybe that a checksum passed. Then they report success.

But upload success does not mean restore success. The upload could have succeeded while the data was already corrupted on disk. The checksum could match a corrupted file. The backup could be complete but missing a critical piece of metadata that the restore process needs. The archive format could be valid but the contents could be internally inconsistent.

This gap is invisible during normal operations. Every backup reports green. Every log shows "completed." The gap only becomes visible on the day you try to restore, and by then you are staring at the consequences of every assumption the tool made on your behalf.

GitLab, January 2017

On January 31, 2017, a GitLab engineer accidentally deleted a production database directory. This is a bad day, but it should not be a catastrophic one. Backups exist for exactly this scenario.

GitLab had five independent backup methods running:

1. Regular database dumps (pg_dump): Had not been working. Silently failed due to a configuration error. No one noticed.
2. Automated snapshots (Azure): Not configured for the production database server.
3. Continuous archiving (WAL-E): Also not configured for the production database server.
4. LVM snapshots: Were being taken, but the process was inconsistent and the snapshots were untested.
5. Disk snapshots (Azure): The only method that partially worked. They recovered from a snapshot that was six hours old, losing six hours of production data across all of GitLab.com.

Five backup methods. Four completely failed. One was six hours stale. None had been tested with a real restore. GitLab, to their enormous credit, live-streamed the recovery and published a detailed post-mortem. Most companies would not be that transparent. But the lesson is stark: the number of backup methods is irrelevant if none of them have been tested with an actual restore.

The post-mortem revealed that the regular database dumps had been silently failing for months. The tool reported no error. It simply did not produce a usable backup. Nobody checked because the process was assumed to work.

The Time Machine discovery

A pattern we hear repeatedly from developers: Time Machine shows a full backup history, green checkmarks, years of data. Then the hard drive fails. They go to restore. And they discover one of several problems:

• The Time Machine backup drive itself has disk errors, and the backup is partially corrupted.
• The backup completed but certain directories were excluded by a rule the developer did not set (or did not remember setting).
• The .git directory was backed up, but it was backed up mid-operation, with locked index files and partially written pack files. The repository is technically present but internally inconsistent.
• The backup is fine, but the restore takes 14 hours over USB, and the developer needed to ship code today.

Time Machine is a good tool. It does what it says it does. But it is a filesystem-level tool. It does not understand git internals, it does not verify repository integrity, and it does not distinguish between "all the files are there" and "the repository actually works."

A real restore test is not "can I download the backup file." A real restore test answers the question: if my machine was gone right now, could I get back to a working state from this backup alone?

Here is what that actually requires:

1. Restore to a clean machine. Not the same machine. Not a directory next to the original. A machine with nothing on it. If the restore depends on something already present on your system, it is not a complete backup.
2. Verify every file. Not just that files exist, but that their contents match. Byte-for-byte comparison against the original. If even one file differs, find out why.
3. Check git history. Run git log, git branch -a, git stash list. Every commit, every branch, every tag, every stash should be present. If the backup only captured the current branch, the rest of your history is gone.
4. Run the tests. If your project has tests, run them. A restore that produces files but breaks the build is not a successful restore. Dependencies, configurations, environment variables, build artifacts: all of these matter.
5. Check uncommitted work. Was there anything in your working directory that was not committed? Was there anything in the staging area? Stashed changes? An in-progress rebase? These are the things most backup tools miss entirely.
6. Measure the time. How long did the restore take? If it takes 8 hours to restore, and your deadline is in 4, the backup exists but is not operationally useful.

Most developers have never done this. Most backup tools have never been subjected to this. And until you do it, every "backup succeeded" message is a statement of faith, not a statement of fact.

# A minimal restore test checklist
# Run this against your restored repo, not the original

# 1. Does the repo exist and have valid objects?
git fsck --full

# 2. Is the complete history present?
git rev-list --count --all

# 3. Are all branches present?
git branch -a

# 4. Are all tags present?
git tag -l

# 5. Are stashes present?
git stash list

# 6. Does the reflog exist?
git reflog show HEAD

# 7. Do the tests pass?
make test  # or: npm test, go test ./...

These are two different things. Most tools do the first. Almost none do the second.

Backup verification answers: "Did the backup process complete without errors?" It checks that the file was created, that the upload finished, that the size is nonzero, that a checksum matches. This is necessary but not sufficient.

Restore verification answers: "Can this backup be turned back into the original data?" It proves that the backup is not just present but functional. That the archive can be opened. That the data inside is consistent. That the git objects form a valid graph. That the files match the originals. That nothing was silently dropped.

The difference matters because there are entire categories of failure that pass backup verification but fail restore verification:

• A zip file that was created successfully but contains a corrupted pack file inside it. The zip is valid. The pack file is not.
• An encrypted archive where the encryption succeeded but the key needed for decryption is stored in a location that will not survive the same failure that triggered the restore.
• An incremental backup that uploaded correctly but depends on a base snapshot that has already been deleted by a retention policy.
• A backup that captured the committed history but silently skipped the working directory because the tool treated uncommitted files as temporary.

Every one of these shows "backup succeeded" in the logs. Every one of these fails on restore day.

A backup system that actually protects you has several properties that most tools lack:

• Git-aware capture. It reads directly from git, not the filesystem. The backup is a valid git artifact, not a folder of files.
• Uncommitted work included. It captures everything in your working state: the staging area, the working directory, stashed changes, in-progress operations. The work you have not committed yet is the work you will miss the most.
• Restore verification built in. It does not just confirm the upload. It proves that the backup can be restored. It verifies the data after encryption, after upload, and before you need it. Every backup is tested before it is trusted.
• No vendor dependency on restore. You can restore without the backup tool's servers being online. You can restore without a subscription. The backup format is something you can open with standard tools if the vendor disappears. Your data belongs to you, and you can get to it with or without anyone else's cooperation.
• Keys you control. Encryption keys stay on your machine. They are not stored by the vendor. They are not managed by a service you do not control. If you lose the key, that is your responsibility, but at least it is your responsibility and not a vendor's business decision.
• Storage you own. The backup goes to storage you control: your own S3-compatible bucket, your own infrastructure. Not a vendor's proprietary storage where your data lives at the vendor's discretion.

These are not exotic requirements. They are the minimum for a backup system that actually works on the day you need it. The problem is that most tools skip the hard parts (restore verification, uncommitted work capture, vendor-independent restore) because those are the parts that require understanding git at the object level, not the filesystem level.

The question is simple: if your machine died right now, could you get back to exactly where you were? Not last week's committed code. Not yesterday's push to remote. Right now. Every file, every branch, every uncommitted change.

If the answer is "I think so," that is not good enough. Test it. Find out. The time to discover your backup does not work is today, not the day you actually need it.