The Unseen Clockmaker: On Setting the Chronometer for Your Daily Backups
There is a quiet, almost meditative satisfaction in building a mechanism that works precisely as intended, without fuss or fanfare. It’s the satisfaction of the clockmaker, who knows that the true art lies not in the ornate face, but in the reliable, meshing gears hidden behind it. For those of us tending to small services, our most crucial clockwork is the daily backup. We trust it implicitly, until the day we need it and discover the mechanism has been still for who knows how long.
The common advice is to "test your backups," which is absolutely correct, but also a bit like saying "ensure your clock tells the right time." It’s a grand, overarching principle that lacks a specific, actionable technique. The question isn't *whether* to test, but *how* to build the test directly into the mechanism itself. My method is simple: I don’t just back up the data; I back up a known, verifiable signal along with it.
Here is the concrete technique. Each night, as your backup script runs, have it append a single, small, highly specific file to the archive. This file should be unique and easily identifiable. I create a plain text file named `verify_backup_YYYY-MM-DD.txt`. Inside it, I write one line: the precise timestamp of the backup’s completion, in UTC, followed by a short checksum of the backup file itself, which any scripting language can generate. This file is your backup's own birth certificate.
The second part of the mechanism is a separate, independent verification script. This script, which runs in the morning, does not assume the backup was successful. Its only job is to look at the most recent backup archive, find the verification file within it, and read its contents. It checks that the date inside matches the archive's date and that the checksum still matches. It then logs its findings—a simple "VERIFICATION SUCCESS" or "VERIFICATION FAILURE"—to a dedicated log file and, crucially, sends a brief status email.
This decouples the act of backing up from the act of verifying. The backup script’s job is to create the archive; the verification script’s job is to forensically examine it. By embedding a known-good artifact and having a separate process interrogate it, you transform an opaque, trusted process into a transparent, verified one. You are no longer hoping the clock is right; you have built a second, simpler clock whose sole purpose is to confirm the first one is ticking. It is a small, boring, and profoundly reliable piece of technology.
Notes & further reading
A few pages I came back to while writing this: