The Keeper's Two Jars: On the Preservation of the Staple and the Spice

On my shelf, I keep two jars. One is made of thick, sturdy glass, sealed airtight. It holds rice, a staple, plain and reliable. Its contents change slowly, predictably. I know its weight, its volume, and its purpose. The other jar is smaller, crafted from clay, with a loose-fitting lid. It contains a special blend of spices, vibrant and potent. It is used infrequently, but its presence is essential for the occasional meal that requires a certain kind of magic. In my work keeping small services running, I’ve come to see these two jars as the perfect analogue for two fundamentally different, equally vital, approaches to preserving state: the snapshot and the event log.

The snapshot is my jar of rice. It is the complete picture, a full backup of a database, a virtual machine image frozen in time. It is monolithic, heavy to lift, and incredibly straightforward. When you restore from a snapshot, you get everything, exactly as it was at a single moment. There’s a brute-force elegance to it. You don't need to know what happened before or after; you have a perfect clone. This is the tool for catastrophic failure, for when the entire system needs to be rolled back to a known good state. It’s the staple. Dependable, life-sustaining, but not particularly refined. It tells you what was, but it is silent on the journey.

The event log, in contrast, is my jar of spices. It is not the state itself, but the history of all the tiny actions that led to the current state. Every user login, every data update, every configuration change is recorded as a discrete, sequential event. It is a stream of moments, not a single moment. Restoring from an event log is a process of rehydration. You start from a known point—often a snapshot—and then you replay the log, applying each event one by one until you have rebuilt the present. This is a more nuanced preservation. It allows you to travel not just to a point in time, but through time. You can see the how and the why.

Neither approach is superior in all circumstances; they serve different needs in the pantry of reliability. The snapshot is your safety net. It’s what you rely on when speed and certainty are paramount. But it is a blunt instrument. The event log is your source of truth. It enables forensic debugging, complex replication, and a deep understanding of system behavior. It is, however, more complex to manage. A corrupted log can be as useless as a jar of spoiled spice.

The wisdom, then, lies not in choosing one over the other, but in understanding their symbiotic relationship. The most resilient systems I’ve tended use both. They take a snapshot of the rice jar at dawn—a solid, dependable baseline. Throughout the day, they meticulously record every pinch and measure added to the spice jar. If the kitchen catches fire, you can always start again with the rice and, if the spice log is safe, meticulously recreate the exact flavor of the day. One jar holds the substance; the other holds the story. And a well-run service, like a well-stocked pantry, needs both the staple and the spice to truly nourish.

Notes & further reading

A few pages I came back to while writing this: