The Steward's Two Buckets: On the Cure and the Antidote

There is a quiet, almost invisible fork in the road for those of us tending to small, crucial services. It appears when you first realize your system is sick. You reach for a tool, and you must choose one of two buckets. In the first is the cure. In the second, the antidote. Both will make the patient well, but they leave the system, and the steward, in profoundly different states.

The cure is the direct fix. The database is full; you run a script to archive old records. The log volume is choking the disk; you increase the retention threshold. The weekly backup job is failing; you adjust the cron schedule. A cure is surgical, logical, and satisfying. It solves the presenting symptom with precision. It feels like engineering. You diagnose, you prescribe, you move on. The system is healthy again, and you have added another line to your internal manual of known remedies.

The antidote, by contrast, is not for the system, but for the condition that allowed the sickness. The database is full? The antidote is not an archiving script, but a meter—a simple, visible gauge on the dashboard that turns yellow at 70% and screams red at 85%. The log volume is a problem? The antidote is a change to the log generation itself, making the verbose, chattering logs optional and the silent, critical ones unmissable. The backup job fails? The antidote is a hook that, on failure, sends a message to a channel you cannot ignore, making the silent failure impossible.

Where the cure treats the patient, the antidote treats the environment. It makes the problem announce itself earlier, louder, and more clearly. It is often less technically elegant than a clever script. It is, frankly, boring work. It is the plumbing of observability, the unglamorous wiring of alarms and sanity checks. A cure makes you a skilled surgeon. An antidote makes you a gardner adjusting the soil.

The peril of the pure cure is the accumulation of silent knowledge. You become the oracle, the only one who knows which incantation to whisper when the disk fills for the third Tuesday in a row. The system relies on your memory. The peril of the pure antidote is paralysis—you can spend all your time building warning systems for problems that never arrive, constructing a cathedral of vigilance around a shed.

The best stewards, I think, carry both buckets. They apply the cure to stop the bleeding today, but they always, always spend the next quiet hour distilling the antidote. They ask not just "How do I fix this?" but "How do I make sure I *cannot not know* this is about to break next time?" It is the difference between being a perpetual firefighter and becoming a gradual clockmaker, slowly tuning a mechanism that tells its own time and, more importantly, rings its own alarms.

Notes & further reading

A few pages I came back to while writing this: