The Blacksmith's Unheated Iron: On the Metal That Holds Stronger Without the Fire

There’s a sacred mantra in our craft: automate everything. It’s the first piece of advice given to anyone running services, the foundational stone of modern ops. We script our deployments, our scaling, our failovers. We build Rube Goldberg machines of automation that, in theory, allow us to sleep soundly while the machines tend to themselves. The goal is a system that requires no human touch, a perfectly self-sustaining clockwork universe. But what if this pursuit of total automation is, in some critical cases, making our systems more brittle, not less?

I am not arguing against automation. It is, unequivocally, a force for good. But I am arguing for the intentional preservation of a little friction. The common advice leaves no room for the carefully considered manual process, branding it as inefficiency to be eliminated. We’ve forgotten that the occasional, purposeful turn of the crank by hand is what keeps the mechanic intimately connected to the machine’s inner workings. When we automate a complex recovery procedure that is run only during a full-blown crisis, we risk creating a "write-only" script—a piece of code so critical yet so rarely executed manually that its failure modes become a terrifying mystery.

Consider a catastrophic database restore. The fully automated solution is a single button press that triggers a multi-stage process: spinning up isolated infrastructure, streaming the latest snapshot, applying WAL logs, and validating the data before a final cutover. It’s elegant. It’s also a potential single point of catastrophic failure. If the script has a subtle bug based on an untested edge case—a full disk, a peculiar network partition, a version mismatch you didn’t know about—it can fail in ways that are incomprehensible in the panic of the moment. The operator, utterly divorced from the intricate steps by layers of abstraction, is left helpless, staring at a failing automation whose logic is now a foreign language.

The counterintuitive argument, then, is this: for your most critical, rarest, and most dangerous procedures, do not automate them completely. Instead, document them meticulously and practice them manually. Keep the iron unheated. A manual runbook, executed by a human who understands the purpose and outcome of each command, builds invaluable institutional knowledge. The operator feels the resistance of the system, sees the intermediate outputs, and develops a gut-level understanding of the recovery process. This manual practice reveals hidden dependencies and assumptions that your automation would silently gloss over until it was too late. The goal isn’t speed; it’s certainty. The time invested in periodically performing the ceremony by hand is not wasted—it’s the premium you pay for a deeper, more resilient form of reliability that no script can ever provide on its own.

Notes & further reading

A few pages I came back to while writing this: