The Scribe's Two Inks: On the Hand That Writes and the One That Blots

There is a quiet ritual performed daily in the engine rooms of our small services. It’s a fundamental act, so basic we rarely think about its dual nature: the act of committing state. We are taught, from our first steps in this craft, the importance of logs, of databases, of transaction files. We are scribes, meticulously recording the story of our system’s life. But we seldom pause to consider that for every ink we use to write, we must keep another, of a different character, to blot.

On one side sits the permanent, indelible ledger. This is the log file that streams to disk, the row inserted into the main database table, the configuration change pushed to the version-controlled file. This ink is for history. It is dark and lasting, meant to be read back, queried, and analyzed. It tells the story of what happened, in the order it happened. Its purpose is to create a persistent truth, an audit trail that can withstand the fog of memory and the chaos of a crash. This is the hand that writes the epic.

But the other ink is temporary, almost spectral. It exists not for history, but for the immediate, precarious present. This is the transaction in memory, the staged config file in a /tmp directory, the intermediary state of a complex operation. Its purpose is to hold the fragile, unfinished thought of the system. This ink is water-soluble; it can be washed away without a trace if the process is interrupted. It allows for a crucial kind of safety: the ability to abandon a change cleanly. This is the hand that drafts, that experiments, that holds the potential for a future commit.

The true art of reliability lies not just in using both inks, but in understanding their distinct purposes and the strict boundary between them. A failure of this boundary is a classic source of subtle corruption. Imagine a scribe who, in a moment of haste, begins drafting a crucial decree directly into the royal ledger. A sudden disturbance, an ink spill, a distracted moment—and the permanent record is left in a half-finished, nonsensical state. The truth is mangled. This is the equivalent of a service writing directly to its production database without the safety net of a transaction that can be rolled back. The commit is atomic, but the logic leading to it is not.

Conversely, a scribe who only ever drafts, who fills scrap after scrap with temporary notes but never commits a final, clean version to the official record, creates a system of profound uncertainty. What is the current state? Which draft was the last good one? This is the service that keeps all its state in memory or in fragile, unacknowledged caches. When it restarts, it awakens with amnesia, unable to reconstruct its purpose. It lacks a source of truth.

The most resilient systems I’ve maintained are those that embody this scribal wisdom. They have a clear, deliberate flow: a thought is composed in the ephemeral ink of a staging area, its logic is validated, and only then is it committed with the permanent ink to the ledger. The blotting hand protects the ledger from the messiness of creation, and the ledger gives the ephemeral hand a final, authoritative purpose. It’s a dance between the tentative and the absolute, a partnership that ensures the story told is both complete and correct.

Notes & further reading

A few pages I came back to while writing this: