The Unnecessary Burden: Why Your Logs Don't Need a Narrative
We’ve all heard the sermon. It’s delivered from conference stages and preached in operations manuals: your logs must tell a story. They should be a coherent narrative, a chronological tale that a human can read from top to bottom and understand the precise sequence of events leading to a failure. We invest in complex structuring, painstakingly adding context and crafting a flowing journal of system events. We do this because we believe it is the path to clarity. But what if this pursuit of a perfect story is, in fact, a beautifully crafted trap?
The argument for narrative logs is seductive. It appeals to our human brains, wired for causality and plot. The problem is, our systems are not human. They are vast, distributed, and operate at a scale and speed that defies linear storytelling. By insisting that logs conform to a narrative structure, we are forcing a human construct onto a non-human process. We are building a library where every book is written in a single, continuous paragraph, without an index. When the fire alarm goes off, you don’t want to read a novel; you want to find the fire exit.
The Tyranny of Context
This obsession with narrative leads directly to the tyranny of over-contextualization. We burden every log line with excessive metadata—user IDs, session tokens, request paths—in the hope that this ‘context’ will make the story clearer. Instead, it often creates a cacophony of noise, obscuring the true signal. It makes logs bloated, difficult to store, and expensive to process. More critically, it assumes the context we preemptively add is the context we will later need. This is a gamble we almost always lose. The question that arises during an incident is invariably the one we didn’t think to pre-load into our log fields.
The counterintuitive, liberating alternative is this: stop trying to tell a story and start building an index. Treat your logs not as a flowing narrative but as a massive, disorganized heap of facts. Embrace the chaos of raw, unstructured, even slightly brutish event data. Then, invest your engineering might not in crafting the story at write-time, but in building powerful, fast tools to sift through the rubble at read-time. The power lies not in the log line itself, but in the ability to rapidly correlate a trillion disparate lines across a thousand systems to find the one pattern that explains everything.
A narrative is static; it is the story you thought was important when you wrote it. An index is dynamic; it answers the question you have right now. In the frantic minutes of a page, you are not a reader enjoying a book. You are an archaeologist with a shovel, frantically digging for one specific bone. Don’t build a museum; build the best shovel you can. Let go of the need for a story, and you might just find the truth faster.
Notes & further reading
A few pages I came back to while writing this: