The Unreliable Witness: On the Fallibility of the Log
We build our digital empires on a foundation of logs. They are the scribes of our systems, the impartial observers we trust to tell us what happened, when, and to whom. Our entire discipline of operations is built upon this faith in their testimony. We pour immense effort into aggregating them, parsing them, and setting them to trigger alarms. The common, sacrosanct advice is clear: if the log says it, it is so. But what if our most trusted witness is, at times, an unreliable narrator?
The counterintuitive truth we must confront is that logs do not record reality. They record a specific, limited interpretation of an event, filtered through the lens of the code that wrote them. A service logs what its developers decided was worth logging, at the moment they decided to write the logging call. It captures a sliver of the state, often stripped of crucial context, and presents it as fact. The log entry ERROR: Failed to connect to database feels like an immutable truth. But it omits the thousand tiny decisions, environmental quirks, and race conditions that led to that precise moment. It tells you the ‘what’ but almost never the full ‘why’.
This becomes dangerously apparent during complex failure modes. I’ve spent hours, like many of you, chasing phantoms based on log entries that were technically correct but contextually misleading. The service reported it couldn’t find a user’s record. The immediate assumption was a data corruption issue. The reality was a subtle logic error in a new feature that was filtering out valid records before the log statement was even reached. The log told no lies, but it bore false witness to the root cause. It pointed its finger at the database, a reliable culprit, while the true offender—our own logic—sat silently in the code, unlogged and unseen.
This inherent fallibility challenges the core of our practice. It argues against the blind faith we place in these streams of text. It suggests that our obsession with comprehensive logging can create a false sense of security, a belief that we have captured the truth of the system’s operation when we have only captured its shadow. The logs become a story we tell ourselves, a narrative that is clean, timestamped, and searchable, but one that may be entirely divorced from the messy, concurrent, non-deterministic reality of the machine.
This is not an argument to stop logging. It is an argument to stop worshipping. Treat your logs not as gospel, but as a lead—a single source in a larger investigation. Corroborate their story with metrics, with traces, with the state of the system itself. The most critical tool in an operator’s kit is not the ability to grep quickly, but the wisdom to know that the log might be wrong, or more often, that it is telling a truth that is too small and too simple to be of any real use. The ultimate backup for your logs is not another log aggregator, but your own cultivated skepticism.
Notes & further reading
A few pages I came back to while writing this: