The Two Wells: On Drawing from Deep Logs and Shallow Streams
There is a quiet fork in the road that every keeper of a small service must eventually face. It is not marked by signposts, but by the subtle, accumulating weight of data. It is the choice between depth and surface, between the deep well and the shallow stream. I am speaking, of course, of the fundamental decision in logging: whether to record the rich, contextual detail of a single event, or the high-frequency, lightweight pulse of many.
One approach is the deep log. It is the practice of writing a sparse, almost rare entry, but when you do, you pour the entire context of the moment into it. A single line becomes a story. It contains the user’s intent, the state of the system, the relevant identifiers, and the outcome—all in a structured, queryable block. This is the well you descend into with a bucket, drawing up a single, complete vessel of understanding. When something goes wrong, you might only have a few of these entries, but each one is a self-contained exhibit. The cost is in the storage and the processing power to write such a dense payload, especially under load. But the reward is clarity. Each log is an island of truth.
The contrasting approach is the shallow stream. Here, you write a torrent of simple, atomic lines. A connection opened. A request was received. A cache was missed. A value was written. Each event is a single, faint star in a vast constellation. To understand the story, you must trace the patterns between them. This stream is lightweight to produce, a gentle, constant murmur in the background of your service. Its strength is in its granularity and its low overhead. Its weakness is that meaning is emergent; it is not inherent in any single line. You need the entire sequence, the whole night sky, to see the shape of the bear or the hunter.
I have tended systems built on both philosophies. The deep log is like a meticulous journal, each entry a full paragraph of reflection. It is invaluable for forensic debugging, for understanding the ‘why’ behind a singular, complex failure. The shallow stream is like a metronome, its constant tick a reassurance that the system lives and breathes. It is perfect for dashboards, for alerting on rate changes, and for witnessing the flow of traffic itself.
The wisdom, I have found, is not in choosing one exclusively, but in knowing which well to dig for which purpose. Let the shallow stream flow for health and metrics. Then, at critical junctures—a user action, a transaction commit, an error state—sink a deep well. Let the stream tell you that something is happening; let the well tell you what it truly means. Together, they form a complete map: one shows the terrain in broad strokes, and the other provides a detailed cross-section of the most important landmarks.
Notes & further reading
A few pages I came back to while writing this: