It always starts the same way. A pager buzzes at 2:14 a.m. The checkout service is returning 500 errors for "some" users, but nobody can say exactly which ones. The dashboard is almost entirely green, except for one graph that turned red an hour ago. The request entered the system looking perfectly healthy, passed through forty microservices, and never came out the other side. There is no stack trace long enough to follow, and no single log entry that explains what happened.
You pour a cup of coffee and get ready to focus. Tonight, you are not just a developer — you are a detective, and the system is a suspect that has learned to stay silent.
In a monolith, everything happens in one place — one process, one stack trace, and one log file to read from start to finish. In microservices, the request is passed from one service to another across the network. The service that reports the failure is almost never the one that caused it. Without observability, you are left questioning forty different services, each of which claims it only saw the request for four milliseconds before passing it along.
MOTIVE — WHY THIS MATTERSYou cannot fix what you cannot see
Here is an uncomfortable truth that every on-call engineer eventually learns: in a distributed system, failures are normal, not an exception. Networks have brief interruptions. Pods get evicted. A downstream dependency adds 200 milliseconds, and a retry storm turns that into a full cascade. The real question is never whether something will break. It is whether you can identify the cause before your users identify it for you, on social media.
Observability is the difference between saying "checkout is slow, we are looking into it" for six hours, and saying "the payment gateway's p99 latency tripled at 02:03, here is the trace, and we are deploying a timeout fix now." The first is a guess. The second is real detective work. The gap between them is measured in lost revenue, lost sleep, and lost trust.
Monitoring tells you the house is on fire.
Observability tells you who lit the match.
THE WITNESSES — THREE PILLARSEveryone saw something different
Every case has witnesses, and a good detective knows that each one only tells part of the story. Observability has three main witnesses, and you need all of them working together.
This is the most detailed witness. It remembers exact words, timestamps, and the occasional exception. But its notes are scattered across forty notebooks in forty different services, and it often writes pages that nobody asked for. It is not very useful until you can line up every entry that belongs to the same request.
This witness saw the whole street from the rooftop. It can tell you that error rates rose, latency doubled, and the queue backed up — the overall shape of the problem. But it never remembers a single face. It is very good at telling you that "something is wrong," but it cannot tell you "which request, and why."
This is the star witness. It followed the request from the front door through every service it touched, timing each stop, and noting exactly where it stumbled. This witness is the one who breaks the case — the connecting thread the other two can only point toward.
None of the three witnesses can solve the case alone. Metrics tell you when to start looking, traces tell you where the request died, and logs tell you why — the exact exception at that exact point. The skill lies in connecting all three with one shared thread: an identifier that every witness is forced to write down.
THE THREAD — CORRELATIONOne ID to connect their testimonies
This is one of the oldest tricks in detective work: give every witness the same case number, and require them to cite it. In distributed tracing, that number is called the trace ID. It is created the instant a request enters the system, and it is then carried across every network hop in the request headers. Each service that touches the request opens a span — its own record of "I received it at T, did this work, and handed it off at T+Δ" — and every span is stamped with the same trace ID.
The modern standard for this is OpenTelemetry (OTel). It is vendor-neutral, and it is the reason your traces, metrics, and logs can finally work together. In .NET, it is built on System.Diagnostics.Activity and the W3C traceparent header, so context rides along on HTTP calls without you passing IDs around by hand. Queues are the exception: a message carries no headers unless you put them there, which is the one place the trail breaks by default.
// Wire the whole app to testify: traces + metrics, auto-propagated.
builder.Services.AddOpenTelemetry()
.ConfigureResource(r => r.AddService("checkout-api"))
.WithTracing(t => t
.AddAspNetCoreInstrumentation() // inbound spans
.AddHttpClientInstrumentation() // outbound hops carry traceparent
.AddSource("Checkout")
.AddOtlpExporter()) // → Azure Monitor / Jaeger / Tempo
.WithMetrics(m => m
.AddAspNetCoreInstrumentation()
.AddOtlpExporter());
// A custom span = one witness statement you control.
using var activity = _source.StartActivity("ReservePayment");
activity?.SetTag("order.id", order.Id);
activity?.SetTag("customer.tier", tier);
// ...on failure, record who did it:
activity?.SetStatus(ActivityStatusCode.Error, "gateway timeout");That is the whole trick: with one round of setup, every request now leaves a trail of breadcrumbs, each one stamped with the same case number. The different services can no longer tell inconsistent stories, because they are all writing down the same ID.
EXHIBIT A — THE TRACEReading the eyewitness testimony
Here is what the star witness gives you: a waterfall chart. Every bar represents one span — one service's part of the request — laid out on a timeline. Reading it from left to right, you can watch the request move through the system. The moment one bar stretches out while the rest stay idle, you have found where the time was lost.
This is the real payoff. Without the trace, you would have to log into six different services and read through logs, guessing at the order of events. With the trace, the guilty span is the longest red bar on the screen. The detective's job becomes much simpler: instead of searching the whole city, you just read the one testimony that followed the request all the way to the end.
THE METHOD — SOLVING THE CASEHow to work the scene
Every experienced investigator follows a procedure. Here is the one that closes distributed-systems cases quickly:
- Start from the alarm, not the code. A metric — error rate, p99 latency, or saturation — tells you when and roughly where the trouble began. Let the RED (Rate, Errors, Duration) or USE signals point you toward the right area before you start investigating in detail.
- Get the trace for a failing request. Filter your traces to the errors in that time window, and open one. The waterfall chart shows the guilty span, without you having to guess the order of services.
- Zoom in from span to log. Every span carries the trace ID, and every log line should too. Jump from the red span straight to its logs — the exact exception and the exact input — instead of searching through forty different files.
- Confirm the pattern, not just one example. One slow trace is only a lead; the metric confirms whether the problem is systemic. Check whether the p99 latency on that dependency changed for everyone, or just for this one request.
- Name it, fix it, and leave a better witness behind. Add the missing span attribute, the timeout, or the retry budget — so that the next detective (probably you, next month) can solve it in one screen instead of six.
FIELD KIT — ON AZURE & KUBERNETESThe detective's toolbelt
- Instrument with OpenTelemetry, and export anywhere. OTel provides the vendor-neutral wiring. Point the OTLP exporter at Azure Monitor / Application Insights in production, or at Jaeger / Grafana Tempo locally — the instrumentation stays the same, only the destination changes.
- Carry context across the message bus, not just over HTTP. The trail breaks the moment an event reaches a queue without its
traceparent. Carry the trace context into Service Bus or Kafka messages, so that asynchronous hops remain part of the same story. This is exactly where an outbox row should also record the trace ID. - Sample your traces intentionally. You cannot keep every single trace at scale, but tail-based sampling lets you keep the interesting ones — the slow requests and the failed ones. Never accidentally sample away the important cases.
- Make your logs structured and correlated. JSON logs with the trace ID as a proper field turn a slow, hopeful search into a one-click jump from a span straight to the evidence.
- Watch the golden signals, and alert on symptoms. Page your team for user-facing problems, such as errors and latency, not for small CPU fluctuations. An alarm should mean something is genuinely wrong, not just a minor, harmless change.
The closing statement
The request that vanished at 2:14 a.m. was never gone. It left a trail the whole way through the system, waiting for someone with the right tools to read it. That trail is what observability buys you: a system that can explain itself when something goes wrong, rather than a wall of dashboards that can only tell you it did.
Do the work up front: set up OpenTelemetry once, put a trace ID on every hop, and make your logs aware of their case number. After that, the next 2 a.m. page stops being a long manhunt. It becomes what good detective work always is: a quiet, methodical reading of the evidence, with a clear answer written down before the coffee even goes cold.
Put on the hat. The system is talking. All you have to do is make it impossible for it to stay silent.
Discussion