It always starts the same way. The pager buzzes at 2:14 a.m. The checkout service is giving 500 errors to "some" users, but nobody can say which ones. The dashboard is almost fully green, except one graph that turned red an hour ago. The request came into the system looking perfectly fine, passed through forty microservices, and never came out. There is no stack trace long enough to follow, and no single log line that explains what happened.
You make a cup of coffee and sit up straight. Tonight you are not only a developer. You are a detective, and the system is a suspect that has learnt to keep quiet.
In a monolith, everything happens in one place: one process, one stack trace, one log file to read from top to bottom. In microservices, the request is passed from one service to the next over the network. The service that reports the failure is almost never the one that caused it. Without observability, you end up questioning forty services, and each one says it saw the request for only four milliseconds before passing it on.
MOTIVE — WHY THIS MATTERSYou cannot fix what you cannot see
Every on-call engineer learns this sooner or later: in a distributed system, failures are normal, not rare. The network drops for a moment. Pods get restarted. One service adds 200 milliseconds, retries pile up, and soon everything is slow. The real question is never whether something will break. It is whether you can find the cause before your users find it and post about it on social media.
Observability is the difference between saying "checkout is slow, we are looking into it" for six hours, and saying "at 02:03 the payment gateway's slowest calls became three times slower, here is the trace, and we are deploying a timeout fix now." The first is a guess. The second is real detective work. The gap between them costs you money, sleep and your users' trust.
Monitoring tells you the house is on fire.
Observability tells you who lit the match.
THE WITNESSES — THREE PILLARSEveryone saw something different
Every case has witnesses, and a good detective knows each one tells only part of the story. Observability has three main witnesses, and you need all three working together.
This witness remembers the most detail: exact words, times and the odd exception. But its notes are scattered across forty notebooks in forty services, and it often writes pages nobody asked for. It is not much use until you can line up every note that belongs to the same request.
This witness watched the whole street from the rooftop. It can tell you that errors went up, responses became twice as slow, and the queue got backed up. It sees the overall shape of the problem. But it never remembers a single face. It is very good at saying "something is wrong", but it cannot say "which request, and why".
This is the star witness. It followed the request from the front door through every service, timed each stop, and noted exactly where it tripped. This is the witness who cracks the case, the thread the other two can only point towards.
None of the three can solve the case alone. Metrics tell you when to start looking, traces tell you where the request died, and logs tell you why: the exact exception at that exact point. The skill is in tying all three together with one shared thread, an ID that every witness must write down.
THE THREAD — CORRELATIONOne ID to connect every statement
This is one of the oldest tricks in detective work: give every witness the same case number and make them quote it. In distributed tracing, that number is called the trace ID. It is created the moment a request enters the system, and it travels in the request headers across every network hop. Each service that handles the request opens a span, its own note saying "I got it at this time, did this work, and passed it on at that time". Every span carries the same trace ID.
The standard way to do this today is OpenTelemetry (OTel). It works with any vendor, and it is the reason your traces, metrics and logs can finally work together. In .NET it is built on System.Diagnostics.Activity and the W3C traceparent header, so the trace ID travels with every HTTP call without you passing it by hand. Queues are different: a message carries no headers unless you add them. That is the one place where the trail breaks by default.
// Wire the whole app to testify: traces + metrics, auto-propagated.
builder.Services.AddOpenTelemetry()
.ConfigureResource(r => r.AddService("checkout-api"))
.WithTracing(t => t
.AddAspNetCoreInstrumentation() // inbound spans
.AddHttpClientInstrumentation() // outbound hops carry traceparent
.AddSource("Checkout")
.AddOtlpExporter()) // → Azure Monitor / Jaeger / Tempo
.WithMetrics(m => m
.AddAspNetCoreInstrumentation()
.AddOtlpExporter());
// A custom span = one witness statement you control.
using var activity = _source.StartActivity("ReservePayment");
activity?.SetTag("order.id", order.Id);
activity?.SetTag("customer.tier", tier);
// ...on failure, record who did it:
activity?.SetStatus(ActivityStatusCode.Error, "gateway timeout");That is the whole trick. After one round of setup, every request leaves a trail of breadcrumbs, each stamped with the same case number. The services can no longer tell different stories, because they are all writing down the same ID.
EXHIBIT A — THE TRACEReading the star witness's statement
This is what the star witness gives you: a waterfall chart. Each bar is one span, one service's part of the request, placed on a timeline. Read it from left to right and you can watch the request move through the system. The moment one bar stretches out while the others sit idle, you have found where the time went.
This is the real reward. Without the trace, you would log in to six services, read their logs and guess the order of events. With the trace, the culprit is simply the longest red bar on the screen. Instead of searching the whole city, you read the one statement that followed the request all the way to the end.
THE METHOD — SOLVING THE CASEHow to work the scene
Every experienced detective follows a routine. Here is the one that closes distributed-system cases quickly:
- Start from the alarm, not the code. A metric, such as the error rate, how long the slowest requests take, or how close a service is to its limit, tells you when and roughly where the trouble started. Let request rate, errors and response time point you to the right area before you dig in.
- Pull the trace of a failed request. Filter your traces to the errors in that time window and open one. The waterfall shows the culprit span, so you do not have to guess the order of services.
- Jump from the span to its logs. Every span carries the trace ID, and every log line should carry it too. Go from the red span straight to its logs, with the exact exception and input, instead of searching forty files.
- Check the pattern, not just one example. One slow trace is only a clue. The metric tells you if the problem is widespread. Check whether that dependency became slow for everyone, or only for this one request.
- Name it, fix it, and leave a better witness behind. Add the missing span detail, the timeout or the retry limit, so the next detective (probably you, next month) can solve it on one screen instead of six.
FIELD KIT — ON AZURE & KUBERNETESThe detective's toolkit
- Add OpenTelemetry once, send the data anywhere. OTel gives you wiring that works with any vendor. Point the OTLP exporter at Azure Monitor / Application Insights in production, or at Jaeger or Grafana Tempo on your laptop. The code stays the same; only the destination changes.
- Carry the trace ID through queues too, not only HTTP. The trail breaks the moment an event goes into a queue without its
traceparent. Put the trace details into your Service Bus or Kafka messages, so background steps stay part of the same story. This is also exactly why an outbox row should store the trace ID. - Choose which traces to keep, on purpose. At scale you cannot keep every trace. Tail-based sampling lets you keep the interesting ones: the slow requests and the failed ones. Make sure you never throw away the important cases by mistake.
- Write structured logs with the trace ID. JSON logs with the trace ID as a proper field turn a slow, hopeful search into a single click from a span straight to the evidence.
- Watch what users feel, and alert on that. Wake your team for problems users can see, like errors and slow pages, not for small CPU changes. An alarm should mean something is really wrong, not a harmless blip.
The closing statement
The request that vanished at 2:14 a.m. was never really gone. It left a trail all the way through the system, waiting for someone with the right tools to read it. That trail is what observability gives you: a system that can explain itself when something breaks, instead of a wall of dashboards that can only tell you that it broke.
Do the work early. Set up OpenTelemetry once, put a trace ID on every hop, and make every log line carry its case number. After that, the next 2 a.m. call is no longer a long manhunt. It becomes what good detective work always is: a calm, step-by-step reading of the evidence, with the answer written down before your coffee goes cold.
Put on the hat. The system is talking. Your job is to make sure it can never stay silent.
Discussion