Distributed tracing shows where a request failed and why, service by service. When one request crosses six services and each logs its piece, correlating entries by timestamp becomes guesswork under load. Trace IDs stitch the full journey together.

Spans and W3C context

Traces contain spans: named operations with duration, attributes, and status. Parent-child relationships form the execution tree. W3C Trace Context propagates context through traceparent headers across HTTP, gRPC, and message queues. Follow that spec from day one.

SDK setup

Build a TracerProvider with an OTLP exporter pointed at the collector or vendor endpoint. BatchSpanProcessor delivers production-grade throughput, unlike SimpleSpanProcessor in tests. Resource attributes declare service name, version, and environment; telemetry without origin identification turns into noise.

Instrumentation: auto or manual

Start with zero-code agents for Node, Python, and Java: they cover popular frameworks and HTTP clients without touching your code. The OpenTelemetry documentation lists the available agents per language. Restrict manual spans to business-critical operations, with stable names and domain attributes (order id, customer plan). Names too generic hurt vendor search.

Sampling: head or tail

Head sampling between 10% and 100% depending on traffic decides collection at the source. Tail sampling keeps every error trace plus slow outliers while dropping routine successes; the collector owns that decision in tail mode. Without tail sampling, the rare trace of a hard bug slips away.

Collector deployment

The agent sidecar plus gateway pattern balances local overhead against central processing. Processors handle batching, filtering, and PII scrubbing before export. Scrubbing at the collector keeps sensitive data away from third-party vendors.

Vendor

Jaeger works for a self-hosted start; Grafana Tempo bets on object storage to make long retention cheap; Datadog and Honeycomb sell managed analysis. OTLP keeps switching cheap: changing backends means swapping the exporter, not reinstrumenting the codebase.

Pitfalls

  • Propagation lost across async boundaries and thread pools; the trace breaks midway
  • Span explosion from one span per database row; aggregate per query, not per row
  • Clock skew between hosts confusing the waterfall view