Tracing a request across Kafka, ClickHouse, and three services

A single API request in a modern stack rarely lives in one process. It hits a gateway, passes through auth middleware, triggers a planning call to an LLM, queries ClickHouse for aggregates, falls back to Postgres for a row lookup, and streams a response back over SSE. Six hops, five different systems, one user waiting.

The waterfall view in Traces lays every one of those hops on a single timeline, positioned by when it started and sized by how long it took. That layout is what makes a slow request legible: instead of six separate log lines you have to mentally stitch together, you see a bar for `query_database` stretching 1.2 seconds while everything else finished in under 400ms combined.

Once a slow span is visually obvious, clicking it hands off directly to the Debugger, which shows exactly what that span executed — the SQL statement, the parameters, the row count returned — closing the loop from 'something is slow' to 'this specific query is slow, and here's why.'

Implementing this level of distributed tracing required careful instrumentation across our entire stack. We had to ensure context propagation worked seamlessly across asynchronous message queues like Kafka and synchronous RPC calls alike, standardizing our metadata injection to avoid dropped traces.

The visibility gained from this end-to-end tracing has fundamentally changed how we approach performance optimization. Instead of guessing which microservice is the bottleneck, our engineers can pinpoint the exact network hop, database query, or third-party API call that needs tuning.