Tracing a request across Kafka, ClickHouse, and three services

A single API request in a modern stack rarely lives in one process. It hits a gateway, passes through auth middleware, triggers a planning call to an LLM, queries ClickHouse for aggregates, falls back to Postgres for a row lookup, and streams a response back over SSE. Six hops, five different systems, one user waiting.

The waterfall view in Traces lays every one of those hops on a single timeline, positioned by when it started and sized by how long it took. That layout is what makes a slow request legible: instead of six separate log lines you have to mentally stitch together, you see a bar for `query_database` stretching 1.2 seconds while everything else finished in under 400ms combined.

Once a slow span is visually obvious, clicking it hands off directly to the Debugger, which shows exactly what that span executed — the SQL statement, the parameters, the row count returned — closing the loop from 'something is slow' to 'this specific query is slow, and here's why.'