Searching Traces
Traces can be found in Grafana's Tempo data source via direct TraceId lookup, Grafana Explore's trace search UI, or the Tempo HTTP API. This page covers all three approaches and explains when to use each.
Method 1: TraceId Lookup (Fastest)
If you know the TraceId (from a log line, alert notification, or error report), direct lookup is instant:
Open Grafana Explore
Navigate to Explore (compass icon) and select the Tempo data source.
Select "TraceQL" or "Search" query type
In the query type dropdown, select "TraceQL". Enter: { traceId = "4bf92f3577b34da6a3ce929d0e0e4736" } — or paste the full TraceId in the "Trace ID" field directly.
View trace detail
Grafana renders the trace waterfall showing all spans with their durations, attributes, and parent-child relationships.
Method 2: Grafana Explore Trace Search
When you don't have a specific TraceId, use the search interface to find traces by criteria:
| Search Field | Description | Example |
|---|---|---|
| Service name | Filter by the root span's service | processengine |
| Span name | Filter by span operation name | workflow.execute |
| Duration | Filter by trace total duration | Min: 5s (find slow traces) |
| Status | Filter by trace status | Error (find failing traces) |
| Tags / Attributes | Filter by span attribute values | tenant.id = "tenant-abc" |
| Time range | Filter by trace start time | Last 1 hour, custom range |
Method 3: Tempo HTTP API
# Search traces by attribute (Tempo Search API)
curl "http://tempo:3200/api/search?tags=service.name%3Dprocessengine+tenant.id%3Dtenant-abc&minDuration=5s&start=1716644400&end=1716648000&limit=20"
# Get trace by ID
curl "http://tempo:3200/api/traces/4bf92f3577b34da6a3ce929d0e0e4736"
# Get tags available for search
curl "http://tempo:3200/api/search/tags"
# Get values for a specific tag
curl "http://tempo:3200/api/search/tag/service.name/values"Reading a Trace Waterfall
In Grafana's trace detail view, the waterfall shows:
- X axis — time from trace start to end
- Each row — one span; indentation shows parent-child relationship
- Bar width — span duration proportional to total trace duration
- Colors — different services have different colors; error spans are red
- Click a span row — expands to show span attributes, events, and links
- Logs button — jumps to Loki Explore with a time-scoped query for that service
The most common trace-finding workflow: you see an error in Loki, copy the traceId field from the log line, paste it into the Tempo TraceId lookup field. If Grafana's Derived Fields are configured correctly, the traceId in Loki log lines appears as a clickable link — clicking it opens the trace automatically in a split panel.