Trace-to-Log Correlation

Log-to-Trace (Loki → Tempo)

Configure Derived Fields in the Loki data source to make TraceIds in log lines clickable:

# grafana-provisioning/datasources/bizfirst-observe.yaml
datasources:
  - name: Loki
    type: loki
    url: http://loki:3100
    jsonData:
      derivedFields:
        - name: TraceID
          matcherRegex: '"traceId":"(\w+)"'   # Matches JSON log format
          url: '$${__value.raw}'
          datasourceName: Tempo               # Opens in Tempo data source
          urlDisplayLabel: "Open in Tempo"

After this configuration:

• Every log line in Grafana's Logs panel that contains a "traceId":"..." field shows a Tempo link button
• Clicking the button opens the Tempo trace detail in a split panel alongside the logs
• The trace waterfall shows the exact execution context of the log line

Trace-to-Log (Tempo → Loki)

Configure the Tempo data source to enable "Logs" links from trace spans:

# grafana-provisioning/datasources/bizfirst-observe.yaml
datasources:
  - name: Tempo
    type: tempo
    url: http://tempo:3200
    uid: tempo-uid
    jsonData:
      tracesToLogs:
        datasourceName: Loki             # Target data source
        tags: ['service.name', 'job']    # Labels to filter Loki stream
        spanStartTimeShift: -5m          # Start time shift for log query
        spanEndTimeShift: 5m             # End time shift for log query
        filterByTraceID: true            # Filter logs by traceId field
        filterBySpanID: false            # Don't filter by spanId (too specific)

      # Also enable node graph (service map)
      nodeGraph:
        enabled: true

After this configuration, from any trace in Grafana:

• Each span row has a Logs button (when the service emits logs to Loki)
• Clicking Logs opens Grafana Explore with a pre-built Loki query scoped to the span's service, time window, and traceId
• The generated LogQL query is: {job="processengine"} |= "4bf92f..." with the span's time range

The Auto-Generated LogQL Query

# Auto-generated by Grafana when clicking "Logs" from a trace span:
{
  service_name="processengine",
  environment="production"
} |= "4bf92f3577b34da6a3ce929d0e0e4736"

# Time range: [span_start - 5m] to [span_end + 5m]
# This finds all logs from the processengine service that reference this TraceId