Portal Community
BizFirst Observe

Metrics Exemplars

Prometheus exemplars are the bridge between metrics and traces. They attach a TraceId to specific high-value metric samples — enabling you to jump from a spike on a Grafana histogram panel directly to the trace that caused it.

What Are Exemplars?

An exemplar is additional metadata attached to an individual metric sample. For BizFirstGO histograms, the OTel SDK automatically attaches the current TraceId as an exemplar to histogram bucket observations — when an active trace context is present. This means every high-latency node execution measurement is linked to its exact trace.

How Exemplars Work

# Prometheus exposition format with exemplar:
bizfirst_node_execution_duration_seconds_bucket{
  node_type="DataFetchNode",
  le="5.0"
} 2211 # {traceID="4bf92f3577b34da6a3ce929d0e0e4736",spanID="00f067aa0ba902b7"} 4.742 1716648000000

# Breakdown:
# 2211          = bucket count (number of observations ≤ 5.0s)
# traceID=...   = exemplar: the trace for the observation that had value 4.742s
# 4.742         = the actual observation value that triggered this exemplar
# timestamp     = when this observation occurred

Enabling Exemplars in Prometheus

# prometheus.yml — enable exemplar storage
global:
  scrape_interval: 15s

# Enable exemplar storage (required for Prometheus 2.25+)
storage:
  exemplars:
    max_exemplars: 100000   # Maximum exemplars stored in memory

Viewing Exemplars in Grafana

To see exemplar dots on Grafana histogram panels:

1

Configure Exemplar Trace Destination in Prometheus Data Source

In Grafana's Prometheus data source settings → Exemplars section → add a link: TraceId field name: traceID, target datasource: Tempo.

2

Enable Exemplars on Grafana Time Series Panel

Edit the panel → Visualization tab → enable "Exemplars" toggle. A ⬥ diamond appears on the chart at data points with exemplars attached.

3

Click an Exemplar Diamond

Clicking a ⬥ diamond on the chart opens the linked trace in Tempo — showing the full execution trace for that specific high-latency observation.

Prometheus Data Source Exemplar Configuration

# grafana-provisioning/datasources/bizfirst-observe.yaml
datasources:
  - name: Prometheus
    type: prometheus
    url: http://prometheus:9090
    jsonData:
      exemplarTraceIdDestinations:
        - name: traceID          # Exemplar label name in Prometheus
          datasourceUid: tempo-uid  # Grafana UID of the Tempo datasource
          urlDisplayLabel: "View in Tempo"
Exemplars are Automatic — No Code Changes

The OTel SDK automatically attaches TraceId exemplars to histogram observations when a trace context is active. Since BizFirstGO's BaseNodeExecutor always runs within an active trace context, every node execution duration measurement is automatically exemplar-linked. No changes to executor code are required to enable this Metrics → Trace correlation path.

← Trace-to-Log Correlation