Portal Community

Provisioning via YAML (Recommended)

Configure data sources as YAML provisioning files so Grafana starts up fully configured without manual UI steps. Place these files in /etc/grafana/provisioning/datasources/:

# grafana-provisioning/datasources/bizfirst-observe.yaml
apiVersion: 1

datasources:

  # ── Loki ────────────────────────────────────────────
  - name: Loki
    uid: loki-uid
    type: loki
    url: http://loki:3100
    access: proxy
    isDefault: false
    jsonData:
      maxLines: 5000
      timeout: 60
      derivedFields:
        - name: TraceID
          matcherRegex: '"traceId":"(\w+)"'
          url: '${__value.raw}'
          datasourceName: Tempo
          datasourceUid: tempo-uid

  # ── Prometheus ──────────────────────────────────────
  - name: Prometheus
    uid: prometheus-uid
    type: prometheus
    url: http://prometheus:9090
    access: proxy
    isDefault: true
    jsonData:
      timeInterval: 15s
      queryTimeout: 60s
      httpMethod: POST
      exemplarTraceIdDestinations:
        - name: traceID
          datasourceUid: tempo-uid

  # ── Tempo ────────────────────────────────────────────
  - name: Tempo
    uid: tempo-uid
    type: tempo
    url: http://tempo:3200
    access: proxy
    isDefault: false
    jsonData:
      tracesToLogs:
        datasourceName: Loki
        datasourceUid: loki-uid
        tags: ['service.name', 'job']
        spanStartTimeShift: -5m
        spanEndTimeShift: 5m
        filterByTraceID: true
      tracesToMetrics:
        datasourceName: Prometheus
        datasourceUid: prometheus-uid
        tags:
          - key: service.name
            value: job
      serviceMap:
        datasourceUid: prometheus-uid
      search:
        hide: false
      nodeGraph:
        enabled: true

Verifying Data Source Connections

After provisioning or manual configuration, verify each data source:

1

Loki — Check label values are returning

Go to Explore → select Loki → Label Browser → confirm that job, environment, tenant_id labels are populated with BizFirstGO service names.

2

Prometheus — Check targets are UP

Go to Explore → select Prometheus → enter up → run query → confirm all BizFirstGO service targets show value 1.

3

Tempo — Check trace search works

Go to Explore → select Tempo → Search tab → select Service Name: processengine → run search → confirm traces appear in results.

Common Data Source Issues

IssueData SourceLikely CauseFix
Label browser emptyLokiNo logs ingested yet, or wrong URLCheck Collector is running and sending to Loki
up shows 0 for servicesPrometheusServices not reachable on scrape portCheck network connectivity and /metrics endpoint
Trace search returns no resultsTempoNo traces ingested, or retention expiredCheck OTel Collector Tempo exporter; trigger a workflow
TraceId links don't workLoki → TempoDerived Fields regex doesn't match log formatAdjust matcherRegex to match your actual log format