Scrape Configuration

Complete BizFirstGO Scrape Config

# prometheus.yml — scrape_configs section
scrape_configs:

  # ── Self-monitoring ──────────────────────────────────
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  # ── OTel Collector self-metrics ──────────────────────
  - job_name: 'otel-collector'
    static_configs:
      - targets: ['otel-collector:8888']

  # ── BizFirstGO Application Services ─────────────────
  - job_name: 'processengine'
    metrics_path: '/metrics'
    scrape_interval: 15s
    static_configs:
      - targets: ['processengine:8080']
        labels:
          product: 'flow-studio'

  - job_name: 'edgestream'
    metrics_path: '/metrics'
    scrape_interval: 15s
    static_configs:
      - targets: ['edgestream:8081']

  - job_name: 'octopus'
    metrics_path: '/metrics'
    scrape_interval: 15s
    static_configs:
      - targets: ['octopus:8082']

  - job_name: 'api-gateway'
    metrics_path: '/metrics'
    scrape_interval: 15s
    static_configs:
      - targets: ['api-gateway:8083']

  # ── Infrastructure ───────────────────────────────────
  - job_name: 'node-exporter'
    static_configs:
      - targets: ['node-exporter:9100']
        labels:
          instance_type: 'host'

  - job_name: 'cadvisor'
    static_configs:
      - targets: ['cadvisor:8080']
        labels:
          instance_type: 'container'

  # ── Alertmanager ─────────────────────────────────────
  - job_name: 'alertmanager'
    static_configs:
      - targets: ['alertmanager:9093']

Kubernetes Service Discovery

In Kubernetes deployments, use annotation-based service discovery instead of static targets. Add annotations to BizFirstGO pod specs:

# kubernetes pod spec annotations
metadata:
  annotations:
    prometheus.io/scrape: "true"
    prometheus.io/port: "8080"
    prometheus.io/path: "/metrics"

# prometheus.yml — Kubernetes SD config
scrape_configs:
  - job_name: 'kubernetes-pods'
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      # Only scrape pods with the annotation
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: "true"
      # Use annotated port
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_port]
        action: replace
        target_label: __address__
        regex: (.+)
        replacement: ${1}
      # Use annotated path
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
        action: replace
        target_label: __metrics_path__
        regex: (.+)
      # Add pod namespace as label
      - source_labels: [__meta_kubernetes_namespace]
        target_label: namespace
      # Add pod name as label
      - source_labels: [__meta_kubernetes_pod_name]
        target_label: pod

Scrape Target Health

Prometheus automatically tracks scrape target health. View target status at http://prometheus:9090/targets. The up metric is automatically created for each target:

# Check target health via PromQL
up{job="processengine"}  # 1 = UP, 0 = DOWN

# Alert when a service goes down
- alert: ServiceDown
  expr: up{job=~"processengine|edgestream|octopus"} == 0
  for: 1m
  labels:
    severity: critical
  annotations:
    summary: "{{ $labels.job }} is DOWN"