Multi-Region Observability

Multi-Region Architecture

# Architecture: one observability stack per region, one central Grafana

# Region: us-east-1
# - OTel Collector (receives from BizFirstGO services in us-east-1)
# - Loki (logs for us-east-1 tenants)
# - Prometheus (metrics for us-east-1 services)
# - Tempo (traces for us-east-1 executions)

# Region: eu-west-1
# - OTel Collector (receives from BizFirstGO services in eu-west-1)
# - Loki (logs for eu-west-1 tenants)
# - Prometheus (metrics for eu-west-1 services)
# - Tempo (traces for eu-west-1 executions)

# Central Grafana (can be in either region or a dedicated control region):
# - Data sources: one Loki per region, one Prometheus/Thanos Querier per region, one Tempo per region
# - Dashboards: use $region template variable to switch between regions
# - Global dashboards: Thanos Querier federates metrics across regions

Prometheus Federation for Cross-Region Metrics

# Central Thanos Querier — queries both regions' Prometheus instances:
thanos query \
  --store=prometheus-us-east-1.observe.internal:10901 \
  --store=prometheus-eu-west-1.observe.internal:10901 \
  --store=thanos-store-gateway-us:10901 \
  --store=thanos-store-gateway-eu:10901 \
  --query.replica-label=region \
  --http-address=0.0.0.0:10902

# In Grafana: one Prometheus data source pointing to the central Thanos Querier:
# url: http://central-thanos-querier:10902
# Now PromQL queries aggregate across all regions automatically.

# Example: error rate across ALL regions:
sum by (region) (rate(bizfirst_workflow_executions_total{status="failed"}[5m]))

Loki Multi-Region Query

# In Grafana, create one Loki data source per region:
# Loki-US: http://loki-us-east-1.observe.internal:3100
# Loki-EU: http://loki-eu-west-1.observe.internal:3100

# Add a $region template variable to dashboards:
# Variable: region, Type: Custom, Values: us-east-1,eu-west-1

# In panel queries, use the variable to select the correct data source:
# Panel data source: Loki-$region (uses the selected region's Loki)

# For cross-region log search (finding a trace that spans regions):
# Use Grafana's "Mixed" data source feature to query multiple Loki instances
# and merge results in one panel

Data Residency Considerations