Install the Stack

Method 1: Docker Compose

The recommended approach for development, staging, and smaller production deployments (up to ~20 tenants).

# Step 1: Get the BizFirst Observe configuration files
# (Assumed to be in your BizFirstGO repository)
cd bizfirstgo/infrastructure/observe/

# Step 2: Create required data directories
mkdir -p data/loki data/prometheus data/tempo data/grafana

# Step 3: Set required environment variables
# Create a .env file (never commit this to git):
cat > .env <<'EOF'
GF_SECURITY_ADMIN_PASSWORD=changeme-production-password
SLACK_WEBHOOK_URL=https://hooks.slack.com/services/YOUR/SLACK/WEBHOOK
PAGERDUTY_KEY=your-pagerduty-integration-key
MINIO_ACCESS_KEY=minioadmin
MINIO_SECRET_KEY=minioadmin
EOF

# Step 4: Start all components
docker compose up -d

# Expected output:
# [+] Running 8/8
#  ✔ Container observe-minio        Started   2.1s
#  ✔ Container observe-otel-collector Started  1.8s
#  ✔ Container observe-loki         Started   3.2s
#  ✔ Container observe-prometheus   Started   2.9s
#  ✔ Container observe-tempo        Started   2.4s
#  ✔ Container observe-alertmanager Started   1.7s
#  ✔ Container observe-node-exporter Started  1.2s
#  ✔ Container observe-grafana      Started   4.1s

# Step 5: Verify all containers are healthy
docker compose ps

# Expected output (all containers should show "healthy" or "running"):
# NAME                     SERVICE           STATUS    PORTS
# observe-grafana          grafana           running   0.0.0.0:3000->3000/tcp
# observe-loki             loki              running   0.0.0.0:3100->3100/tcp
# observe-otel-collector   otel-collector    running   0.0.0.0:4317->4317/tcp, 0.0.0.0:4318->4318/tcp
# observe-prometheus       prometheus        running   0.0.0.0:9090->9090/tcp
# observe-tempo            tempo             running   0.0.0.0:3200->3200/tcp
# observe-alertmanager     alertmanager      running   0.0.0.0:9093->9093/tcp

# Step 6: Check startup logs for errors
docker compose logs --tail=20 loki
docker compose logs --tail=20 prometheus
docker compose logs --tail=20 grafana

# Grafana healthy output includes:
# level=info msg="HTTP Server Listen" logger=server address=[::]:3000

# Loki healthy output includes:
# level=info msg="Loki started" version=2.9.x

# Prometheus healthy output includes:
# msg="Server is ready to receive web requests."

Quick Component Health Check

# Verify each component responds:
curl -s http://localhost:3000/api/health | jq .
# Expected: {"commit":"...","database":"ok","version":"10.x.x"}

curl -s http://localhost:3100/ready
# Expected: ready

curl -s http://localhost:9090/-/ready
# Expected: Prometheus Server is Ready.

curl -s http://localhost:3200/ready
# Expected: ready

curl -s http://localhost:4317 2>&1 | head -1
# Expected: any response (gRPC endpoint — won't respond to HTTP curl)

Method 2: Kubernetes (Helm)

For production deployments on Kubernetes. Uses the community Helm chart umbrella stack.

# Step 1: Add Helm repositories
helm repo add grafana https://grafana.github.io/helm-charts
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

# Step 2: Create namespace
kubectl create namespace observe

# Step 3: Install kube-prometheus-stack (Prometheus + Grafana + Alertmanager)
helm install kube-prom prometheus-community/kube-prometheus-stack \
  --namespace observe \
  --values prometheus-values.yaml \
  --wait

# Step 4: Install Loki stack
helm install loki grafana/loki-stack \
  --namespace observe \
  --values loki-values.yaml \
  --wait

# Step 5: Install Tempo
helm install tempo grafana/tempo \
  --namespace observe \
  --values tempo-values.yaml \
  --wait

# Step 6: Install OTel Collector
helm install otel-collector open-telemetry/opentelemetry-collector \
  --namespace observe \
  --values otel-collector-values.yaml \
  --wait

# Verify all pods are running:
kubectl get pods -n observe

# Expected output (all pods should show Running / 1/1 or 2/2):
# NAME                                  READY   STATUS    RESTARTS   AGE
# grafana-7d9f6b5c8-x2j4p              1/1     Running   0          2m
# loki-0                                1/1     Running   0          2m
# prometheus-kube-prom-prometheus-0     2/2     Running   0          2m
# tempo-0                               1/1     Running   0          2m
# otel-collector-6b7d9f4c-r8k2m        1/1     Running   0          2m
# alertmanager-0                        2/2     Running   0          2m

Access URLs After Installation