Prometheus Metric Retention

Prometheus Local Retention

# Set via command-line flags when starting Prometheus:
prometheus \
  --storage.tsdb.path=/var/lib/prometheus \
  --storage.tsdb.retention.time=90d \         # Keep 90 days of data locally
  --storage.tsdb.retention.size=100GB \       # OR: delete oldest when > 100GB
  --storage.tsdb.wal-compression \            # Enable WAL compression
  --web.enable-lifecycle                      # Allow hot reload

# In Docker Compose:
services:
  prometheus:
    image: prom/prometheus:latest
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
      - '--storage.tsdb.retention.time=90d'
      - '--storage.tsdb.retention.size=100GB'
      - '--web.enable-lifecycle'
    volumes:
      - ./data/prometheus:/prometheus

Prometheus TSDB Storage Sizing

Thanos for Long-Term Metric Storage

Prometheus alone is not suitable for retention beyond 1-2 years at high cardinality. Thanos extends Prometheus with object storage and HA:

# Thanos architecture for BizFirstGO:
# 1. Thanos Sidecar runs alongside Prometheus — uploads TSDB blocks to S3
# 2. Thanos Store Gateway reads historical blocks from S3
# 3. Thanos Querier federates queries across Prometheus + Store Gateway

# prometheus.yml — remote write to Thanos Receive (alternative approach):
remote_write:
  - url: http://thanos-receive:19291/api/v1/receive
    queue_config:
      max_samples_per_send: 5000
      max_shards: 10
      capacity: 10000

# Thanos Sidecar (runs alongside Prometheus):
thanos sidecar \
  --tsdb.path=/prometheus \
  --prometheus.url=http://localhost:9090 \
  --objstore.config-file=/etc/thanos/s3.yaml \
  --http-address=0.0.0.0:10902 \
  --grpc-address=0.0.0.0:10901

# S3 object store config for Thanos:
# /etc/thanos/s3.yaml
type: S3
config:
  bucket: bizfirst-prometheus-thanos
  endpoint: s3.amazonaws.com
  region: us-east-1
  access_key: ${AWS_ACCESS_KEY}
  secret_key: ${AWS_SECRET_KEY}