BizFirst Observe
Loki Distributed Mode
Loki's microservices deployment mode splits the monolith into individually scalable components: Distributor (ingest fan-out), Ingester (write WAL), Querier (query execution), Query Frontend (caching), and Compactor (retention). Use when single-node Loki cannot keep up with log volume.
Microservices Components
| Component | Role | Scales With |
|---|---|---|
| Distributor | Receives log pushes; validates; fans out to ingesters via hash ring | Ingest rate (logs/second) |
| Ingester | Buffers logs in WAL; flushes chunks to object storage | Active stream count + ingest rate |
| Query Frontend | Splits large queries into shards; caches results; queues | Query concurrency |
| Querier | Executes LogQL against object storage and ingester caches | Query parallelism |
| Compactor | Compacts chunks; enforces retention; manages index | Single instance (usually) |
| Ruler | Evaluates alerting rules against Loki data | Alert rule count |
Kubernetes Helm Deployment (Loki Distributed)
# Add Grafana Helm repo (if not already done):
helm repo add grafana https://grafana.github.io/helm-charts
helm repo update
# Install Loki in distributed mode:
helm install loki grafana/loki-distributed \
--namespace observe \
--values loki-distributed-values.yaml
# loki-distributed-values.yaml:
loki:
auth_enabled: true # Multi-tenancy required for enterprise
storage:
type: s3
s3:
endpoint: s3.amazonaws.com
region: us-east-1
bucketnames: bizfirst-loki-chunks
access_key_id: ${AWS_ACCESS_KEY}
secret_access_key: ${AWS_SECRET_KEY}
ingester:
replicas: 3 # 3 ingesters for write HA
persistence:
enabled: true
size: 10Gi
distributor:
replicas: 2 # 2 distributors for ingest HA
querier:
replicas: 2 # 2 queriers for query parallelism
queryFrontend:
replicas: 2
compactor:
enabled: true
serviceAccount:
create: trueScaling Guidelines
| Daily Log Volume | Ingesters | Queriers | Distributors |
|---|---|---|---|
| < 10 GB/day | 1 (single-binary) | 1 | 1 |
| 10–50 GB/day | 3 | 2 | 2 |
| 50–200 GB/day | 6 | 4 | 3 |
| > 200 GB/day | 12+ | 8+ | 6+ |
Ingester Ring Requires Coordination
Loki ingesters use a consistent hash ring to distribute stream ownership. When scaling ingesters up or down, the ring rebalances automatically — but rapid scaling changes can cause brief ingestion gaps. Scale gradually (add 1-2 ingesters at a time) and wait for the ring to stabilize (check loki_ring_members metric) before scaling further.