Resiliency & Monitoring

Application-Level Capacity Guard

MinioCapacityGuard is the first line of defence. Before every write, it reads the MinIO Prometheus metrics endpoint, computes disk usage percentage, and throws StorageCapacityException (HTTP 507) if usage meets or exceeds the configured threshold.

Prometheus Metrics

MinIO exposes a Prometheus-compatible metrics endpoint at {Endpoint}/minio/v2/metrics/cluster. The capacity guard reads two metrics:

Enable public Prometheus access in your MinIO Docker configuration:

environment:
  MINIO_PROMETHEUS_AUTH_TYPE: public

Alert Thresholds

Three alert levels are defined for disk usage. Each requires a different response:

Prometheus Alert Rules

groups:
  - name: bizfirst-storage
    rules:

      - alert: StorageDiskHigh
        expr: |
          sum(minio_node_disk_used_bytes) /
          sum(minio_node_disk_total_bytes) * 100 >= 70
        for: 5m
        labels:
          severity: warning
          team: infra
        annotations:
          summary: "MinIO disk usage above 70%"
          description: "Current usage: {{ $value | humanizePercentage }}"

      - alert: StorageDiskCritical
        expr: |
          sum(minio_node_disk_used_bytes) /
          sum(minio_node_disk_total_bytes) * 100 >= 85
        for: 2m
        labels:
          severity: high
          team: infra
        annotations:
          summary: "MinIO disk usage above 85% — action required"
          description: "Current usage: {{ $value | humanizePercentage }}"

      - alert: StorageDiskEmergency
        expr: |
          sum(minio_node_disk_used_bytes) /
          sum(minio_node_disk_total_bytes) * 100 >= 95
        for: 1m
        labels:
          severity: critical
          team: infra
        annotations:
          summary: "MinIO disk at 95% — writes will stop at 90%"
          description: "Current usage: {{ $value | humanizePercentage }}"

      - alert: StorageHighErrorRate
        expr: rate(minio_s3_errors_total[5m]) > 0.05
        for: 3m
        labels:
          severity: warning
        annotations:
          summary: "MinIO S3 error rate elevated"

      - alert: StorageBackupMissed
        expr: time() - bizfirst_last_backup_timestamp > 90000
        for: 0m
        labels:
          severity: high
        annotations:
          summary: "Storage backup has not run in over 25 hours"

Grafana Dashboard

Import the following panel configuration into Grafana (connected to your Prometheus data source):

Health Endpoints

# Quick health check from the application container
curl -f http://minio:9000/minio/health/live && echo "MinIO is healthy"

Incident Runbooks

Runbook 1 — Disk at 85–90% (StorageDiskCritical alert)

mc admin info local

mc du local/bizfirst-files --depth 2

mc ilm tier ls local
mc admin lifecycle start local/bizfirst-files

Runbook 2 — StorageCapacityException in application logs

Application writes are already blocked. The disk is at or above WriteStopThresholdPercent (default 90%).

1. Immediately review disk usage via MinIO console or mc admin info local.
2. If growth is from a runaway process, stop the process first.
3. Delete or archive large unnecessary objects to bring usage below the threshold.
4. The capacity guard will automatically resume writes once usage drops below the threshold on the next write attempt.

Runbook 3 — MinIO container down

1. Check container status: docker ps -a | grep minio
2. Inspect logs: docker logs bizfirst-minio --tail 100
3. Common causes: out of disk space, port conflict, corrupted data volume.
4. Restart: docker start bizfirst-minio
5. If data is corrupted, restore from last restic backup. See Point-in-Time Restore.

Disaster Recovery Targets