Portal Community
BizFirstAI Portal / InfraHub / Storage S3 / Maintenance

Maintenance

Day-to-day operations for InfraHub Storage — capacity checks, backup verification, MinIO console, log review, and routine housekeeping.

On This Page

  1. Daily Checklist
  2. Weekly Checklist
  3. Capacity Management
  4. Backup Verification
  5. MinIO Console
  6. Log Review
  7. MinIO Upgrades
  8. Access Key Rotation

Daily Checklist

CheckCommand / ActionPass Condition
MinIO container runningdocker ps | grep bizfirst-minioStatus = Up
Health endpoint respondscurl -f http://minio:9000/minio/health/liveHTTP 200
Disk usage below 80%Grafana dashboard or mc admin info local< 80% used
Backup ran last nightCheck backup container logs or Loki alert"Backup completed" in logs
No capacity exceptions in app logsSearch Loki for StorageCapacityExceptionZero occurrences

Weekly Checklist

CheckCommand / ActionPurpose
List backup snapshotsdocker exec bizfirst-backup restic snapshotsConfirm 7 daily snapshots exist
Verify backup integritydocker exec bizfirst-backup restic checkDetect pack file corruption
Review lifecycle rule statusmc ilm ls local/bizfirst-filesConfirm rules are still Enabled
Review error rate trendGrafana — S3 error rate panelDetect gradual degradation
Check disk growth rateGrafana — Disk Used GB trend over 7 daysProject time to 80% threshold

Capacity Management

Check Current Disk Usage

# Summary via MinIO admin
mc admin info local

# Per-bucket usage
mc du local/bizfirst-files

# Per-prefix usage (top-level tenant directories)
mc du local/bizfirst-files --depth 1

Identify Large or Unexpected Objects

# List objects sorted by size (largest first)
mc find local/bizfirst-files --larger 50MiB | sort

# Count objects per tenant prefix
mc ls local/bizfirst-files --recursive | grep "tenant_" | awk '{print $5}' | \
  sed 's|/.*||' | sort | uniq -c | sort -rn

Trigger Lifecycle Processing Manually

# Force lifecycle rule evaluation (MinIO 2023+)
mc admin lifecycle start local/bizfirst-files

Expanding Storage Capacity

Stop MinIO

docker stop bizfirst-minio

Resize the Docker volume or attach a larger disk

On Linux, extend the underlying LVM volume or attach a new disk and mount it. Update docker-compose.yml volume mapping if needed.

Restart MinIO

docker start bizfirst-minio

Verify new capacity

mc admin info local

Backup Verification

List Snapshots

docker exec bizfirst-backup restic snapshots --tag bizfirst-storage

Integrity Check (fast)

docker exec bizfirst-backup restic check

Integrity Check with Data Sampling (thorough — run monthly)

docker exec bizfirst-backup restic check --read-data-subset=20%

Test Restore to Temporary Directory

# Restore latest snapshot to /tmp/test-restore (does not affect live data)
docker exec bizfirst-backup restic restore latest \
  --target /tmp/test-restore \
  --tag bizfirst-storage

# Confirm files are present
ls /tmp/test-restore/data/

# Clean up
rm -rf /tmp/test-restore

MinIO Console

The MinIO web console is available at http://minio:9001 (or http://localhost:9001 in local dev). Log in with the MINIO_ROOT_USER / MINIO_ROOT_PASSWORD credentials.

Console SectionUse for
BucketsView bucket list, object counts, storage used
Buckets → BrowseInspect individual objects, view metadata, download files
Buckets → LifecycleView and edit lifecycle rules without using mc
MonitoringReal-time metrics — disk usage, request rate, error rate
Identity → UsersCreate and manage access keys for the application
Identity → PoliciesConfigure IAM-style bucket access policies
Console access in production Restrict the MinIO console port (9001) to internal networks only. Do not expose it on a public IP. Use a VPN or bastion host for remote access.

Log Review

Application Logs — Key Patterns to Monitor

Log MessageSourceAction
Disk capacity check failed — write proceeding without guardMinioCapacityGuardPrometheus endpoint unreachable — check network
Write rejected — disk at X%MinioCapacityGuardDisk at threshold — immediate capacity action
[S3ObjectService.Upload] AWS ErrorS3ObjectServiceS3 API error — check credentials and bucket name
[S3ObjectService.Exists] AWS ErrorS3ObjectServiceConnectivity issue — check MinIO container health

MinIO Container Logs

# Last 100 lines
docker logs bizfirst-minio --tail 100

# Follow in real-time
docker logs bizfirst-minio -f

# Filter for errors
docker logs bizfirst-minio 2>&1 | grep -i error

Backup Container Logs

# Check last backup run
docker logs bizfirst-backup --tail 50

MinIO Upgrades

Always test upgrades in a staging environment first MinIO releases can include breaking changes to the metrics API or behaviour. Verify that the capacity guard and backup scripts continue to work after an upgrade.

Take a backup snapshot before upgrading

docker exec bizfirst-backup restic backup /data --tag pre-upgrade

Pull the new MinIO image

docker pull minio/minio:RELEASE.YYYY-MM-DD...

Stop and recreate the container

docker compose up -d --no-deps minio

Verify health and metrics

curl -f http://minio:9000/minio/health/live
curl http://minio:9000/minio/v2/metrics/cluster | grep minio_node_disk

Access Key Rotation

Rotate MinIO access keys periodically or after a suspected credential compromise.

Create new access key in MinIO Console

Navigate to Identity → Users → serviceaccount → Create Access Key. Copy the new key ID and secret.

Update environment variables

Update MINIO_ACCESS_KEY / MINIO_SECRET_KEY in your secret store (Kubernetes Secret, Docker .env, Vault).

Restart the application

The new credentials are picked up on the next container restart (FileStorageSettings is bound at startup).

Delete the old access key

In MinIO Console, delete the old key to prevent its further use.

Update backup container

Update B2_APPLICATION_KEY_ID / B2_APPLICATION_KEY in the backup container environment if Backblaze B2 keys were also rotated.