Alert Management
Grafana Unified Alerting provides a single interface for creating, managing, and routing alerts across all three data sources — Loki, Prometheus, and Tempo. Contact points (Slack, email, PagerDuty) and notification policies control where alerts are sent.
Alert Lifecycle
| State | Meaning | Color |
|---|---|---|
| Normal | Alert condition is not met | Green |
| Pending | Condition met but for duration not yet elapsed | Yellow |
| Firing | Condition met for the full for duration — notification sent | Red |
| Resolved | Was firing, now returned to Normal — resolution notification sent | Green |
| No Data | Query returns no data (possible if service is down) | Gray |
Configuring Contact Points
# Grafana alerting provisioning — contact points
# grafana-provisioning/alerting/contact-points.yaml
apiVersion: 1
contactPoints:
- name: slack-platform
receivers:
- uid: slack-platform-uid
type: slack
settings:
url: "${SLACK_WEBHOOK_URL}"
channel: "#platform-alerts"
title: |
{{ if eq .Status "firing" }}🔴{{ else }}✅{{ end }} {{ .CommonLabels.alertname }}
text: |
{{ range .Alerts }}
*Summary:* {{ .Annotations.summary }}
*Description:* {{ .Annotations.description }}
*Severity:* {{ .Labels.severity }}
{{ end }}
- name: pagerduty-critical
receivers:
- uid: pagerduty-uid
type: pagerduty
settings:
integrationKey: "${PAGERDUTY_KEY}"
class: "BizFirstGO"
- name: email-platform
receivers:
- uid: email-uid
type: email
settings:
addresses: "platform@bizfirstai.com"Notification Policies
# grafana-provisioning/alerting/notification-policies.yaml
apiVersion: 1
policies:
- orgId: 1
receiver: email-platform # Default receiver
group_by: ['alertname', 'team']
group_wait: 30s
group_interval: 5m
repeat_interval: 4h
routes:
- receiver: pagerduty-critical
matchers:
- severity = critical
continue: false
- receiver: slack-platform
matchers:
- team = platform
continue: trueCreating a Silence (Maintenance Window)
During planned maintenance, silence alerts to prevent noise:
Navigate to Alerting → Silences
Click "Add silence" in the Grafana Alerting section.
Set matchers and duration
Add matchers to scope the silence: e.g., environment = staging to silence all staging alerts. Set start and end time for the maintenance window.
Add a comment
Add "Reason" and your name. Required by compliance — creates an audit trail of why this silence was created.
A silence suppresses alert notifications — it does not resolve the alert. When the silence expires, if the alert is still firing, notifications resume immediately. Always verify the underlying issue is resolved before letting a silence expire on a critical alert.