Alerting

Recommended Alert Rules

expr: |
  100 * (
    sum(rate(interaction_timeout_total[5m]))
    /
    sum(rate(interaction_published_total[5m]))
  ) > 15
annotations:
  summary: "Interaction timeout rate critically high ({{ $value | printf \"%.1f\" }}%)"
  description: "More than 15% of interactions are timing out. This indicates a delivery failure or systemic user absence."

expr: |
  100 * (
    sum(rate(interaction_timeout_total[5m]))
    /
    sum(rate(interaction_published_total[5m]))
  ) > 5
annotations:
  summary: "Interaction timeout rate elevated ({{ $value | printf \"%.1f\" }}%)"
  description: "More than 5% of interactions are timing out. Monitor for further increase."

expr: |
  histogram_quantile(0.95,
    sum(rate(interaction_response_time_ms_bucket{type="approval"}[15m]))
    by (le)
  ) > 14400000  # 4 hours in ms
annotations:
  summary: "Approval response P95 exceeds 4 hours"
  description: "95% of approvals are taking longer than 4 hours. Review pending approvals and escalation policy."

expr: |
  sum(interaction_in_flight) > 100
annotations:
  summary: "More than 100 interactions are currently in-flight"
  description: "A large number of interactions are pending without response. Check InteractionMonitor and delivery pipeline."

expr: |
  100 * (
    sum(rate(interaction_blocked_total[5m]))
    /
    (sum(rate(interaction_published_total[5m])) + sum(rate(interaction_blocked_total[5m])))
  ) > 10
labels:
  team: platform
annotations:
  summary: "More than 10% of interactions are being blocked by pre-send hooks"
  description: "Review rate limit or pre-send hook configuration. Hook blocking by: {{ $labels.blocked_by }}"

Alert Rules YAML — Full File

groups:
  - name: edge-interact
    rules:
      - alert: HighInteractionTimeoutRate
        expr: |
          100 * (sum(rate(interaction_timeout_total[5m])) / sum(rate(interaction_published_total[5m]))) > 15
        for: 5m
        labels: { severity: critical, team: platform }
        annotations:
          summary: "Interaction timeout rate critically high"

      - alert: ElevatedInteractionTimeoutRate
        expr: |
          100 * (sum(rate(interaction_timeout_total[5m])) / sum(rate(interaction_published_total[5m]))) > 5
        for: 5m
        labels: { severity: warning, team: platform }
        annotations:
          summary: "Interaction timeout rate elevated"

      - alert: SlowApprovalResponseTime
        expr: |
          histogram_quantile(0.95, sum(rate(interaction_response_time_ms_bucket{type="approval"}[15m])) by (le)) > 14400000
        for: 15m
        labels: { severity: warning, team: platform }
        annotations:
          summary: "Approval P95 response time exceeds 4h"

      - alert: GrowingInFlightInteractions
        expr: sum(interaction_in_flight) > 100
        for: 10m
        labels: { severity: warning, team: platform }
        annotations:
          summary: "More than 100 in-flight interactions"

      - alert: HighBlockedInteractionRate
        expr: |
          100 * (sum(rate(interaction_blocked_total[5m])) / (sum(rate(interaction_published_total[5m])) + sum(rate(interaction_blocked_total[5m])))) > 10
        for: 5m
        labels: { severity: warning, team: platform }
        annotations:
          summary: "High interaction block rate"

Alert Routing

In AlertManager, route EdgeInteract alerts to the platform team Slack channel and PagerDuty for critical severity:

# alertmanager.yml
route:
  routes:
    - match:
        team: platform
        severity: critical
      receiver: pagerduty-platform
    - match:
        team: platform
      receiver: slack-platform

receivers:
  - name: pagerduty-platform
    pagerduty_configs:
      - routing_key: "<PAGERDUTY_KEY>"
        description: "{{ .CommonAnnotations.summary }}"

  - name: slack-platform
    slack_configs:
      - api_url: "<SLACK_WEBHOOK_URL>"
        channel: "#platform-alerts"
        text: "{{ .CommonAnnotations.summary }}\n{{ .CommonAnnotations.description }}"

Alert Threshold Reference