Label Hygiene — Sensitive Data Handling

BizFirstGO Approved Loki Labels

These are the only labels that should appear on BizFirstGO log streams. All other fields belong in the log body (structured JSON), not as labels:

Label	Example Values	Why It's Safe
`job`	processengine, edgestream, octopus	Bounded set (service names)
`tenant_id`	tenant-abc, tenant-xyz	Internal IDs — not PII; bounded by tenant count
`environment`	production, staging, development	Bounded set (3-5 values max)
`level`	info, warn, error, debug	Bounded set (5 values)
`region`	us-east-1, eu-west-1	Bounded set (cloud region names)

Labels That Must Never Appear

Forbidden Label	Risk	Safe Alternative
`user_id` (as label)	PII — user IDs may be email addresses or SSOs	Put userId in log body as a JSON field
`email`	PII — email address directly in label index	Never log email as label or body (use opaque ID)
`execution_id`	High cardinality — millions of unique executions	Put executionId in log body as a JSON field
`request_id`	High cardinality	Put requestId in log body
`ip_address`	PII in some jurisdictions + high cardinality	Never log IP addresses; or log hashed/truncated

OTel Collector Label Control

# otel-collector-config.yaml — control which attributes become Loki labels
processors:
  resource/bizfirst-labels:
    attributes:
      # Approved labels — set from OTel resource attributes:
      - key: job
        value: ${SERVICE_NAME}
        action: insert
      - key: tenant_id
        from_attribute: tenant.id
        action: insert
      - key: environment
        from_attribute: deployment.environment
        action: insert

      # REMOVE any accidentally included PII attributes:
      - key: user.email
        action: delete
      - key: user.id
        action: delete
      - key: http.client_ip
        action: delete

exporters:
  loki:
    endpoint: http://loki:3100/loki/api/v1/push
    labels:
      # ONLY these OTel resource attributes become Loki labels:
      resource_labels:
        - job
        - tenant_id
        - environment
        - level
      # Everything else goes into the log body's structured metadata

Prometheus Label Hygiene

# Prometheus metric label rules — enforced at the service level (MetricsRegistry.cs):

# GOOD: bounded labels
bizfirst_workflow_executions_total{
  status="success",         # bounded: success/failed/cancelled
  workflow_type="approval", # bounded: known workflow types
  tenant_id="tenant-abc"   # bounded: internal tenant identifier
}

# BAD: unbounded labels that would appear in Prometheus
# These create a new time series per unique value:
bizfirst_workflow_executions_total{
  user_email="user@company.com",  # FORBIDDEN — PII + unbounded
  execution_id="exec-abc123"      # FORBIDDEN — unbounded cardinality
}

# To audit label cardinality in Prometheus:
# Query: topk(20, count by (__name__)({__name__=~".+"}))
# Shows the 20 metric families with the most series

Label Mistakes Are Hard to Fix

If a high-cardinality or PII label is accidentally added to Prometheus metrics, removing it requires dropping the entire metric family and restarting. Prometheus does not support renaming labels retroactively. Similarly in Loki, removing a label from existing streams requires deleting and re-ingesting — which is not practical. Define your label set in advance and enforce it at code review.

← OTel Collector Redaction Next: Trace Attribute Filtering →