Sensitive Data Overview
Observability systems are a common vector for accidental PII exposure. Log lines can contain form field values, trace spans can capture HTTP headers with authentication tokens, and metrics labels can encode user identifiers. BizFirstGO implements a defense-in-depth approach — sanitization at emission, redaction at the collector, and access control at the storage layer.
Sensitive Data Risk Categories
| Risk Category | Examples | Where It Appears | Defense Layer |
|---|---|---|---|
| Authentication credentials | Passwords, API keys, OAuth tokens, JWT secrets | Log messages, span attributes, HTTP headers | INodeLogger sanitizer + OTel Collector redaction |
| Personal identifiers (PII) | Email addresses, SSN, phone numbers, national IDs | Workflow input/output data in logs | INodeLogger sanitizer + field-level redaction |
| Financial data | Credit card numbers, bank account numbers | Payment workflow logs | INodeLogger sanitizer (regex patterns) |
| Health data (PHI) | Diagnoses, prescription data, patient IDs | Healthcare workflow logs | Allowlist logging (log what is safe, not what exists) |
| Cross-tenant data | Tenant A seeing Tenant B's metrics or logs | Grafana dashboards, Loki queries | Loki multi-tenancy + Grafana access control |
Defense-in-Depth Architecture
Source: INodeLogger sanitization (BizFirstGO)
Before a log line is emitted, BizFirstGO's INodeLogger passes the message through LogSanitizer.cs, which applies regex patterns to mask known PII patterns. This is the first and most effective defense — data never leaves the service if it matches a mask pattern.
Transport: OTel Collector redaction processor
The OTel Collector's redaction processor applies a second layer of attribute-level scrubbing to spans and logs passing through. Catches anything the service-level sanitizer missed — acts as a safety net.
Storage: Loki multi-tenancy enforcement
Loki with auth_enabled: true enforces tenant isolation at the storage layer. Tenant A's logs are stored in a separate keyspace and cannot be queried by Tenant B — even if a Grafana dashboard is misconfigured.
Access: Grafana role-based access control
Grafana roles (Viewer/Editor/Admin) and team folder permissions control which humans can see which dashboards. Viewers cannot use Explore to make ad-hoc queries. Cross-tenant dashboard access is blocked by folder permissions.
Remediation: GDPR deletion API
If sensitive data is discovered in Loki despite the above layers, Loki's delete API allows targeted deletion of specific log lines matching a pattern — without deleting the entire tenant's logs.
The BizFirstGO LogSanitizer and OTel Collector redaction processor must be configured before the first production workflow execution. Sensitive data written to Loki cannot be efficiently "un-written" — it can be deleted, but the deletion process is slow and may leave traces in compacted chunks. Prevention is always better than remediation.