Log Ingestion from BizFirstGO — Loki Log Aggregation

Ingestion Path

BizFirstGO Service (Serilog + OTel)

Serilog emits structured JSON log events. The OTel Serilog sink converts these to OTel LogRecord format and forwards to the OTel SDK's log pipeline. The SDK batches and sends to the Collector via OTLP/gRPC.

OTel Collector — OTLP Receiver

Collector receives the log batch on port 4317. Resource attributes (service.name, deployment.environment) become Loki labels. Processors apply enrichment and PII redaction.

OTel Collector — Loki Exporter → Loki Push API

The Loki exporter formats the logs as Loki push API JSON and sends to POST /loki/api/v1/push on the Loki server. Loki ingests into the appropriate stream based on the label set.

OTel Collector Loki Exporter Configuration

# otel-collector-config.yaml — Loki exporter section
exporters:
  loki:
    endpoint: http://loki:3100/loki/api/v1/push
    default_labels_enabled:
      exporter: false    # Don't add "exporter" label
      job: true          # Add "job" label from resource
      instance: true     # Add "instance" label
      level: true        # Add "level" label from log severity
    labels:
      attributes:
        - service.name    # Becomes label: service_name
        - deployment.environment  # Becomes label: environment
      resource:
        - service.version
      record:
        - tenant.id       # Becomes label: tenant_id from log attribute

# Multi-tenant: send X-Scope-OrgID header per tenant
    headers:
      X-Scope-OrgID: "${TENANT_ID}"  # Or extract from resource attribute

BizFirstGO Log Schema

All BizFirstGO services emit logs in this structured JSON format. The schema is enforced by ObservabilityServiceExtensions.ConfigureLogging():

{
  "Timestamp": "2026-05-25T14:32:01.1234567Z",
  "Level": "Error",
  "Message": "Node execution failed: DataFetchNode timeout after 5000ms",
  "MessageTemplate": "Node execution failed: {NodeType} timeout after {TimeoutMs}ms",
  "Properties": {
    "NodeType": "DataFetchNode",
    "TimeoutMs": 5000,
    "executionId": "exec-d1e2f3a4",
    "workflowId": "wf-8a4c2f91",
    "tenantId": "tenant-abc-123",
    "nodeKey": "data-fetch-01",
    "traceId": "4bf92f3577b34da6a3ce929d0e0e4736",
    "spanId": "00f067aa0ba902b7",
    "ServiceName": "processengine",
    "Environment": "production"
  },
  "Exception": {
    "Type": "TimeoutException",
    "Message": "The operation timed out.",
    "StackTrace": "..."
  }
}

Label Extraction from Log Attributes

The OTel Collector's Loki exporter maps OTel resource/log attributes to Loki labels. The mapping is:

OTel Attribute	Loki Label Name	Source
`service.name`	`job`	OTel Resource
`deployment.environment`	`environment`	OTel Resource
`tenant.id`	`tenant_id`	OTel Log Record attribute
Severity (OTel LogRecord)	`level`	OTel LogRecord severity
`service.version`	`version`	OTel Resource

Out-of-Order Log Handling

Loki requires log lines within a stream to arrive in timestamp order. If your BizFirstGO services run as multiple replicas or if logs are buffered and retried, you may see "out of order" ingestion errors. Loki 2.4+ supports configurable out-of-order ingestion with unordered_writes: true in the ingester config — enable this for any production deployment where log delivery order cannot be guaranteed.

← Deploying Loki Next: Label Strategy →