Instrumentation Layer
Every BizFirstGO service is pre-instrumented with the OpenTelemetry SDK. This page explains what the SDK captures automatically, what is captured manually, and how telemetry is emitted to the OTel Collector.
OpenTelemetry SDK Integration
All BizFirstGO backend services register the OTel SDK during startup via ObservabilityServiceExtensions.AddBizFirstObservability(). This single call configures all three signal types — logs, metrics, and traces — with sensible defaults for the BizFirstGO runtime.
// ProcessEngine startup — ObservabilityServiceExtensions.cs
builder.Services.AddBizFirstObservability(options =>
{
options.ServiceName = Environment.GetEnvironmentVariable("OTEL_SERVICE_NAME") ?? "processengine";
options.OtlpEndpoint = Environment.GetEnvironmentVariable("OTEL_EXPORTER_OTLP_ENDPOINT") ?? "http://otel-collector:4317";
options.EnableTracing = true;
options.EnableMetrics = true;
options.EnableLogging = true;
options.SamplingRate = 1.0; // 100% in development; adjust for production
});Auto-Instrumentation Coverage
The OTel SDK automatically instruments the following without any code changes in individual services:
| Library / Framework | What is Captured | Signal |
|---|---|---|
| ASP.NET Core | HTTP request duration, status codes, route templates | Traces + Metrics |
| HttpClient | Outbound HTTP calls with URL, method, status | Traces |
| Entity Framework Core | Database query duration and SQL text (sanitized) | Traces |
| SQL Client | Connection pool metrics, query durations | Metrics + Traces |
| Serilog | Structured log events forwarded to OTel log pipeline | Logs |
| gRPC (server + client) | RPC method duration, status codes | Traces + Metrics |
Manual Instrumentation — BizFirstGO Specifics
In addition to auto-instrumentation, BizFirstGO adds manual instrumentation at key workflow lifecycle points:
Workflow Execution Spans
When ProcessEngine begins executing a workflow, it creates a root span named workflow.execute with the following attributes:
Activity span = ActivitySource.StartActivity("workflow.execute");
span?.SetTag("workflow.id", workflowId);
span?.SetTag("workflow.name", workflowName);
span?.SetTag("tenant.id", tenantId);
span?.SetTag("execution.id", executionId);
span?.SetTag("triggered_by", triggeredBy); // manual | scheduler | webhookNode Execution Spans
Each node executor creates a child span within the workflow execution span:
// BaseNodeExecutor.cs — automatically wraps all node executions
using var nodeSpan = ActivitySource.StartActivity("node.execute", ActivityKind.Internal, parentContext);
nodeSpan?.SetTag("node.key", nodeKey);
nodeSpan?.SetTag("node.type", nodeType);
nodeSpan?.SetTag("node.name", displayName);
nodeSpan?.SetTag("execution.id", executionId);HIL Suspension and Resume Spans
// When a workflow suspends for human-in-the-loop
span?.SetTag("hil.reason", suspensionReason);
span?.SetTag("hil.actor", assignedActorId);
span?.AddEvent("hil.suspended", DateTimeOffset.UtcNow);
// When resumed
span?.AddEvent("hil.resumed", DateTimeOffset.UtcNow, new ActivityTagsCollection {
["hil.duration_seconds"] = (resumedAt - suspendedAt).TotalSeconds,
["hil.outcome"] = outcome // approved | rejected | timeout
});Structured Log Format
BizFirstGO services emit structured JSON logs. Every log line includes the OTel correlation fields so that logs can be linked to their corresponding traces:
{
"timestamp": "2026-05-25T14:32:01.123Z",
"level": "Information",
"message": "Node executed successfully",
"service.name": "processengine",
"tenant_id": "tenant-abc-123",
"workflow_id": "wf-8a4c2f91",
"execution_id": "exec-d1e2f3a4",
"node_key": "approval-node-01",
"node_type": "ApprovalNode",
"traceId": "4bf92f3577b34da6a3ce929d0e0e4736",
"spanId": "00f067aa0ba902b7",
"duration_ms": 142
}Every log line emitted during a request context must include traceId and spanId. The Serilog OTel sink handles this automatically when trace context is active. If you see log lines without TraceId, it indicates the code is running outside a traced activity context — investigate the call path.
Metrics Registered at Startup
The MetricsRegistry class registers all BizFirstGO OTel meters at service startup:
| Metric Name | Type | Labels | Description |
|---|---|---|---|
bizfirst_workflow_executions_total | Counter | tenant_id, status | Total workflow executions |
bizfirst_node_execution_duration_seconds | Histogram | node_type, tenant_id | Node execution latency distribution |
bizfirst_hil_pending_count | Gauge | tenant_id | Number of HIL tasks awaiting action |
bizfirst_hil_suspension_duration_seconds | Histogram | tenant_id, outcome | Time workflow was suspended waiting for human |
bizfirst_edgestream_messages_total | Counter | topic, tenant_id | Messages processed by EdgeStream |
bizfirst_active_connections | Gauge | service | Active SignalR / WebSocket connections |
OTLP Export Configuration
The SDK exports all three signal types via OTLP/gRPC to the OTel Collector. The endpoint is configured via environment variables — the same configuration pattern works for all BizFirstGO services:
# Environment variables for OTLP export
OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4317
OTEL_EXPORTER_OTLP_PROTOCOL=grpc
OTEL_SERVICE_NAME=processengine
OTEL_RESOURCE_ATTRIBUTES=deployment.environment=production,service.version=3.2.1,tenant.cluster=us-east-1
# Per-signal overrides (optional)
OTEL_EXPORTER_OTLP_LOGS_ENDPOINT=http://otel-collector:4317
OTEL_EXPORTER_OTLP_METRICS_ENDPOINT=http://otel-collector:4317
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=http://otel-collector:4317Custom ExecutionNode executors that extend BaseNodeExecutor automatically receive trace spans, duration metrics, and structured logging. You do not need to add any observability code to your executor class unless you want to add custom span attributes or custom metrics beyond the defaults.