Observability Specification

Status: v1.12 draft (to be elaborated before production hardening — the rules marked normative below are already binding) Owner: datapipelines.co core Depends on: all other specs Last updated: 2026-09-17

1. Purpose

Observability for datapipelines.co covers logs, metrics, traces, and health endpoints — the signals an operator uses to detect, diagnose, and resolve issues in production.

This spec is lighter than the others because observability is cross-cutting and mostly follows industry-standard patterns. It documents the specific decisions made for v1 and the conventions modules follow. Dashboards, alert rules, SLOs and log-retention policy are still to be elaborated before production hardening — this remains a draft in that sense.

Four parts of it are normative today and binding on implementation:

2. Design Principles

1. Structured logs only. Every log entry is JSON with stable field names. No println, no ad-hoc string formatting in logs.
2. Correlation IDs everywhere. Every request carries a correlation ID (DP-Correlation-Id, REST API §3.4, registered in the DP- header registry §3.6) through every log line, trace span, SSE event payload and MCP tool result. Pipeline executions extend this to per-node correlation. Correlation ID is never a metric tag (§4.3).
3. Metrics via Micrometer. Spring Boot's default micrometer integration; no custom metrics framework.
4. Tracing via OpenTelemetry. Vendor-neutral; instrumented into every module.
5. Health endpoints for orchestrators. Root-level /health and /ready for k8s liveness/readiness probes. The payload contract is owned by REST API §11; §6 here restates it only as far as needed for operators.
6. No sensitive data in logs — enforced at the encoder, not by convention. Passwords, API keys, JDBC URLs with credentials, encryption keys, query result values are scrubbed by the logging pipeline itself, so no logger call site can bypass the rule (§9.2).

3. Logging

3.1 Format

JSON via logstash-logback-encoder. Every log entry:

{
  "@timestamp": "2026-08-05T14:30:00.123Z",
  "level": "INFO",
  "logger": "co.datapipelines.dag.PipelineExecutor",
  "thread": "executor-worker-3",
  "message": "Node completed",
  "correlation_id": "uuid",
  "execution_id": "uuid",
  "pipeline_id": "uuid",
  "node_id": "fetch_orders",
  "duration_ms": 1266,
  "rows_out": 12453
}

Standard fields: @timestamp, level, logger, thread, message. Context fields added via MDC (SLF4J's Mapped Diagnostic Context).

3.2 Levels

3.3 Correlation ID propagation

• Set on inbound request from the DP-Correlation-Id header, or generated if absent; echoed back on the response (REST API §3.4).
• Stored in MDC at request start, cleared at request end.
• Propagated through async work via Spring's TaskDecorator (for thread pools) and Kotlin coroutine context (for executor work).
• Included in every log line in the request's call tree.
• Pipeline executions add execution_id and propagate it to every node-execution log.

Normative — propagation past the HTTP boundary. A correlation ID that stops at the response header is useless for the two asynchronous surfaces this system exposes:

3.4 What's logged per module

3.4A The pool-retirement events (094)

A datasource that is edited or deleted has its pool RETIRED, not closed on the spot: it leaves the live map immediately, stops handing out connections, and closes once the statements already running on it finish (Datasources §5.2). Four events name that lifecycle, and the third is the only one that is not routine.

The WARN is the one to alert on. Every other line is the mechanism working. pool_hard_closed means a statement outlived datapipelines.datasources.retire-ceiling-seconds (Configuration §3.26) — either a genuinely hung query, or a ceiling set below what this deployment's statements really take. It carries the number of connections it took down, which the matching datapipelines.datasource.pool.hard_closed counter deliberately does not (an unbounded tag value, §4.3).

3.4B The mail events (137)

The notices of Auth §5A.8 leave two kinds of trace: the AUDIT rows mail.sent / mail.failed (Enums §15), and these structured log lines. None carries an address beyond its domain, and none ever carries a body.

send_failed is the one to alert on — a welcome mail that failed is a user who cannot log in until an admin resets. The admin screen shows the failure too; this line is for the operator who watches logs rather than screens.

3.4C The LAKE instance events (152)

A LAKE datasource's pool owns one embedded DuckDB instance per pool generation (Datasources §5.2): a retained owner connection is opened at pool build, initialized once, and released after the Hikari pool has shut down. Nine events name that lifecycle and its failures; the ERROR is the one that means a pool needs rebuilding, and the WARNs are what the operator should read.

instance_owner_lost is the one to alert on: it is a datasource whose engine went away underneath a live pool, and it does not self-heal — retire-and-rebuild (a datasource save, a table registration, or the reconcile) is the remedy, and the message says so. view_failed is the existing per-table isolation working (the table's row carries the error; the UI shows it); a burst of them after a registry change is a bad location or credential, not the engine. Nothing here is an audit row.

3.5 Log destination

• Stdout by default — collected by container runtime (Docker / k8s) and shipped to the operator's log aggregator (CloudWatch, Stackdriver, Loki, ELK, etc.).
• No file logging in container. Operators choose the aggregation strategy.
• Local dev: human-readable console output (via logback-spring.xml dev profile).

4. Metrics

4.1 Metric naming

All application metrics are prefixed with datapipelines.. Framework-supplied metrics keep the names their instrumentation gives them — notably Spring Boot's http.server.requests and HikariCP's hikaricp.*, which are not renamed into the datapipelines. namespace (renaming them would break every off-the-shelf dashboard and Boot's own auto-configuration).

Tag sets below are the complete, normative set for each metric — adding a tag is a spec change, not an implementation detail (§4.3).

Result delivery, SSE and idempotency (REST API §7, D9):

4.2 Exposure

• Micrometer's Prometheus endpoint at /actuator/prometheus — served on a separate management port (management.server.port, framework wiring key), never on the application port. Scrapers reach it over the cluster-internal network; it is NOT in auth.md §8.3's public list and no /actuator path is reachable on the app port without auth. Rationale (2026-08-07 security review MEDIUM-7): §4.3's metric tags include pipeline_id, datasource_name and template_id — an unauthenticated metrics endpoint on the app port would publish the internal inventory that §6's bare-UP/DOWN health design exists to protect.
• Note the asymmetry: /health, /ready and /info are served at the root of the application port, not under /actuator and not under /api/v1 (§6).
• Other backends (Datadog, New Relic, CloudWatch) selectable via Micrometer registry config.
• Actuator/management exposure is configured with Spring Boot's own management.* keys — datapipelines defines no key of its own for it (configuration.md §3.14 registers those framework paths; §3.15 holds this product's own observability keys).

4.3 Cardinality discipline

High-cardinality tags avoided:

• ❌ Don't tag by user_id, execution_id, correlation_id, exception class, method name, or any raw URI/SQL string.
• ✅ Do tag by pipeline_id, node_id, datasource_name, template_id (bounded counts), and by closed enum-valued dimensions (status, reason, outcome, format).
• Per-execution metrics live in the execution record (database), not in the metrics system.

Normative rule: every tag value must come from a closed set — an enum, or an entity ID whose population is bounded by what the deployment stores. A tag whose values are derived from code shape (class, method, stack frame) or from input text is forbidden; that detail belongs in the log line and the trace span, which are indexed for it. §8.1 applies this rule to error counting.

5. Tracing

5.1 OpenTelemetry

OpenTelemetry SDK initialized in app. Auto-instrumentation via the opentelemetry-spring-boot-starter:

• HTTP server spans for every inbound request (REST + MCP).
• JDBC spans for every DB query (via opentelemetry-jdbc-instrumentation).
• HikariCP spans for connection acquisition.
• Custom spans for template rendering and node execution.

5.2 Span attributes

Every span carries:

• correlation_id — request correlation.
• user_id (on auth'd requests).
• pipeline_id, execution_id, node_id (on pipeline-execution spans).
• datasource_name (on datasource-related spans).

5.3 Exporter

• OTLP exporter (HTTP/gRPC to collector) — vendor-neutral.
• Operator configures the collector endpoint (OTEL_EXPORTER_OTLP_ENDPOINT).
• No vendor lock-in.

5.4 Sampling

• Default: probabilistic sampling at 10% for general traffic, 100% for pipeline executions (high-value, low-volume).
• Configurable via otel.traces.sampler.* properties.

6. Health Endpoints

Authority: the payload and the paths are defined by REST API §11. This section is the operator-facing reading of that contract; where the two ever differ, rest-api wins.

All three endpoints are served at the root — /health, /ready, /info — not under /api/v1 and not under /actuator. Probe configuration (k8s livenessProbe/readinessProbe) targets those paths directly.

6.1 /health

Liveness probe. Returns 200 OK with the service status. No auth required.

{
  "status": "UP",
  "version": "1.2.3",
  "components": {
    "database": "UP",
    "redis": "UP",
    "h2_factory": "UP"
  }
}

• version is top-level, not a component.
• Component keys are snake_case and the set is exactly these three: database (metadata DB connectivity), redis (result store / idempotency / event log connectivity), h2_factory (can create a staging H2 instance).
• No diskSpace component. Boot's default disk-space indicator is disabled: nothing in this architecture writes to local disk (logs go to stdout §3.5, staging is in-memory H2, results live in Redis), so a disk-space signal would report on something the service does not depend on — and, being DOWN on a full container filesystem, would restart a perfectly healthy pod.

6.2 /ready

Readiness probe. Returns 200 OK when the service can serve traffic (everything in /health plus warm-up complete), 503 otherwise.

Returns 503 during startup (until Spring Boot signals ready) and during shutdown (draining) — the drain window is where in-flight executions are given up to execution-timeout-seconds before stragglers are cancelled (deployment.md, D7).

6.3 /info

Build info at the application-port root. No auth required. Key names are the contract:

Bare values only — no hostnames, no paths, same discipline as /health (§6.4).

6.4 Actuator security

Only /health, /ready and /info are exposed without auth on the application port — nothing under /actuator is routable there. /actuator/health (and, with the metrics registry, /actuator/prometheus) is exposed on the separate management port only (§4.2), confined to the cluster network by deployment topology; every other actuator endpoint is disabled. Health output carries no credential, hostname or JDBC URL — component values are bare UP/DOWN strings precisely so an unauthenticated probe surface cannot leak topology (§9.2).

7. Audit Log

Already covered in Auth spec §10 (event catalog §10.1, log shape §10.2). The audit log is separate from the general application log:

• Append-only table in the metadata DB.
• Structured events (auth events, admin actions) — the catalog in Auth §10.1 is authoritative; there are no password or lockout events, because authentication is OIDC-only.
• Retention governed by datapipelines.audit.retention-days (configuration.md).

The audit log captures who did what when for compliance / forensic purposes. The general log captures what happened in the system for debugging.

8. Error Reporting

8.1 Uncaught exceptions

Uncaught exceptions in any thread / coroutine:

• Logged at ERROR with full stack trace, correlation ID, and (where applicable) execution_id / node_id.
• Counted by datapipelines.errors.total{domain} — one tag only. domain is the first segment of the error code that the failure maps to (pipeline, template, datasource, auth, result, rate_limit, idempotency; internal for a failure with no mapped code), per the {domain}.{entity}.{failure} scheme in enums §16.
• Exception class and method are deliberately not tags: both are unbounded, code-shape-derived dimensions that §4.3 forbids. That detail is already carried by the ERROR log line's stack trace and the trace span's exception event, which are the right places to search it.
• For pipeline-execution coroutines, fails the execution with the mapped pipeline.* code (dag-executor §8.2); a cancelled execution is execution_aborted, counted by datapipelines.executions.aborted{reason} (§4.1), not by errors.total.

8.2 External error reporting (optional)

• Sentry / Bugsnag / Rollbar integration via Spring Boot's spring-boot-starter-actuator + vendor SDK.
• Configured per deployment (disabled by default in v1).

9. Configuration & Redaction

Inbound links land here from MCP Server §6.3; correlation-ID propagation itself is specified in §3.3.

9.1 Configuration keys

configuration.md is the single authority for config keys — YAML path, env var, default and description all live there (D8). This spec references keys by name and never restates a default. The observability keys are defined in configuration.md §3.14:

Two families deliberately have no datapipelines-namespaced key, because their frameworks already own the surface and duplicating it would create a second authority:

• Log levels — Spring Boot's own logging.level.* (e.g. logging.level.co.datapipelines.dag=DEBUG). §3.2 states which levels carry which meaning.
• Actuator/Prometheus exposure and path — Spring Boot's management.* (§4.2).

Sampling rates (§5.4) are set with the OpenTelemetry SDK's own otel.traces.sampler.* properties.

9.2 Redaction — normative

Redaction is not configurable and not opt-in. There is no redaction.enabled key: a switch that can turn secret-scrubbing off is a switch that will be off in some deployment. The mechanism is two layers inside the logging pipeline, so that no logger call site can bypass it — a developer cannot leak a secret by choosing the wrong logging idiom, only by inventing a key name that is not on the list.

1. Field filter in the JSON encoder. logstash-logback-encoder is configured with a field-name filter over structured fields (MDC entries, key-value pairs, StructuredArguments, and marker-attached objects). Any field whose key matches the sensitive-key list is emitted as "***" — the key is kept (its presence is itself diagnostic), the value never is.
2. A MessageConverter over the rendered message. Registered in logback-spring.xml in place of the stock %message/%msg converter, it scans the formatted message text for key=value and "key": "value" occurrences of the same list and rewrites the value to ***. This catches the case the field filter cannot: a secret interpolated into a message string or arriving inside an exception message (a driver's SQLException quoting the JDBC URL is the realistic one).

Both layers read the same sensitive-key list (case-insensitive, matched on the whole key and on *_<key> / <key>_* compounds):

Never redacted, by design: correlation_id, execution_id, pipeline_id, node_id, datasource_name, user_id. These are the fields that make an incident diagnosable; scrubbing them would defeat the purpose of structured logging.

Row values are never logged at all — not at INFO, and at TRACE only under the per-row diagnostic logger that §3.2 marks as disabled in production. Redaction is a backstop for accidents, not a licence to log result data.

9.3 Redaction beyond logs — error payloads

The same secrecy rule binds anything that leaves the process, not just log output. Normative: an error details map must never carry jdbc_url, a password, or a raw connection string — in an SSE event payload, in a REST error envelope, or in an MCP tool result. Failures that are about a datasource identify it by datasource_name and, where useful, an underlying_error string that has itself passed the redaction list.

REST API §6.4.4 node_failed already complies: its details carries datasource_name + underlying_error and no URL. The rule is stated here so the next event type added does not have to rediscover it.

This is a construction rule, not a filter — the redacting encoder covers logs, and error envelopes are built by hand, so the two must be kept honest independently.

10. What's Out of Scope for v1

• Distributed tracing across pipelines (e.g., tracing a pipeline execution that triggers another pipeline) — v1 has no pipeline-to-pipeline calls, so not needed.
• Custom dashboards — operator responsibility (Grafana templates provided as documentation, not shipped code).
• Per-tenant metrics — single-tenant v1.
• Real-time alerting — operator responsibility (Alertmanager / equivalent).

Appendix A: Change Log