Observability Specification

Status: v1.47 draft (to be elaborated before production hardening — the rules marked normative below are already binding) Owner: datapipelines.co core Depends on: all other specs Last updated: 2026-11-01

1. Purpose

Observability for datapipelines.co covers logs, metrics, traces, and health endpoints — the signals an operator uses to detect, diagnose, and resolve issues in production.

This spec is lighter than the others because observability is cross-cutting and mostly follows industry-standard patterns. It documents the specific decisions made for v1 and the conventions modules follow. Dashboards, alert rules, SLOs and log-retention policy are still to be elaborated before production hardening — this remains a draft in that sense.

Four parts of it are normative today and binding on implementation:

2. Design Principles

1. Structured logs only. Every log entry is JSON with stable field names. No println, no ad-hoc string formatting in logs.
2. Correlation IDs everywhere. Every request carries a correlation ID (DP-Correlation-Id, REST API §3.4, registered in the DP- header registry §3.6) through every log line, trace span, SSE event payload and MCP tool result. Pipeline executions extend this to per-node correlation. Correlation ID is never a metric tag (§4.3).
3. Metrics via Micrometer. Spring Boot's default micrometer integration; no custom metrics framework.
4. Tracing via OpenTelemetry is a design target. Vendor-neutral instrumentation and export remain deferred (§5); the current build does not enable them.
5. Health endpoints for orchestrators. Root-level /health and /ready for k8s liveness/readiness probes. The payload contract is owned by REST API §11; §6 here restates it only as far as needed for operators.
6. No sensitive data in logs — enforced at the encoder, not by convention. Passwords, API keys, JDBC URLs with credentials, encryption keys, query result values are scrubbed by the logging pipeline itself, so no logger call site can bypass the rule (§9.2).

3. Logging

3.1 Format

JSON via Spring Boot's structured logging (logging.structured.format's logstash format) with the house redacting customizer — no new dependency (#337). Every log entry:

{
  "@timestamp": "2026-09-30T14:30:00.123456789-04:00",
  "level": "INFO",
  "logger": "co.datapipelines.dag.PipelineExecutor",
  "thread": "dag-executor-3",
  "message": "Node completed",
  "correlation_id": "uuid",
  "execution_id": "uuid",
  "pipeline_id": "uuid",
  "node_id": "fetch_orders",
  "duration_ms": 1266,
  "rows_out": 12453
}

Standard fields: @timestamp, level, logger, thread, message — the logstash format's logger_name/thread_name are renamed to logger/thread by the binding. MDC entries and SLF4J key-value pairs are flattened at the root (so correlation_id and execution_id sit beside the standard fields, not nested), which is what makes the example's context fields the same shape the code writes. The format also carries @version, level_value, tags (marker names) and stack_trace (when a throwable is logged); they are additional, never a replacement for the standard fields.

The switch is datapipelines.observability.logging.format (configuration.md §3.15): json selects this line, console selects the human-readable development format (§3.5). It is resolved before logging initialises, and the value set is closed — anything else refuses startup. Under json the binding also switches Spring's banner off (spring.main.banner-mode=off, bound after any operator setting): the banner prints to stdout before logging exists, and stdout under json is a stream of records — every non-blank line of a normal startup is one JSON object (#337-b, pinned by a normal-boot test). Under console the banner stays.

3.2 Levels

3.3 Correlation ID propagation

• Set on inbound request from the DP-Correlation-Id header, or generated if absent; echoed back on the response (REST API §3.4).
• Stored in MDC at request start, cleared at request end.
• Propagated through async work: a coroutine ThreadContextElement built from the REQUEST's ids carries correlation_id and execution_id across every suspension of the execution's coroutines (installed once in PipelineExecutor.execute, re-added explicitly at the per-node deadline scope, whose fresh CoroutineScope inherits no elements), and every pool whose threads run inside node work captures the submitter's MDC at submission and restores the thread's state after — the script-evaluation, selector-statement (and its abandon thread), template-render, template-parse and cancel-re-issue threads (#337).
• Included in every log line in the request's call tree — as the correlation_id / execution_id members under json, and as correlation_id=<id> / execution_id=<id> fields before the message under console (- when the event has no such context, never an id left over from an earlier event; the console names exactly these two MDC keys and never dumps the MDC) (#337-b).
• Pipeline executions add execution_id and propagate it to every node-execution log.

Deliberately NOT propagated, by design (#337) — a line there must not wear a caller's id, because the unit of work outlives or spans callers: the persistence batching writer's threads and the shutdown drain (one batch spans many callers), the scheduled jobs' scheduler (dp-scheduled) and the SSE log/heartbeat schedulers (no request), and the mail worker (a claim row is picked up long after the request that queued it returned).

Normative — propagation past the HTTP boundary. A correlation ID that stops at the response header is useless for the two asynchronous surfaces this system exposes:

3.4 What's logged per module

The tables below catalogue thirteen namespaces: audit, dashboard, datasource, endpoint, execution, pipeline, shutdown, parameter, lake, mail, persistence, scheduler and retention. All thirteen currently catalogued namespaces are enforced by ObservabilityEventCatalogParityTest (#439, #443): the event=<namespace>.<name> literals under modules/<module>/src/main/kotlin and the backticked names in these tables' event column must agree in both directions. Per-namespace non-vacuity floors prevent an empty extraction from passing. This is a source-text check (comments count), not an AST or runtime inventory; uncatalogued code namespaces remain outside it. Computed event identities in files logging an enforced namespace are refused. The two exact retained audit_log write failed formats in AuditLogger.kt (§3.4G) carry an audit-row identity in event={} rather than a structured-log identity; each exception is pinned to that file and required exactly once. audit_event={} is a separate field.

3.4A The pool-retirement events (094)

A datasource that is edited or deleted has its pool RETIRED, not closed on the spot: it leaves the live map immediately, stops handing out connections, and closes once the statements already running on it finish (Datasources §5.2). Four pool events name that lifecycle; pool_hard_closed is its non-routine outcome. This section also catalogues bootstrap and invalidation failures under datasource.*.

The pool-retirement WARN pool_hard_closed is the one to alert on for lost connections. Bootstrap credential ERRORs need operator repair; the resync WARN reports a credential replacement. The invalidation failure lines retain exception-message text (the malformed-message line caps it at 200 characters); they do not follow §3.4G's class/SQLSTATE-only shape, and their remaining review is tracked in #329. pool_hard_closed means a statement outlived datapipelines.datasources.retire-ceiling-seconds (Configuration §3.26) — either a genuinely hung query, or a ceiling set below what this deployment's statements really take. It carries the number of connections it took down, which the matching datapipelines.datasource.pool.hard_closed counter deliberately does not (an unbounded tag value, §4.3).

3.4B The mail events (137)

The notices of Auth §5A.8 leave two kinds of trace: the AUDIT rows mail.sent / mail.failed (Enums §15), and these structured log lines. None carries an address beyond its domain, and none ever carries a body.

send_failed is the one to alert on — a welcome mail that failed is a user who cannot log in until an admin resets. The admin screen shows the failure too; this line is for the operator who watches logs rather than screens.

3.4C The LAKE instance events (152)

A LAKE datasource's pool owns one embedded DuckDB instance per pool generation (Datasources §5.2): a retained owner connection is opened at pool build, initialized once, and released after the Hikari pool has shut down. Nine events name that lifecycle and its failures; the ERROR is the one that means a pool needs rebuilding, and the WARNs are what the operator should read.

instance_owner_lost is the one to alert on: it is a datasource whose engine went away underneath a live pool, and it does not self-heal — retire-and-rebuild (a datasource save, a table registration, or the reconcile) is the remedy, and the message says so. view_failed is the existing per-table isolation working (the table's row carries the error; the UI shows it); a burst of them after a registry change is a bad location or credential, not the engine. Nothing here is an audit row.

3.4D The promoter lens event (178)

A promoter sees only released, not-yet-promoted pipelines and templates (Auth §11A.1); deciding that needs the higher environment's inventory, read through a per-workspace cache (datapipelines.deployment.promotion.inventory-cache-ttl-seconds, Configuration §3.19). When the target cannot be read the lens FAILS CLOSED — every promoter read is empty, the screens say why — and one event says so per window.

Sustained repeats every window mean the sender's promotion target is down or misconfigured: promoters see empty lists everywhere until it is back, and the promotion page shows the same code. Not an audit row.

3.4E The scheduler events (#9)

The scheduler (Scheduler) logs its own work under scheduler.*. A person's actions on a schedule are AUDIT rows (Enums §15, schedule.*), and every run keeps its own trail in the database — these lines are for the operator watching the instance, never the history of record. Ids only; never a payload or a parameter value.

| INFO | scheduler.notification.enqueued (#442b) | A run transition (or a block) wrote its notice: one event row and its deliveries, in the transition's own transaction | schedule_id, run_id, notification_id, notice (the checkbox), outcome, queued, refused (<code>:<count>, or 0) | | WARN | scheduler.notification.backlog_full (#442b) | More than notifications.backlog-max deliveries were pending: the notice's remaining deliveries were written failed / backlog_full | notification_id, backlog_max | | INFO | scheduler.notification.sent (#442b) | The SMTP relay accepted one delivery — not proof it reached an inbox | delivery_id, notification_id, schedule_id, run_id, notice, domain (the recipient's domain only), attempts, code (-) | | INFO | scheduler.notification.retry (#442b) | A send provably never reached the relay (connect, 4xx, credentials); the delivery is pending again after the backoff | the same fields; code is the sanitized reason | | WARN | scheduler.notification.failed (#442b) | A delivery is finally failed: refused (5xx, invalid address), out of tries, or found with mail off or its domain outside a narrowed allowlist at send | the same fields | | WARN | scheduler.notification.uncertain (#442b) | A delivery may have been sent — the connection failed after the message went out, or a claim a dead instance left was recovered (code=interrupted). Never retried automatically; a person may resend | the same fields (a recovered claim adds message) | | ERROR | scheduler.notification.render_failed (#442b) | A notice could not be rendered (a defect); the delivery is failed / render_failed, nothing was sent | delivery_id, schedule_id, run_id, error (the exception class) | | ERROR | scheduler.notification.port_failed (#442b) | The mail port threw instead of answering (a defect); the delivery is uncertain / interrupted | delivery_id, schedule_id, run_id, error (the exception class) | | WARN | scheduler.notification.settle_lost (#442b) | A send finished after its claim had already been recovered as interrupted; the recovered state stands | delivery_id, schedule_id, run_id, message |

The notification lines carry ids, the notice and the recipient's DOMAIN; never an address, a subject or a body. A notice's history of record is the run's trail (notification_enqueued / _sent / _failed / _uncertain, whose details carry no domain either) and its delivery rows (Metadata DB §4.40).

A scheduler.schedule_blocked needs a person. A steady scheduler.run_not_started means max-concurrent-runs is too low for this instance's schedules, or runs are too long for their cadence — datapipelines.scheduler.capacity.retries (§4.1) is the leading signal.

3.4F The legacy endpoint events (#274, #286)

A published_endpoints row whose stored path fails today's grammar is RETIRED, never fatal (Metadata DB §4.13): never served, listed flagged, unpublishing it is the fix. The first two lines say so to an operator; both carry a COUNT, never a path. The remaining rows describe serving, registry invalidation and key-budget failures.

Neither legacy-row line is an alert: the API console lists every legacy row with its reason. A steady promotion_legacy_omitted means an endpoint that will not reach the target until someone republishes it at a legal path.

The invalidation WARNs retain exception-message text; only the malformed-message diagnostic is capped. They are not §3.4G's class/SQLSTATE-only shape (#329). A sustained key-budget saturation rate means new keys are being admitted unmetered.

3.4G The persistence events (#266)

The batching writers in front of the audit log, the execution-event record and the replay log (DAG Executor §10.1, Auth §10.1A, Configuration §3.32) log under persistence.*; writer is the store (audit, execution_events, replay_log). An item is named by its ids only (item, failed_items: execution and event id, or audit event name, user id and key id) — never a payload or details (§9.2).

A failure is named by cause (the exception's simple class name) and sql_state (the first non-null SQLSTATE among at most 16 causes, unchanged only if it matches ASCII ^[0-9A-Za-z]{1,5}$; invalid for any other shape, without trying later states, and none when no state is found) — never by the exception's message or its cause chain, which a store fills with the refused row (Postgres's DETAIL and CONTEXT; PgJDBC's batch exception quotes the bound values). 23xxx is a row the store refuses, 08xxx a connection lost (#266b). The emitter and the audit sink keep their own lines too — Durable event … not written (kind), SSE event log append failed … (replay will be incomplete), audit_log write failed … kind=… (on the direct path … cause=… sql_state=…) — the ones operators have always searched for. A steady persistence.saturated or a rising datapipelines.persistence.fallbacks{reason=timeout} (§4.1) means the store cannot keep up; batch_retried with kind=poison means rows the store refuses, which no retry will fix.

The same class/SQLState rule applies to both lake outcome-recorder failure paths (§3.4C), the write-back rollback warning and the example-content import failure line (#329, first slice): their error field uses FailureShape.cause, with sql_state=none when absent, and no exception message or throwable attachment. This slice leaves the remaining #329 inventory for separate review.

3.4H The scheduled jobs' thread (#316)

Every @Scheduled job — the stale-execution sweep (every 15 s), the pool reaper (every 5 s), the dashboard refresh sweep (every 60 s) and the hourly retention sweep — runs on ONE thread of its own, dp-scheduled (SweepSchedulingConfiguration's taskScheduler), and nothing else runs there. Their lines therefore carry thread: dp-scheduled (§3.1): execution.swept / execution.sweep_failed, execution.events_purged / execution.event_retention_failed, keys.purged, the audit.retention* lines and retention.step_failed (§7), the dashboard and parameter housekeeping lines (§3.4J, §3.4M), and §3.4A's datasource.pool_hard_closed. Until #316 they carried dp-sse-log: the jobs ran on the SSE log streamer's thread (REST API §10.3's replays), so a slow tick stalled every replay and a burst of replays delayed the sweep.

3.4I The executor's retention events (#316 review)

The executor's two scheduled jobs on the dp-scheduled thread (§3.4H) — the stale-execution sweep (StaleExecutionSweeper, every 15 s) and the hourly retention sweep over execution_events (ExecutionEventRetention) — log these. The two failure lines name the store failure by error (its class) and sql_state, never its message (#321, the §3.4G rule; error= is the scheduled-job mould's field name — audit.retention_failed — where §3.4G's writers say cause=). Catalogued at the 316 merge: §3.4H cited two of these names before any table defined them, and the docs audit (whose execution event family joined in the same commit — execution is also a permission family) refused the citation.

3.4J The dashboard refresh events (#10 L2)

The dashboard runtime (Dashboards §5) logs at these points. The WARN/ERROR failure rows name their cause by class (error=) and, for a store, by SQLState — never by message (the 321 rule, §3.4G). The two DEBUG stream-write diagnostics attach a throwable; its message and stack remain diagnostic text, so this catalogue does not claim every dashboard log message is sanitized. No row below deliberately logs a selection, binding value, result row or execution message as a field.

3.4K The live-write outage events (#336)

pipeline_executions.heartbeat_at is the stale-execution sweep's primary signal (Metadata DB §8.3): a store that refuses the live writes makes a LIVE execution look dead to the sweep on another instance. JdbcExecutionProgress logs its write failures by §3.2's degradation rule — once per execution per outage, on the healthy → failing transition, with an INFO when a write answers again — because a heartbeating execution writes at a fixed cadence and a per-refusal line would be a second outage in the log pipeline. The failures between the transition lines are counted by the metric, not logged.

A refusal is named by error (its class) and sql_state (FailureShape, the §3.4G rule), never the driver's message.

3.4L The selector-lease cleanup events (#336)

ReadOnlyStatementLease's cleanup endings (Datasources §5.3) log one WARN per refused ending — a statement close, a connection close, or a statement cancel in abandon(). The cancelled-connection path's discard runs unconditionally; the WARN is evidence, not a reason to keep a connection in service.

3.4M The parameter evaluation events (#376)

The parameter-set evaluation history (REST API §21.5, Metadata DB §8.1 retention, §8.5 sweep) logs at these points. Every failure line names its cause by class (error=) and, for a store, by SQLState (sql_state=) — never by message (the 321 rule, §3.4G); NO line carries a selection, a binding value, a row or a query's text. A history write that fails is logged at ERROR and the evaluation carries on unrecorded: that is the #376 design — the record is a side channel of the evaluation, never its master — so a steady presence of parameter.evaluation_record_failed means the metadata store is refusing the history writes while evaluations still answer.

3.4N The audit retention events (#310, #443)

Audit retention is the last step of the hourly fixed-delay retention sweep on dp-scheduled (§3.4H, §7). Each tick reads one cutoff from the database clock and deletes bounded batches. These lines describe that tick's committed deletions, including a tick that subsequently fails; they are not audit-row contents. Failures carry class and SQLSTATE under §3.4G's rule. A continuing incomplete/failed pattern means retention is falling behind or the metadata store is unavailable.

audit.write is catalogued in §3.4G; these three rows complete the four-name audit namespace.

3.4O The execution shutdown drain events (#443)

These lines occur once per execution-drain stop, after scheduler admission closes and before the web server's graceful shutdown and persistence drain (§3.4G). Readiness flips before cancellation; the drain then cancels local executions through their ordinary statement/job path and waits for deregistration, retrying cancellation during the wait. The default flush bound is 20 s, with 100 ms polls. This is a lifecycle sequence, not a periodic job. The persistence and scheduled-job shutdown outcomes remain in §3.4G/H.

3.5 Log destination

• Stdout by default — collected by container runtime (Docker / k8s) and shipped to the operator's log aggregator (CloudWatch, Stackdriver, Loki, ELK, etc.).
• No file logging in container. Operators choose the aggregation strategy.
• Local dev: human-readable console output — the console value of datapipelines.observability.logging.format (configuration.md §3.15), the value deploy/env/defaults.env ships for local runs (#337).

4. Metrics

4.1 Metric naming

All application metrics are prefixed with datapipelines.. Framework-supplied metrics keep the names their instrumentation gives them — notably Spring Boot's http.server.requests and HikariCP's hikaricp.*, which are not renamed into the datapipelines. namespace (renaming them would break every off-the-shelf dashboard and Boot's own auto-configuration).

Tag sets below are the complete, normative set for each metric — adding a tag is a spec change, not an implementation detail (§4.3).

Result delivery, SSE and idempotency (REST API §7, D9):

Scheduler (#9, Scheduler) — the scheduler's state is watched here, not through a health component (§6.1):

Persistence batching (#266, §3.4G) — one set per batching writer, every metric tagged store (audit/execution_events/replay_log), bound by WebMetrics.bindPersistence. The audit set is bound even while the audit writer is switched off (the default, #266b) and then reads zero — zero batches means the direct path, not a missing meter:

4.2 Exposure

• Micrometer's Prometheus endpoint at /actuator/prometheus — served on a separate management port (management.server.port, framework wiring key), never on the application port. Scrapers reach it over the cluster-internal network; it is NOT in auth.md §8.3's public list and no /actuator path is reachable on the app port without auth. Rationale (2026-08-07 security review MEDIUM-7): §4.3's metric tags include pipeline_id, datasource_name and template_id — an unauthenticated metrics endpoint on the app port would publish the internal inventory that §6's bare-UP/DOWN health design exists to protect.
• Note the asymmetry: /health, /ready and /info are served at the root of the application port, not under /actuator and not under /api/v1 (§6).
• Other backends (Datadog, New Relic, CloudWatch) selectable via Micrometer registry config.
• Actuator/management exposure is configured with Spring Boot's own management.* keys — datapipelines defines no key of its own for it (configuration.md §3.14 registers those framework paths; §3.15 holds this product's own observability keys).

4.3 Cardinality discipline

High-cardinality tags avoided:

• ❌ Don't tag by user_id, execution_id, correlation_id, exception class, method name, or any raw URI/SQL string.
• ✅ Do tag by pipeline_id, node_id, datasource_name, template_id (bounded counts), and by closed enum-valued dimensions (status, reason, outcome, format).
• Per-execution metrics live in the execution record (database), not in the metrics system.

Normative rule: every tag value must come from a closed set — an enum, or an entity ID whose population is bounded by what the deployment stores. A tag whose values are derived from code shape (class, method, stack frame) or from input text is forbidden; that detail belongs in the log line and the trace span, which are indexed for it. §8.1 applies this rule to error counting.

5. Tracing

5.1 OpenTelemetry

Implementation status: the tracing wiring below is a design target, not shipped behavior. The application has no OpenTelemetry SDK/exporter or consumer for the reserved tracing switches; setting them does not enable tracing. See Configuration §3.15 and the deferred export work in #270.

OpenTelemetry SDK initialized in app. Auto-instrumentation via the opentelemetry-spring-boot-starter:

• HTTP server spans for every inbound request (REST + MCP).
• JDBC spans for every DB query (via opentelemetry-jdbc-instrumentation).
• HikariCP spans for connection acquisition.
• Custom spans for template rendering and node execution.

5.2 Span attributes

Every span carries:

• correlation_id — request correlation.
• user_id (on auth'd requests).
• pipeline_id, execution_id, node_id (on pipeline-execution spans).
• datasource_name (on datasource-related spans).

5.3 Exporter

• OTLP exporter (HTTP/gRPC to collector) — vendor-neutral.
• Operator configures the collector endpoint (OTEL_EXPORTER_OTLP_ENDPOINT).
• No vendor lock-in.

5.4 Sampling

• Default: probabilistic sampling at 10% for general traffic, 100% for pipeline executions (high-value, low-volume).
• Configurable via otel.traces.sampler.* properties.

6. Health Endpoints

Authority: the payload and the paths are defined by REST API §11. This section is the operator-facing reading of that contract; where the two ever differ, rest-api wins.

All three endpoints are served at the root — /health, /ready, /info — not under /api/v1 and not under /actuator. Probe configuration (k8s livenessProbe/readinessProbe) targets those paths directly.

6.1 /health

Liveness probe. Returns 200 OK with the service status. No auth required.

{
  "status": "UP",
  "version": "1.2.3",
  "components": {
    "database": "UP",
    "redis": "UP",
    "h2_factory": "UP"
  }
}

• version is top-level, not a component.
• Component keys are snake_case and the set is exactly these three: database (metadata DB connectivity), redis (result store / idempotency / event log connectivity), h2_factory (can create a staging H2 instance).
• No scheduler component. db-scheduler's own indicator is disabled (management.health.db-scheduler.enabled: false, Configuration §3.29): it reports DOWN on an API-mode instance, which never starts the scheduler, and a stalled dispatcher is not a reason to restart a pod. Watch §4.1's scheduler metrics instead.
• No diskSpace component. Boot's default disk-space indicator is disabled: nothing in this architecture writes to local disk (logs go to stdout §3.5, staging is in-memory H2, results live in Redis), so a disk-space signal would report on something the service does not depend on — and, being DOWN on a full container filesystem, would restart a perfectly healthy pod.

6.2 /ready

Readiness probe. Returns 200 OK when the service can serve traffic (everything in /health plus warm-up complete), 503 otherwise.

Returns 503 during startup (until Spring Boot signals ready) and during shutdown (draining) — the drain window is where in-flight executions are given up to execution-timeout-seconds before stragglers are cancelled (deployment.md, D7).

6.3 /info

Build info at the application-port root. No auth required. Key names are the contract:

Bare values only — no hostnames, no paths, same discipline as /health (§6.4).

6.4 Actuator security

Only /health, /ready and /info are exposed without auth on the application port — nothing under /actuator is routable there. /actuator/health (and, with the metrics registry, /actuator/prometheus) is exposed on the separate management port only (§4.2), confined to the cluster network by deployment topology; every other actuator endpoint is disabled. Health output carries no credential, hostname or JDBC URL — component values are bare UP/DOWN strings precisely so an unauthenticated probe surface cannot leak topology (§9.2).

7. Audit Log

Already covered in Auth spec §10 (event catalog §10.1, log shape §10.2). The audit log is separate from the general application log:

• Append-only table in the metadata DB.
• Structured events (auth events, admin actions) — the catalog in Auth §10.1 is authoritative; there are no password or lockout events, because authentication is OIDC-only.
• Retention governed by datapipelines.audit.retention-days (configuration.md)
• Delivery (#266). A row is committed before AuditEventSink.log returns — by default the direct INSERT (on the caller's own connection inside its transaction); with datapipelines.persistence.audit.enabled: true, batched outside a transaction (Auth §10.1A). The same holds for every execution event (DAG Executor §10.1).
• The loss window, stated. No awaited row — audit or event — is ever acknowledged before it is durable, so a crash cannot lose a row whose caller proceeded: a caller interrupted mid-write had not proceeded (for an execution, the outcome is "not completed", never "completed without a record"). The batching writers' non-awaited submit path is the one exception — its queue at the moment of a crash, and whatever the shutdown drain could not write by shutdown-drain-ms (persistence.drain_incomplete, failures{kind=drain_lost}) — and it has no production caller yet (the dashboard's refresh event, D27, will be its first).

Retention (#310). The retention sweep's last step (AuditLogRetention, hourly, Auth §10.3, Metadata DB §8.2) deletes the rows older than the database's NOW() minus datapipelines.audit.retention-days (30–3650, default 365), in batches of at most 5,000, at most 50 batches or 2 seconds a tick. A tick that deleted anything logs one INFO line event=audit.retention purged=<n> cutoff=<iso> batches=<k>; a tick that stopped with rows left logs a WARN event=audit.retention_incomplete … reason=batch_ceiling|time_budget and the next tick continues; a metadata-DB fault logs a WARN event=audit.retention_failed purged=<n> batches=<k> cutoff=<iso> error=<class> sql_state=<state> with what it had already committed — the class and SQLState, never the store's message (§3.4G). Counts and the cutoff only — never a row's content. The meter is datapipelines.audit.retention.purged (§4.1). A step of the sweep that throws anything else logs ERROR event=retention.step_failed step=<execution_events|keys|audit_log> and the tick continues with the next step.

The audit log captures who did what when for compliance / forensic purposes. The general log captures what happened in the system for debugging.

What the audit log records, and what it does not (#164). It records authentication, administration, lifecycle verbs and MCP tool calls (mcp.tool.called, and mcp.tool.write for the mutating ones). REST reads — GET on /api/v1/* and on published endpoints — produce NO audit row, by design: an MCP key is refused on every REST route (endpoint.key_kind_refused, Auth §7.7), so an agent's reads travel over MCP, where every tool call is a row. "What did the agent see" is answered by the mcp.tool.called rows, not by REST. The one write an agent performs over REST, the visualization test screenshot upload (credentialed by the single-use DP-Upload-Token, never a key), is audited as visualization.test.screenshot_uploaded (Auth §10.1), with the run's starter as the actor.

8. Error Reporting

8.1 Uncaught exceptions

Uncaught exceptions in any thread / coroutine:

• Logged at ERROR with full stack trace, correlation ID, and (where applicable) execution_id / node_id.
• Counted by datapipelines.errors.total{domain} — one tag only. domain is the first segment of the error code that the failure maps to (pipeline, template, datasource, auth, result, rate_limit, idempotency; internal for a failure with no mapped code), per the {domain}.{entity}.{failure} scheme in enums §16.
• Exception class and method are deliberately not tags: both are unbounded, code-shape-derived dimensions that §4.3 forbids. That detail is already carried by the ERROR log line's stack trace and the trace span's exception event, which are the right places to search it.
• For pipeline-execution coroutines, fails the execution with the mapped pipeline.* code (dag-executor §8.2); a cancelled execution is execution_aborted, counted by datapipelines.executions.aborted{reason} (§4.1), not by errors.total.

A client disconnect is not an uncaught exception (#562): when the client has gone away mid-response, both Throwable backstops (ApiExceptionHandler, UiExceptionHandler) log one DEBUG line, disconnected {method} {uri}: {exception class}, with no stack trace, write no response (no envelope, no page, no toast), and it is not counted as an error. Exactly two exceptions are recognised, anywhere in the first eight links of the cause chain: AsyncRequestNotUsableException (a streamed answer whose client went away) and Tomcat's ClientAbortException (an aborted fetch). Both arise only from writing to the client's own response. A broken-pipe or connection-reset IOException is recognised only when one of them wraps it. On its own, or inside anything else, it is a failure on one of our connections (the pool, a datasource, object storage, an HTTP client), as is an EOFException: those keep the ERROR line, the stack and the 500. Every other exception that reaches the backstops keeps the ERROR line and stack above.

8.2 External error reporting (optional)

• Sentry / Bugsnag / Rollbar integration via Spring Boot's spring-boot-starter-actuator + vendor SDK.
• Configured per deployment (disabled by default in v1).

9. Configuration & Redaction

Inbound links land here from MCP Server §6.3; correlation-ID propagation itself is specified in §3.3.

9.1 Configuration keys

configuration.md is the single authority for config keys — YAML path, env var, default and description all live there (D8). This spec references keys by name and never restates a default. The observability keys are defined in configuration.md §3.15:

Two families deliberately have no datapipelines-namespaced key, because their frameworks already own the surface and duplicating it would create a second authority:

• Log levels — Spring Boot's own logging.level.* (e.g. logging.level.co.datapipelines.dag=DEBUG). §3.2 states which levels carry which meaning.
• Actuator/Prometheus exposure and path — Spring Boot's management.* (§4.2).

Sampling rates (§5.4) are set with the OpenTelemetry SDK's own otel.traces.sampler.* properties.

9.2 Redaction — normative

Redaction is not configurable and not opt-in. There is no redaction.enabled key: a switch that can turn secret-scrubbing off is a switch that will be off in some deployment. The mechanism is two layers inside the logging pipeline, so that no logger call site can bypass it — a developer cannot leak a secret by choosing the wrong logging idiom, only by inventing a key name that is not on the list. (#337 builds both on Spring Boot's structured-logging customizer point — no logstash-logback-encoder, no logback-spring.xml converter entry; the console format's one redaction word is registered by the house encoder itself; every behavioural rule of this section is unchanged.)

1. Field filter in the JSON encoder. The house JSON members customizer applies a value processor to EVERY member the formatter writes — MDC entries, SLF4J key-value pairs, and any member added later. Any member whose key matches the sensitive-key list is emitted as "***" — the key is kept (its presence is itself diagnostic), the value never is.

2. The rendered text, message and stack trace. The same redactor rewrites the two shapes a secret takes inside free text — key=value and "key": "value" — wherever text is rendered: on the structured format's message member, in the stack-trace printer's output (a driver's SQLException quoting the JDBC URL is the realistic one), and under the console format in the pattern's redaction word (%dpRedact, a house converter over the rendered message, correlation fields and exception word). Both formats call the SAME single recognizer (RedactionScanner, reached through LogRedactor.scrubText), so they cannot drift. Recognition is one left-to-right pass over the ORIGINAL text, and overlapping spans merge: every key start is tried against the original characters, spans that overlap become one region, and the region is replaced once, by the replacement of the span that started it — so a secret-shaped fragment INSIDE a value ({"password":"x secret=abc\"tail"}, password="{\"api_key\":\"…\"}") is part of that value and can never end the outer value early, and a nested match whose own value runs past the end of the one holding it ("secret": secret= tail) extends the mask to its end. Nothing is rewritten until the pass is over. The work is linear in the text length and a failed match is never rescanned (a repeated password_password_… token with no value costs the same per character as any other text); the tests hold it to a counted number of scan steps rather than a timing. The shapes are matched case-insensitively in both forms ({"PASSWORD":"…"} is masked like password=…), by the same key rule, and the WHOLE value is masked with its delimiters kept: a quoted value (double- or single-quoted) runs to its first UNESCAPED closing quote, so an escaped quote or backslash inside it (\", \\) never ends it; a quote never closed masks to the end of its line only; a bare token ends at whitespace or a delimiter; the JSON form also masks a bare scalar ("secret":12345). The stack trace is redacted as a whole before its 8192-character output bound is applied, so a value the bound would have cut is already masked.

   Known limits of the text layer (the field filter in layer 1 has none of them — a member is judged by its key): a value that is an object or array under a sensitive key (its own members are matched by their own keys); a JSON key whose quotes are themselves escaped (\"password\": — text that was JSON-encoded before it was logged); Python-style single-quoted keys ('password': 'x'); the unquoted colon form (password: x); and a secret under a key that is not on the list. Text matching over-matches on purpose — dbPassword= and "client-secret" are masked though they are not *_<key> compounds — and the member matcher is boundary-strict (db_password_v2 matches, dbpassword does not).

Both layers read the same sensitive-key list (case-insensitive, matched on the whole key and on *_<key> / <key>_* compounds):

Never redacted, by design: correlation_id, execution_id, pipeline_id, node_id, datasource_name, user_id. These are the fields that make an incident diagnosable; scrubbing them would defeat the purpose of structured logging.

Row values are never logged at all — not at INFO, and at TRACE only under the per-row diagnostic logger that §3.2 marks as disabled in production. Redaction is a backstop for accidents, not a licence to log result data.

9.3 Redaction beyond logs — error payloads

The same secrecy rule binds anything that leaves the process, not just log output. Normative: an error details map must never carry jdbc_url, a password, or a raw connection string — in an SSE event payload, in a REST error envelope, or in an MCP tool result. Failures that are about a datasource identify it by datasource_name and, where useful, an underlying_error string that has itself passed the redaction list.

REST API §6.4.4 node_failed already complies: its details carries datasource_name + underlying_error and no URL. The rule is stated here so the next event type added does not have to rediscover it.

This is a construction rule, not a filter — the redacting encoder covers logs, and error envelopes are built by hand, so the two must be kept honest independently.

10. What's Out of Scope for v1

• Distributed tracing across pipelines (e.g., tracing a pipeline execution that triggers another pipeline) — v1 has no pipeline-to-pipeline calls, so not needed.
• Custom dashboards — operator responsibility (Grafana templates provided as documentation, not shipped code).
• Per-tenant metrics — single-tenant v1.
• Real-time alerting — operator responsibility (Alertmanager / equivalent).

Appendix A: Change Log