Operations manual

Configuration Reference

Status: v1.23 (single source of truth for every config key) Owner: datapipelines.co core Last updated: 2026-09-17


1. Purpose

Every configuration key for datapipelines.co, in one place. Environment variables, YAML paths, defaults, and descriptions. A developer or operator should never need to search across 15 specs to find a config key.

Authority rule: this document is the ONLY place a configuration key is defined. Other specs reference keys by name and link here — they never restate defaults or introduce keys of their own. A key that does not appear in this document does not exist. (Enforced by scripts/docs-audit.sh.)

Naming rules:

  • YAML paths carry explicit units as suffixes: -seconds, -minutes, -hours, -days, -ms, -bytes, -mb, -rows.
  • Env var names are derived mechanically: datapipelines. prefix → DATAPIPELINES_, then the YAML path upper-snake-cased. Example: datapipelines.executor.node-query-timeout-secondsDATAPIPELINES_EXECUTOR_NODE_QUERY_TIMEOUT_SECONDS. No abbreviations, no exceptions — the env var is always derivable from the YAML path.

2. Required Configuration

The app will not start without these. Fail-fast on missing values.

YAML path Env var Description
spring.datasource.url SPRING_DATASOURCE_URL Metadata DB JDBC URL. Example: jdbc:postgresql://host:5432/datapipelines
spring.datasource.username SPRING_DATASOURCE_USERNAME Metadata DB username
spring.datasource.password SPRING_DATASOURCE_PASSWORD Metadata DB password
datapipelines.redis.host DATAPIPELINES_REDIS_HOST Redis host (results, idempotency, post-completion event log)
datapipelines.jwt.secret DATAPIPELINES_JWT_SECRET Internal JWT signing secret. ≥ 32 bytes random, base64-encoded
datapipelines.db.encryption-key DATAPIPELINES_DB_ENCRYPTION_KEY AES-256 data key for datasource password encryption. Exactly 32 bytes, base64-encoded. Required — there is no fallback source. It is key version 1, forever: every credential written under it carries 0x01 as its first byte (Datasources §7.1). Rotation and KMS-backed sources are §3.20's key-provider seam.

OIDC provider configuration is also required — at least one provider must be configured in datapipelines.auth.oidc.providers (in application.yml). Each provider requires client-id, client-secret, and issuer-uri. The client-id and client-secret are typically referenced from env vars. See Auth spec §11.1 for the full format.

The specific env var names depend on which provider(s) the deployment chooses. Examples:

Env var pattern For
GOOGLE_CLIENT_ID, GOOGLE_CLIENT_SECRET Google Workspace
MICROSOFT_CLIENT_ID, MICROSOFT_CLIENT_SECRET Microsoft Entra ID
OKTA_CLIENT_ID, OKTA_CLIENT_SECRET Okta
KEYCLOAK_CLIENT_ID, KEYCLOAK_CLIENT_SECRET Keycloak

The deployment defines these env var names in application.yml — they're not hardcoded by the app. (OIDC provider env vars are the one deliberate exception to the naming derivation rule in §1, since the deployment names them.)


3. Optional Configuration (with defaults)

3.1 Redis

YAML path Default Description
datapipelines.redis.port 6379 Redis port
datapipelines.redis.password (none) Redis password

Env vars are derived per §1 (e.g. DATAPIPELINES_REDIS_PORT) and are omitted from the tables below for brevity.

3.2 Executor

YAML path Default Description
datapipelines.executor.max-parallel-nodes 4 Max parallel nodes within one execution
datapipelines.executor.max-concurrent-executions-per-user 10 Per-user concurrent execution limit
datapipelines.executor.max-concurrent-executions-per-instance 100 Instance-wide concurrent execution limit — per instance (050/R2): N replicas admit N × this in total (Deployment §6.2)
datapipelines.executor.max-concurrent-executions-global unset Deprecated alias for max-concurrent-executions-per-instance (one release, 050/R2). Set alone → its value runs and startup logs one WARN naming the new key; set together with the new key and differing → startup refuses. The limit was always per JVM — the old name was false at N replicas
datapipelines.executor.node-query-timeout-seconds 60 Per-node JDBC query timeout. A datasource's own query_timeout_seconds, when set, overrides this for nodes on that datasource (Datasources §5)
datapipelines.executor.execution-timeout-seconds 600 Overall execution timeout
datapipelines.executor.node-timeout-seconds 300 The per-node WALL-CLOCK deadline the executor enforces: RENDER → CONNECT → EXECUTE → STAGE → MATERIALIZE, staging included. Overridable per node by node.settings.timeout_seconds (pipeline-contract §4.11). Unlike node-query-timeout-seconds, which is the DRIVER's bound on one execute* call, this one is the executor's own and fires whatever the driver does
datapipelines.executor.node-timeout-max-seconds 900 The ceiling a node's own settings.timeout_seconds may not exceed. A pipeline declaring more is refused at SAVE with pipeline.validation.node_timeout_invalid — refused rather than clamped, so an author who asks for 4 hours is not left debugging a silent 15 minutes
datapipelines.executor.node-query-timeout-max-seconds 900 (156, #2) The ceiling a pipeline's or a node's own settings.query_timeout_seconds may not exceed (pipeline-contract §4.11/§5.3) — refused at SAVE with pipeline.validation.pipeline_query_timeout_invalid / pipeline.validation.node_query_timeout_invalid, same reasoning as node-timeout-max-seconds. Independent of that key: the two settings bound different budgets
datapipelines.executor.node-query-timeout-seconds-by-dialect.<dialect> LAKE: 180; every other dialect unset (156, #2) The operator's per-dialect default statement timeout, consulted when neither the node, the pipeline nor the datasource declares one — the innermost fallback before the flat node-query-timeout-seconds above. LAKE ships higher because a LAKE engine re-reads cold from object storage, and less memory the engine can keep means a longer cold read (§E below). Each value is validated positive and ≤ node-query-timeout-max-seconds at boot, fail-fast. One explicit YAML line per overridden dialect, never a bare env var for a dialect not already named here — Spring's relaxed binding cannot reliably reverse-map an arbitrary ..._BY_DIALECT_<X> environment variable onto this map's dashed key (measured, ExecutorPropertiesDialectMapBindingTest); the shipped form is lake: ${DATAPIPELINES_EXECUTOR_NODE_QUERY_TIMEOUT_SECONDS_BY_DIALECT_LAKE:180}
datapipelines.executor.cancel-grace-seconds 5 After a node's deadline fires, how long the executor waits for the cancelled statement to actually return before abandoning it. Past it the node fails on schedule and the leaked statement is logged once with the execution id (event=node.statement_abandoned). Never a wait on the query itself — a driver that ignores cancel() cannot extend a node's budget, only leak a connection until its own pool reclaims it
datapipelines.executor.source-fetch-size 1000 The JDBC fetchSize set on every DQL source cursor — what makes a source node stream instead of materialising its whole result inside the driver. pgjdbc uses a server-side cursor only with autoCommit=false and fetchSize > 0, so the executor also takes a Postgres source connection out of autocommit for the read and commits unconditionally on the way out (so a multi-statement template's side effects persist exactly as they did under autocommit); the pool restores autocommit when the lease returns. MySQL is the exception the driver forces: Connector/J streams only at Integer.MIN_VALUE, which the executor passes for that dialect and this key does not affect. 0 turns streaming off and leaves every source statement exactly as pre-108 code left it — the escape hatch for the one shape that can behave differently: a DQL template whose SQL is several statements, since pgjdbc's server-side-cursor path uses the extended query protocol, which carries only one
datapipelines.executor.progress-write-interval-seconds 5 The floor between two THROTTLED live-progress writes for one execution. A staging drain reports per batch — thousands of times for a large node — and one UPDATE per batch would put a metadata-DB write on the insert path. Node boundaries are not throttled: a node starting or finishing is the event a watcher is waiting for, and delaying it would leave the screen naming the wrong node
datapipelines.executor.progress-sample-interval-seconds 1 The floor between two periodic node_progress samples of one node operation (REST API §6.4.9, 149). A sample is also taken, unthrottled, at each FIRST entry into a measured state and at the operation's end, so this key bounds only how often the cumulative counts refresh — never whether a state is seen. Every sample is a live SSE event, an execution_events row and a Redis-log entry, so raising it lowers event volume on long nodes; at the default a node emits at most one periodic sample per second plus its (at most six) first-entry samples and one terminal sample
datapipelines.executor.heartbeat-seconds 15 How often a running execution stamps pipeline_executions.heartbeat_at (V21). The crash sweep reaps a RUNNING row whose stamp is older than three of these — one missed beat is a slow tick, two a loaded box, three an instance that is gone — so this also sets how fast a dead instance's rows are reaped: ~45 s plus one 15 s sweep tick, against the sixty MINUTES datapipelines.executions.stale-timeout-minutes alone gave. That key survives as the backstop for rows a pre-V21 instance left with no stamp at all

Three budgets, one precedence (the same table appears in pipeline-contract §4.11 and dag-executor §5.3):

Bound Setting Scope Enforced by
Execution datapipelines.executor.execution-timeout-seconds (600) the whole execution the executor
Node node.settings.timeout_seconds, else datapipelines.executor.node-timeout-seconds (300) one node, wall clock, all five phases the executor
Statement node.settings.query_timeout_seconds (156, §5.3), else pipeline settings.query_timeout_seconds, else the datasource's query_timeout_seconds, else node-query-timeout-seconds-by-dialect.<dialect> above, else datapipelines.executor.node-query-timeout-seconds (60) one execute* call the JDBC driver

Instances are moderate, so the statement bound must be sized for a COLD read, not a warm one (156, #2). This deployment is expected to run as several instances of moderate size, not one box with all the memory a query could want — so a LAKE engine sharing that box's memory with everything else keeps less of the object store cached, and a query that lands on an instance whose cache is cold re-reads from S3 at network speed instead of from RAM. The per-dialect operator default above exists for exactly that reason, sized to the cold read of the largest scan a deployment runs, not the warm one a developer's laptop measures. When cold reads on a given box routinely approach the default, the right levers are raising node-query-timeout-seconds-by-dialect.lake (this section) or raising the engine's own memory-limit (§3.25) so more of the data stays cached between queries — see also #141 on the shared-engine memory budget these two knobs both bound against.

Read downward. The statement bound is the driver's and drivers honour it unevenly — measured on the shipped drivers (108 §1), which is precisely why the middle row exists. The precedence is a recommendation, not a cross-key constraint: nothing refuses a configuration that inverts it, because every inversion is harmless. A node deadline above the execution's is simply never reached — the outer bound fires first and reports pipeline.execution.timeout. One below the statement timeout is stronger, not broken: the executor stops the node before the driver would have, which is the whole point of owning a bound above the driver's. What a cross-key refusal would reliably do instead is turn "I lowered execution-timeout-seconds for this deployment" into a startup crash. node-timeout-max-seconds is deliberately NOT constrained against the execution timeout — it bounds what an author may ASK for, not what a run may take.

3.3 Staging (tempdb)

YAML path Default Description
datapipelines.staging.h2.mode PostgreSQL H2 compatibility mode
datapipelines.staging.h2.max-memory-mb 1024 The memory-guard THRESHOLD each execution compares against the whole JVM's used heap — a shared, sampled measurement, not an isolated per-execution budget (see below; Staging §8.2). A pipeline's settings.tempdb.config.max_memory_mb, when present, changes the threshold for that execution (Pipeline Contract §5) — never which allocations are measured
datapipelines.staging.h2.insert-batch-size 1000 Rows per INSERT batch when staging source data
datapipelines.staging.h2.result-batch-size 10000 Rows per fetch batch when reading staged data out
datapipelines.staging.h2.query-timeout-seconds 60 H2 query timeout
datapipelines.staging.h2.max-connections 4 Cap on the operational H2 connections one execution's staging pool may hold open at once (Staging §9). A ceiling, not an allocation: the pool starts with one connection and grows on demand. 1 is legal and is the diagnosis/comparison setting — every tempdb operation then queues behind one connection

max-connections is capacity, not parallelism. It bounds how many tempdb operations of ONE execution can be inside H2 at the same instant; it creates no extra eligible DAG nodes (executor.max-parallel-nodes decides those) and promises no speedup — H2 keeps its own transaction, row and catalog locks, so two nodes touching one table still serialise inside the engine. Set it lower than max-parallel-nodes and nodes queue safely for a connection; set it higher and the surplus is never opened. Aggregate cost: every connection is one H2 session with its own query working memory, and max-concurrent-executions-per-instance executions can each hold up to this many — the staged tables themselves are shared by the connections of one execution and are not multiplied. Session state (SET SCHEMA, SET @var, local temporary tables, an open transaction) is per operation and is reset when a connection is returned; a node must not rely on it from another node — express data flow through ordinary tables and depends_on, whatever the cap (Staging §9.2).

max-memory-mb is a per-execution THRESHOLD over a JVM-WIDE measurement — not an isolated per-execution budget. The guard samples used heap for the WHOLE JVM (in-process totalMemory − freeMemory, Staging §8.2), so every concurrent execution's check reads the same shared number: one heavy execution can trip a lighter one's check, and the guard is a sampled circuit breaker — not a hard reservation, and no guarantee against OOM. Size from the demand side instead: N concurrent executions really do hold N tempdbs' tables, indexes and query working memory at once, so worst-case heap DEMAND stays max-memory-mb × datapipelines.executor.max-concurrent-executions-per-instance per instance (050/R2: the multiplier is per-instance; N replicas multiply it again — Deployment §6.6), and -Xmx plus the container limit must cover that with baseline headroom — an underestimate ends in the §8.4 JVM backstop, not in a clean pipeline.staging.memory_limit_exceeded. A per-execution accounting redesign is a separately decided follow-up, not this guard's promise.

Startup says the arithmetic out loud (108 §C). ConfigValidator logs ONE warn line at startup when max-concurrent-executions-per-instance × max-memory-mb > 0.8 × the JVM's max heap, naming all three numbers and the ratio. It is a WARNING and not a refusal, deliberately: the budget is a ceiling each execution MAY reach, not one it will, and a deployment whose pipelines stage tens of megabytes is right to run 100 slots against a 1 GB budget — refusing that would break every default installation. What an operator cannot do is notice the arithmetic unaided, since the two keys live in different sections of this document and neither used to mention the other. Above the ratio, the per-execution limit would be enforced by an OOM rather than by pipeline.staging.memory_limit_exceeded.

A pipeline's settings.tempdb.config.max_memory_mb override is clamped to ≤ this value — it may lower the operator's ceiling for that pipeline, never raise it. Save-time validation only checks > 0, so without the clamp an author could declare an arbitrarily large budget and disable the only ceiling the executor's withConnection paths have (DAG Executor §9).

3.4 Auth

YAML path Default Description
datapipelines.auth.base-url (none) The deployment's exact external origin, e.g. https://dp.example.com (scheme + host [+ port], no trailing slash). OIDC redirect URIs are built absolutely from it (Auth §5.2) — never from request headers. Startup fails when unset while any OIDC provider is configured.
datapipelines.auth.jwt.ttl-hours 8 Session JWT TTL
datapipelines.auth.allowlist.domains (empty) Comma-separated allowed email domains. Binds to List<String> (comma-split; empty string = empty list = open provisioning)
datapipelines.auth.api-keys.cache-ttl-seconds 60 Cache TTL for validated API keys and user is_active checks (Auth §11.4)
datapipelines.auth.api-keys.default-scopes read Default scope for new API keys
datapipelines.auth.rate-limit.login-per-minute 10 Per-IP login attempts per minute (OIDC and local)
datapipelines.auth.trusted-proxies (empty) CIDRs of proxies whose X-Forwarded-For names the real client — the login limiter and every audit source_ip resolve through it (Deployment §6.2). Empty (the default) = the header is ignored entirely and the direct peer is the client — a bare deployment behaves exactly as before. Each entry must parse as a CIDR (a bare IP is a host CIDR, e.g. 10.0.0.5 = 10.0.0.5/32); anything else refuses startup. Resolution is spoof-safe: an untrusted peer cannot forge a client by setting the header
datapipelines.auth.bootstrap-admin-email (none) Bootstrap admin: when a user with exactly this email is provisioned via OIDC first login, is_admin is set true (idempotent, audit-logged as auth.user.admin_granted with actor bootstrap). The ONLY way a fresh deployment gets its first admin — "first login wins" is explicitly rejected (Auth §4.4)
datapipelines.auth.local.enabled false Optional local username/password accounts (Auth §5A) — a second sign-in method for deployments without an IdP. Disabled = the deployment behaves exactly as OIDC-only
datapipelines.auth.local.bootstrap-password-hash (none) Initial credential for the FIRST ADMIN ONLY (the bootstrap-admin-email account), as a pre-computed Argon2id hash — the preferred form (produce one with the hashPassword Gradle task, Deployment §5). Seeding is create-if-absent and idempotent, forces a first-login change, and never applies to ordinary users — passwords are not a config medium
datapipelines.auth.local.bootstrap-password (none) Plaintext alternative to the hash form, accepted for zero-setup demos: always sets must_change_password, is never logged, and startup is refused when both forms are set
datapipelines.auth.local.lockout.max-failures 5 Consecutive failed local logins that lock the account — per-account, complementing the per-IP rate-limit.login-per-minute, which cannot stop a slow spray against one account
datapipelines.auth.local.lockout.duration-minutes 15 How long a locked account refuses local login. An admin unlock or password reset clears the lock early
datapipelines.auth.cookie-secure (empty) The Secure flag on the cookies this deployment mints (dp_session, dp_oauth2_authz, dp_csrf). Empty = derived from base-url's scheme: an explicit http:// base-url drops the flag so local login works over plain HTTP, anything else keeps it. true/false pin it. The hardened posture ships true (§3.23) — a hardened deployment runs https, and a dropped flag there puts the session cookie on the wire
datapipelines.auth.allow-local-only false The operator's explicit acknowledgement that a hardened deployment has no OIDC provider configured and authenticates with local password accounts only. Inert under development. Under hardened, startup is refused unless either an OIDC provider is fully configured or this is true, and setting it logs the acknowledgement (§3.23, §7)

3.5 Results

Every completed execution's caller result is stored in Redis and read through the result cursor (REST API §7).

YAML path Default Description
datapipelines.result.ttl-default-seconds 300 Result TTL when the client sends no DP-Result-TTL-Seconds header
datapipelines.result.ttl-min-seconds 60 Lower clamp for client-requested TTL
datapipelines.result.ttl-max-seconds 3600 Upper clamp for client-requested TTL
datapipelines.result.max-size-bytes 104857600 Hard cap on a caller result (100 MB). Exceeding it fails the execution with result.too_large
datapipelines.result.page-size-rows 1000 Rows in the inline first page of data_ready, and the default limit for cursor reads
datapipelines.result.page-max-rows 100000 Upper bound on the cursor limit parameter

3.6 SSE

YAML path Default Description
datapipelines.sse.heartbeat-interval-seconds 15 SSE heartbeat comment interval
datapipelines.sse.disconnect-grace-seconds 30 Grace period after client disconnect before the in-flight execution is cancelled (REST API §6.8)
datapipelines.sse.max-streams-per-user 50 Concurrent SSE streams per user

3.7 Rate Limiting

Limits are per user (an API key inherits its owner's budget — minting more keys does not raise the limit).

YAML path Default Description
datapipelines.rate-limit.requests-per-second 100 Per-user requests per second
datapipelines.rate-limit.requests-per-minute 1000 Per-user requests per minute

There is deliberately no fail-open key. When the limiter's Redis is unreachable the limiter refuses the request (429 rate_limit.unavailable, REST API §12.3); a per-deployment toggle would only ever be reached for at exactly the moment the refusal is the correct answer.

3.8 Idempotency

YAML path Default Description
datapipelines.idempotency.ttl-seconds 86400 Retention of Idempotency-Key records in Redis

3.9 Templates

YAML path Default Description
datapipelines.templates.cache-size 500 Parsed-template cache entries
datapipelines.templates.render-timeout-ms 5000 Hard limit on a single template render
datapipelines.templates.max-body-chars 262144 Max template body length accepted at save (256K chars); over-cap bodies are rejected with template.validation.syntax_error before parsing (Templates §4.2) — bounds parse cost and heap against an adversarial body

3.10 UI

YAML path Default Description
datapipelines.ui.theme dark Design system theme name. Validated at startup against the vendored themes in modules/web/src/main/resources/static/vendor/design-system/

3.11 Execution History

YAML path Default Description
datapipelines.executions.error-detail full How much of a failure's detail travels to every surface that carries an error (the SSE node_failed/pipeline_failed error objects, pipeline_executions.error_json, the editor's failure panel, MCP). full includes the exception chain with stack frames and the rendered SQL in :name form; structured omits both. Choose structured for a deployment whose pipeline authors are not trusted to see driver internals (DAG executor §8.4)
datapipelines.executions.event-retention-days 7 How long to keep execution_events rows (Postgres, the durable record) past their execution's completion — enforced by the hourly retention job (050/T60, safe for N replicas: one idempotent DELETE; pipeline_executions rows are never touched). The post-completion Redis event log lives for 1 hour, not configurable
datapipelines.executions.stale-timeout-minutes 60 Mark RUNNING executions older than this as ABORTED (crash sweep)

3.12 Audit

YAML path Default Description
datapipelines.audit.retention-days 365 Retention of audit-log rows

3.13 Server

YAML path Env var Default Description
server.port SERVER_PORT 8080 HTTP port
spring.datasource.hikari.maximum-pool-size SPRING_DATASOURCE_HIKARI_MAXIMUM_POOL_SIZE 10 Metadata DB connection pool size

3.14 Framework wiring keys

These framework key paths appear in application.yml as internal wiring. They are listed here so the §1 authority sentence stays literally true. With the single exception of management.server.port (below), operators do not set them and deployments must not override them:

Path Why it exists
spring.application.name Log/metric attribution (datapipelines)
spring.data.redis.host / .port / .password Bridge binding the canonical DATAPIPELINES_REDIS_* env vars onto Spring Boot's Redis autoconfiguration (see the binding note under §5)
spring.flyway.enabled / .locations / .baseline-on-migrate Migration wiring — Flyway always runs on startup (Deployment §8.2)
management.endpoints.web.exposure.include "health" — served on the management port only; prometheus joins it when the metrics registry lands (Observability §6.4). Must not be set to "" or exclude: "*": Spring's @ConditionalOnAvailableEndpoint keys bean creation off exposure, so an empty include falls back to Boot's default set (re-exposing health on whatever port serves actuator), and excluding everything deletes the HealthEndpoint bean the root /health controller injects — context startup fails outright.
management.health.diskspace.enabled false — the health contract has no disk component (REST API §11.1)

Operator-tunable exception:

YAML path Env var Default Description
management.server.port MANAGEMENT_SERVER_PORT 9090 Separate management port serving /actuator/health (and later /actuator/prometheus) — never publish it; nothing actuator is routable on the application port.
management.server.address MANAGEMENT_SERVER_ADDRESS 127.0.0.1 Loopback by default — the management surface is silently-public on no deployment shape. Kubernetes scraping (scraper → pod IP) requires explicitly setting 0.0.0.0, paired with a NetworkPolicy restricting the port to the monitoring namespace (Deployment §9). Explicit opt-open, never default-open.

3.15 Observability

YAML path Env var Default Description
datapipelines.observability.tracing.enabled DATAPIPELINES_OBSERVABILITY_TRACING_ENABLED false Enable OpenTelemetry tracing
datapipelines.observability.tracing.endpoint OTEL_EXPORTER_OTLP_ENDPOINT (none) OTLP collector endpoint (standard OTel env var, exception to §1 derivation)
datapipelines.observability.logging.format DATAPIPELINES_OBSERVABILITY_LOGGING_FORMAT json (prod), console (dev) Log output format

3.16 Pipelines

YAML path Default Description
datapipelines.pipelines.max-composition-depth 5 Deepest admitted chain of pipelines executing pipelines (a PIPELINE node spawning a child execution). Enforced at save time and again at runtime; must be ≥ 1

3.17 Workspaces

Workspaces are created by super admins (auth.md §4.2/§5.6, §12). The one workspace the product ships is demo, and a user with no membership becomes a viewer of it on first login.

YAML path Default Description
datapipelines.workspaces.member-datasources-enabled true May a workspace ADMIN register a datasource bound to their own workspace? false makes datasource registration a super-admin-only act instance-wide. Visibility is still the grant either way (auth.md §11A)

Removed in RBAC round 1 (provisioning-mode, open-join). Capability moved onto the workspace membership, and with it went the modes that decided who could create a workspace: auto-per-user (a personal workspace per login), self-serve (anyone creates) and closed (admin only), plus open-join (anyone self-joins). Both keys are refused BY NAME at startup (§7) rather than ignored — a deployment that still says auto-per-user is a deployment expecting a personal workspace per user, and silently giving it something else is how an operator finds out from a user. Delete the key; there is no replacement to set, because the behaviour is no longer a knob.

3.18 Bootstrap

Config-declared content applied at startup (design 2026-08-16-sample-data §6/§6.1). Both keys name files already on the container's filesystem — the app never fetches an artifact at runtime; downloading and verifying artifacts is a deployment step (D5) — and for both, unset (or empty) means the feature is off. There is no separate enable flag to disagree with the path.

Both values are comma-separated LISTS of paths (one entry per sample-data family; the demo profiles compose the list from the active families). A single value with no comma is the one-file shape older deployments ship — the list semantics are backward compatible by construction. Empty entries (a leading comma when a family is off, or a whitespace-only entry) are dropped, so a list built by shell conditional expansion never turns the feature on by accident. The files are processed in declared order.

YAML path Default Description
datapipelines.bootstrap.datasources-file (none) Path(s) to YAML file(s) of datasource definitions registered create-if-absent at startup (Datasources §8A). Set-but-unreadable, unparseable, or carrying an entry that fails §9 validation = fail-fast startup. Registration is per entry, so overlapping files are idempotent
datapipelines.bootstrap.examples-file (none) Path(s) to JSON file(s) of example templates and pipelines seeded into each personal workspace at auto-per-user provisioning, through the same import services as POST /pipelines/import and POST /templates/import. Shape: {"templates": [...], "pipelines": [...]}, each array element exactly what its import endpoint takes. Seeding runs ALL templates (file by file) before ANY pipeline, so a pipeline in one file may reference a template seeded from another. Set-but-unreadable or unparseable = fail-fast startup; an entry that fails import validation fails the provisioning login

Cross-key rule: datasources-file set while datapipelines.auth.bootstrap-admin-email is unset is a startup refusal naming both keys. Bootstrap-registered datasources are created_by that user, and registration runs before anyone has logged in, so the row is pre-provisioned from that address (Auth §4.4). examples-file carries no such rule: seeding runs at first login, under the identity of the user logging in.

Cross-key rule (examples-file): examples-file set while datapipelines.workspaces.provisioning-mode is anything but auto-per-user — including its shipped default self-serve — is a startup refusal naming both keys. Seeding runs only when first login provisions a personal workspace; under any other mode the file is read and validated at startup and then never seeded, silently. Set the mode, or unset the file.

3.19 Deployment

The deployment-role settings (Versioning §5.5) — grouped one-per-concern like auth, executor, staging.

Renamed in 075. datapipelines.deployment.name — the deployment LABEL — is now datapipelines.env (DATAPIPELINES_ENV), documented in §3.23 beside the posture it is deliberately separate from. The old key is honoured for one release: set alone it still names the deployment and startup logs event=config.deployment_name_deprecated; set together with datapipelines.env and disagreeing, startup is refused naming both. See Environments.

YAML path Env var Default Description
datapipelines.deployment.authoring-enabled DATAPIPELINES_DEPLOYMENT_AUTHORING_ENABLED true (false under the hardened posture — §3.23) The authoring capability. When false, every pipeline/template authoring write (create, update/draft, release, discard, delete) is refused with pipeline.authoring.disabled / template.authoring.disabled — fail-closed, naming the reason. Reads, execution and import are unaffected (promotion imports RELEASED versions; that is the one writer a receiver must accept). Startup REFUSES if drafts still exist while this is false (§7)

Promotion (the deployment-to-deployment channel)

Promotion is one deployment writing RELEASED content into exactly one configured higher environment (Versioning §10). The credential is a server key, not a principal (Versioning §10.6): no users row for the credential and no scope-matrix entry. A deployment may hold the receiver half, the sender half, both, or neither.

The receiver half is now a KEY, and the config value below is DEPRECATED (091). Mint a server-kind API key on the API screen (Auth §7.7) and give it to the sending deployment as its target.server-key. A stored key is everything a config value cannot be: expiring, revocable, listed beside every other credential, rotatable without a restart, and stamped with a last-used time. datapipelines.deployment.promotion.server-key is still accepted for one release so an upgrade does not break an existing pair — and it WARNs at boot when set (§7). Both credentials open the same routes; the receiver tries the configured value first, then the key store.

YAML path Env var Default Description
datapipelines.deployment.promotion.server-key DATAPIPELINES_DEPLOYMENT_PROMOTION_SERVER_KEY (empty) Receiver half — DEPRECATED (091), removed next release. The pre-shared secret an inbound promotion may present in the DP-Promotion-Key header, superseded by a server-kind API key. Empty is not "promotion is open" — with no configured value AND no live server key, every push is refused; fail closed. Compared in constant time, before the key store is consulted; never logged. Consulted on /api/v1/promotion/** and nowhere else, and it grants no read access outside that pair. Set ⇒ one WARN at boot
datapipelines.deployment.promotion.target.base-url DATAPIPELINES_DEPLOYMENT_PROMOTION_TARGET_URL (empty) Sender half. The single higher environment's base URL (e.g. https://uat.example.com). Empty = this deployment promotes nowhere and the promotion screen says so. Exactly one target — multi-target promotion is deliberately not a feature
datapipelines.deployment.promotion.target.server-key DATAPIPELINES_DEPLOYMENT_PROMOTION_TARGET_KEY (empty) Sender half. The SAME secret the target holds under its own server-key. A base-url set without this key refuses startup (§7): the pair is meaningless apart

On the SENDER, the value is whatever the receiver issued: the dpk_<id>.<secret> plaintext of the receiver's server key, shown exactly once at minting. Rotation is then a receiver-side act with no restart anywhere: mint a second server key, set it on the sender, revoke the first. For the deprecated config value, generate it the way every other secret here is — openssl rand -base64 32 — and set it identically on both sides; rotation means restarting both. Either way no user account is involved, so offboarding a human can never break production promotion — a server key's owner is the admin who minted it, and the promotion acts as the system service account regardless (Auth §4.5).

Both keys are bearer secrets. They are never logged, never in an error message, never in an audit details map, and never in a toString(). A receiver records only a truncated SHA-256 fingerprint of the key a push presented, so two keys are distinguishable across a rotation without either being carried.

3.20 Credential key provider

Where the AES data keys for datasources.credential_encrypted come from (Datasources §7.1). The seam exists so a customer's AWS/GCP/Azure/Vault key store is an implementation of a contract, not a change to the crypto — docs/key-providers.md is the guide an implementer works from. Every stored credential carries the key VERSION it was sealed under as its first byte, which is what makes rotation lazy-safe.

YAML path Env var Default Description
datapipelines.db.key-provider DATAPIPELINES_DB_KEY_PROVIDER env Which provider supplies data keys. env is the only one this build ships and is today's behaviour, so a deployment that predates the seam needs no config edit. An unknown name refuses startup, listing what is shipped (§7)
datapipelines.db.encryption-keys (per entry, e.g. DATAPIPELINES_DB_ENCRYPTION_KEYS_2) (unset) env only. Additional keys as version: base64, for a rotation — e.g. 2: ${DATAPIPELINES_DB_ENCRYPTION_KEY_V2}. Each value is exactly 32 bytes base64-encoded; each version is an integer 1..255. Version 1 may not appear here: that version is datapipelines.db.encryption-key and has exactly one spelling
datapipelines.db.encryption-key-current DATAPIPELINES_DB_ENCRYPTION_KEY_CURRENT the highest configured version env only. Which configured version NEW encryptions use. Existing rows keep decrypting under the version they carry, so flipping this rotates lazily rather than all at once (Datasources §7.3)

Every configured key is a bearer secret. None of them is logged, put in an error message, put in an audit details map, or rendered by a toString() — the §7 violations name the property and the defect, never the value.

Rotating. Generate a key (openssl rand -base64 32), add it as version N+1, set encryption-key-current: N+1, restart. New writes carry N+1; rows written under earlier versions keep decrypting, and are rewritten under the current key the next time their password is saved. Keep the old keys configured until no row still carries their version — Datasources §7.3 has the one SQL query that answers that.

3.21 Organisation

The organisation's own facts — its currency, when its fiscal year starts, which day its week starts on, which timezone "today" means. They are configuration, not pipeline data: identical for every pipeline in the deployment, changed about once a decade, and a value that differs between deployments must be visible in the deployment's yml rather than copied into every body.

Every key here enters every execution's Context as an org_* key — the yml path minus the datapipelines.org. prefix, with dots and dashes as _ (Pipeline Contract §7.2). A node binds :org_currency_symbol; a CALCULATOR node references $org_fiscal_start_date. None of them is a secret, and a pipeline may override any of them by declaring a parameter of the same name.

YAML path Default Description
datapipelines.org.currency.name Dollar The currency amounts are reported in. Context key org_currency_name
datapipelines.org.currency.symbol $ Rendered beside amounts. Context key org_currency_symbol
datapipelines.org.fiscal-start-date 01-01 MM-DD — the day the fiscal year starts; 01-01 is the calendar year. Month names are refused (§7). 02-29 is accepted and resolves to 02-28 in a non-leap year. Context key org_fiscal_start_date
datapipelines.org.week-start monday monday | sunday — which day a week starts on for week bucketing. Context key org_week_start
datapipelines.org.timezone UTC An IANA zone id. current_date is evaluated in it, so a deployment in Sydney on a UTC host still sees its own today. Context key org_timezone

Env vars follow §1's derivation rule: DATAPIPELINES_ORG_CURRENCY_NAME, DATAPIPELINES_ORG_CURRENCY_SYMBOL, DATAPIPELINES_ORG_FISCAL_START_DATE, DATAPIPELINES_ORG_WEEK_START, DATAPIPELINES_ORG_TIMEZONE.

Restart to change. There is no runtime override and no per-user setting: an org value that could change mid-run would make two nodes of one execution disagree about what year it is. A bad value stops the server (§7) rather than silently defaulting — a wrong fiscal start is a wrong number in every report the deployment produces.

3.22 Published endpoints

The timeout bounds for a released pipeline published as a GET endpoint under /api/x (round 074). timeout-default-seconds is what a publish that names no timeout is stored with; the min/max pair clamps every stored published_endpoints.timeout_seconds at write time, so retuning the bounds later never makes an existing row unreadable — it only changes what the next publish may ask for.

YAML path Default Description
datapipelines.endpoints.timeout-default-seconds 30 Timeout a publish that omits one is stored with
datapipelines.endpoints.timeout-min-seconds 1 Lower clamp for a published endpoint's timeout
datapipelines.endpoints.timeout-max-seconds 300 Upper clamp, and the longest a serve may block before it answers 202 and leaves the execution running

There is deliberately no page-rows-max key here. The DP-Result-Page-Rows request header is one contract across a published endpoint and POST /pipelines/{id}/execute, and both clamp it to datapipelines.result.page-max-rows (§3.5). A second key that had to equal the first would be a second authority for one bound, and an operator who retuned one and not the other would get two different clamps on one documented header.

3.23 Environment and posture

The two variables an organisation sets (round 075). Environments is the page written for the DevOps engineer who deploys this; this section is the key reference.

The environment's NAME belongs to the org; the POSTURE belongs to the product. Organisations have dev, qa, uat, perf, sandbox-eu, prod — any names, any count — and the product must never branch on one: a single-server user honestly writes prod as their label, and the first if (env == "prod") anywhere locks them out of authoring on the only server they have. What the product branches on is a closed, two-valued posture with the documented semantics below, and the org maps each of its environments onto one.

YAML path Env var Default Description
datapipelines.env DATAPIPELINES_ENV local The org's label for this deployment. Grammar: [a-z0-9][a-z0-9_-]{0,31} — lowercase letters, digits, _ and -, at most 32 characters. It is the boot line's identity and the source_env a promotion receiver records; nothing branches on it (pinned by a guard test) and it is deliberately not on /info, which is permitAll
datapipelines.posture DATAPIPELINES_POSTURE (none) The product's stance: development or hardened. There is no shipped default. §7 supplies development only when datapipelines.env is local; any other environment with no posture refuses to start, naming both variables — explicit beats guessed
datapipelines.demo DATAPIPELINES_DEMO (empty) The sample-data families to load out of the box, comma-separated: nyc, trade. Empty = off. Demo is a flag, not an environment: it is out-of-the-box evaluation inside a development environment, and a non-empty value is REFUSED under hardened

The posture table (normative)

Rule development hardened
datapipelines.deployment.authoring-enabled default true false — a promotion receiver; explicitly settable either way
datapipelines.demo allowed refused at boot
local bootstrap password (datapipelines.auth.local.bootstrap-password[-hash]) allowed refused at boot. Local accounts themselves stay allowed — a seeded credential is what has no place in a hardened deployment
loopback metadata DB / Redis allowed refused at boot
datapipelines.auth.cookie-secure default derived from base-url's scheme true
OIDC optional required unless DATAPIPELINES_AUTH_ALLOW_LOCAL_ONLY=true (an explicit acknowledgement, logged)
boot line event=config.posture env=<env> posture=development authoring=on demo=nyc,trade same shape

The posture IS the Spring profile

spring.profiles.active is derived from DATAPIPELINES_POSTURE in application.yml, and application-development.yml / application-hardened.yml carry the posture's non-secret defaults (the first two rows above). One variable therefore gives every loader — Compose, a bare java -jar, systemd, Kubernetes, ECS, Nomad — the right defaults, and nothing the product needs lives only in a launcher. Setting SPRING_PROFILES_ACTIVE yourself is unnecessary; a value that disagrees with the posture refuses startup (§7) rather than silently loading one posture's defaults while every posture RULE judges the other. 075 renamed the dev profile to development; a deployment still asking for dev is refused, not ignored.

Nothing lives only in a compose file

The contract is environment variables, in two files: deploy/env/defaults.env (tracked — every non-secret variable this build binds, with the value it ships) and deploy/secrets.env (git-ignored — every credential, plus every override that belongs to this deployment). deploy/secrets.env.example is the template for the second and names every variable you may set. scripts/compose-env-audit.sh — which runs on every ./gradlew build — fails when those files, deploy/compose.yml and application.yml disagree about which variables exist or what they default to, and when a variable is declared in both tracked files or in neither. Settings are tracked in the repo; secrets never are.

3.24 Lake datasource engine limits (dp-lake)

A LAKE datasource runs its queries on the app's own box — DuckDB is embedded in the app process, so its memory, threads and spill disk are operator knobs, not someone else's infrastructure. These are datasource properties (properties.dialect.*, set per datasource through the REST API or a bootstrap datasources file — Datasources §12.1), not datapipelines.* keys: they do not belong in application.yml or an env file, they are validated per key at save time (an unknown key or a bad value is refused), and a change takes effect at the datasource's next pool build, with no app restart. The existing execution bounds apply unchanged beside them: the result row cap and node-query-timeout-seconds.

Scope: one engine per datasource pool, shared by its connections (152). Each LAKE datasource pool build opens ONE embedded DuckDB instance that every pooled connection joins (Datasources §5.2); the three properties below are applied once to that instance and are ONE budget for the pool — properties.hikari.maximumPoolSize does not multiply them, and concurrent queries on the datasource share them. Size the box for memory_limit × the number of LAKE datasources on the instance (each has its own engine), with headroom for a retiring pool's engine while it drains after a datasource save. The engine's file cache lives with that instance, so a warm lake stays warm across HikariCP's routine connection replacement; a datasource save or a lake-table registration rebuilds the pool and starts a fresh engine, cold.

Datasource property Default Semantics
properties.dialect.memory_limit datapipelines.duckdb.memory-limit if set, else 25 % of the container's memory, clamped to 64 MiB – 4 GiB The engine's memory budget, e.g. 512MB or 2GB (B, KB, MB, GB, TB — a % is refused). Precedence: this per-datasource property, if set, always wins; otherwise the deployment's operator default (§3.25) applies verbatim; otherwise the default reads the container limit the container-aware JVM reports (OperatingSystemMXBean.totalMemorySize, the same figure DuckDB's own default reads), takes a quarter of it because the engine shares the box with the JVM, and hard-caps at 4 GiB so a large host does not hand one lake query enough to evict the app itself. An explicit value — this property or the operator default — is not capped
properties.dialect.threads (engine default — no statement emitted) The engine's worker threads — a positive integer, e.g. 4. Unset means DuckDB chooses (its own default tracks the box's cores)
properties.dialect.temp_directory Anonymous in-memory URL: unique datapipelines-lake-<UUID> directory under absolute JVM java.io.tmpdir (normally /tmp); named/file URL: retain engine setting Where oversized operators spill, e.g. /data/spill. Must be an absolute path under the app's data volume — inside the container the path means nothing unless the volume backs it — with no quotes, backslashes, whitespace or control characters (it is interpolated into a SET statement, and a value that would need escaping is refused, matching the lake-table location grammar)

For the supported anonymous in-memory URLs (jdbc:duckdb: and jdbc:duckdb::memory:), the default spill directory belongs to one engine initialization: pooled connections share it, while independent in-memory pool generations receive different paths. DuckDB creates it lazily when spilling and removes it on normal instance shutdown. The parent directory must exist, be writable by the application user, and have enough disk space for concurrent queries; the Helm chart provides disk-backed /tmp. A bare JVM can select the parent with -Djava.io.tmpdir=/writable/path. An explicit override is used exactly as supplied: dedicate that directory to the engine, keep unrelated files out, and avoid sharing it between independent engines or overlapping generations. Named in-memory and file-backed URLs can join an existing JVM-wide engine; when no override is supplied, its current spill setting is left untouched. Configure a writable path explicitly for those URLs, consistently across every pool sharing that engine. A crash or a driver that refuses shutdown can leave spill files behind; reclaim them only after the owning process has stopped.

threads is deliberately left unset, and the measurement is why — though not the way the hypothesis expected (108 §C). In-process DuckDB shares the JVM's CPUs with the executor, so the obvious guard is to cap the engine's worker threads; max(2, cores/2) was the proposal. Measured on a 10-core box as the median of 5 PAIRED runs (each pair times the scan alone and then immediately under load, so a drifting box moves both numbers):

threads 2 staging drains 8 staging drains
unset (all 10) 1.07× slower 1.11× slower
SET threads = 5 0.92× 1.13×

A 0.92× is a scan that ran FASTER under load than alone, which is not a result — it is the noise floor. On a box shared with other work the contention effect is smaller than the measurement's own variance, and an earlier single-run shape produced swings from 1.13× to 1.82× for the same cell. So the honest reading is: no evidence for a thread cap, and none against one either — the effect this knob would manage is not resolvable here. The default therefore stays unset (DuckDB's own scheduler handles oversubscription), an operator who needs the engine bounded sets threads explicitly, and anyone who wants to settle it should re-run scripts/measure/03-pressure.sh on a dedicated box.

What the runs DO agree on: a lake read on a busy box loses on the order of 10 % or more, and no threads value observed changed that.

Two statements are emitted on every lake connection regardless of these keys: SET memory_limit = '<explicit-or-default>' (there is always a budget) and SET preserve_insertion_order = false — insertion order costs memory and temp-file discipline the engine would otherwise spend on a guarantee a read-only lake never asks for. It is not a knob: a lake is a read connector, and making the trade configurable would only let an operator buy back a guarantee no query path uses.

3.25 DuckDB extension directory (dp-lake)

A LAKE datasource whose data is on S3 needs DuckDB's httpfs and aws extensions, and the Iceberg catalog kinds additionally need iceberg (which itself requires avro). Where those binaries come from is this key. It is an operator-level key — one value for the whole deployment, restart-to-change — deliberately NOT a per-datasource properties.dialect.* entry: which extension files the engine may load is a deployment security posture, not row data.

YAML path Env var Default Description
datapipelines.duckdb.extension-directory DATAPIPELINES_DUCKDB_EXTENSION_DIRECTORY (empty) A directory holding pre-populated DuckDB extensions in the engine's own layout (<dir>/v<core-version>/<platform>/<name>.duckdb_extension). Must be an absolute path with no quotes, backslashes, whitespace or control characters (it is interpolated into a SET statement; a value that would need escaping is refused, matching the temp_directory grammar)
datapipelines.duckdb.memory-limit DATAPIPELINES_DUCKDB_MEMORY_LIMIT (empty) (153, #136) The deployment-wide default memory_limit every LAKE engine build gets when a datasource declares no properties.dialect.memory_limit of its own — likewise operator-level, restart-to-change, deliberately not a datasource property: it is a capacity decision about the box, not row data. Empty keeps §3.24's derived 25 %-of-container default exactly. A non-empty value is a DuckDB size string, the same grammar as properties.dialect.memory_limit (512MB, 2GB, …; a % is refused), used verbatim, with no cap — the operator's own number. Validated at startup: an invalid value fails fast at context start naming this key and the grammar, rather than silently falling back to the derived default. Precedence is §3.24's per-datasource properties.dialect.memory_limit first, then this key, then the derived default

Operator instruction: set DATAPIPELINES_DUCKDB_MEMORY_LIMIT=8GB in deploy/secrets.env (or the deployment's own env file) and restart the app — the value is read once at wiring, not per pool build, so every LAKE datasource without its own properties.dialect.memory_limit picks it up on its next pool build after the restart (a fresh cold engine; the first query after restart is cold, the next warm). Confirm it took effect over a LAKE connection with SELECT current_setting('memory_limit') (reports the exact configured value, e.g. 8.0GB) or SELECT * FROM duckdb_memory() (the engine's own memory-accounting view, docs/staging.md §"Memory accounting" — its buffer manager budget reflects the same limit).

Set (the shipped image): every lake connection runs SET extension_directory = '<dir>' followed by bare LOADs — LOAD httpfs; LOAD aws for catalog.kind: s3, plus LOAD avro; LOAD iceberg for the Iceberg kinds — and never an INSTALL. In DuckDB v1.5.5 LOAD strictly loads already-present files (no download code path runs at all — measured in the 089 §7.3 spike against this exact base image with --network none), so a lake pool connects with zero egress. The published image bundles the four extensions for DuckDB core v1.5.5 under /opt/duckdb/extensions/v1.5.5/<platform>/, the <platform> following the image build's architecture (linux_amd64 or linux_arm64 — the Dockerfile maps BuildKit's TARGETARCH; hard-coding one architecture leaves the other LOAD-only against an empty directory, a pool-init failure at connect — found by the 089 live gate), and exports this variable from the Dockerfile; deploy/compose.yml defaults to the same path. The bundle adds ~108 MB uncompressed to the image (~39 MB downloaded at build). A hardened dp-lake deployment with the bundled directory needs no egress to extensions.duckdb.org at all.

Unset (a bare java -jar, a developer machine): the adapter keeps its explicit INSTALL+LOAD pairs, unchanged. The first INSTALL of each extension downloads it from extensions.duckdb.org — once per extension per container filesystem (the engine's default ~/.duckdb/extensions cache), so an ephemeral container re-downloads on every fresh start.

Version coupling: the bundled binaries are valid for exactly one DuckDB core version. The pinned duckdb_jdbc is 1.5.5.1, whose bundled core reports v1.5.5 — the directory's v1.5.5 component is the CORE version, not the JDBC patch version. A duckdb_jdbc upgrade must re-download all four extensions and rename the directory accordingly (the Dockerfile pins both in one ARG).

3.26 Datasource pools

How long a retired connection pool may keep connections out before it is closed regardless (Datasources §5.2, round 094). A datasource that is edited or deleted has its pool taken out of the live map at once and soft-evicted; the pool itself is closed when the statements already running on it finish — or at this ceiling, whichever comes first, so a genuinely hung statement cannot pin a deleted datasource's pool forever. Each hard close logs one WARN naming the datasource and the connections it took down, and increments datapipelines.datasource.pool.hard_closed (Observability §4.1).

YAML path Default Description
datapipelines.datasources.retire-ceiling-seconds derived Seconds a retired pool may hold connections before it is hard-closed. derived means datapipelines.executor.node-query-timeout-seconds + 30 (so 90 with the shipped defaults) — the longest a well-behaved node statement can run, plus slack for it to return its connection. Set it explicitly only when this deployment's datasources carry their own longer query_timeout_seconds

Why the default is derived rather than a number. The ceiling and the node query timeout are the same fact seen twice: a deployment that raises node-query-timeout-seconds to 600 and leaves a literal 90 here would start hard-closing pools out from under statements that are still legitimately running, and nothing would tell it. Deriving makes the two move together; an explicit value opts out.

3.27 Mail

Outbound mail (137, Auth §5A.8) — the SMTP server this deployment sends its two notices through: the welcome / password-reset mail to a local user (the login URL and the one-time password) and the "New user" notice to sys-ops (no password). Any SMTP server: traditional host, port, username, password. Postmark's SMTP endpoint is one such server, and its message-stream header is the only vendor-shaped knob.

Mail is enabled exactly when it is configured — there is no enabled key. Mail is on iff host AND from are both set. A flag that could disagree with the fields it summarises is the YAML-boolean trap the house has already logged; "configured" is the only honest definition of "on" for an outbound adapter. Every half-configured shape refuses startup naming the field (§7).

YAML path Default Description
datapipelines.mail.host (none) SMTP host. Empty = mail off
datapipelines.mail.port 587 SMTP port — the submission port (RFC 6409) every STARTTLS-capable server listens on
datapipelines.mail.username (none) SMTP username. Empty = the relay takes no credentials. Postmark: the server token
datapipelines.mail.password (none) SMTP password. Never logged, never in the properties' toString. Postmark: the server token
datapipelines.mail.starttls true STARTTLS on the submission connection — required, not opportunistic: true refuses a server that does not upgrade rather than falling back to plaintext. The hardened posture refuses false (§3.23, §7)
datapipelines.mail.from (none) The sender — a bare address or Display Name <address>. Half of "enabled"
datapipelines.mail.reply-to (none) Reply-To on every message — the human a user writes back to. Empty = from
datapipelines.mail.ops-to (none) The sys-ops sink for the "New user" notice — one address or a comma list. Empty = no new-user notices, even with mail on. Set without host = refused (a sink nobody can reach is a misconfiguration, not a preference)
datapipelines.mail.message-stream (none) Postmark's X-PM-Message-Stream header. Set = added to every message; empty = nothing is added

Cross-key rule: mail on requires datapipelines.auth.base-url (§3.4) — the welcome mail carries the login URL, which is built from it and never from a request's origin (Auth §5.2).

What the admin screen shows follows from the derivation: with mail on, the create-user and reset-password actions no longer display the one-time password — it went to the user, and the screen says so with the send's outcome; with mail off they show it as before (UI §4.12).


4. Precedence

Resolution order for any key, highest first:

  1. Environment variable (via the ${ENV:default} placeholder).
  2. Active profile YAML — the POSTURE file, application-development.yml or application-hardened.yml (§3.23).
  3. Base application.yml default.

Under a loader that assembles env files (docker compose --env-file, set -a; . file), the files are read in order and the LAST one wins; app.sh passes deploy/env/defaults.env, then deploy/secrets.env — secrets last, so a value an operator sets there always wins. Two files, that order, under every loader.

Two documented per-entity overrides sit above global config at runtime (they are data, not config):

  • Pipeline settings.tempdb.config.max_memory_mb overrides datapipelines.staging.h2.max-memory-mb for that pipeline.
  • Datasource query_timeout_seconds overrides datapipelines.executor.node-query-timeout-seconds for nodes on that datasource.

5. Full application.yml Template

Complete — a deployment assembled from this block gets the framework wiring (§3.14) too. Omitting the management: block in particular re-serves the actuator on the application port, which is exactly the exposure the management port exists to prevent.

spring:
  application:
    name: datapipelines
  profiles:
    # §3.23 — the POSTURE selects the profile that carries its defaults. Empty means no
    # profile, and the base defaults below ARE the `development` column.
    active: ${DATAPIPELINES_POSTURE:}
  datasource:
    url: ${SPRING_DATASOURCE_URL}
    username: ${SPRING_DATASOURCE_USERNAME}
    password: ${SPRING_DATASOURCE_PASSWORD}
    hikari:
      maximum-pool-size: ${SPRING_DATASOURCE_HIKARI_MAXIMUM_POOL_SIZE:10}
  data:
    redis:                       # §3.14 bridge — operators set only DATAPIPELINES_REDIS_*
      host: ${DATAPIPELINES_REDIS_HOST}
      port: ${DATAPIPELINES_REDIS_PORT:6379}
      password: ${DATAPIPELINES_REDIS_PASSWORD:}
  flyway:
    enabled: true
    locations: classpath:db/migration
    baseline-on-migrate: false

server:
  port: ${SERVER_PORT:8080}

management:                      # §3.14 — actuator on the management port ONLY
  server:
    port: ${MANAGEMENT_SERVER_PORT:9090}
    address: ${MANAGEMENT_SERVER_ADDRESS:127.0.0.1}   # loopback default; 0.0.0.0 only with a NetworkPolicy
  endpoints:
    web:
      exposure:
        include: "health"        # never "" and never exclude:"*" — see §3.14
  health:
    diskspace:
      enabled: false

datapipelines:
  redis:
    host: ${DATAPIPELINES_REDIS_HOST}
    port: ${DATAPIPELINES_REDIS_PORT:6379}
    password: ${DATAPIPELINES_REDIS_PASSWORD:}

  jwt:
    secret: ${DATAPIPELINES_JWT_SECRET}

  db:
    key-provider: ${DATAPIPELINES_DB_KEY_PROVIDER:env}
    encryption-key: ${DATAPIPELINES_DB_ENCRYPTION_KEY}
    # Rotation (§3.20) — unset by default; a declared empty map would bind over the environment.
    # encryption-keys:
    #   2: ${DATAPIPELINES_DB_ENCRYPTION_KEY_V2}
    # encryption-key-current: 2

  auth:
    oidc:
      providers:
        # client-id defaulting to empty = the entry is IGNORED with a WARN (§7);
        # a local-accounts-only deployment starts with zero OIDC providers.
        - name: google
          client-id: ${GOOGLE_CLIENT_ID:}
          client-secret: ${GOOGLE_CLIENT_SECRET:}
          issuer-uri: https://accounts.google.com
          display-name: "Sign in with Google"
        - name: microsoft
          client-id: ${MICROSOFT_CLIENT_ID:}
          client-secret: ${MICROSOFT_CLIENT_SECRET:}
          issuer-uri: https://login.microsoftonline.com/common/v2.0
          display-name: "Sign in with Microsoft"
        # Add more providers as needed (Okta, Auth0, Keycloak, etc.)
    jwt:
      ttl-hours: ${DATAPIPELINES_AUTH_JWT_TTL_HOURS:8}
    allowlist:
      domains: ${DATAPIPELINES_AUTH_ALLOWLIST_DOMAINS:}
    api-keys:
      cache-ttl-seconds: ${DATAPIPELINES_AUTH_API_KEYS_CACHE_TTL_SECONDS:60}
      default-scopes: ${DATAPIPELINES_AUTH_API_KEYS_DEFAULT_SCOPES:read}
    rate-limit:
      login-per-minute: ${DATAPIPELINES_AUTH_RATE_LIMIT_LOGIN_PER_MINUTE:10}
    trusted-proxies: ${DATAPIPELINES_AUTH_TRUSTED_PROXIES:}
    local:
      enabled: ${DATAPIPELINES_AUTH_LOCAL_ENABLED:false}
      bootstrap-password-hash: ${DATAPIPELINES_AUTH_LOCAL_BOOTSTRAP_PASSWORD_HASH:}
      bootstrap-password: ${DATAPIPELINES_AUTH_LOCAL_BOOTSTRAP_PASSWORD:}
      lockout:
        max-failures: ${DATAPIPELINES_AUTH_LOCAL_LOCKOUT_MAX_FAILURES:5}
        duration-minutes: ${DATAPIPELINES_AUTH_LOCAL_LOCKOUT_DURATION_MINUTES:15}
    cookie-secure: ${DATAPIPELINES_AUTH_COOKIE_SECURE:}
    allow-local-only: ${DATAPIPELINES_AUTH_ALLOW_LOCAL_ONLY:false}

  executor:
    max-parallel-nodes: ${DATAPIPELINES_EXECUTOR_MAX_PARALLEL_NODES:4}
    max-concurrent-executions-per-user: ${DATAPIPELINES_EXECUTOR_MAX_CONCURRENT_EXECUTIONS_PER_USER:10}
    # Per-INSTANCE ceiling (050/R2); the deprecated `max-concurrent-executions-global` alias is
    # deliberately absent here — it binds only when an operator still sets it.
    max-concurrent-executions-per-instance: ${DATAPIPELINES_EXECUTOR_MAX_CONCURRENT_EXECUTIONS_PER_INSTANCE:100}
    node-query-timeout-seconds: ${DATAPIPELINES_EXECUTOR_NODE_QUERY_TIMEOUT_SECONDS:60}
    execution-timeout-seconds: ${DATAPIPELINES_EXECUTOR_EXECUTION_TIMEOUT_SECONDS:600}
    node-timeout-seconds: ${DATAPIPELINES_EXECUTOR_NODE_TIMEOUT_SECONDS:300}
    node-timeout-max-seconds: ${DATAPIPELINES_EXECUTOR_NODE_TIMEOUT_MAX_SECONDS:900}
    cancel-grace-seconds: ${DATAPIPELINES_EXECUTOR_CANCEL_GRACE_SECONDS:5}
    source-fetch-size: ${DATAPIPELINES_EXECUTOR_SOURCE_FETCH_SIZE:1000}
    progress-write-interval-seconds: ${DATAPIPELINES_EXECUTOR_PROGRESS_WRITE_INTERVAL_SECONDS:5}
    progress-sample-interval-seconds: ${DATAPIPELINES_EXECUTOR_PROGRESS_SAMPLE_INTERVAL_SECONDS:1}
    heartbeat-seconds: ${DATAPIPELINES_EXECUTOR_HEARTBEAT_SECONDS:15}

  pipelines:
    max-composition-depth: ${DATAPIPELINES_PIPELINES_MAX_COMPOSITION_DEPTH:5}

  deployment:
    authoring-enabled: ${DATAPIPELINES_DEPLOYMENT_AUTHORING_ENABLED:true}
    promotion:
      server-key: ${DATAPIPELINES_DEPLOYMENT_PROMOTION_SERVER_KEY:}
      target:
        base-url: ${DATAPIPELINES_DEPLOYMENT_PROMOTION_TARGET_URL:}
        server-key: ${DATAPIPELINES_DEPLOYMENT_PROMOTION_TARGET_KEY:}

  workspaces:
    member-datasources-enabled: ${DATAPIPELINES_WORKSPACES_MEMBER_DATASOURCES_ENABLED:true}

  staging:
    h2:
      mode: ${DATAPIPELINES_STAGING_H2_MODE:PostgreSQL}
      max-memory-mb: ${DATAPIPELINES_STAGING_H2_MAX_MEMORY_MB:1024}
      insert-batch-size: ${DATAPIPELINES_STAGING_H2_INSERT_BATCH_SIZE:1000}
      result-batch-size: ${DATAPIPELINES_STAGING_H2_RESULT_BATCH_SIZE:10000}
      query-timeout-seconds: ${DATAPIPELINES_STAGING_H2_QUERY_TIMEOUT_SECONDS:60}
      max-connections: ${DATAPIPELINES_STAGING_H2_MAX_CONNECTIONS:4}

  result:
    ttl-default-seconds: ${DATAPIPELINES_RESULT_TTL_DEFAULT_SECONDS:300}
    ttl-min-seconds: ${DATAPIPELINES_RESULT_TTL_MIN_SECONDS:60}
    ttl-max-seconds: ${DATAPIPELINES_RESULT_TTL_MAX_SECONDS:3600}
    max-size-bytes: ${DATAPIPELINES_RESULT_MAX_SIZE_BYTES:104857600}
    page-size-rows: ${DATAPIPELINES_RESULT_PAGE_SIZE_ROWS:1000}
    page-max-rows: ${DATAPIPELINES_RESULT_PAGE_MAX_ROWS:100000}

  sse:
    heartbeat-interval-seconds: ${DATAPIPELINES_SSE_HEARTBEAT_INTERVAL_SECONDS:15}
    disconnect-grace-seconds: ${DATAPIPELINES_SSE_DISCONNECT_GRACE_SECONDS:30}
    max-streams-per-user: ${DATAPIPELINES_SSE_MAX_STREAMS_PER_USER:50}

  rate-limit:
    requests-per-second: ${DATAPIPELINES_RATE_LIMIT_REQUESTS_PER_SECOND:100}
    requests-per-minute: ${DATAPIPELINES_RATE_LIMIT_REQUESTS_PER_MINUTE:1000}

  idempotency:
    ttl-seconds: ${DATAPIPELINES_IDEMPOTENCY_TTL_SECONDS:86400}

  templates:
    cache-size: ${DATAPIPELINES_TEMPLATES_CACHE_SIZE:500}
    render-timeout-ms: ${DATAPIPELINES_TEMPLATES_RENDER_TIMEOUT_MS:5000}

  ui:
    theme: ${DATAPIPELINES_UI_THEME:dark}

  executions:
    event-retention-days: ${DATAPIPELINES_EXECUTIONS_EVENT_RETENTION_DAYS:7}
    stale-timeout-minutes: ${DATAPIPELINES_EXECUTIONS_STALE_TIMEOUT_MINUTES:60}

  audit:
    retention-days: ${DATAPIPELINES_AUDIT_RETENTION_DAYS:365}

  observability:
    tracing:
      enabled: ${DATAPIPELINES_OBSERVABILITY_TRACING_ENABLED:false}
    logging:
      format: ${DATAPIPELINES_OBSERVABILITY_LOGGING_FORMAT:json}

  bootstrap:
    datasources-file: ${DATAPIPELINES_BOOTSTRAP_DATASOURCES_FILE:}
    examples-file: ${DATAPIPELINES_BOOTSTRAP_EXAMPLES_FILE:}

  # §3.21 — organisation facts. Appended AFTER the whole datapipelines: tree, never inserted
  # between its children: a 2-space block placed mid-tree re-parents whatever follows it.
  org:
    currency:
      name: ${DATAPIPELINES_ORG_CURRENCY_NAME:Dollar}
      symbol: ${DATAPIPELINES_ORG_CURRENCY_SYMBOL:$}
    fiscal-start-date: ${DATAPIPELINES_ORG_FISCAL_START_DATE:01-01}
    week-start: ${DATAPIPELINES_ORG_WEEK_START:monday}
    timezone: ${DATAPIPELINES_ORG_TIMEZONE:UTC}
  endpoints:
    timeout-default-seconds: ${DATAPIPELINES_ENDPOINTS_TIMEOUT_DEFAULT_SECONDS:30}
    timeout-min-seconds: ${DATAPIPELINES_ENDPOINTS_TIMEOUT_MIN_SECONDS:1}
    timeout-max-seconds: ${DATAPIPELINES_ENDPOINTS_TIMEOUT_MAX_SECONDS:300}
  # §3.23 — appended after the whole datapipelines: tree, never inserted between two of
  # its children (a 2-space block placed mid-tree closes its predecessor and silently
  # re-parents whatever follows it).
  env: ${DATAPIPELINES_ENV:local}
  posture: ${DATAPIPELINES_POSTURE:}
  demo: ${DATAPIPELINES_DEMO:}

  # §3.25 — the LAKE engine's bundled extension directory. Empty = INSTALL+LOAD with egress
  # (a bare jar has nothing bundled); the shipped image exports the variable itself.
  duckdb:
    extension-directory: ${DATAPIPELINES_DUCKDB_EXTENSION_DIRECTORY:}
    # §3.25 — the operator default memory_limit every LAKE engine build gets when a
    # datasource declares none. Empty = the derived 25%-of-container default.
    memory-limit: ${DATAPIPELINES_DUCKDB_MEMORY_LIMIT:}

  # §3.27 — outbound mail. NO `enabled` key: on exactly when host AND from are set.
  mail:
    host: ${DATAPIPELINES_MAIL_HOST:}
    port: ${DATAPIPELINES_MAIL_PORT:587}
    username: ${DATAPIPELINES_MAIL_USERNAME:}
    password: ${DATAPIPELINES_MAIL_PASSWORD:}
    starttls: ${DATAPIPELINES_MAIL_STARTTLS:true}
    from: ${DATAPIPELINES_MAIL_FROM:}
    reply-to: ${DATAPIPELINES_MAIL_REPLY_TO:}
    ops-to: ${DATAPIPELINES_MAIL_OPS_TO:}
    message-stream: ${DATAPIPELINES_MAIL_MESSAGE_STREAM:}

Note: OIDC provider config is in the app's own YAML namespace (datapipelines.auth.oidc.providers), NOT in Spring Security's native spring.security.oauth2.client.* namespace. Our OidcConfig bean reads this list and builds ClientRegistration objects programmatically. See Auth spec §5.2.

Internal binding note (Redis, 2026-08-07): Spring Boot's Redis autoconfiguration reads spring.data.redis.*, so application.yml carries an internal bridge (spring.data.redis.host: ${DATAPIPELINES_REDIS_HOST} etc.) mapping the canonical datapipelines.redis.* keys onto it. This introduces NO operator-facing keys — operators set only the DATAPIPELINES_REDIS_* variables defined here. The bridge is an implementation detail and may be replaced by an explicit LettuceConnectionFactory bound to @ConfigurationProperties("datapipelines.redis").


6. Posture Profiles (application-development.yml / application-hardened.yml)

The POSTURE's non-secret defaults (§3.23). spring.profiles.active is derived from DATAPIPELINES_POSTURE in application.yml, so setting that one variable loads the right file under every loader — Compose, a bare java -jar, systemd, Kubernetes. Setting SPRING_PROFILES_ACTIVE yourself is unnecessary, and a value that disagrees with the posture refuses startup (§7).

These files carry POSTURE, never INFRASTRUCTURE. Before 075 this section documented application-dev.yml, which was really a LAPTOP file: localhost:5434, console logging, one developer's host ports. An org's dev environment on Kubernetes runs the development posture too and must not inherit any of that. The laptop's values are ordinary lines in deploy/env/defaults.env, which every loader reads (DEVELOPMENT.md §2/§4).

And they carry the posture's defaults ALONE (081). No env file may name DATAPIPELINES_DEPLOYMENT_AUTHORING_ENABLED or DATAPIPELINES_AUTH_COOKIE_SECURE: there is one tracked settings file, it is loaded for every posture, and an environment variable outranks a profile — so a value there would be one posture's answer imposed on both. 075 did exactly that and a hardened stack booted with authoring=on. deploy/compose.yml passes those two in Compose's valueless form (DATAPIPELINES_AUTH_COOKIE_SECURE:, nothing after the colon), the only form that leaves a variable unset rather than empty — an empty variable outranks a profile just as a set one does. An operator can still override either in deploy/secrets.env. PostureDefaultsSpecDriftTest and compose-env-audit.sh check 7 hold the two halves.

No literal secrets, in either file (2026-08-07 security review): both ship inside every production jar (src/main/resources), so a working literal there means one stray SPRING_PROFILES_ACTIVE on a production manifest runs real infrastructure on publicly-known keys — forgeable admin JWTs and decryptable datasource credentials. Every secret is a placeholder resolved from the deployment's own environment, in every posture.

# application-development.yml
datapipelines:
  deployment:
    authoring-enabled: ${DATAPIPELINES_DEPLOYMENT_AUTHORING_ENABLED:true}
  auth:
    cookie-secure: ${DATAPIPELINES_AUTH_COOKIE_SECURE:}
# application-hardened.yml
datapipelines:
  deployment:
    authoring-enabled: ${DATAPIPELINES_DEPLOYMENT_AUTHORING_ENABLED:false}
  auth:
    cookie-secure: ${DATAPIPELINES_AUTH_COOKIE_SECURE:true}

Everything else the posture means is a §7 REFUSAL, which no YAML file can carry: demo, a seeded bootstrap credential, loopback infrastructure and a missing OIDC provider are all refused under hardened. The full table is §3.23.

The laptop's own settings, for reference (DEVELOPMENT.md §2/§4) — ordinary lines in deploy/env/defaults.env, read FIRST like every other default, with no inversion of the secrets-last rule:

DATAPIPELINES_ENV=local
DATAPIPELINES_POSTURE=development
SPRING_DATASOURCE_URL=jdbc:postgresql://localhost:5434/datapipelines
SPRING_DATASOURCE_USERNAME=datapipelines
DATAPIPELINES_REDIS_HOST=localhost
DATAPIPELINES_REDIS_PORT=6381
DATAPIPELINES_OBSERVABILITY_LOGGING_FORMAT=console
DATAPIPELINES_AUTH_ALLOWLIST_DOMAINS=

The two passwords are absent on purpose: they are secrets, they live only in deploy/secrets.env, and deploy/compose.laptop-infra.yml starts its Postgres and Redis from those same two variables (each keeping its historical fixed dev value as the interpolation default). Before 081 that file hard-coded datapipelines and no Redis password at all, and a third env file had to be loaded after the secrets to undo them — the one inversion this design used to have. DATAPIPELINES_AUTH_BASE_URL is likewise absent: it is this deployment's own identity and lives in secrets.env, where the scaffold writes http://localhost:<port>.

Dev host ports (2026-08-12): the laptop's Postgres listens on host port 5434 and Redis on 6381 — not the universal defaults 5432/6379, which collide with other local stacks on developer machines (Postgres.app/brew default to 5432). The host mapping lives in deploy/compose.laptop-infra.yml; SPRING_DATASOURCE_URL and DATAPIPELINES_REDIS_PORT in deploy/env/defaults.env must stay in sync with it and with DEVELOPMENT.md §2/§4. In production, SPRING_DATASOURCE_URL and DATAPIPELINES_REDIS_* are operator-set and unaffected.


7. Config Validation

On startup, the app validates:

  • All required keys present → fail-fast with a clear error if missing.
  • DATAPIPELINES_JWT_SECRET ≥ 32 bytes decoded.
  • DATAPIPELINES_DB_ENCRYPTION_KEY is exactly 32 bytes decoded.
  • Key provider (§3.20): datapipelines.db.key-provider names a provider this build ships (a violation lists them and points at docs/key-providers.md), and that provider's own settings are present and well-formed. For env: every entry of datapipelines.db.encryption-keys has an integer version in 1..255 that is not 1, and a value that decodes to exactly 32 bytes; datapipelines.db.encryption-key-current is an integer naming a configured version (the violation lists which versions ARE configured). An unknown provider name short-circuits the rest — reporting env's key rules to an operator who asked for aws-kms is noise they cannot act on.
  • DATAPIPELINES_UI_THEME matches a vendored theme directory.
  • At least one authentication method: a fully-configured OIDC provider (non-empty client-id, client-secret, and issuer-uri) or datapipelines.auth.local.enabled=true. A provider entry with an empty client-id is ignored with a WARN — it does not count, and it is not a violation on its own.
  • datapipelines.auth.local.bootstrap-password and datapipelines.auth.local.bootstrap-password-hash are never both set; either seed requires datapipelines.auth.local.enabled=true AND datapipelines.auth.bootstrap-admin-email — each violation names both keys.
  • datapipelines.auth.local.lockout.max-failures and datapipelines.auth.local.lockout.duration-minutes are positive integers.
  • The deprecated executor alias datapipelines.executor.max-concurrent-executions-global (050/R2): set alone → its value runs and startup logs one WARN naming max-concurrent-executions-per-instance; set together with the new key and differing → startup REFUSES naming both keys.
  • result.ttl-min-secondsresult.ttl-default-secondsresult.ttl-max-seconds.
  • endpoints.timeout-min-secondsendpoints.timeout-default-secondsendpoints.timeout-max-seconds, and the minimum is ≥ 1 (§3.22). Unlike the result TTLs, an ABSENT key is not a violation here — the binder default applies — so only a present, out-of-order triple refuses startup.
  • datapipelines.workspaces.provisioning-mode is one of auto-per-user | self-serve | closed.
  • datapipelines.workspaces.open-join: true together with closed provisioning is refused, naming both keys (§3.17) — open-join is a self-serve knob, and under closed it would let any authenticated user self-join any workspace, the exact surface closed exists to close.
  • Every datapipelines.auth.trusted-proxies entry parses as a CIDR (a bare IP is a host CIDR); anything else refuses startup at the auth module's resolver construction — a typo'd range must not silently widen proxy trust.
  • datapipelines.bootstrap.datasources-file is not set without datapipelines.auth.bootstrap-admin-email (§3.18) — the violation names both keys.
  • datapipelines.bootstrap.examples-file is not set while datapipelines.workspaces.provisioning-mode is anything but auto-per-user (§3.18) — the violation names both keys. Only auto-per-user provisions the personal workspace the examples are seeded into, so any other mode (the shipped default included) leaves the configured file permanently unseeded. A mode that is misspelled is reported by the mode check alone, not twice.
  • No OIDC provider is named bootstrap, local or system. Those are the users.provider values the system writes for identities it creates itself (§6.1 bootstrap actor, §5A local accounts, Auth §4.5 system service account), and a provider's configured name is written to that column verbatim — an external provider under any of them would be indistinguishable from them, and for system that indistinguishability is the whole of the account's safety argument. The reservation is case-insensitive and applies to an entry with a blank client-id too.
  • Environment name (§3.23): datapipelines.env matches [a-z0-9][a-z0-9_-]{0,31}. The deprecated datapipelines.deployment.name alias: set alone it still names the deployment and startup logs event=config.deployment_name_deprecated; set together with datapipelines.env and disagreeing, startup is REFUSED naming both keys.
  • Posture (§3.23): datapipelines.posture is development or hardened. A named environment with no posture refuses to start, naming both variables — the sole exception is the environment named local, which resolves to development. Explicit beats guessed: the product cannot infer a stance from a name it is forbidden to branch on.
  • Posture / profile alignment (§3.23): spring.profiles.active is derived from the posture, so setting SPRING_PROFILES_ACTIVE to a DIFFERENT posture refuses startup naming both — otherwise one posture's defaults would load while every posture rule below judged the other. The dev profile was renamed development in 075; a deployment still asking for dev is refused rather than silently getting no posture file at all.
  • The hardened posture (§3.23), each refusal naming the variable and the posture: datapipelines.demo must be empty; no local bootstrap credential may be set (bootstrap-password or bootstrap-password-hash — local accounts themselves stay allowed); spring.datasource.url and datapipelines.redis.host must not be loopback (039's dev-profile guard, inverted and reused — "dev convenience must never touch production infrastructure" is the same fact as "a hardened deployment does not run against a laptop's database"); and at least one OIDC provider must be fully configured unless datapipelines.auth.allow-local-only=true, which is logged as event=config.auth_local_only.
  • Organisation (§3.21): datapipelines.org.fiscal-start-date is MM-DD and a day the calendar has — 02-30 and 13-01 are refused, and a month name (SEP-15) is refused with a message naming the MM-DD form; datapipelines.org.week-start is monday or sunday; datapipelines.org.timezone is an IANA zone id (a fixed offset such as +02:00 is not one); datapipelines.org.currency.name and .symbol are non-blank. All four report together — every value is in every Context, so a wrong one is a wrong number in every report the deployment produces.
  • Redis auth warning: when datapipelines.redis.password is empty and datapipelines.redis.host is not loopback, log a structured WARN (production Redis holds materialized caller results — Deployment §7.3).
  • datapipelines.deployment.promotion.server-key set ⇒ WARN (091): the value is deprecated in favour of a server-kind API key and is removed next release. Presence only — the warning never carries the secret.
  • datapipelines.deployment.promotion.target.base-url is not set without datapipelines.deployment.promotion.target.server-key (§3.19) — the violation names both keys. The target's pre-shared key is what authenticates the push, so a target without one would have every promotion refused at the far end, at the end of a UI action a human took. The reverse is not a violation: a server-key with no target is an ordinary receiver.
  • Mail (§3.27, 137): datapipelines.mail.host and datapipelines.mail.from are set together or not at all — one without the other refuses startup naming the missing one; datapipelines.mail.ops-to without host is refused (a sink nobody can reach); mail on (both set) without datapipelines.auth.base-url is refused (the welcome mail's login URL); port is a port in 1..65535; from is an address (a bare address or Display Name <address>). Under the hardened posture, with mail on: starttls must be true, and username set requires password set — each refusal naming the variable and the posture. The SMTP password is carried as PRESENCE only; no violation ever prints it.
  • The boot line (§3.23): event=config.posture env=<env> posture=<posture> authoring=<on|off> demo=<families> is logged once (the label's only consumer — no code branches on it, pinned by a guard test). When a promotion receiver key is configured AND datapipelines.deployment.authoring-enabled=true, log a structured WARN — a promotion receiver should not author (Versioning D7), though a one-box deployment may legitimately be both. And when authoring is DISABLED while draft pipeline/template versions still exist, startup FAILS naming them: someone authored on a receiver and version alignment may already be broken (Versioning §5.5/§9.3).

The validator's own test suite must assert that the documented laptop setup (deploy/env/defaults.env + deploy/secrets.env, §6) passes the production rules — so a broken local value gets fixed at the data, never by weakening the check.

Validation runs in @PostConstruct of a ConfigValidator bean. Failures stop startup with a clear log message listing every missing/invalid key.


Appendix A: Change Log

Date Version Author Change
2026-09-17 v1.23 155 / #122 staging memory guard scope §3.3: max-memory-mb re-described truthfully — the threshold is per-execution (the pipeline override changes the number one execution compares, never what is measured), but the MEASUREMENT is the whole JVM's used heap, sampled, so the guard is a shared circuit breaker, not an isolated budget, a reservation, or an OOM guarantee (Staging §8.2 says the same; one cross-link, not two copies). Sizing guidance rewritten from the demand side: worst-case heap demand stays max-memory-mb × max-concurrent-executions-per-instance per instance and -Xmx/container memory must cover it. No behaviour change; the 108 §C startup warn line is unchanged.
2026-09-17 v1.22 153 operator memory-limit config (#136) New §3.25 row: datapipelines.duckdb.memory-limit (DATAPIPELINES_DUCKDB_MEMORY_LIMIT, default empty). Deployment-wide default memory_limit for every LAKE engine build when a datasource sets none, restart-to-change, validated at startup with properties.dialect.memory_limit's own grammar. §3.24's memory_limit row gains the three-way precedence sentence (datasource property > this key > derived 25%). §3.25's title and intro widened to cover both operator keys. §5 template block appended after extension-directory (same block, not a new one).
2026-09-17 v1.21 Writable LAKE spill default (#133) §3.24: unique engine-owned absolute path under java.io.tmpdir; writable disk requirement, explicit override and cleanup semantics.
2026-09-17 v1.20 152 LAKE engine scope (#128) §3.24 gains the resource-scope paragraph: a LAKE datasource pool owns ONE embedded DuckDB instance shared by all its pooled connections, so memory_limit / threads / temp_directory are one budget per datasource pool (not per connection, not multiplied by maximumPoolSize), the sizing rule is per LAKE datasource on the instance plus a draining generation's headroom, and the file cache survives Hikari's connection replacement but not a pool rebuild. No key added or changed. Status line re-synced to the changelog (it read v1.1 above rows that ended at v1.18)
2026-09-16 v1.19 149 / #125 node_progress cadence §3.2 gains datapipelines.executor.progress-sample-interval-seconds (1): the floor between two periodic node_progress samples of one operation; first entries into a state and the terminal sample are never throttled by it. Mirrored in application.yml, defaults.env, secrets.env.example, compose.yml; WebPropertiesSpecDriftTest pins the default.
2026-09-14 v1.18 137 mail notices New §3.27 Mail: datapipelines.mail.host / port (587) / username / password / starttls (true, required-not-opportunistic) / from / reply-to (empty = from) / ops-to (comma list) / message-stream (Postmark's optional header). No enabled key — mail is on exactly when host and from are both set. §5 template block appended after duckdb:; §7 gains the shape rules (host↔from, ops-to without host, mail on without auth.base-url, port range, from is an address) and the two hardened refusals (starttls must be true; a username needs a password) — checks 23 and 24
2026-09-08 v1.17 091 keys §3.19: datapipelines.deployment.promotion.server-key is DEPRECATED in favour of a server-kind API key (Auth §7.7) — mintable by an admin on the API screen, expiring, revocable, rotatable without a restart, and visible in the key list. Both credentials are accepted for one release; the configured value is compared FIRST (so a deployment that has not migrated pays no database read) and its presence raises one WARN at boot. §7 gains that rule (check 23). Fail-closed is unchanged and now has two halves: no configured value AND no live server key ⇒ every push refused.
2026-09-07 v1.16 089 dp-lake §D extension bundling New §3.25 DuckDB extension directory (dp-lake): datapipelines.duckdb.extension-directory (DATAPIPELINES_DUCKDB_EXTENSION_DIRECTORY, default empty). Set — as the shipped image does via the Dockerfile, which now bundles httpfs/aws/iceberg/avro for DuckDB core v1.5.5 for the build's architecture (linux_amd64 or linux_arm64 via TARGETARCH, +108 MB uncompressed, ~39 MB at build) — lake connections SET extension_directory and emit bare LOADs, never an INSTALL, so a hardened dp-lake deployment needs no egress to extensions.duckdb.org. Unset keeps the INSTALL+LOAD behavior developer machines rely on. §5 template block appended after demo:
2026-09-07 v1.15 089 dp-lake phases B–D New §3.24 Lake datasource engine limits (dp-lake): the properties.dialect.memory_limit / threads / temp_directory datasource properties (datasource-row keys, NOT datapipelines.* env keys — validated per key at save), the default memory limit (25 % of the container's memory as the cgroup-aware JVM reports it, hard-capped at 4 GiB, floored at 64 MiB), and the always-on preserve_insertion_order = false. Phases B/C add no operator keys: per-table views and registry-backed introspection are behavior, not configuration
2026-09-06 v1.14 081 one env file §3.23, §4 and §6 rewritten to two files. The contract is deploy/env/defaults.env (tracked, every non-secret variable with its shipped value) then deploy/secrets.env (git-ignored, credentials and this deployment's own overrides), in that order under every loader; deploy/secrets.env.example is the template and names every variable. The five files 075 shipped under deploy/env/ are deleted, and with them the laptop's load-order inversion — deploy/compose.laptop-infra.yml now starts Postgres and Redis from SPRING_DATASOURCE_PASSWORD and DATAPIPELINES_REDIS_PASSWORD. §6: the posture's two default rows live ONLY in the profile ymls, and no env file may name them. compose-env-audit.sh gains the one-authority proof (each bound variable declared in exactly one tracked file) and the valueless-form check.
2026-09-05 v1.13 075 environments and posture New §3.23 Environment and posture: datapipelines.env (DATAPIPELINES_ENV, default local — the ORG's label, renamed from §3.19's datapipelines.deployment.name, which survives one release as a WARNing alias), datapipelines.posture (development | hardened, no default — only the env named local gets one for free) and datapipelines.demo (a FLAG: the sample-data families; refused under hardened), with the normative posture table. §3.4 gains cookie-secure (empty = derived from base-url's scheme; hardened ships true) and allow-local-only (the explicit no-OIDC acknowledgement). §6 is now the POSTURE profiles — application-dev.yml was really a laptop file and its infrastructure moved to deploy/env/laptop.env; application-development.yml and application-hardened.yml carry the posture defaults, and spring.profiles.active is derived from DATAPIPELINES_POSTURE. §7 gains four rules (env grammar + alias, posture required, posture/profile alignment, the hardened refusals) and loses the dev-profile guard, which those refusals subsume. §4 and §5 updated. Environments is the new operator page.
2026-09-05 v1.12 074 published endpoints New §3.22 Published endpoints: datapipelines.endpoints.timeout-default-seconds (30) / -min-seconds (1) / -max-seconds (300); one new §7 rule (min ≤ default ≤ max, min ≥ 1 — the four report together). Deliberately no page-rows-max: DP-Result-Page-Rows clamps to result.page-max-rows on both surfaces (R-EP4). §5 template block appended after org:.
2026-09-04 v1.11 072 calculators New §3.21 Organisation: datapipelines.org.currency.name / .symbol, fiscal-start-date (MM-DD), week-start, timezone — five keys that enter every execution Context as org_* (calculators design §0.1/§0.2, Pipeline Contract §7.2). §5 template block and one new §7 validation rule (all four org checks report together; a month name in fiscal-start-date is refused with a message naming MM-DD)
2026-09-04 v1.10 068 key-provider seam New §3.20 credential key provider: datapipelines.db.key-provider (default env, so no deployment needs a config edit), plus the env provider's optional encryption-keys rotation map and encryption-key-current. §2's encryption-key row now states that it is key version 1, forever. §5 template and §7 updated — one new validation rule: the provider name must be one this build ships, and that provider's own settings must be present and well-formed (an unknown name short-circuits the rest).
2026-09-02 v1.9 051 auth/config sweep Added §3.4 datapipelines.auth.trusted-proxies (CIDR list, default empty = header ignored; the login limiter and every auth source_ip resolve the client through it — R8/T46, deployment.md §6.2) with the §5 template line and two §7 rules (each entry must parse as a CIDR or startup is refused; enforced at the auth module's resolver construction). §3.17: open-join: true + closed provisioning now refused at startup, naming both keys (T45 — the self-join branch gates on open-join alone, so the pair would re-open the membership surface closed exists to keep admin-only); §7 gains the rule
2026-08-05 v1.0 initial draft Complete configuration reference: 6 required keys + OIDC, ~30 optional keys, full application.yml template, dev profile, startup validation
2026-08-07 v1.1 consistency campaign Authority + naming-derivation rules (§1); §3 tables and §5 YAML reconciled (unit-suffixed names win); added result.* (D9), sse.disconnect-grace-seconds (D7), idempotency, templates, audit, staging result-batch-size, login rate-limit keys; removed large-result-threshold-bytes and redis.ttl-seconds (superseded by result.*); rate limits per-user; encryption key required with no fallback; precedence section. See SPEC-REVIEW-2026-08
2026-08-12 v1.2 dev infra ports §6 dev profile now targets host ports 5434 (Postgres) / 6381 (Redis) instead of the colliding defaults 5432/6379; added the dev-host-ports note. Operator-facing keys and production defaults unchanged.
2026-08-17 v1.3 pipeline composition Added §3.16 datapipelines.pipelines.max-composition-depth (default 5) — the depth guard for PIPELINE-node composition
2026-08-26 v1.3.1 workspaces (slice 019) Added §3.17 Workspaces: datapipelines.workspaces.provisioning-mode (auto-per-user | self-serve | closed, default self-serve), open-join (default false), member-datasources-enabled (default true). (This row was written later than the section — v1.4's note promised the backfill, but the row itself never landed; recorded and landed 2026-08-29, 025 D3. Keys unchanged by the delay.)
2026-08-28 v1.4 sample data, slice A Added §3.18 Bootstrap: datapipelines.bootstrap.datasources-file and datapipelines.bootstrap.examples-file (both unset = off), the cross-key rule pairing datasources-file with datapipelines.auth.bootstrap-admin-email, and the matching §7 validation bullet and §5 template block. Backfills the §3.17 Workspaces row this log was missing (added 2026-08-26 with slice 019, keys unchanged here)
2026-08-29 v1.5 local password auth Added §3.4 datapipelines.auth.local.*: enabled, bootstrap-password-hash / bootstrap-password (first-admin seed only, forced first-login change), lockout.max-failures / lockout.duration-minutes — plus the §5 template block. The §7 "at least one OIDC provider" rule becomes "at least one authentication method"; a provider entry with an empty client-id is now ignored with a WARN instead of counting (the stock google entry binds empty when its env vars are unset, so a local-accounts-only deployment starts with zero providers). rate-limit.login-per-minute description widened to OIDC and local
2026-09-02 v1.7 048 bootstrap seeding fixes Two §7 rules added, no new keys: (a) the examples-file cross-key rule pairing it with datapipelines.workspaces.provisioning-mode = auto-per-user (021/F5 — the pair validated green while the seeder was structurally unreachable, on the shipped default); (b) bootstrap and local reserved as OIDC provider names (021/F8 — the users.provider placeholders were squatting in an operator-configurable namespace with nothing reserving them). §3.18 gains the matching cross-key paragraph
2026-09-02 v1.8 050 multi-instance round 2 §3.2: max-concurrent-executions-global renamed max-concurrent-executions-per-instance (the limit was always per JVM — the old name false at N replicas; 050/R2), old key kept as a one-release deprecated alias (alone → WARN naming the new key; both set and differing → §7 refusal); §3.2 heap note and §5 template updated to the per-instance multiplier; §7 gains the alias rule. §3.11: event-retention-days now bound and scheduled (the hourly retention job, M2's sibling)
2026-09-01 v1.6 039 deployment role Added §3.19 Deployment: datapipelines.deployment.name (label only — logged once at boot beside the authoring state; nothing branches on it, pinned by a guard test; deliberately not on /info) and datapipelines.deployment.authoring-enabled (default true; false turns the deployment into a promotion receiver whose authoring writes refuse with *.authoring.disabled; startup refuses if drafts exist while disabled). The reserved datapipelines.deployment.promotion.* sub-block is deliberately NOT declared here — it ships with promotion (Versioning §10.6's fenced sample), per this doc's shipped-keys-only rule. Matching §7 bullets (the one-sided receiver-also-authors WARN; the refuse-on-existing-drafts rule) and §5 template block
2026-09-10 v1.9 112 RBAC round 1 §3.17: datapipelines.workspaces.provisioning-mode and datapipelines.workspaces.open-join removed (D-R11 — capability moved onto the workspace membership, so who may create a workspace is a role, not a mode). Both are refused BY NAME at startup rather than ignored, and both are gone from the §5 template, application.yml and the shipped .env — a placeholder with a default is a set key to Spring's relaxed binding, so leaving them there would have refused the product's own boot. §7 loses the two rules that read them (the mode-value check and the open-join-under-closed check) and the examples-file cross-key rule (example seeding now runs once, when DemoWorkspaceSeeder creates demo); it gains one rule that names both removed keys. member-datasources-enabled survives, re-described: it gates whether a workspace ADMIN may register a datasource bound to their own workspace

This is the documentation packaged with the running version. The same files live in the repository on GitHub.