Configuration Reference
Status: v1.23 (single source of truth for every config key) Owner: datapipelines.co core Last updated: 2026-09-17
1. Purpose
Every configuration key for datapipelines.co, in one place. Environment variables, YAML paths, defaults, and descriptions. A developer or operator should never need to search across 15 specs to find a config key.
Authority rule: this document is the ONLY place a configuration key is defined. Other specs reference keys by name and link here — they never restate defaults or introduce keys of their own. A key that does not appear in this document does not exist. (Enforced by scripts/docs-audit.sh.)
Naming rules:
- YAML paths carry explicit units as suffixes:
-seconds,-minutes,-hours,-days,-ms,-bytes,-mb,-rows. - Env var names are derived mechanically:
datapipelines.prefix →DATAPIPELINES_, then the YAML path upper-snake-cased. Example:datapipelines.executor.node-query-timeout-seconds→DATAPIPELINES_EXECUTOR_NODE_QUERY_TIMEOUT_SECONDS. No abbreviations, no exceptions — the env var is always derivable from the YAML path.
2. Required Configuration
The app will not start without these. Fail-fast on missing values.
| YAML path | Env var | Description |
|---|---|---|
spring.datasource.url |
SPRING_DATASOURCE_URL |
Metadata DB JDBC URL. Example: jdbc:postgresql://host:5432/datapipelines |
spring.datasource.username |
SPRING_DATASOURCE_USERNAME |
Metadata DB username |
spring.datasource.password |
SPRING_DATASOURCE_PASSWORD |
Metadata DB password |
datapipelines.redis.host |
DATAPIPELINES_REDIS_HOST |
Redis host (results, idempotency, post-completion event log) |
datapipelines.jwt.secret |
DATAPIPELINES_JWT_SECRET |
Internal JWT signing secret. ≥ 32 bytes random, base64-encoded |
datapipelines.db.encryption-key |
DATAPIPELINES_DB_ENCRYPTION_KEY |
AES-256 data key for datasource password encryption. Exactly 32 bytes, base64-encoded. Required — there is no fallback source. It is key version 1, forever: every credential written under it carries 0x01 as its first byte (Datasources §7.1). Rotation and KMS-backed sources are §3.20's key-provider seam. |
OIDC provider configuration is also required — at least one provider must be configured in datapipelines.auth.oidc.providers (in application.yml). Each provider requires client-id, client-secret, and issuer-uri. The client-id and client-secret are typically referenced from env vars. See Auth spec §11.1 for the full format.
The specific env var names depend on which provider(s) the deployment chooses. Examples:
| Env var pattern | For |
|---|---|
GOOGLE_CLIENT_ID, GOOGLE_CLIENT_SECRET |
Google Workspace |
MICROSOFT_CLIENT_ID, MICROSOFT_CLIENT_SECRET |
Microsoft Entra ID |
OKTA_CLIENT_ID, OKTA_CLIENT_SECRET |
Okta |
KEYCLOAK_CLIENT_ID, KEYCLOAK_CLIENT_SECRET |
Keycloak |
The deployment defines these env var names in application.yml — they're not hardcoded by the app. (OIDC provider env vars are the one deliberate exception to the naming derivation rule in §1, since the deployment names them.)
3. Optional Configuration (with defaults)
3.1 Redis
| YAML path | Default | Description |
|---|---|---|
datapipelines.redis.port |
6379 |
Redis port |
datapipelines.redis.password |
(none) | Redis password |
Env vars are derived per §1 (e.g.
DATAPIPELINES_REDIS_PORT) and are omitted from the tables below for brevity.
3.2 Executor
| YAML path | Default | Description |
|---|---|---|
datapipelines.executor.max-parallel-nodes |
4 |
Max parallel nodes within one execution |
datapipelines.executor.max-concurrent-executions-per-user |
10 |
Per-user concurrent execution limit |
datapipelines.executor.max-concurrent-executions-per-instance |
100 |
Instance-wide concurrent execution limit — per instance (050/R2): N replicas admit N × this in total (Deployment §6.2) |
datapipelines.executor.max-concurrent-executions-global |
unset |
Deprecated alias for max-concurrent-executions-per-instance (one release, 050/R2). Set alone → its value runs and startup logs one WARN naming the new key; set together with the new key and differing → startup refuses. The limit was always per JVM — the old name was false at N replicas |
datapipelines.executor.node-query-timeout-seconds |
60 |
Per-node JDBC query timeout. A datasource's own query_timeout_seconds, when set, overrides this for nodes on that datasource (Datasources §5) |
datapipelines.executor.execution-timeout-seconds |
600 |
Overall execution timeout |
datapipelines.executor.node-timeout-seconds |
300 |
The per-node WALL-CLOCK deadline the executor enforces: RENDER → CONNECT → EXECUTE → STAGE → MATERIALIZE, staging included. Overridable per node by node.settings.timeout_seconds (pipeline-contract §4.11). Unlike node-query-timeout-seconds, which is the DRIVER's bound on one execute* call, this one is the executor's own and fires whatever the driver does |
datapipelines.executor.node-timeout-max-seconds |
900 |
The ceiling a node's own settings.timeout_seconds may not exceed. A pipeline declaring more is refused at SAVE with pipeline.validation.node_timeout_invalid — refused rather than clamped, so an author who asks for 4 hours is not left debugging a silent 15 minutes |
datapipelines.executor.node-query-timeout-max-seconds |
900 |
(156, #2) The ceiling a pipeline's or a node's own settings.query_timeout_seconds may not exceed (pipeline-contract §4.11/§5.3) — refused at SAVE with pipeline.validation.pipeline_query_timeout_invalid / pipeline.validation.node_query_timeout_invalid, same reasoning as node-timeout-max-seconds. Independent of that key: the two settings bound different budgets |
datapipelines.executor.node-query-timeout-seconds-by-dialect.<dialect> |
LAKE: 180; every other dialect unset |
(156, #2) The operator's per-dialect default statement timeout, consulted when neither the node, the pipeline nor the datasource declares one — the innermost fallback before the flat node-query-timeout-seconds above. LAKE ships higher because a LAKE engine re-reads cold from object storage, and less memory the engine can keep means a longer cold read (§E below). Each value is validated positive and ≤ node-query-timeout-max-seconds at boot, fail-fast. One explicit YAML line per overridden dialect, never a bare env var for a dialect not already named here — Spring's relaxed binding cannot reliably reverse-map an arbitrary ..._BY_DIALECT_<X> environment variable onto this map's dashed key (measured, ExecutorPropertiesDialectMapBindingTest); the shipped form is lake: ${DATAPIPELINES_EXECUTOR_NODE_QUERY_TIMEOUT_SECONDS_BY_DIALECT_LAKE:180} |
datapipelines.executor.cancel-grace-seconds |
5 |
After a node's deadline fires, how long the executor waits for the cancelled statement to actually return before abandoning it. Past it the node fails on schedule and the leaked statement is logged once with the execution id (event=node.statement_abandoned). Never a wait on the query itself — a driver that ignores cancel() cannot extend a node's budget, only leak a connection until its own pool reclaims it |
datapipelines.executor.source-fetch-size |
1000 |
The JDBC fetchSize set on every DQL source cursor — what makes a source node stream instead of materialising its whole result inside the driver. pgjdbc uses a server-side cursor only with autoCommit=false and fetchSize > 0, so the executor also takes a Postgres source connection out of autocommit for the read and commits unconditionally on the way out (so a multi-statement template's side effects persist exactly as they did under autocommit); the pool restores autocommit when the lease returns. MySQL is the exception the driver forces: Connector/J streams only at Integer.MIN_VALUE, which the executor passes for that dialect and this key does not affect. 0 turns streaming off and leaves every source statement exactly as pre-108 code left it — the escape hatch for the one shape that can behave differently: a DQL template whose SQL is several statements, since pgjdbc's server-side-cursor path uses the extended query protocol, which carries only one |
datapipelines.executor.progress-write-interval-seconds |
5 |
The floor between two THROTTLED live-progress writes for one execution. A staging drain reports per batch — thousands of times for a large node — and one UPDATE per batch would put a metadata-DB write on the insert path. Node boundaries are not throttled: a node starting or finishing is the event a watcher is waiting for, and delaying it would leave the screen naming the wrong node |
datapipelines.executor.progress-sample-interval-seconds |
1 |
The floor between two periodic node_progress samples of one node operation (REST API §6.4.9, 149). A sample is also taken, unthrottled, at each FIRST entry into a measured state and at the operation's end, so this key bounds only how often the cumulative counts refresh — never whether a state is seen. Every sample is a live SSE event, an execution_events row and a Redis-log entry, so raising it lowers event volume on long nodes; at the default a node emits at most one periodic sample per second plus its (at most six) first-entry samples and one terminal sample |
datapipelines.executor.heartbeat-seconds |
15 |
How often a running execution stamps pipeline_executions.heartbeat_at (V21). The crash sweep reaps a RUNNING row whose stamp is older than three of these — one missed beat is a slow tick, two a loaded box, three an instance that is gone — so this also sets how fast a dead instance's rows are reaped: ~45 s plus one 15 s sweep tick, against the sixty MINUTES datapipelines.executions.stale-timeout-minutes alone gave. That key survives as the backstop for rows a pre-V21 instance left with no stamp at all |
Three budgets, one precedence (the same table appears in pipeline-contract §4.11 and dag-executor §5.3):
| Bound | Setting | Scope | Enforced by |
|---|---|---|---|
| Execution | datapipelines.executor.execution-timeout-seconds (600) |
the whole execution | the executor |
| Node | node.settings.timeout_seconds, else datapipelines.executor.node-timeout-seconds (300) |
one node, wall clock, all five phases | the executor |
| Statement | node.settings.query_timeout_seconds (156, §5.3), else pipeline settings.query_timeout_seconds, else the datasource's query_timeout_seconds, else node-query-timeout-seconds-by-dialect.<dialect> above, else datapipelines.executor.node-query-timeout-seconds (60) |
one execute* call |
the JDBC driver |
Instances are moderate, so the statement bound must be sized for a COLD read, not a warm one (156, #2). This deployment is expected to run as several instances of moderate size, not one box with all the memory a query could want — so a LAKE engine sharing that box's memory with everything else keeps less of the object store cached, and a query that lands on an instance whose cache is cold re-reads from S3 at network speed instead of from RAM. The per-dialect operator default above exists for exactly that reason, sized to the cold read of the largest scan a deployment runs, not the warm one a developer's laptop measures. When cold reads on a given box routinely approach the default, the right levers are raising node-query-timeout-seconds-by-dialect.lake (this section) or raising the engine's own memory-limit (§3.25) so more of the data stays cached between queries — see also #141 on the shared-engine memory budget these two knobs both bound against.
Read downward. The statement bound is the driver's and drivers honour it unevenly — measured on the shipped drivers (108 §1), which is precisely why the middle row exists. The precedence is a recommendation, not a cross-key constraint: nothing refuses a configuration that inverts it, because every inversion is harmless. A node deadline above the execution's is simply never reached — the outer bound fires first and reports pipeline.execution.timeout. One below the statement timeout is stronger, not broken: the executor stops the node before the driver would have, which is the whole point of owning a bound above the driver's. What a cross-key refusal would reliably do instead is turn "I lowered execution-timeout-seconds for this deployment" into a startup crash. node-timeout-max-seconds is deliberately NOT constrained against the execution timeout — it bounds what an author may ASK for, not what a run may take.
3.3 Staging (tempdb)
| YAML path | Default | Description |
|---|---|---|
datapipelines.staging.h2.mode |
PostgreSQL |
H2 compatibility mode |
datapipelines.staging.h2.max-memory-mb |
1024 |
The memory-guard THRESHOLD each execution compares against the whole JVM's used heap — a shared, sampled measurement, not an isolated per-execution budget (see below; Staging §8.2). A pipeline's settings.tempdb.config.max_memory_mb, when present, changes the threshold for that execution (Pipeline Contract §5) — never which allocations are measured |
datapipelines.staging.h2.insert-batch-size |
1000 |
Rows per INSERT batch when staging source data |
datapipelines.staging.h2.result-batch-size |
10000 |
Rows per fetch batch when reading staged data out |
datapipelines.staging.h2.query-timeout-seconds |
60 |
H2 query timeout |
datapipelines.staging.h2.max-connections |
4 |
Cap on the operational H2 connections one execution's staging pool may hold open at once (Staging §9). A ceiling, not an allocation: the pool starts with one connection and grows on demand. 1 is legal and is the diagnosis/comparison setting — every tempdb operation then queues behind one connection |
max-connectionsis capacity, not parallelism. It bounds how many tempdb operations of ONE execution can be inside H2 at the same instant; it creates no extra eligible DAG nodes (executor.max-parallel-nodesdecides those) and promises no speedup — H2 keeps its own transaction, row and catalog locks, so two nodes touching one table still serialise inside the engine. Set it lower thanmax-parallel-nodesand nodes queue safely for a connection; set it higher and the surplus is never opened. Aggregate cost: every connection is one H2 session with its own query working memory, andmax-concurrent-executions-per-instanceexecutions can each hold up to this many — the staged tables themselves are shared by the connections of one execution and are not multiplied. Session state (SET SCHEMA,SET @var, local temporary tables, an open transaction) is per operation and is reset when a connection is returned; a node must not rely on it from another node — express data flow through ordinary tables anddepends_on, whatever the cap (Staging §9.2).
max-memory-mbis a per-execution THRESHOLD over a JVM-WIDE measurement — not an isolated per-execution budget. The guard samples used heap for the WHOLE JVM (in-processtotalMemory − freeMemory, Staging §8.2), so every concurrent execution's check reads the same shared number: one heavy execution can trip a lighter one's check, and the guard is a sampled circuit breaker — not a hard reservation, and no guarantee against OOM. Size from the demand side instead: N concurrent executions really do hold N tempdbs' tables, indexes and query working memory at once, so worst-case heap DEMAND staysmax-memory-mb×datapipelines.executor.max-concurrent-executions-per-instanceper instance (050/R2: the multiplier is per-instance; N replicas multiply it again — Deployment §6.6), and-Xmxplus the container limit must cover that with baseline headroom — an underestimate ends in the §8.4 JVM backstop, not in a cleanpipeline.staging.memory_limit_exceeded. A per-execution accounting redesign is a separately decided follow-up, not this guard's promise.Startup says the arithmetic out loud (108 §C).
ConfigValidatorlogs ONE warn line at startup whenmax-concurrent-executions-per-instance × max-memory-mb > 0.8 ×the JVM's max heap, naming all three numbers and the ratio. It is a WARNING and not a refusal, deliberately: the budget is a ceiling each execution MAY reach, not one it will, and a deployment whose pipelines stage tens of megabytes is right to run 100 slots against a 1 GB budget — refusing that would break every default installation. What an operator cannot do is notice the arithmetic unaided, since the two keys live in different sections of this document and neither used to mention the other. Above the ratio, the per-execution limit would be enforced by an OOM rather than bypipeline.staging.memory_limit_exceeded.A pipeline's
settings.tempdb.config.max_memory_mboverride is clamped to ≤ this value — it may lower the operator's ceiling for that pipeline, never raise it. Save-time validation only checks> 0, so without the clamp an author could declare an arbitrarily large budget and disable the only ceiling the executor'swithConnectionpaths have (DAG Executor §9).
3.4 Auth
| YAML path | Default | Description |
|---|---|---|
datapipelines.auth.base-url |
(none) | The deployment's exact external origin, e.g. https://dp.example.com (scheme + host [+ port], no trailing slash). OIDC redirect URIs are built absolutely from it (Auth §5.2) — never from request headers. Startup fails when unset while any OIDC provider is configured. |
datapipelines.auth.jwt.ttl-hours |
8 |
Session JWT TTL |
datapipelines.auth.allowlist.domains |
(empty) | Comma-separated allowed email domains. Binds to List<String> (comma-split; empty string = empty list = open provisioning) |
datapipelines.auth.api-keys.cache-ttl-seconds |
60 |
Cache TTL for validated API keys and user is_active checks (Auth §11.4) |
datapipelines.auth.api-keys.default-scopes |
read |
Default scope for new API keys |
datapipelines.auth.rate-limit.login-per-minute |
10 |
Per-IP login attempts per minute (OIDC and local) |
datapipelines.auth.trusted-proxies |
(empty) | CIDRs of proxies whose X-Forwarded-For names the real client — the login limiter and every audit source_ip resolve through it (Deployment §6.2). Empty (the default) = the header is ignored entirely and the direct peer is the client — a bare deployment behaves exactly as before. Each entry must parse as a CIDR (a bare IP is a host CIDR, e.g. 10.0.0.5 = 10.0.0.5/32); anything else refuses startup. Resolution is spoof-safe: an untrusted peer cannot forge a client by setting the header |
datapipelines.auth.bootstrap-admin-email |
(none) | Bootstrap admin: when a user with exactly this email is provisioned via OIDC first login, is_admin is set true (idempotent, audit-logged as auth.user.admin_granted with actor bootstrap). The ONLY way a fresh deployment gets its first admin — "first login wins" is explicitly rejected (Auth §4.4) |
datapipelines.auth.local.enabled |
false |
Optional local username/password accounts (Auth §5A) — a second sign-in method for deployments without an IdP. Disabled = the deployment behaves exactly as OIDC-only |
datapipelines.auth.local.bootstrap-password-hash |
(none) | Initial credential for the FIRST ADMIN ONLY (the bootstrap-admin-email account), as a pre-computed Argon2id hash — the preferred form (produce one with the hashPassword Gradle task, Deployment §5). Seeding is create-if-absent and idempotent, forces a first-login change, and never applies to ordinary users — passwords are not a config medium |
datapipelines.auth.local.bootstrap-password |
(none) | Plaintext alternative to the hash form, accepted for zero-setup demos: always sets must_change_password, is never logged, and startup is refused when both forms are set |
datapipelines.auth.local.lockout.max-failures |
5 |
Consecutive failed local logins that lock the account — per-account, complementing the per-IP rate-limit.login-per-minute, which cannot stop a slow spray against one account |
datapipelines.auth.local.lockout.duration-minutes |
15 |
How long a locked account refuses local login. An admin unlock or password reset clears the lock early |
datapipelines.auth.cookie-secure |
(empty) | The Secure flag on the cookies this deployment mints (dp_session, dp_oauth2_authz, dp_csrf). Empty = derived from base-url's scheme: an explicit http:// base-url drops the flag so local login works over plain HTTP, anything else keeps it. true/false pin it. The hardened posture ships true (§3.23) — a hardened deployment runs https, and a dropped flag there puts the session cookie on the wire |
datapipelines.auth.allow-local-only |
false |
The operator's explicit acknowledgement that a hardened deployment has no OIDC provider configured and authenticates with local password accounts only. Inert under development. Under hardened, startup is refused unless either an OIDC provider is fully configured or this is true, and setting it logs the acknowledgement (§3.23, §7) |
3.5 Results
Every completed execution's caller result is stored in Redis and read through the result cursor (REST API §7).
| YAML path | Default | Description |
|---|---|---|
datapipelines.result.ttl-default-seconds |
300 |
Result TTL when the client sends no DP-Result-TTL-Seconds header |
datapipelines.result.ttl-min-seconds |
60 |
Lower clamp for client-requested TTL |
datapipelines.result.ttl-max-seconds |
3600 |
Upper clamp for client-requested TTL |
datapipelines.result.max-size-bytes |
104857600 |
Hard cap on a caller result (100 MB). Exceeding it fails the execution with result.too_large |
datapipelines.result.page-size-rows |
1000 |
Rows in the inline first page of data_ready, and the default limit for cursor reads |
datapipelines.result.page-max-rows |
100000 |
Upper bound on the cursor limit parameter |
3.6 SSE
| YAML path | Default | Description |
|---|---|---|
datapipelines.sse.heartbeat-interval-seconds |
15 |
SSE heartbeat comment interval |
datapipelines.sse.disconnect-grace-seconds |
30 |
Grace period after client disconnect before the in-flight execution is cancelled (REST API §6.8) |
datapipelines.sse.max-streams-per-user |
50 |
Concurrent SSE streams per user |
3.7 Rate Limiting
Limits are per user (an API key inherits its owner's budget — minting more keys does not raise the limit).
| YAML path | Default | Description |
|---|---|---|
datapipelines.rate-limit.requests-per-second |
100 |
Per-user requests per second |
datapipelines.rate-limit.requests-per-minute |
1000 |
Per-user requests per minute |
There is deliberately no fail-open key. When the limiter's Redis is unreachable the limiter refuses the request (429 rate_limit.unavailable, REST API §12.3); a per-deployment toggle would only ever be reached for at exactly the moment the refusal is the correct answer.
3.8 Idempotency
| YAML path | Default | Description |
|---|---|---|
datapipelines.idempotency.ttl-seconds |
86400 |
Retention of Idempotency-Key records in Redis |
3.9 Templates
| YAML path | Default | Description |
|---|---|---|
datapipelines.templates.cache-size |
500 |
Parsed-template cache entries |
datapipelines.templates.render-timeout-ms |
5000 |
Hard limit on a single template render |
datapipelines.templates.max-body-chars |
262144 |
Max template body length accepted at save (256K chars); over-cap bodies are rejected with template.validation.syntax_error before parsing (Templates §4.2) — bounds parse cost and heap against an adversarial body |
3.10 UI
| YAML path | Default | Description |
|---|---|---|
datapipelines.ui.theme |
dark |
Design system theme name. Validated at startup against the vendored themes in modules/web/src/main/resources/static/vendor/design-system/ |
3.11 Execution History
| YAML path | Default | Description |
|---|---|---|
datapipelines.executions.error-detail |
full |
How much of a failure's detail travels to every surface that carries an error (the SSE node_failed/pipeline_failed error objects, pipeline_executions.error_json, the editor's failure panel, MCP). full includes the exception chain with stack frames and the rendered SQL in :name form; structured omits both. Choose structured for a deployment whose pipeline authors are not trusted to see driver internals (DAG executor §8.4) |
datapipelines.executions.event-retention-days |
7 |
How long to keep execution_events rows (Postgres, the durable record) past their execution's completion — enforced by the hourly retention job (050/T60, safe for N replicas: one idempotent DELETE; pipeline_executions rows are never touched). The post-completion Redis event log lives for 1 hour, not configurable |
datapipelines.executions.stale-timeout-minutes |
60 |
Mark RUNNING executions older than this as ABORTED (crash sweep) |
3.12 Audit
| YAML path | Default | Description |
|---|---|---|
datapipelines.audit.retention-days |
365 |
Retention of audit-log rows |
3.13 Server
| YAML path | Env var | Default | Description |
|---|---|---|---|
server.port |
SERVER_PORT |
8080 |
HTTP port |
spring.datasource.hikari.maximum-pool-size |
SPRING_DATASOURCE_HIKARI_MAXIMUM_POOL_SIZE |
10 |
Metadata DB connection pool size |
3.14 Framework wiring keys
These framework key paths appear in application.yml as internal wiring. They are listed here so the §1 authority sentence stays literally true. With the single exception of management.server.port (below), operators do not set them and deployments must not override them:
| Path | Why it exists |
|---|---|
spring.application.name |
Log/metric attribution (datapipelines) |
spring.data.redis.host / .port / .password |
Bridge binding the canonical DATAPIPELINES_REDIS_* env vars onto Spring Boot's Redis autoconfiguration (see the binding note under §5) |
spring.flyway.enabled / .locations / .baseline-on-migrate |
Migration wiring — Flyway always runs on startup (Deployment §8.2) |
management.endpoints.web.exposure.include |
"health" — served on the management port only; prometheus joins it when the metrics registry lands (Observability §6.4). Must not be set to "" or exclude: "*": Spring's @ConditionalOnAvailableEndpoint keys bean creation off exposure, so an empty include falls back to Boot's default set (re-exposing health on whatever port serves actuator), and excluding everything deletes the HealthEndpoint bean the root /health controller injects — context startup fails outright. |
management.health.diskspace.enabled |
false — the health contract has no disk component (REST API §11.1) |
Operator-tunable exception:
| YAML path | Env var | Default | Description |
|---|---|---|---|
management.server.port |
MANAGEMENT_SERVER_PORT |
9090 |
Separate management port serving /actuator/health (and later /actuator/prometheus) — never publish it; nothing actuator is routable on the application port. |
management.server.address |
MANAGEMENT_SERVER_ADDRESS |
127.0.0.1 |
Loopback by default — the management surface is silently-public on no deployment shape. Kubernetes scraping (scraper → pod IP) requires explicitly setting 0.0.0.0, paired with a NetworkPolicy restricting the port to the monitoring namespace (Deployment §9). Explicit opt-open, never default-open. |
3.15 Observability
| YAML path | Env var | Default | Description |
|---|---|---|---|
datapipelines.observability.tracing.enabled |
DATAPIPELINES_OBSERVABILITY_TRACING_ENABLED |
false |
Enable OpenTelemetry tracing |
datapipelines.observability.tracing.endpoint |
OTEL_EXPORTER_OTLP_ENDPOINT |
(none) | OTLP collector endpoint (standard OTel env var, exception to §1 derivation) |
datapipelines.observability.logging.format |
DATAPIPELINES_OBSERVABILITY_LOGGING_FORMAT |
json (prod), console (dev) |
Log output format |
3.16 Pipelines
| YAML path | Default | Description |
|---|---|---|
datapipelines.pipelines.max-composition-depth |
5 |
Deepest admitted chain of pipelines executing pipelines (a PIPELINE node spawning a child execution). Enforced at save time and again at runtime; must be ≥ 1 |
3.17 Workspaces
Workspaces are created by super admins (auth.md §4.2/§5.6, §12). The one workspace the product ships is demo, and a user with no membership becomes a viewer of it on first login.
| YAML path | Default | Description |
|---|---|---|
datapipelines.workspaces.member-datasources-enabled |
true |
May a workspace ADMIN register a datasource bound to their own workspace? false makes datasource registration a super-admin-only act instance-wide. Visibility is still the grant either way (auth.md §11A) |
Removed in RBAC round 1 (provisioning-mode, open-join). Capability moved onto the workspace membership, and with it went the modes that decided who could create a workspace: auto-per-user (a personal workspace per login), self-serve (anyone creates) and closed (admin only), plus open-join (anyone self-joins). Both keys are refused BY NAME at startup (§7) rather than ignored — a deployment that still says auto-per-user is a deployment expecting a personal workspace per user, and silently giving it something else is how an operator finds out from a user. Delete the key; there is no replacement to set, because the behaviour is no longer a knob.
3.18 Bootstrap
Config-declared content applied at startup (design 2026-08-16-sample-data §6/§6.1). Both keys name files already on the container's filesystem — the app never fetches an artifact at runtime; downloading and verifying artifacts is a deployment step (D5) — and for both, unset (or empty) means the feature is off. There is no separate enable flag to disagree with the path.
Both values are comma-separated LISTS of paths (one entry per sample-data family; the demo profiles compose the list from the active families). A single value with no comma is the one-file shape older deployments ship — the list semantics are backward compatible by construction. Empty entries (a leading comma when a family is off, or a whitespace-only entry) are dropped, so a list built by shell conditional expansion never turns the feature on by accident. The files are processed in declared order.
| YAML path | Default | Description |
|---|---|---|
datapipelines.bootstrap.datasources-file |
(none) | Path(s) to YAML file(s) of datasource definitions registered create-if-absent at startup (Datasources §8A). Set-but-unreadable, unparseable, or carrying an entry that fails §9 validation = fail-fast startup. Registration is per entry, so overlapping files are idempotent |
datapipelines.bootstrap.examples-file |
(none) | Path(s) to JSON file(s) of example templates and pipelines seeded into each personal workspace at auto-per-user provisioning, through the same import services as POST /pipelines/import and POST /templates/import. Shape: {"templates": [...], "pipelines": [...]}, each array element exactly what its import endpoint takes. Seeding runs ALL templates (file by file) before ANY pipeline, so a pipeline in one file may reference a template seeded from another. Set-but-unreadable or unparseable = fail-fast startup; an entry that fails import validation fails the provisioning login |
Cross-key rule: datasources-file set while datapipelines.auth.bootstrap-admin-email is unset is a startup refusal naming both keys. Bootstrap-registered datasources are created_by that user, and registration runs before anyone has logged in, so the row is pre-provisioned from that address (Auth §4.4). examples-file carries no such rule: seeding runs at first login, under the identity of the user logging in.
Cross-key rule (examples-file): examples-file set while datapipelines.workspaces.provisioning-mode is anything but auto-per-user — including its shipped default self-serve — is a startup refusal naming both keys. Seeding runs only when first login provisions a personal workspace; under any other mode the file is read and validated at startup and then never seeded, silently. Set the mode, or unset the file.
3.19 Deployment
The deployment-role settings (Versioning §5.5) — grouped one-per-concern like auth, executor, staging.
Renamed in 075.
datapipelines.deployment.name— the deployment LABEL — is nowdatapipelines.env(DATAPIPELINES_ENV), documented in §3.23 beside the posture it is deliberately separate from. The old key is honoured for one release: set alone it still names the deployment and startup logsevent=config.deployment_name_deprecated; set together withdatapipelines.envand disagreeing, startup is refused naming both. See Environments.
| YAML path | Env var | Default | Description |
|---|---|---|---|
datapipelines.deployment.authoring-enabled |
DATAPIPELINES_DEPLOYMENT_AUTHORING_ENABLED |
true (false under the hardened posture — §3.23) |
The authoring capability. When false, every pipeline/template authoring write (create, update/draft, release, discard, delete) is refused with pipeline.authoring.disabled / template.authoring.disabled — fail-closed, naming the reason. Reads, execution and import are unaffected (promotion imports RELEASED versions; that is the one writer a receiver must accept). Startup REFUSES if drafts still exist while this is false (§7) |
Promotion (the deployment-to-deployment channel)
Promotion is one deployment writing RELEASED content into exactly one configured higher environment (Versioning §10). The credential is a server key, not a principal (Versioning §10.6): no users row for the credential and no scope-matrix entry. A deployment may hold the receiver half, the sender half, both, or neither.
The receiver half is now a KEY, and the config value below is DEPRECATED (091). Mint a
server-kind API key on the API screen (Auth §7.7) and give it to the sending deployment as itstarget.server-key. A stored key is everything a config value cannot be: expiring, revocable, listed beside every other credential, rotatable without a restart, and stamped with a last-used time.datapipelines.deployment.promotion.server-keyis still accepted for one release so an upgrade does not break an existing pair — and it WARNs at boot when set (§7). Both credentials open the same routes; the receiver tries the configured value first, then the key store.
| YAML path | Env var | Default | Description |
|---|---|---|---|
datapipelines.deployment.promotion.server-key |
DATAPIPELINES_DEPLOYMENT_PROMOTION_SERVER_KEY |
(empty) | Receiver half — DEPRECATED (091), removed next release. The pre-shared secret an inbound promotion may present in the DP-Promotion-Key header, superseded by a server-kind API key. Empty is not "promotion is open" — with no configured value AND no live server key, every push is refused; fail closed. Compared in constant time, before the key store is consulted; never logged. Consulted on /api/v1/promotion/** and nowhere else, and it grants no read access outside that pair. Set ⇒ one WARN at boot |
datapipelines.deployment.promotion.target.base-url |
DATAPIPELINES_DEPLOYMENT_PROMOTION_TARGET_URL |
(empty) | Sender half. The single higher environment's base URL (e.g. https://uat.example.com). Empty = this deployment promotes nowhere and the promotion screen says so. Exactly one target — multi-target promotion is deliberately not a feature |
datapipelines.deployment.promotion.target.server-key |
DATAPIPELINES_DEPLOYMENT_PROMOTION_TARGET_KEY |
(empty) | Sender half. The SAME secret the target holds under its own server-key. A base-url set without this key refuses startup (§7): the pair is meaningless apart |
On the SENDER, the value is whatever the receiver issued: the dpk_<id>.<secret> plaintext of the receiver's server key, shown exactly once at minting. Rotation is then a receiver-side act with no restart anywhere: mint a second server key, set it on the sender, revoke the first. For the deprecated config value, generate it the way every other secret here is — openssl rand -base64 32 — and set it identically on both sides; rotation means restarting both. Either way no user account is involved, so offboarding a human can never break production promotion — a server key's owner is the admin who minted it, and the promotion acts as the system service account regardless (Auth §4.5).
Both keys are bearer secrets. They are never logged, never in an error message, never in an audit details map, and never in a toString(). A receiver records only a truncated SHA-256 fingerprint of the key a push presented, so two keys are distinguishable across a rotation without either being carried.
3.20 Credential key provider
Where the AES data keys for datasources.credential_encrypted come from (Datasources §7.1). The seam exists so a customer's AWS/GCP/Azure/Vault key store is an implementation of a contract, not a change to the crypto — docs/key-providers.md is the guide an implementer works from. Every stored credential carries the key VERSION it was sealed under as its first byte, which is what makes rotation lazy-safe.
| YAML path | Env var | Default | Description |
|---|---|---|---|
datapipelines.db.key-provider |
DATAPIPELINES_DB_KEY_PROVIDER |
env |
Which provider supplies data keys. env is the only one this build ships and is today's behaviour, so a deployment that predates the seam needs no config edit. An unknown name refuses startup, listing what is shipped (§7) |
datapipelines.db.encryption-keys |
(per entry, e.g. DATAPIPELINES_DB_ENCRYPTION_KEYS_2) |
(unset) | env only. Additional keys as version: base64, for a rotation — e.g. 2: ${DATAPIPELINES_DB_ENCRYPTION_KEY_V2}. Each value is exactly 32 bytes base64-encoded; each version is an integer 1..255. Version 1 may not appear here: that version is datapipelines.db.encryption-key and has exactly one spelling |
datapipelines.db.encryption-key-current |
DATAPIPELINES_DB_ENCRYPTION_KEY_CURRENT |
the highest configured version | env only. Which configured version NEW encryptions use. Existing rows keep decrypting under the version they carry, so flipping this rotates lazily rather than all at once (Datasources §7.3) |
Every configured key is a bearer secret. None of them is logged, put in an error message, put in an audit details map, or rendered by a toString() — the §7 violations name the property and the defect, never the value.
Rotating. Generate a key (openssl rand -base64 32), add it as version N+1, set encryption-key-current: N+1, restart. New writes carry N+1; rows written under earlier versions keep decrypting, and are rewritten under the current key the next time their password is saved. Keep the old keys configured until no row still carries their version — Datasources §7.3 has the one SQL query that answers that.
3.21 Organisation
The organisation's own facts — its currency, when its fiscal year starts, which day its week starts on, which timezone "today" means. They are configuration, not pipeline data: identical for every pipeline in the deployment, changed about once a decade, and a value that differs between deployments must be visible in the deployment's yml rather than copied into every body.
Every key here enters every execution's Context as an org_* key — the yml path minus the datapipelines.org. prefix, with dots and dashes as _ (Pipeline Contract §7.2). A node binds :org_currency_symbol; a CALCULATOR node references $org_fiscal_start_date. None of them is a secret, and a pipeline may override any of them by declaring a parameter of the same name.
| YAML path | Default | Description |
|---|---|---|
datapipelines.org.currency.name |
Dollar |
The currency amounts are reported in. Context key org_currency_name |
datapipelines.org.currency.symbol |
$ |
Rendered beside amounts. Context key org_currency_symbol |
datapipelines.org.fiscal-start-date |
01-01 |
MM-DD — the day the fiscal year starts; 01-01 is the calendar year. Month names are refused (§7). 02-29 is accepted and resolves to 02-28 in a non-leap year. Context key org_fiscal_start_date |
datapipelines.org.week-start |
monday |
monday | sunday — which day a week starts on for week bucketing. Context key org_week_start |
datapipelines.org.timezone |
UTC |
An IANA zone id. current_date is evaluated in it, so a deployment in Sydney on a UTC host still sees its own today. Context key org_timezone |
Env vars follow §1's derivation rule: DATAPIPELINES_ORG_CURRENCY_NAME, DATAPIPELINES_ORG_CURRENCY_SYMBOL, DATAPIPELINES_ORG_FISCAL_START_DATE, DATAPIPELINES_ORG_WEEK_START, DATAPIPELINES_ORG_TIMEZONE.
Restart to change. There is no runtime override and no per-user setting: an org value that could change mid-run would make two nodes of one execution disagree about what year it is. A bad value stops the server (§7) rather than silently defaulting — a wrong fiscal start is a wrong number in every report the deployment produces.
3.22 Published endpoints
The timeout bounds for a released pipeline published as a GET endpoint under /api/x (round 074). timeout-default-seconds is what a publish that names no timeout is stored with; the min/max pair clamps every stored published_endpoints.timeout_seconds at write time, so retuning the bounds later never makes an existing row unreadable — it only changes what the next publish may ask for.
| YAML path | Default | Description |
|---|---|---|
datapipelines.endpoints.timeout-default-seconds |
30 |
Timeout a publish that omits one is stored with |
datapipelines.endpoints.timeout-min-seconds |
1 |
Lower clamp for a published endpoint's timeout |
datapipelines.endpoints.timeout-max-seconds |
300 |
Upper clamp, and the longest a serve may block before it answers 202 and leaves the execution running |
There is deliberately no page-rows-max key here. The DP-Result-Page-Rows request header is one contract across a published endpoint and POST /pipelines/{id}/execute, and both clamp it to datapipelines.result.page-max-rows (§3.5). A second key that had to equal the first would be a second authority for one bound, and an operator who retuned one and not the other would get two different clamps on one documented header.
3.23 Environment and posture
The two variables an organisation sets (round 075). Environments is the page written for the DevOps engineer who deploys this; this section is the key reference.
The environment's NAME belongs to the org; the POSTURE belongs to the product. Organisations have dev, qa, uat, perf, sandbox-eu, prod — any names, any count — and the product must never branch on one: a single-server user honestly writes prod as their label, and the first if (env == "prod") anywhere locks them out of authoring on the only server they have. What the product branches on is a closed, two-valued posture with the documented semantics below, and the org maps each of its environments onto one.
| YAML path | Env var | Default | Description |
|---|---|---|---|
datapipelines.env |
DATAPIPELINES_ENV |
local |
The org's label for this deployment. Grammar: [a-z0-9][a-z0-9_-]{0,31} — lowercase letters, digits, _ and -, at most 32 characters. It is the boot line's identity and the source_env a promotion receiver records; nothing branches on it (pinned by a guard test) and it is deliberately not on /info, which is permitAll |
datapipelines.posture |
DATAPIPELINES_POSTURE |
(none) | The product's stance: development or hardened. There is no shipped default. §7 supplies development only when datapipelines.env is local; any other environment with no posture refuses to start, naming both variables — explicit beats guessed |
datapipelines.demo |
DATAPIPELINES_DEMO |
(empty) | The sample-data families to load out of the box, comma-separated: nyc, trade. Empty = off. Demo is a flag, not an environment: it is out-of-the-box evaluation inside a development environment, and a non-empty value is REFUSED under hardened |
The posture table (normative)
| Rule | development |
hardened |
|---|---|---|
datapipelines.deployment.authoring-enabled default |
true |
false — a promotion receiver; explicitly settable either way |
datapipelines.demo |
allowed | refused at boot |
local bootstrap password (datapipelines.auth.local.bootstrap-password[-hash]) |
allowed | refused at boot. Local accounts themselves stay allowed — a seeded credential is what has no place in a hardened deployment |
| loopback metadata DB / Redis | allowed | refused at boot |
datapipelines.auth.cookie-secure default |
derived from base-url's scheme |
true |
| OIDC | optional | required unless DATAPIPELINES_AUTH_ALLOW_LOCAL_ONLY=true (an explicit acknowledgement, logged) |
| boot line | event=config.posture env=<env> posture=development authoring=on demo=nyc,trade |
same shape |
The posture IS the Spring profile
spring.profiles.active is derived from DATAPIPELINES_POSTURE in application.yml, and application-development.yml / application-hardened.yml carry the posture's non-secret defaults (the first two rows above). One variable therefore gives every loader — Compose, a bare java -jar, systemd, Kubernetes, ECS, Nomad — the right defaults, and nothing the product needs lives only in a launcher. Setting SPRING_PROFILES_ACTIVE yourself is unnecessary; a value that disagrees with the posture refuses startup (§7) rather than silently loading one posture's defaults while every posture RULE judges the other. 075 renamed the dev profile to development; a deployment still asking for dev is refused, not ignored.
Nothing lives only in a compose file
The contract is environment variables, in two files: deploy/env/defaults.env (tracked — every non-secret variable this build binds, with the value it ships) and deploy/secrets.env (git-ignored — every credential, plus every override that belongs to this deployment). deploy/secrets.env.example is the template for the second and names every variable you may set. scripts/compose-env-audit.sh — which runs on every ./gradlew build — fails when those files, deploy/compose.yml and application.yml disagree about which variables exist or what they default to, and when a variable is declared in both tracked files or in neither. Settings are tracked in the repo; secrets never are.
3.24 Lake datasource engine limits (dp-lake)
A LAKE datasource runs its queries on the app's own box — DuckDB is embedded in the app process, so its memory, threads and spill disk are operator knobs, not someone else's infrastructure. These are datasource properties (properties.dialect.*, set per datasource through the REST API or a bootstrap datasources file — Datasources §12.1), not datapipelines.* keys: they do not belong in application.yml or an env file, they are validated per key at save time (an unknown key or a bad value is refused), and a change takes effect at the datasource's next pool build, with no app restart. The existing execution bounds apply unchanged beside them: the result row cap and node-query-timeout-seconds.
Scope: one engine per datasource pool, shared by its connections (152). Each LAKE datasource pool build opens ONE embedded DuckDB instance that every pooled connection joins (Datasources §5.2); the three properties below are applied once to that instance and are ONE budget for the pool — properties.hikari.maximumPoolSize does not multiply them, and concurrent queries on the datasource share them. Size the box for memory_limit × the number of LAKE datasources on the instance (each has its own engine), with headroom for a retiring pool's engine while it drains after a datasource save. The engine's file cache lives with that instance, so a warm lake stays warm across HikariCP's routine connection replacement; a datasource save or a lake-table registration rebuilds the pool and starts a fresh engine, cold.
| Datasource property | Default | Semantics |
|---|---|---|
properties.dialect.memory_limit |
datapipelines.duckdb.memory-limit if set, else 25 % of the container's memory, clamped to 64 MiB – 4 GiB |
The engine's memory budget, e.g. 512MB or 2GB (B, KB, MB, GB, TB — a % is refused). Precedence: this per-datasource property, if set, always wins; otherwise the deployment's operator default (§3.25) applies verbatim; otherwise the default reads the container limit the container-aware JVM reports (OperatingSystemMXBean.totalMemorySize, the same figure DuckDB's own default reads), takes a quarter of it because the engine shares the box with the JVM, and hard-caps at 4 GiB so a large host does not hand one lake query enough to evict the app itself. An explicit value — this property or the operator default — is not capped |
properties.dialect.threads |
(engine default — no statement emitted) | The engine's worker threads — a positive integer, e.g. 4. Unset means DuckDB chooses (its own default tracks the box's cores) |
properties.dialect.temp_directory |
Anonymous in-memory URL: unique datapipelines-lake-<UUID> directory under absolute JVM java.io.tmpdir (normally /tmp); named/file URL: retain engine setting |
Where oversized operators spill, e.g. /data/spill. Must be an absolute path under the app's data volume — inside the container the path means nothing unless the volume backs it — with no quotes, backslashes, whitespace or control characters (it is interpolated into a SET statement, and a value that would need escaping is refused, matching the lake-table location grammar) |
For the supported anonymous in-memory URLs (jdbc:duckdb: and jdbc:duckdb::memory:), the default spill directory belongs to one engine initialization: pooled connections share it, while independent in-memory pool generations receive different paths. DuckDB creates it lazily when spilling and removes it on normal instance shutdown. The parent directory must exist, be writable by the application user, and have enough disk space for concurrent queries; the Helm chart provides disk-backed /tmp. A bare JVM can select the parent with -Djava.io.tmpdir=/writable/path. An explicit override is used exactly as supplied: dedicate that directory to the engine, keep unrelated files out, and avoid sharing it between independent engines or overlapping generations. Named in-memory and file-backed URLs can join an existing JVM-wide engine; when no override is supplied, its current spill setting is left untouched. Configure a writable path explicitly for those URLs, consistently across every pool sharing that engine. A crash or a driver that refuses shutdown can leave spill files behind; reclaim them only after the owning process has stopped.
threads is deliberately left unset, and the measurement is why — though not the way the
hypothesis expected (108 §C). In-process DuckDB shares the JVM's CPUs with the executor, so the
obvious guard is to cap the engine's worker threads; max(2, cores/2) was the proposal. Measured on
a 10-core box as the median of 5 PAIRED runs (each pair times the scan alone and then immediately
under load, so a drifting box moves both numbers):
threads |
2 staging drains | 8 staging drains |
|---|---|---|
| unset (all 10) | 1.07× slower | 1.11× slower |
SET threads = 5 |
0.92× | 1.13× |
A 0.92× is a scan that ran FASTER under load than alone, which is not a result — it is the noise
floor. On a box shared with other work the contention effect is smaller than the measurement's own
variance, and an earlier single-run shape produced swings from 1.13× to 1.82× for the same cell. So
the honest reading is: no evidence for a thread cap, and none against one either — the effect
this knob would manage is not resolvable here. The default therefore stays unset (DuckDB's own
scheduler handles oversubscription), an operator who needs the engine bounded sets threads
explicitly, and anyone who wants to settle it should re-run
scripts/measure/03-pressure.sh on a dedicated box.
What the runs DO agree on: a lake read on a busy box loses on the order of 10 % or more, and no
threads value observed changed that.
Two statements are emitted on every lake connection regardless of these keys: SET memory_limit = '<explicit-or-default>' (there is always a budget) and SET preserve_insertion_order = false — insertion order costs memory and temp-file discipline the engine would otherwise spend on a guarantee a read-only lake never asks for. It is not a knob: a lake is a read connector, and making the trade configurable would only let an operator buy back a guarantee no query path uses.
3.25 DuckDB extension directory (dp-lake)
A LAKE datasource whose data is on S3 needs DuckDB's httpfs and aws extensions, and the Iceberg catalog kinds additionally need iceberg (which itself requires avro). Where those binaries come from is this key. It is an operator-level key — one value for the whole deployment, restart-to-change — deliberately NOT a per-datasource properties.dialect.* entry: which extension files the engine may load is a deployment security posture, not row data.
| YAML path | Env var | Default | Description |
|---|---|---|---|
datapipelines.duckdb.extension-directory |
DATAPIPELINES_DUCKDB_EXTENSION_DIRECTORY |
(empty) | A directory holding pre-populated DuckDB extensions in the engine's own layout (<dir>/v<core-version>/<platform>/<name>.duckdb_extension). Must be an absolute path with no quotes, backslashes, whitespace or control characters (it is interpolated into a SET statement; a value that would need escaping is refused, matching the temp_directory grammar) |
datapipelines.duckdb.memory-limit |
DATAPIPELINES_DUCKDB_MEMORY_LIMIT |
(empty) | (153, #136) The deployment-wide default memory_limit every LAKE engine build gets when a datasource declares no properties.dialect.memory_limit of its own — likewise operator-level, restart-to-change, deliberately not a datasource property: it is a capacity decision about the box, not row data. Empty keeps §3.24's derived 25 %-of-container default exactly. A non-empty value is a DuckDB size string, the same grammar as properties.dialect.memory_limit (512MB, 2GB, …; a % is refused), used verbatim, with no cap — the operator's own number. Validated at startup: an invalid value fails fast at context start naming this key and the grammar, rather than silently falling back to the derived default. Precedence is §3.24's per-datasource properties.dialect.memory_limit first, then this key, then the derived default |
Operator instruction: set DATAPIPELINES_DUCKDB_MEMORY_LIMIT=8GB in deploy/secrets.env (or the deployment's own env file) and restart the app — the value is read once at wiring, not per pool build, so every LAKE datasource without its own properties.dialect.memory_limit picks it up on its next pool build after the restart (a fresh cold engine; the first query after restart is cold, the next warm). Confirm it took effect over a LAKE connection with SELECT current_setting('memory_limit') (reports the exact configured value, e.g. 8.0GB) or SELECT * FROM duckdb_memory() (the engine's own memory-accounting view, docs/staging.md §"Memory accounting" — its buffer manager budget reflects the same limit).
Set (the shipped image): every lake connection runs SET extension_directory = '<dir>' followed by bare LOADs — LOAD httpfs; LOAD aws for catalog.kind: s3, plus LOAD avro; LOAD iceberg for the Iceberg kinds — and never an INSTALL. In DuckDB v1.5.5 LOAD strictly loads already-present files (no download code path runs at all — measured in the 089 §7.3 spike against this exact base image with --network none), so a lake pool connects with zero egress. The published image bundles the four extensions for DuckDB core v1.5.5 under /opt/duckdb/extensions/v1.5.5/<platform>/, the <platform> following the image build's architecture (linux_amd64 or linux_arm64 — the Dockerfile maps BuildKit's TARGETARCH; hard-coding one architecture leaves the other LOAD-only against an empty directory, a pool-init failure at connect — found by the 089 live gate), and exports this variable from the Dockerfile; deploy/compose.yml defaults to the same path. The bundle adds ~108 MB uncompressed to the image (~39 MB downloaded at build). A hardened dp-lake deployment with the bundled directory needs no egress to extensions.duckdb.org at all.
Unset (a bare java -jar, a developer machine): the adapter keeps its explicit INSTALL+LOAD pairs, unchanged. The first INSTALL of each extension downloads it from extensions.duckdb.org — once per extension per container filesystem (the engine's default ~/.duckdb/extensions cache), so an ephemeral container re-downloads on every fresh start.
Version coupling: the bundled binaries are valid for exactly one DuckDB core version. The pinned duckdb_jdbc is 1.5.5.1, whose bundled core reports v1.5.5 — the directory's v1.5.5 component is the CORE version, not the JDBC patch version. A duckdb_jdbc upgrade must re-download all four extensions and rename the directory accordingly (the Dockerfile pins both in one ARG).
3.26 Datasource pools
How long a retired connection pool may keep connections out before it is closed regardless (Datasources §5.2, round 094). A datasource that is edited or deleted has its pool taken out of the live map at once and soft-evicted; the pool itself is closed when the statements already running on it finish — or at this ceiling, whichever comes first, so a genuinely hung statement cannot pin a deleted datasource's pool forever. Each hard close logs one WARN naming the datasource and the connections it took down, and increments datapipelines.datasource.pool.hard_closed (Observability §4.1).
| YAML path | Default | Description |
|---|---|---|
datapipelines.datasources.retire-ceiling-seconds |
derived |
Seconds a retired pool may hold connections before it is hard-closed. derived means datapipelines.executor.node-query-timeout-seconds + 30 (so 90 with the shipped defaults) — the longest a well-behaved node statement can run, plus slack for it to return its connection. Set it explicitly only when this deployment's datasources carry their own longer query_timeout_seconds |
Why the default is derived rather than a number. The ceiling and the node query timeout are the same fact seen twice: a deployment that raises node-query-timeout-seconds to 600 and leaves a literal 90 here would start hard-closing pools out from under statements that are still legitimately running, and nothing would tell it. Deriving makes the two move together; an explicit value opts out.
3.27 Mail
Outbound mail (137, Auth §5A.8) — the SMTP server this deployment sends its two notices through: the welcome / password-reset mail to a local user (the login URL and the one-time password) and the "New user" notice to sys-ops (no password). Any SMTP server: traditional host, port, username, password. Postmark's SMTP endpoint is one such server, and its message-stream header is the only vendor-shaped knob.
Mail is enabled exactly when it is configured — there is no enabled key. Mail is on iff host AND from are both set. A flag that could disagree with the fields it summarises is the YAML-boolean trap the house has already logged; "configured" is the only honest definition of "on" for an outbound adapter. Every half-configured shape refuses startup naming the field (§7).
| YAML path | Default | Description |
|---|---|---|
datapipelines.mail.host |
(none) | SMTP host. Empty = mail off |
datapipelines.mail.port |
587 |
SMTP port — the submission port (RFC 6409) every STARTTLS-capable server listens on |
datapipelines.mail.username |
(none) | SMTP username. Empty = the relay takes no credentials. Postmark: the server token |
datapipelines.mail.password |
(none) | SMTP password. Never logged, never in the properties' toString. Postmark: the server token |
datapipelines.mail.starttls |
true |
STARTTLS on the submission connection — required, not opportunistic: true refuses a server that does not upgrade rather than falling back to plaintext. The hardened posture refuses false (§3.23, §7) |
datapipelines.mail.from |
(none) | The sender — a bare address or Display Name <address>. Half of "enabled" |
datapipelines.mail.reply-to |
(none) | Reply-To on every message — the human a user writes back to. Empty = from |
datapipelines.mail.ops-to |
(none) | The sys-ops sink for the "New user" notice — one address or a comma list. Empty = no new-user notices, even with mail on. Set without host = refused (a sink nobody can reach is a misconfiguration, not a preference) |
datapipelines.mail.message-stream |
(none) | Postmark's X-PM-Message-Stream header. Set = added to every message; empty = nothing is added |
Cross-key rule: mail on requires datapipelines.auth.base-url (§3.4) — the welcome mail carries the login URL, which is built from it and never from a request's origin (Auth §5.2).
What the admin screen shows follows from the derivation: with mail on, the create-user and reset-password actions no longer display the one-time password — it went to the user, and the screen says so with the send's outcome; with mail off they show it as before (UI §4.12).
4. Precedence
Resolution order for any key, highest first:
- Environment variable (via the
${ENV:default}placeholder). - Active profile YAML — the POSTURE file,
application-development.ymlorapplication-hardened.yml(§3.23). - Base
application.ymldefault.
Under a loader that assembles env files (docker compose --env-file, set -a; . file), the files are read in order and the LAST one wins; app.sh passes deploy/env/defaults.env, then deploy/secrets.env — secrets last, so a value an operator sets there always wins. Two files, that order, under every loader.
Two documented per-entity overrides sit above global config at runtime (they are data, not config):
- Pipeline
settings.tempdb.config.max_memory_mboverridesdatapipelines.staging.h2.max-memory-mbfor that pipeline. - Datasource
query_timeout_secondsoverridesdatapipelines.executor.node-query-timeout-secondsfor nodes on that datasource.
5. Full application.yml Template
Complete — a deployment assembled from this block gets the framework wiring (§3.14) too. Omitting the management: block in particular re-serves the actuator on the application port, which is exactly the exposure the management port exists to prevent.
spring:
application:
name: datapipelines
profiles:
# §3.23 — the POSTURE selects the profile that carries its defaults. Empty means no
# profile, and the base defaults below ARE the `development` column.
active: ${DATAPIPELINES_POSTURE:}
datasource:
url: ${SPRING_DATASOURCE_URL}
username: ${SPRING_DATASOURCE_USERNAME}
password: ${SPRING_DATASOURCE_PASSWORD}
hikari:
maximum-pool-size: ${SPRING_DATASOURCE_HIKARI_MAXIMUM_POOL_SIZE:10}
data:
redis: # §3.14 bridge — operators set only DATAPIPELINES_REDIS_*
host: ${DATAPIPELINES_REDIS_HOST}
port: ${DATAPIPELINES_REDIS_PORT:6379}
password: ${DATAPIPELINES_REDIS_PASSWORD:}
flyway:
enabled: true
locations: classpath:db/migration
baseline-on-migrate: false
server:
port: ${SERVER_PORT:8080}
management: # §3.14 — actuator on the management port ONLY
server:
port: ${MANAGEMENT_SERVER_PORT:9090}
address: ${MANAGEMENT_SERVER_ADDRESS:127.0.0.1} # loopback default; 0.0.0.0 only with a NetworkPolicy
endpoints:
web:
exposure:
include: "health" # never "" and never exclude:"*" — see §3.14
health:
diskspace:
enabled: false
datapipelines:
redis:
host: ${DATAPIPELINES_REDIS_HOST}
port: ${DATAPIPELINES_REDIS_PORT:6379}
password: ${DATAPIPELINES_REDIS_PASSWORD:}
jwt:
secret: ${DATAPIPELINES_JWT_SECRET}
db:
key-provider: ${DATAPIPELINES_DB_KEY_PROVIDER:env}
encryption-key: ${DATAPIPELINES_DB_ENCRYPTION_KEY}
# Rotation (§3.20) — unset by default; a declared empty map would bind over the environment.
# encryption-keys:
# 2: ${DATAPIPELINES_DB_ENCRYPTION_KEY_V2}
# encryption-key-current: 2
auth:
oidc:
providers:
# client-id defaulting to empty = the entry is IGNORED with a WARN (§7);
# a local-accounts-only deployment starts with zero OIDC providers.
- name: google
client-id: ${GOOGLE_CLIENT_ID:}
client-secret: ${GOOGLE_CLIENT_SECRET:}
issuer-uri: https://accounts.google.com
display-name: "Sign in with Google"
- name: microsoft
client-id: ${MICROSOFT_CLIENT_ID:}
client-secret: ${MICROSOFT_CLIENT_SECRET:}
issuer-uri: https://login.microsoftonline.com/common/v2.0
display-name: "Sign in with Microsoft"
# Add more providers as needed (Okta, Auth0, Keycloak, etc.)
jwt:
ttl-hours: ${DATAPIPELINES_AUTH_JWT_TTL_HOURS:8}
allowlist:
domains: ${DATAPIPELINES_AUTH_ALLOWLIST_DOMAINS:}
api-keys:
cache-ttl-seconds: ${DATAPIPELINES_AUTH_API_KEYS_CACHE_TTL_SECONDS:60}
default-scopes: ${DATAPIPELINES_AUTH_API_KEYS_DEFAULT_SCOPES:read}
rate-limit:
login-per-minute: ${DATAPIPELINES_AUTH_RATE_LIMIT_LOGIN_PER_MINUTE:10}
trusted-proxies: ${DATAPIPELINES_AUTH_TRUSTED_PROXIES:}
local:
enabled: ${DATAPIPELINES_AUTH_LOCAL_ENABLED:false}
bootstrap-password-hash: ${DATAPIPELINES_AUTH_LOCAL_BOOTSTRAP_PASSWORD_HASH:}
bootstrap-password: ${DATAPIPELINES_AUTH_LOCAL_BOOTSTRAP_PASSWORD:}
lockout:
max-failures: ${DATAPIPELINES_AUTH_LOCAL_LOCKOUT_MAX_FAILURES:5}
duration-minutes: ${DATAPIPELINES_AUTH_LOCAL_LOCKOUT_DURATION_MINUTES:15}
cookie-secure: ${DATAPIPELINES_AUTH_COOKIE_SECURE:}
allow-local-only: ${DATAPIPELINES_AUTH_ALLOW_LOCAL_ONLY:false}
executor:
max-parallel-nodes: ${DATAPIPELINES_EXECUTOR_MAX_PARALLEL_NODES:4}
max-concurrent-executions-per-user: ${DATAPIPELINES_EXECUTOR_MAX_CONCURRENT_EXECUTIONS_PER_USER:10}
# Per-INSTANCE ceiling (050/R2); the deprecated `max-concurrent-executions-global` alias is
# deliberately absent here — it binds only when an operator still sets it.
max-concurrent-executions-per-instance: ${DATAPIPELINES_EXECUTOR_MAX_CONCURRENT_EXECUTIONS_PER_INSTANCE:100}
node-query-timeout-seconds: ${DATAPIPELINES_EXECUTOR_NODE_QUERY_TIMEOUT_SECONDS:60}
execution-timeout-seconds: ${DATAPIPELINES_EXECUTOR_EXECUTION_TIMEOUT_SECONDS:600}
node-timeout-seconds: ${DATAPIPELINES_EXECUTOR_NODE_TIMEOUT_SECONDS:300}
node-timeout-max-seconds: ${DATAPIPELINES_EXECUTOR_NODE_TIMEOUT_MAX_SECONDS:900}
cancel-grace-seconds: ${DATAPIPELINES_EXECUTOR_CANCEL_GRACE_SECONDS:5}
source-fetch-size: ${DATAPIPELINES_EXECUTOR_SOURCE_FETCH_SIZE:1000}
progress-write-interval-seconds: ${DATAPIPELINES_EXECUTOR_PROGRESS_WRITE_INTERVAL_SECONDS:5}
progress-sample-interval-seconds: ${DATAPIPELINES_EXECUTOR_PROGRESS_SAMPLE_INTERVAL_SECONDS:1}
heartbeat-seconds: ${DATAPIPELINES_EXECUTOR_HEARTBEAT_SECONDS:15}
pipelines:
max-composition-depth: ${DATAPIPELINES_PIPELINES_MAX_COMPOSITION_DEPTH:5}
deployment:
authoring-enabled: ${DATAPIPELINES_DEPLOYMENT_AUTHORING_ENABLED:true}
promotion:
server-key: ${DATAPIPELINES_DEPLOYMENT_PROMOTION_SERVER_KEY:}
target:
base-url: ${DATAPIPELINES_DEPLOYMENT_PROMOTION_TARGET_URL:}
server-key: ${DATAPIPELINES_DEPLOYMENT_PROMOTION_TARGET_KEY:}
workspaces:
member-datasources-enabled: ${DATAPIPELINES_WORKSPACES_MEMBER_DATASOURCES_ENABLED:true}
staging:
h2:
mode: ${DATAPIPELINES_STAGING_H2_MODE:PostgreSQL}
max-memory-mb: ${DATAPIPELINES_STAGING_H2_MAX_MEMORY_MB:1024}
insert-batch-size: ${DATAPIPELINES_STAGING_H2_INSERT_BATCH_SIZE:1000}
result-batch-size: ${DATAPIPELINES_STAGING_H2_RESULT_BATCH_SIZE:10000}
query-timeout-seconds: ${DATAPIPELINES_STAGING_H2_QUERY_TIMEOUT_SECONDS:60}
max-connections: ${DATAPIPELINES_STAGING_H2_MAX_CONNECTIONS:4}
result:
ttl-default-seconds: ${DATAPIPELINES_RESULT_TTL_DEFAULT_SECONDS:300}
ttl-min-seconds: ${DATAPIPELINES_RESULT_TTL_MIN_SECONDS:60}
ttl-max-seconds: ${DATAPIPELINES_RESULT_TTL_MAX_SECONDS:3600}
max-size-bytes: ${DATAPIPELINES_RESULT_MAX_SIZE_BYTES:104857600}
page-size-rows: ${DATAPIPELINES_RESULT_PAGE_SIZE_ROWS:1000}
page-max-rows: ${DATAPIPELINES_RESULT_PAGE_MAX_ROWS:100000}
sse:
heartbeat-interval-seconds: ${DATAPIPELINES_SSE_HEARTBEAT_INTERVAL_SECONDS:15}
disconnect-grace-seconds: ${DATAPIPELINES_SSE_DISCONNECT_GRACE_SECONDS:30}
max-streams-per-user: ${DATAPIPELINES_SSE_MAX_STREAMS_PER_USER:50}
rate-limit:
requests-per-second: ${DATAPIPELINES_RATE_LIMIT_REQUESTS_PER_SECOND:100}
requests-per-minute: ${DATAPIPELINES_RATE_LIMIT_REQUESTS_PER_MINUTE:1000}
idempotency:
ttl-seconds: ${DATAPIPELINES_IDEMPOTENCY_TTL_SECONDS:86400}
templates:
cache-size: ${DATAPIPELINES_TEMPLATES_CACHE_SIZE:500}
render-timeout-ms: ${DATAPIPELINES_TEMPLATES_RENDER_TIMEOUT_MS:5000}
ui:
theme: ${DATAPIPELINES_UI_THEME:dark}
executions:
event-retention-days: ${DATAPIPELINES_EXECUTIONS_EVENT_RETENTION_DAYS:7}
stale-timeout-minutes: ${DATAPIPELINES_EXECUTIONS_STALE_TIMEOUT_MINUTES:60}
audit:
retention-days: ${DATAPIPELINES_AUDIT_RETENTION_DAYS:365}
observability:
tracing:
enabled: ${DATAPIPELINES_OBSERVABILITY_TRACING_ENABLED:false}
logging:
format: ${DATAPIPELINES_OBSERVABILITY_LOGGING_FORMAT:json}
bootstrap:
datasources-file: ${DATAPIPELINES_BOOTSTRAP_DATASOURCES_FILE:}
examples-file: ${DATAPIPELINES_BOOTSTRAP_EXAMPLES_FILE:}
# §3.21 — organisation facts. Appended AFTER the whole datapipelines: tree, never inserted
# between its children: a 2-space block placed mid-tree re-parents whatever follows it.
org:
currency:
name: ${DATAPIPELINES_ORG_CURRENCY_NAME:Dollar}
symbol: ${DATAPIPELINES_ORG_CURRENCY_SYMBOL:$}
fiscal-start-date: ${DATAPIPELINES_ORG_FISCAL_START_DATE:01-01}
week-start: ${DATAPIPELINES_ORG_WEEK_START:monday}
timezone: ${DATAPIPELINES_ORG_TIMEZONE:UTC}
endpoints:
timeout-default-seconds: ${DATAPIPELINES_ENDPOINTS_TIMEOUT_DEFAULT_SECONDS:30}
timeout-min-seconds: ${DATAPIPELINES_ENDPOINTS_TIMEOUT_MIN_SECONDS:1}
timeout-max-seconds: ${DATAPIPELINES_ENDPOINTS_TIMEOUT_MAX_SECONDS:300}
# §3.23 — appended after the whole datapipelines: tree, never inserted between two of
# its children (a 2-space block placed mid-tree closes its predecessor and silently
# re-parents whatever follows it).
env: ${DATAPIPELINES_ENV:local}
posture: ${DATAPIPELINES_POSTURE:}
demo: ${DATAPIPELINES_DEMO:}
# §3.25 — the LAKE engine's bundled extension directory. Empty = INSTALL+LOAD with egress
# (a bare jar has nothing bundled); the shipped image exports the variable itself.
duckdb:
extension-directory: ${DATAPIPELINES_DUCKDB_EXTENSION_DIRECTORY:}
# §3.25 — the operator default memory_limit every LAKE engine build gets when a
# datasource declares none. Empty = the derived 25%-of-container default.
memory-limit: ${DATAPIPELINES_DUCKDB_MEMORY_LIMIT:}
# §3.27 — outbound mail. NO `enabled` key: on exactly when host AND from are set.
mail:
host: ${DATAPIPELINES_MAIL_HOST:}
port: ${DATAPIPELINES_MAIL_PORT:587}
username: ${DATAPIPELINES_MAIL_USERNAME:}
password: ${DATAPIPELINES_MAIL_PASSWORD:}
starttls: ${DATAPIPELINES_MAIL_STARTTLS:true}
from: ${DATAPIPELINES_MAIL_FROM:}
reply-to: ${DATAPIPELINES_MAIL_REPLY_TO:}
ops-to: ${DATAPIPELINES_MAIL_OPS_TO:}
message-stream: ${DATAPIPELINES_MAIL_MESSAGE_STREAM:}
Note: OIDC provider config is in the app's own YAML namespace (
datapipelines.auth.oidc.providers), NOT in Spring Security's nativespring.security.oauth2.client.*namespace. OurOidcConfigbean reads this list and buildsClientRegistrationobjects programmatically. See Auth spec §5.2.
Internal binding note (Redis, 2026-08-07): Spring Boot's Redis autoconfiguration reads
spring.data.redis.*, soapplication.ymlcarries an internal bridge (spring.data.redis.host: ${DATAPIPELINES_REDIS_HOST}etc.) mapping the canonicaldatapipelines.redis.*keys onto it. This introduces NO operator-facing keys — operators set only theDATAPIPELINES_REDIS_*variables defined here. The bridge is an implementation detail and may be replaced by an explicitLettuceConnectionFactorybound to@ConfigurationProperties("datapipelines.redis").
6. Posture Profiles (application-development.yml / application-hardened.yml)
The POSTURE's non-secret defaults (§3.23). spring.profiles.active is derived from DATAPIPELINES_POSTURE in application.yml, so setting that one variable loads the right file under every loader — Compose, a bare java -jar, systemd, Kubernetes. Setting SPRING_PROFILES_ACTIVE yourself is unnecessary, and a value that disagrees with the posture refuses startup (§7).
These files carry POSTURE, never INFRASTRUCTURE. Before 075 this section documented application-dev.yml, which was really a LAPTOP file: localhost:5434, console logging, one developer's host ports. An org's dev environment on Kubernetes runs the development posture too and must not inherit any of that. The laptop's values are ordinary lines in deploy/env/defaults.env, which every loader reads (DEVELOPMENT.md §2/§4).
And they carry the posture's defaults ALONE (081). No env file may name DATAPIPELINES_DEPLOYMENT_AUTHORING_ENABLED or DATAPIPELINES_AUTH_COOKIE_SECURE: there is one tracked settings file, it is loaded for every posture, and an environment variable outranks a profile — so a value there would be one posture's answer imposed on both. 075 did exactly that and a hardened stack booted with authoring=on. deploy/compose.yml passes those two in Compose's valueless form (DATAPIPELINES_AUTH_COOKIE_SECURE:, nothing after the colon), the only form that leaves a variable unset rather than empty — an empty variable outranks a profile just as a set one does. An operator can still override either in deploy/secrets.env. PostureDefaultsSpecDriftTest and compose-env-audit.sh check 7 hold the two halves.
No literal secrets, in either file (2026-08-07 security review): both ship inside every production jar (src/main/resources), so a working literal there means one stray SPRING_PROFILES_ACTIVE on a production manifest runs real infrastructure on publicly-known keys — forgeable admin JWTs and decryptable datasource credentials. Every secret is a placeholder resolved from the deployment's own environment, in every posture.
# application-development.yml
datapipelines:
deployment:
authoring-enabled: ${DATAPIPELINES_DEPLOYMENT_AUTHORING_ENABLED:true}
auth:
cookie-secure: ${DATAPIPELINES_AUTH_COOKIE_SECURE:}
# application-hardened.yml
datapipelines:
deployment:
authoring-enabled: ${DATAPIPELINES_DEPLOYMENT_AUTHORING_ENABLED:false}
auth:
cookie-secure: ${DATAPIPELINES_AUTH_COOKIE_SECURE:true}
Everything else the posture means is a §7 REFUSAL, which no YAML file can carry: demo, a seeded bootstrap credential, loopback infrastructure and a missing OIDC provider are all refused under hardened. The full table is §3.23.
The laptop's own settings, for reference (DEVELOPMENT.md §2/§4) — ordinary lines in deploy/env/defaults.env, read FIRST like every other default, with no inversion of the secrets-last rule:
DATAPIPELINES_ENV=local
DATAPIPELINES_POSTURE=development
SPRING_DATASOURCE_URL=jdbc:postgresql://localhost:5434/datapipelines
SPRING_DATASOURCE_USERNAME=datapipelines
DATAPIPELINES_REDIS_HOST=localhost
DATAPIPELINES_REDIS_PORT=6381
DATAPIPELINES_OBSERVABILITY_LOGGING_FORMAT=console
DATAPIPELINES_AUTH_ALLOWLIST_DOMAINS=
The two passwords are absent on purpose: they are secrets, they live only in deploy/secrets.env, and deploy/compose.laptop-infra.yml starts its Postgres and Redis from those same two variables (each keeping its historical fixed dev value as the interpolation default). Before 081 that file hard-coded datapipelines and no Redis password at all, and a third env file had to be loaded after the secrets to undo them — the one inversion this design used to have. DATAPIPELINES_AUTH_BASE_URL is likewise absent: it is this deployment's own identity and lives in secrets.env, where the scaffold writes http://localhost:<port>.
Dev host ports (2026-08-12): the laptop's Postgres listens on host port 5434 and Redis on 6381 — not the universal defaults 5432/6379, which collide with other local stacks on developer machines (Postgres.app/brew default to 5432). The host mapping lives in
deploy/compose.laptop-infra.yml;SPRING_DATASOURCE_URLandDATAPIPELINES_REDIS_PORTindeploy/env/defaults.envmust stay in sync with it and with DEVELOPMENT.md §2/§4. In production,SPRING_DATASOURCE_URLandDATAPIPELINES_REDIS_*are operator-set and unaffected.
7. Config Validation
On startup, the app validates:
- All required keys present → fail-fast with a clear error if missing.
DATAPIPELINES_JWT_SECRET≥ 32 bytes decoded.DATAPIPELINES_DB_ENCRYPTION_KEYis exactly 32 bytes decoded.- Key provider (§3.20):
datapipelines.db.key-providernames a provider this build ships (a violation lists them and points atdocs/key-providers.md), and that provider's own settings are present and well-formed. Forenv: every entry ofdatapipelines.db.encryption-keyshas an integer version in1..255that is not1, and a value that decodes to exactly 32 bytes;datapipelines.db.encryption-key-currentis an integer naming a configured version (the violation lists which versions ARE configured). An unknown provider name short-circuits the rest — reportingenv's key rules to an operator who asked foraws-kmsis noise they cannot act on. DATAPIPELINES_UI_THEMEmatches a vendored theme directory.- At least one authentication method: a fully-configured OIDC provider (non-empty
client-id,client-secret, andissuer-uri) ordatapipelines.auth.local.enabled=true. A provider entry with an emptyclient-idis ignored with a WARN — it does not count, and it is not a violation on its own. datapipelines.auth.local.bootstrap-passwordanddatapipelines.auth.local.bootstrap-password-hashare never both set; either seed requiresdatapipelines.auth.local.enabled=trueANDdatapipelines.auth.bootstrap-admin-email— each violation names both keys.datapipelines.auth.local.lockout.max-failuresanddatapipelines.auth.local.lockout.duration-minutesare positive integers.- The deprecated executor alias
datapipelines.executor.max-concurrent-executions-global(050/R2): set alone → its value runs and startup logs one WARN namingmax-concurrent-executions-per-instance; set together with the new key and differing → startup REFUSES naming both keys. result.ttl-min-seconds≤result.ttl-default-seconds≤result.ttl-max-seconds.endpoints.timeout-min-seconds≤endpoints.timeout-default-seconds≤endpoints.timeout-max-seconds, and the minimum is ≥ 1 (§3.22). Unlike the result TTLs, an ABSENT key is not a violation here — the binder default applies — so only a present, out-of-order triple refuses startup.datapipelines.workspaces.provisioning-modeis one ofauto-per-user|self-serve|closed.datapipelines.workspaces.open-join: truetogether withclosedprovisioning is refused, naming both keys (§3.17) — open-join is aself-serveknob, and underclosedit would let any authenticated user self-join any workspace, the exact surfaceclosedexists to close.- Every
datapipelines.auth.trusted-proxiesentry parses as a CIDR (a bare IP is a host CIDR); anything else refuses startup at the auth module's resolver construction — a typo'd range must not silently widen proxy trust. datapipelines.bootstrap.datasources-fileis not set withoutdatapipelines.auth.bootstrap-admin-email(§3.18) — the violation names both keys.datapipelines.bootstrap.examples-fileis not set whiledatapipelines.workspaces.provisioning-modeis anything butauto-per-user(§3.18) — the violation names both keys. Onlyauto-per-userprovisions the personal workspace the examples are seeded into, so any other mode (the shipped default included) leaves the configured file permanently unseeded. A mode that is misspelled is reported by the mode check alone, not twice.- No OIDC provider is named
bootstrap,localorsystem. Those are theusers.providervalues the system writes for identities it creates itself (§6.1 bootstrap actor, §5A local accounts, Auth §4.5 system service account), and a provider's configured name is written to that column verbatim — an external provider under any of them would be indistinguishable from them, and forsystemthat indistinguishability is the whole of the account's safety argument. The reservation is case-insensitive and applies to an entry with a blankclient-idtoo. - Environment name (§3.23):
datapipelines.envmatches[a-z0-9][a-z0-9_-]{0,31}. The deprecateddatapipelines.deployment.namealias: set alone it still names the deployment and startup logsevent=config.deployment_name_deprecated; set together withdatapipelines.envand disagreeing, startup is REFUSED naming both keys. - Posture (§3.23):
datapipelines.postureisdevelopmentorhardened. A named environment with no posture refuses to start, naming both variables — the sole exception is the environment namedlocal, which resolves todevelopment. Explicit beats guessed: the product cannot infer a stance from a name it is forbidden to branch on. - Posture / profile alignment (§3.23):
spring.profiles.activeis derived from the posture, so settingSPRING_PROFILES_ACTIVEto a DIFFERENT posture refuses startup naming both — otherwise one posture's defaults would load while every posture rule below judged the other. Thedevprofile was renameddevelopmentin 075; a deployment still asking fordevis refused rather than silently getting no posture file at all. - The
hardenedposture (§3.23), each refusal naming the variable and the posture:datapipelines.demomust be empty; no local bootstrap credential may be set (bootstrap-passwordorbootstrap-password-hash— local accounts themselves stay allowed);spring.datasource.urlanddatapipelines.redis.hostmust not be loopback (039's dev-profile guard, inverted and reused — "dev convenience must never touch production infrastructure" is the same fact as "a hardened deployment does not run against a laptop's database"); and at least one OIDC provider must be fully configured unlessdatapipelines.auth.allow-local-only=true, which is logged asevent=config.auth_local_only. - Organisation (§3.21):
datapipelines.org.fiscal-start-dateisMM-DDand a day the calendar has —02-30and13-01are refused, and a month name (SEP-15) is refused with a message naming theMM-DDform;datapipelines.org.week-startismondayorsunday;datapipelines.org.timezoneis an IANA zone id (a fixed offset such as+02:00is not one);datapipelines.org.currency.nameand.symbolare non-blank. All four report together — every value is in every Context, so a wrong one is a wrong number in every report the deployment produces. - Redis auth warning: when
datapipelines.redis.passwordis empty anddatapipelines.redis.hostis not loopback, log a structured WARN (production Redis holds materialized caller results — Deployment §7.3). datapipelines.deployment.promotion.server-keyset ⇒ WARN (091): the value is deprecated in favour of aserver-kind API key and is removed next release. Presence only — the warning never carries the secret.datapipelines.deployment.promotion.target.base-urlis not set withoutdatapipelines.deployment.promotion.target.server-key(§3.19) — the violation names both keys. The target's pre-shared key is what authenticates the push, so a target without one would have every promotion refused at the far end, at the end of a UI action a human took. The reverse is not a violation: aserver-keywith no target is an ordinary receiver.- Mail (§3.27, 137):
datapipelines.mail.hostanddatapipelines.mail.fromare set together or not at all — one without the other refuses startup naming the missing one;datapipelines.mail.ops-towithouthostis refused (a sink nobody can reach); mail on (both set) withoutdatapipelines.auth.base-urlis refused (the welcome mail's login URL);portis a port in1..65535;fromis an address (a bare address orDisplay Name <address>). Under thehardenedposture, with mail on:starttlsmust betrue, andusernameset requirespasswordset — each refusal naming the variable and the posture. The SMTP password is carried as PRESENCE only; no violation ever prints it. - The boot line (§3.23):
event=config.posture env=<env> posture=<posture> authoring=<on|off> demo=<families>is logged once (the label's only consumer — no code branches on it, pinned by a guard test). When a promotion receiver key is configured ANDdatapipelines.deployment.authoring-enabled=true, log a structured WARN — a promotion receiver should not author (Versioning D7), though a one-box deployment may legitimately be both. And when authoring is DISABLED while draft pipeline/template versions still exist, startup FAILS naming them: someone authored on a receiver and version alignment may already be broken (Versioning §5.5/§9.3).
The validator's own test suite must assert that the documented laptop setup (deploy/env/defaults.env + deploy/secrets.env, §6) passes the production rules — so a broken local value gets fixed at the data, never by weakening the check.
Validation runs in @PostConstruct of a ConfigValidator bean. Failures stop startup with a clear log message listing every missing/invalid key.
Appendix A: Change Log
| Date | Version | Author | Change |
|---|---|---|---|
| 2026-09-17 | v1.23 | 155 / #122 staging memory guard scope | §3.3: max-memory-mb re-described truthfully — the threshold is per-execution (the pipeline override changes the number one execution compares, never what is measured), but the MEASUREMENT is the whole JVM's used heap, sampled, so the guard is a shared circuit breaker, not an isolated budget, a reservation, or an OOM guarantee (Staging §8.2 says the same; one cross-link, not two copies). Sizing guidance rewritten from the demand side: worst-case heap demand stays max-memory-mb × max-concurrent-executions-per-instance per instance and -Xmx/container memory must cover it. No behaviour change; the 108 §C startup warn line is unchanged. |
| 2026-09-17 | v1.22 | 153 operator memory-limit config (#136) | New §3.25 row: datapipelines.duckdb.memory-limit (DATAPIPELINES_DUCKDB_MEMORY_LIMIT, default empty). Deployment-wide default memory_limit for every LAKE engine build when a datasource sets none, restart-to-change, validated at startup with properties.dialect.memory_limit's own grammar. §3.24's memory_limit row gains the three-way precedence sentence (datasource property > this key > derived 25%). §3.25's title and intro widened to cover both operator keys. §5 template block appended after extension-directory (same block, not a new one). |
| 2026-09-17 | v1.21 | Writable LAKE spill default (#133) | §3.24: unique engine-owned absolute path under java.io.tmpdir; writable disk requirement, explicit override and cleanup semantics. |
| 2026-09-17 | v1.20 | 152 LAKE engine scope (#128) | §3.24 gains the resource-scope paragraph: a LAKE datasource pool owns ONE embedded DuckDB instance shared by all its pooled connections, so memory_limit / threads / temp_directory are one budget per datasource pool (not per connection, not multiplied by maximumPoolSize), the sizing rule is per LAKE datasource on the instance plus a draining generation's headroom, and the file cache survives Hikari's connection replacement but not a pool rebuild. No key added or changed. Status line re-synced to the changelog (it read v1.1 above rows that ended at v1.18) |
| 2026-09-16 | v1.19 | 149 / #125 node_progress cadence | §3.2 gains datapipelines.executor.progress-sample-interval-seconds (1): the floor between two periodic node_progress samples of one operation; first entries into a state and the terminal sample are never throttled by it. Mirrored in application.yml, defaults.env, secrets.env.example, compose.yml; WebPropertiesSpecDriftTest pins the default. |
| 2026-09-14 | v1.18 | 137 mail notices | New §3.27 Mail: datapipelines.mail.host / port (587) / username / password / starttls (true, required-not-opportunistic) / from / reply-to (empty = from) / ops-to (comma list) / message-stream (Postmark's optional header). No enabled key — mail is on exactly when host and from are both set. §5 template block appended after duckdb:; §7 gains the shape rules (host↔from, ops-to without host, mail on without auth.base-url, port range, from is an address) and the two hardened refusals (starttls must be true; a username needs a password) — checks 23 and 24 |
| 2026-09-08 | v1.17 | 091 keys | §3.19: datapipelines.deployment.promotion.server-key is DEPRECATED in favour of a server-kind API key (Auth §7.7) — mintable by an admin on the API screen, expiring, revocable, rotatable without a restart, and visible in the key list. Both credentials are accepted for one release; the configured value is compared FIRST (so a deployment that has not migrated pays no database read) and its presence raises one WARN at boot. §7 gains that rule (check 23). Fail-closed is unchanged and now has two halves: no configured value AND no live server key ⇒ every push refused. |
| 2026-09-07 | v1.16 | 089 dp-lake §D extension bundling | New §3.25 DuckDB extension directory (dp-lake): datapipelines.duckdb.extension-directory (DATAPIPELINES_DUCKDB_EXTENSION_DIRECTORY, default empty). Set — as the shipped image does via the Dockerfile, which now bundles httpfs/aws/iceberg/avro for DuckDB core v1.5.5 for the build's architecture (linux_amd64 or linux_arm64 via TARGETARCH, +108 MB uncompressed, ~39 MB at build) — lake connections SET extension_directory and emit bare LOADs, never an INSTALL, so a hardened dp-lake deployment needs no egress to extensions.duckdb.org. Unset keeps the INSTALL+LOAD behavior developer machines rely on. §5 template block appended after demo: |
| 2026-09-07 | v1.15 | 089 dp-lake phases B–D | New §3.24 Lake datasource engine limits (dp-lake): the properties.dialect.memory_limit / threads / temp_directory datasource properties (datasource-row keys, NOT datapipelines.* env keys — validated per key at save), the default memory limit (25 % of the container's memory as the cgroup-aware JVM reports it, hard-capped at 4 GiB, floored at 64 MiB), and the always-on preserve_insertion_order = false. Phases B/C add no operator keys: per-table views and registry-backed introspection are behavior, not configuration |
| 2026-09-06 | v1.14 | 081 one env file | §3.23, §4 and §6 rewritten to two files. The contract is deploy/env/defaults.env (tracked, every non-secret variable with its shipped value) then deploy/secrets.env (git-ignored, credentials and this deployment's own overrides), in that order under every loader; deploy/secrets.env.example is the template and names every variable. The five files 075 shipped under deploy/env/ are deleted, and with them the laptop's load-order inversion — deploy/compose.laptop-infra.yml now starts Postgres and Redis from SPRING_DATASOURCE_PASSWORD and DATAPIPELINES_REDIS_PASSWORD. §6: the posture's two default rows live ONLY in the profile ymls, and no env file may name them. compose-env-audit.sh gains the one-authority proof (each bound variable declared in exactly one tracked file) and the valueless-form check. |
| 2026-09-05 | v1.13 | 075 environments and posture | New §3.23 Environment and posture: datapipelines.env (DATAPIPELINES_ENV, default local — the ORG's label, renamed from §3.19's datapipelines.deployment.name, which survives one release as a WARNing alias), datapipelines.posture (development | hardened, no default — only the env named local gets one for free) and datapipelines.demo (a FLAG: the sample-data families; refused under hardened), with the normative posture table. §3.4 gains cookie-secure (empty = derived from base-url's scheme; hardened ships true) and allow-local-only (the explicit no-OIDC acknowledgement). §6 is now the POSTURE profiles — application-dev.yml was really a laptop file and its infrastructure moved to deploy/env/laptop.env; application-development.yml and application-hardened.yml carry the posture defaults, and spring.profiles.active is derived from DATAPIPELINES_POSTURE. §7 gains four rules (env grammar + alias, posture required, posture/profile alignment, the hardened refusals) and loses the dev-profile guard, which those refusals subsume. §4 and §5 updated. Environments is the new operator page. |
| 2026-09-05 | v1.12 | 074 published endpoints | New §3.22 Published endpoints: datapipelines.endpoints.timeout-default-seconds (30) / -min-seconds (1) / -max-seconds (300); one new §7 rule (min ≤ default ≤ max, min ≥ 1 — the four report together). Deliberately no page-rows-max: DP-Result-Page-Rows clamps to result.page-max-rows on both surfaces (R-EP4). §5 template block appended after org:. |
| 2026-09-04 | v1.11 | 072 calculators | New §3.21 Organisation: datapipelines.org.currency.name / .symbol, fiscal-start-date (MM-DD), week-start, timezone — five keys that enter every execution Context as org_* (calculators design §0.1/§0.2, Pipeline Contract §7.2). §5 template block and one new §7 validation rule (all four org checks report together; a month name in fiscal-start-date is refused with a message naming MM-DD) |
| 2026-09-04 | v1.10 | 068 key-provider seam | New §3.20 credential key provider: datapipelines.db.key-provider (default env, so no deployment needs a config edit), plus the env provider's optional encryption-keys rotation map and encryption-key-current. §2's encryption-key row now states that it is key version 1, forever. §5 template and §7 updated — one new validation rule: the provider name must be one this build ships, and that provider's own settings must be present and well-formed (an unknown name short-circuits the rest). |
| 2026-09-02 | v1.9 | 051 auth/config sweep | Added §3.4 datapipelines.auth.trusted-proxies (CIDR list, default empty = header ignored; the login limiter and every auth source_ip resolve the client through it — R8/T46, deployment.md §6.2) with the §5 template line and two §7 rules (each entry must parse as a CIDR or startup is refused; enforced at the auth module's resolver construction). §3.17: open-join: true + closed provisioning now refused at startup, naming both keys (T45 — the self-join branch gates on open-join alone, so the pair would re-open the membership surface closed exists to keep admin-only); §7 gains the rule |
| 2026-08-05 | v1.0 | initial draft | Complete configuration reference: 6 required keys + OIDC, ~30 optional keys, full application.yml template, dev profile, startup validation |
| 2026-08-07 | v1.1 | consistency campaign | Authority + naming-derivation rules (§1); §3 tables and §5 YAML reconciled (unit-suffixed names win); added result.* (D9), sse.disconnect-grace-seconds (D7), idempotency, templates, audit, staging result-batch-size, login rate-limit keys; removed large-result-threshold-bytes and redis.ttl-seconds (superseded by result.*); rate limits per-user; encryption key required with no fallback; precedence section. See SPEC-REVIEW-2026-08 |
| 2026-08-12 | v1.2 | dev infra ports | §6 dev profile now targets host ports 5434 (Postgres) / 6381 (Redis) instead of the colliding defaults 5432/6379; added the dev-host-ports note. Operator-facing keys and production defaults unchanged. |
| 2026-08-17 | v1.3 | pipeline composition | Added §3.16 datapipelines.pipelines.max-composition-depth (default 5) — the depth guard for PIPELINE-node composition |
| 2026-08-26 | v1.3.1 | workspaces (slice 019) | Added §3.17 Workspaces: datapipelines.workspaces.provisioning-mode (auto-per-user | self-serve | closed, default self-serve), open-join (default false), member-datasources-enabled (default true). (This row was written later than the section — v1.4's note promised the backfill, but the row itself never landed; recorded and landed 2026-08-29, 025 D3. Keys unchanged by the delay.) |
| 2026-08-28 | v1.4 | sample data, slice A | Added §3.18 Bootstrap: datapipelines.bootstrap.datasources-file and datapipelines.bootstrap.examples-file (both unset = off), the cross-key rule pairing datasources-file with datapipelines.auth.bootstrap-admin-email, and the matching §7 validation bullet and §5 template block. Backfills the §3.17 Workspaces row this log was missing (added 2026-08-26 with slice 019, keys unchanged here) |
| 2026-08-29 | v1.5 | local password auth | Added §3.4 datapipelines.auth.local.*: enabled, bootstrap-password-hash / bootstrap-password (first-admin seed only, forced first-login change), lockout.max-failures / lockout.duration-minutes — plus the §5 template block. The §7 "at least one OIDC provider" rule becomes "at least one authentication method"; a provider entry with an empty client-id is now ignored with a WARN instead of counting (the stock google entry binds empty when its env vars are unset, so a local-accounts-only deployment starts with zero providers). rate-limit.login-per-minute description widened to OIDC and local |
| 2026-09-02 | v1.7 | 048 bootstrap seeding fixes | Two §7 rules added, no new keys: (a) the examples-file cross-key rule pairing it with datapipelines.workspaces.provisioning-mode = auto-per-user (021/F5 — the pair validated green while the seeder was structurally unreachable, on the shipped default); (b) bootstrap and local reserved as OIDC provider names (021/F8 — the users.provider placeholders were squatting in an operator-configurable namespace with nothing reserving them). §3.18 gains the matching cross-key paragraph |
| 2026-09-02 | v1.8 | 050 multi-instance round 2 | §3.2: max-concurrent-executions-global renamed max-concurrent-executions-per-instance (the limit was always per JVM — the old name false at N replicas; 050/R2), old key kept as a one-release deprecated alias (alone → WARN naming the new key; both set and differing → §7 refusal); §3.2 heap note and §5 template updated to the per-instance multiplier; §7 gains the alias rule. §3.11: event-retention-days now bound and scheduled (the hourly retention job, M2's sibling) |
| 2026-09-01 | v1.6 | 039 deployment role | Added §3.19 Deployment: datapipelines.deployment.name (label only — logged once at boot beside the authoring state; nothing branches on it, pinned by a guard test; deliberately not on /info) and datapipelines.deployment.authoring-enabled (default true; false turns the deployment into a promotion receiver whose authoring writes refuse with *.authoring.disabled; startup refuses if drafts exist while disabled). The reserved datapipelines.deployment.promotion.* sub-block is deliberately NOT declared here — it ships with promotion (Versioning §10.6's fenced sample), per this doc's shipped-keys-only rule. Matching §7 bullets (the one-sided receiver-also-authors WARN; the refuse-on-existing-drafts rule) and §5 template block |
| 2026-09-10 | v1.9 | 112 RBAC round 1 | §3.17: datapipelines.workspaces.provisioning-mode and datapipelines.workspaces.open-join removed (D-R11 — capability moved onto the workspace membership, so who may create a workspace is a role, not a mode). Both are refused BY NAME at startup rather than ignored, and both are gone from the §5 template, application.yml and the shipped .env — a placeholder with a default is a set key to Spring's relaxed binding, so leaving them there would have refused the product's own boot. §7 loses the two rules that read them (the mode-value check and the open-join-under-closed check) and the examples-file cross-key rule (example seeding now runs once, when DemoWorkspaceSeeder creates demo); it gains one rule that names both removed keys. member-datasources-enabled survives, re-described: it gates whether a workspace ADMIN may register a datasource bound to their own workspace |