Configuration Reference

Status: v1.71 (operator reference; source-traced configuration authority) Owner: datapipelines.co core Last updated: 2026-11-01

1. Purpose

This is the official reference for configuring the shipped application. Each row explains what a setting controls; §5 records the exact packaged values and environment placeholders, §6 records posture overrides, and §8 identifies the source bindings and consumers used to verify the descriptions. The same document is packaged for the browser documentation and agent documentation lookup.

Authority rule: define application configuration keys here. Other specifications link here for defaults and use their own documents for workflows, API schemas and pipeline definitions. Existing group-specific drift tests and the whole-reference checks in §9 enforce the mechanical parts of this contract.

How to use this reference:

• Start with §2 for credentials and connectivity, then select the environment/posture in §3.23. Follow Deployment or Environments for the loader you use.
• A table default is the application default unless a profile or deployment override is explicitly stated. deploy/env/defaults.env includes laptop-specific overrides; it is not the default of a bare jar. Empty, absent, zero and false are distinct where the row says so.
• Use the actual environment placeholder in §5, or an explicit Env var cell. Most names use upper snake case, but bootstrap, promotion, providers and dashboard runtime have deliberate exceptions. Do not invent an environment name from a dotted path. Map/list members use YAML; [] and <dialect> describe variable members, not literal key names.
• Application properties are read during startup/binding. Restart the application (recreate the container when changing its environment) to apply configuration changes; persisted object settings, user preferences and stored API-key rotation have their own lifecycle described in the relevant row.
• Keep passwords, signing/encryption keys, client secrets, bootstrap credentials and promotion keys in the deployment's secret source. Examples contain placeholders, never working credentials. Do not put secret values on command lines or in support transcripts.

Scope: §§2–7 cover application properties, including framework wiring and reserved settings that currently have no runtime effect. §10 covers launcher/JVM inputs and points to the deployment chart and build options. Pipeline parameters, datasource definitions, dashboard definitions and HTTP/MCP request arguments are data contracts, not application startup configuration: see Pipeline contract, Datasources, Parameter sets, REST API and MCP server. §3.24 retains the existing datasource-specific LAKE limits because they participate in application-limit precedence.

2. Required Configuration

The app will not start without these. Fail-fast on missing values.

Provider configuration is also required — at least one OIDC or OAuth 2.0 provider must be configured in datapipelines.auth.oidc.providers (in application.yml), unless local accounts are enabled. An OIDC provider requires client-id, client-secret, and issuer-uri; an OAuth 2.0 provider (protocol: oauth2, GitHub, #655) requires client-id, client-secret, its three endpoints, user-name-attribute and scopes instead (§3.4). The client-id and client-secret are typically referenced from env vars. See Auth spec §11.1 for the full format.

The specific env var names depend on which provider(s) the deployment chooses. Examples:

The deployment defines these env var names in application.yml — they're not hardcoded by the app. (Provider variable names are deployment-chosen; consult the exact shipped examples in §5.)

3. Optional Configuration (with defaults)

3.1 Redis

Where an Env var column is omitted, §5 gives the exact shipped placeholder (for example, DATAPIPELINES_REDIS_PORT).

3.2 Executor

Three budgets, one precedence (the same table appears in pipeline-contract §4.11 and dag-executor §5.3):

Instances are moderate, so the statement bound must be sized for a COLD read, not a warm one (156, #2). This deployment is expected to run as several instances of moderate size, not one box with all the memory a query could want — so a LAKE engine sharing that box's memory with everything else keeps less of the object store cached, and a query that lands on an instance whose cache is cold re-reads from S3 at network speed instead of from RAM. The per-dialect operator default above exists for exactly that reason, sized to the cold read of the largest scan a deployment runs, not the warm one a developer's laptop measures. When cold reads on a given box routinely approach the default, the right levers are raising node-query-timeout-seconds-by-dialect.lake (this section) or raising the engine's own memory-limit (§3.25) so more of the data stays cached between queries — see also #141 on the shared-engine memory budget these two knobs both bound against.

Read downward. The statement bound is the driver's and drivers honour it unevenly — measured on the shipped drivers (108 §1), which is precisely why the middle row exists. The operator defaults are not required to follow execution ≥ node ≥ statement: a node deadline above the execution's is never reached because the outer bound fires first, and a lower node deadline stops work before the driver's statement timeout. node-timeout-max-seconds is not constrained against the execution timeout. A node's explicit settings.query_timeout_seconds is different: save-time SettingsRules refuses it above that node's effective wall-clock deadline (its own timeout_seconds, or the operator default, capped by node-timeout-max-seconds). The runtime also clamps node wall-clock deadlines to node-timeout-max-seconds, including already-stored pipelines after an operator lowers the ceiling.

3.3 Staging (tempdb)

max-connections is capacity, not parallelism. It bounds how many tempdb operations of ONE execution can be inside H2 at the same instant; it creates no extra eligible DAG nodes (executor.max-parallel-nodes decides those) and promises no speedup — H2 keeps its own transaction, row and catalog locks, so two nodes touching one table still serialise inside the engine. Set it lower than max-parallel-nodes and nodes queue safely for a connection; set it higher and the surplus is never opened. Aggregate cost: every connection is one H2 session with its own query working memory, and max-concurrent-executions-per-instance executions can each hold up to this many — the staged tables themselves are shared by the connections of one execution and are not multiplied. Session state (SET SCHEMA, SET @var, local temporary tables, an open transaction) is per operation and is reset when a connection is returned; a node must not rely on it from another node — express data flow through ordinary tables and depends_on, whatever the cap (Staging §9.2).

max-memory-mb is a per-execution THRESHOLD over a JVM-WIDE measurement — not an isolated per-execution budget. The guard samples used heap for the WHOLE JVM (in-process MemoryMXBean.heapMemoryUsage.used, Staging §8.2), so every concurrent execution's check reads the same shared number: one heavy execution can trip a lighter one's check, and the guard is a sampled circuit breaker — not a hard reservation, and no guarantee against OOM. Concurrent executions retain separate staging data, but multiplying this threshold by the admission limit is neither measured heap demand nor an enforced memory ceiling. Size -Xmx and the container limit from observed concurrent workloads, with headroom for the rest of the application and native DuckDB/JavaScript memory (Deployment §6.6). A per-execution accounting redesign is a separately decided follow-up, not this guard's promise.

Startup capacity warning (108 §C). ConfigValidator warns when max-concurrent-executions-per-instance × max-memory-mb > 0.8 × the JVM's max heap, naming the numbers and ratio. This is a sizing heuristic, not a startup refusal, a reservation or a prediction that OOM must occur. The runtime measurement remains JVM-wide as described above; validate capacity using the deployment's actual workloads.

A pipeline's settings.tempdb.config.max_memory_mb override is clamped to ≤ this value — it may lower the operator's ceiling for that pipeline, never raise it. Save-time validation only checks > 0, so without the clamp an author could declare an arbitrarily large budget and disable the only ceiling the executor's withConnection paths have (DAG Executor §9).

3.4 Auth

Provider-field defaults below describe a new list entry. The shipped YAML declares Google and GitHub entries with their own endpoints, display names and picture hosts; both remain disabled until their client IDs are set. When overriding the provider list in a higher-precedence configuration source, supply the complete desired list: Spring's binder replaces the list, rather than merging each entry with the shipped list by index.

3.5 Results

Every completed execution's caller result is stored in Redis and read through the result cursor (REST API §7).

3.6 SSE

3.7 Rate Limiting

Limits are per user (an API key inherits its owner's budget — minting more keys does not raise the limit).

There is deliberately no fail-open key. When the limiter's Redis is unreachable the limiter refuses the request (429 rate_limit.unavailable, REST API §12.3); a per-deployment toggle would only ever be reached for at exactly the moment the refusal is the correct answer.

3.8 Idempotency

3.9 Templates

3.10 UI

3.11 Execution History

Not a key: the scheduled jobs' statement bounds (#326, #636). The stale sweep's UPDATE and the dashboard refresh sweep's UPDATE are cancelled after 10 s and the hourly retention's metadata statements (the execution_events DELETE, the audit retention's cutoff read and batches, the keys purge's FK read and two DELETEs, the dashboard refreshes' DELETE, the parameter evaluations' sweep UPDATE and batch DELETE) after 25 s — a JDBC queryTimeout, failing the tick with SQLState 57014 (Metadata DB §8.1, §8.3, §8.4, §8.5). They are code constants beside the cadences they serve (SWEEP_STATEMENT_TIMEOUT_SECONDS, REFRESH_SWEEP_STATEMENT_TIMEOUT_SECONDS, RETENTION_STATEMENT_TIMEOUT_SECONDS), as the cadences are, not keys: each is set by its own cadence and the 30 s shutdown phase, not by a deployment.

3.12 Audit

3.13 Server

3.14 Framework wiring keys

These framework key paths appear in application.yml as internal wiring. They are listed here so the §1 authority sentence stays literally true. Except for the management listener settings below, these are shipped framework wiring, not ordinary tuning controls. Preserve them when supplying an external configuration file:

Operator-tunable exception:

3.15 Observability

3.16 Pipelines

3.17 Workspaces

Workspaces are created by super admins (auth.md §4.2/§5.6, §12). The one workspace the product ships is demo, and a user with no membership becomes a viewer of it on first login.

Removed in RBAC round 1 (provisioning-mode, open-join). Capability moved onto the workspace membership, and with it went the modes that decided who could create a workspace: auto-per-user (a personal workspace per login), self-serve (anyone creates) and closed (admin only), plus open-join (anyone self-joins). Both keys are refused BY NAME at startup (§7) rather than ignored — a deployment that still says auto-per-user is a deployment expecting a personal workspace per user, and silently giving it something else is how an operator finds out from a user. Delete the key; there is no replacement to set, because the behaviour is no longer a knob.

3.18 Bootstrap

Config-declared content applied at startup (design 2026-08-16-sample-data §6/§6.1). Both keys name files already on the container's filesystem — the app never fetches an artifact at runtime; downloading and verifying artifacts is a deployment step (D5) — and for both, unset (or empty) means the feature is off. There is no separate enable flag to disagree with the path.

Both values are comma-separated LISTS of paths (one entry per sample-data family; the demo profiles compose the list from the active families). A single value with no comma is the one-file shape older deployments ship — the list semantics are backward compatible by construction. Empty entries (a leading comma when a family is off, or a whitespace-only entry) are dropped, so a list built by shell conditional expansion never turns the feature on by accident. The files are processed in declared order.

Cross-key rule: datasources-file set while datapipelines.auth.bootstrap-admin-email is unset is a startup refusal naming both keys. Bootstrap-registered datasources are created_by that user, and registration runs before anyone has logged in, so the row is pre-provisioned from that address (Auth §4.4). examples-file carries no such rule: demo content seeding uses the system actor at startup.

demo-api-key (v1.29): its explicit environment variable is DATAPIPELINES_DEMO_API_KEY. Blank is the kill switch: on restart, the seeder retracts managed demo endpoints and revokes managed demo keys. The shipped default is the committed public value the demo-data page renders. The validator exempts that exact default under hardened; an operator-selected non-default value under hardened is refused at boot (§7). Seeding publishes no endpoints and no key when no demo family produced pipelines for it; it also retracts previously managed rows in that case.

No cross-key rule for examples-file (dropped in v1.9 with the provisioning modes, §3.17). The rule refused the file while provisioning-mode was anything but auto-per-user, because only that mode provisioned the personal workspace the examples were seeded into. Since RBAC round 1 the seeder runs when demo is created, in every deployment, so there is no mode left for the file to be unreachable under and ConfigValidator carries no such check. The file is still read and structurally checked at startup (fail-fast on an unreadable or unparseable file).

3.19 Deployment

The deployment-role settings (Versioning §5.5) — grouped one-per-concern like auth, executor, staging.

Renamed in 075. datapipelines.deployment.name — the deployment LABEL — is now datapipelines.env (DATAPIPELINES_ENV), documented in §3.23 beside the posture it is deliberately separate from. The old key is honoured for one release: set alone it still names the deployment and startup logs event=config.deployment_name_deprecated; set together with datapipelines.env and disagreeing, startup is refused naming both. See Environments.

Promotion (the deployment-to-deployment channel)

Promotion is one deployment writing RELEASED content into exactly one configured higher environment (Versioning §10). The credential is a server-kind API key (or the deprecated shared config secret below), presented as DP-Promotion-Key. PromotionServerKeyFilter creates a principal with the promotion_receiver key role: a stored key uses its key identity, while the legacy config secret uses the system actor. A deployment may hold the receiver half, the sender half, both, or neither.

The receiver half is now a KEY, and the config value below is DEPRECATED (091). Mint a server-kind API key on the API screen (Auth §7.7) and give it to the sending deployment as its target.server-key. A stored key is everything a config value cannot be: expiring, revocable, listed beside every other credential, rotatable without a restart, and stamped with a last-used time. datapipelines.deployment.promotion.server-key is still accepted for one release so an upgrade does not break an existing pair — and it WARNs at boot when set (§7). Both credentials open the same routes; the receiver tries the configured value first, then the key store.

On the SENDER, the value is whatever the receiver issued: the dpk_<id>.<secret> plaintext of the receiver's server key, shown exactly once at minting. To rotate, mint a second server key on the receiver, update the sender's secret and restart the sender, verify promotion, then revoke the first key. The receiver needs no restart for stored-key minting or revocation. For the deprecated config secret, generate it with openssl rand -base64 32, configure the same value at both ends and restart both. Stored server keys authenticate as their own key identities, not the human who created them; revocation, expiration, a deactivated identity or a deactivated pinned workspace can refuse the credential.

Both keys are bearer secrets. They are never logged, never in an error message, never in an audit details map, and never in a toString(). A receiver records only a truncated SHA-256 fingerprint of the key a push presented, so two keys are distinguishable across a rotation without either being carried.

3.20 Credential key provider

Where the AES data keys for datasources.credential_encrypted come from (Datasources §7.1). The same encryptor opens legacy sealed login-key copies through SecretSealer; newly minted keys do not store such copies. Keep old keys while any ciphertext still needs their version. The seam permits a customer's AWS/GCP/Azure/Vault key store implementation without changing the crypto — Key providers is the implementer's guide; only the env implementation ships. Every ciphertext carries the key version as its first byte, which makes rotation lazy-safe.

Every configured key is a bearer secret. None of them is logged, put in an error message, put in an audit details map, or rendered by a toString() — the §7 violations name the property and the defect, never the value.

Rotating. Generate a key (openssl rand -base64 32), add it as version N+1, set encryption-key-current: N+1, restart. New writes carry N+1; rows written under earlier versions keep decrypting, and are rewritten under the current key the next time their password is saved. Keep the old keys configured until no row still carries their version — Datasources §7.3 has the one SQL query that answers that.

3.21 Organisation

The organisation's own facts — its currency, when its fiscal year starts, which day its week starts on, which timezone "today" means. They are configuration, not pipeline data: identical for every pipeline in the deployment, changed about once a decade, and a value that differs between deployments must be visible in the deployment's yml rather than copied into every body.

Every key here enters every execution's Context as an org_* key — the yml path minus the datapipelines.org. prefix, with dots and dashes as _ (Pipeline Contract §7.2). A node binds :org_currency_symbol; a CALCULATOR node references $org_fiscal_start_date. None of them is a secret, and a pipeline may override any of them by declaring a parameter of the same name.

Env vars follow §1's derivation rule: DATAPIPELINES_ORG_CURRENCY_NAME, DATAPIPELINES_ORG_CURRENCY_SYMBOL, DATAPIPELINES_ORG_FISCAL_START_DATE, DATAPIPELINES_ORG_WEEK_START, DATAPIPELINES_ORG_TIMEZONE.

Restart to change. There is no runtime override and no per-user setting: an org value that could change mid-run would make two nodes of one execution disagree about what year it is. A bad value stops the server (§7) rather than silently defaulting — a wrong fiscal start is a wrong number in every report the deployment produces.

3.22 Published endpoints

The timeout bounds for a released pipeline published as a GET endpoint at /api/<category>/<version>/<path…> (round 074, re-rooted by 172's R-EP5). timeout-default-seconds is what a publish that names no timeout is stored with. The min/max pair clamps requested values at publish time and clamps the stored value again when serving. Retuning the bounds therefore changes the wait budget of existing endpoints after restart without rewriting their rows. The wait expiring returns 202; it does not cancel the pipeline execution, which keeps its own executor deadlines.

There is deliberately no page-rows-max key here. The DP-Result-Page-Rows request header is one contract across a published endpoint and POST /pipelines/{id}/execute, and both clamp it to datapipelines.result.page-max-rows (§3.5). A second key that had to equal the first would be a second authority for one bound, and an operator who retuned one and not the other would get two different clamps on one documented header.

3.23 Environment and posture

The two variables an organisation sets (round 075). Environments is the page written for the DevOps engineer who deploys this; this section is the key reference.

The environment's NAME belongs to the org; the POSTURE belongs to the product. Organisations have dev, qa, uat, perf, sandbox-eu, prod — any names, any count — and the product must never branch on one: a single-server user honestly writes prod as their label, and the first if (env == "prod") anywhere locks them out of authoring on the only server they have. What the product branches on is a closed, two-valued posture with the documented semantics below, and the org maps each of its environments onto one.

The posture table (normative)

The posture IS the Spring profile

spring.profiles.active is derived from DATAPIPELINES_POSTURE in application.yml, and application-development.yml / application-hardened.yml carry the posture's non-secret defaults (the first two rows above). One variable therefore gives every loader — Compose, a bare java -jar, systemd, Kubernetes, ECS, Nomad — the right defaults, and nothing the product needs lives only in a launcher. Setting SPRING_PROFILES_ACTIVE yourself is unnecessary; a value that disagrees with the posture refuses startup (§7) rather than silently loading one posture's defaults while every posture RULE judges the other. 075 renamed the dev profile to development; a deployment still asking for dev is refused, not ignored.

Nothing lives only in a compose file

The contract is environment variables, in two files: deploy/env/defaults.env (tracked — every non-secret variable this build binds, with the value it ships) and deploy/secrets.env (git-ignored — every credential, plus every override that belongs to this deployment). deploy/secrets.env.example is the template for the second and names every variable you may set. scripts/compose-env-audit.sh — which runs on every ./gradlew build — fails when those files, deploy/compose.yml and application.yml disagree about which variables exist or what they default to, and when a variable is declared in both tracked files or in neither. Settings are tracked in the repo; secrets never are.

3.24 Lake datasource engine limits (dp-lake)

A LAKE datasource runs its queries on the app's own box — DuckDB is embedded in the app process, so its memory, threads and spill disk are operator knobs, not someone else's infrastructure. These are datasource properties (properties.dialect.*, set per datasource through the REST API or a bootstrap datasources file — Datasources §12.1), not datapipelines.* keys: they do not belong in application.yml or an env file, they are validated per key at save time (an unknown key or a bad value is refused), and a change takes effect at the datasource's next pool build, with no app restart. The existing execution bounds apply unchanged beside them: the result row cap and node-query-timeout-seconds.

Scope: one engine per datasource pool, shared by its connections (152). Each LAKE datasource pool build opens ONE embedded DuckDB instance that every pooled connection joins (Datasources §5.2); the three properties below are applied once to that instance and are ONE budget for the pool — properties.hikari.maximumPoolSize does not multiply them, and concurrent queries on the datasource share them. Size the box for memory_limit × the number of LAKE datasources on the instance (each has its own engine), with headroom for a retiring pool's engine while it drains after a datasource save. The engine's file cache lives with that instance, so a warm lake stays warm across HikariCP's routine connection replacement; a datasource save or a lake-table registration rebuilds the pool and starts a fresh engine, cold.

For the supported anonymous in-memory URLs (jdbc:duckdb: and jdbc:duckdb::memory:), the default spill directory belongs to one engine initialization: pooled connections share it, while independent in-memory pool generations receive different paths. DuckDB creates it lazily when spilling and removes it on normal instance shutdown. The parent directory must exist, be writable by the application user, and have enough disk space for concurrent queries; the Helm chart provides disk-backed /tmp. A bare JVM can select the parent with -Djava.io.tmpdir=/writable/path. An explicit override is used exactly as supplied: dedicate that directory to the engine, keep unrelated files out, and avoid sharing it between independent engines or overlapping generations. Named in-memory and file-backed URLs can join an existing JVM-wide engine; when no override is supplied, its current spill setting is left untouched. Configure a writable path explicitly for those URLs, consistently across every pool sharing that engine. A crash or a driver that refuses shutdown can leave spill files behind; reclaim them only after the owning process has stopped.

threads is deliberately left unset, and the measurement is why — though not the way the hypothesis expected (108 §C). In-process DuckDB shares the JVM's CPUs with the executor, so the obvious guard is to cap the engine's worker threads; max(2, cores/2) was the proposal. Measured on a 10-core box as the median of 5 PAIRED runs (each pair times the scan alone and then immediately under load, so a drifting box moves both numbers):

A 0.92× is a scan that ran FASTER under load than alone, which is not a result — it is the noise floor. On a box shared with other work the contention effect is smaller than the measurement's own variance, and an earlier single-run shape produced swings from 1.13× to 1.82× for the same cell. So the honest reading is: no evidence for a thread cap, and none against one either — the effect this knob would manage is not resolvable here. The default therefore stays unset (DuckDB's own scheduler handles oversubscription), an operator who needs the engine bounded sets threads explicitly, and anyone who wants to settle it should re-run scripts/measure/03-pressure.sh on a dedicated box.

What the runs DO agree on: a lake read on a busy box loses on the order of 10 % or more, and no threads value observed changed that.

Every LAKE engine initialization emits SET memory_limit = '<explicit-or-default>' (there is always a budget) and SET preserve_insertion_order = false. These apply to the shared instance for the pool; opening another pooled connection does not create a separate memory budget. Preserving insertion order is not configurable through the datasource properties.

3.25 DuckDB extension directory (dp-lake)

A LAKE datasource whose data is on S3 needs DuckDB's httpfs and aws extensions, and the Iceberg catalog kinds additionally need iceberg (which itself requires avro). Where those binaries come from is this key. It is an operator-level key — one value for the whole deployment, restart-to-change — deliberately NOT a per-datasource properties.dialect.* entry: which extension files the engine may load is a deployment security posture, not row data.

Operator instruction: set DATAPIPELINES_DUCKDB_MEMORY_LIMIT=8GB in deploy/secrets.env (or the deployment's own env file) and restart the app — the value is read once at wiring, not per pool build, so every LAKE datasource without its own properties.dialect.memory_limit picks it up on its next pool build after the restart (a fresh cold engine; the first query after restart is cold, the next warm). Confirm it took effect over a LAKE connection with SELECT current_setting('memory_limit') (reports the exact configured value, e.g. 8.0GB) or SELECT * FROM duckdb_memory() (the engine's own memory-accounting view, docs/staging.md §"Memory accounting" — its buffer manager budget reflects the same limit). Helm operators (deploy/helm/) set these two keys through the chart's duckdb.memoryLimit and duckdb.extensionDirectory values instead of an env file — the chart renders them as these env vars when non-empty (an empty value renders no entry, inheriting the image's bundled extension directory), and an extraEnv double mapping of either is refused at render time (#140, deployment.md §6.4).

Set (the shipped image): a lake with catalog.kind configured initializes its engine with SET extension_directory = '<dir>' followed by bare LOADs — LOAD httpfs; LOAD aws for catalog.kind: s3, plus LOAD avro; LOAD iceberg for the Iceberg kinds — and never an INSTALL. The catalog adapter emits none when catalog.kind is absent; registered Iceberg tables have their own extension setup, described below. Loading the pre-populated files needs no extension-download egress; this does not remove the network access required by a remote catalog or its data. The published image bundles the four extensions for DuckDB core v1.5.5 under /opt/duckdb/extensions/v1.5.5/<platform>/, the <platform> following the image build's architecture (linux_amd64 or linux_arm64 — the Dockerfile maps BuildKit's TARGETARCH; hard-coding one architecture leaves the other LOAD-only against an empty directory, a pool-init failure at connect — found by the 089 live gate), and exports this variable from the Dockerfile; deploy/compose.yml defaults to the same path. The bundle adds ~108 MB uncompressed to the image (~39 MB downloaded at build). A hardened dp-lake deployment with the bundled directory needs no egress to extensions.duckdb.org at all.

Unset (a bare java -jar, a developer machine): when catalog.kind is configured, the adapter emits explicit INSTALL+LOAD pairs for its required extensions. A missing extension is downloaded into DuckDB's extension cache; a fresh ephemeral filesystem has no previous download to reuse. Independently of catalog.kind, a registered Iceberg table's view initialization loads avro and iceberg, using the same configured-directory/INSTALL choice. A lake with neither catalog.kind nor registered Iceberg tables emits no extension setup statements.

Version coupling: the bundled binaries are valid for exactly one DuckDB core version. The pinned duckdb_jdbc is 1.5.5.1, whose bundled core reports v1.5.5 — the directory's v1.5.5 component is the CORE version, not the JDBC patch version. A duckdb_jdbc upgrade must re-download all four extensions and rename the directory accordingly (the Dockerfile pins both in one ARG).

3.26 Datasource pools

How long a retired connection pool may keep connections out before it is closed regardless (Datasources §5.2, round 094). A datasource that is edited or deleted has its pool taken out of the live map at once and soft-evicted; the pool itself is closed when the statements already running on it finish — or at this ceiling, whichever comes first, so a genuinely hung statement cannot pin a deleted datasource's pool forever. Each hard close logs one WARN naming the datasource and the connections it took down, and increments datapipelines.datasource.pool.hard_closed (Observability §4.1).

Why the default is derived rather than a number. The ceiling and the node query timeout are the same fact seen twice: a deployment that raises node-query-timeout-seconds to 600 and leaves a literal 90 here would start hard-closing pools out from under statements that are still legitimately running, and nothing would tell it. Deriving makes the two move together; an explicit value opts out.

3.27 Mail

Outbound mail (Auth §5A.8) — the SMTP transport for welcome/password-reset mail, new-user notices to sys-ops, and scheduler notifications when separately enabled (§3.29). It uses host, port, username, password and STARTTLS; Postmark's message-stream header is optional. SMTP connect time is bounded to 10 seconds and read/write time to 30 seconds in SpringMailSender; these bounds are code constants, not additional settings.

Mail is enabled exactly when it is configured — there is no enabled key. Mail is on iff host AND from are both set. A flag that could disagree with the fields it summarises is the YAML-boolean trap the house has already logged; "configured" is the only honest definition of "on" for an outbound adapter. Every half-configured shape refuses startup naming the field (§7).

Cross-key rule: mail on requires datapipelines.auth.base-url (§3.4) — the welcome mail carries the login URL, which is built from it and never from a request's origin (Auth §5.2).

What the admin screen shows follows from the derivation: with mail on, the create-user and reset-password actions no longer display the one-time password — it went to the user, and the screen says so with the send's outcome; with mail off they show it as before (UI §4.12).

3.28 Script evaluation (7b, #7)

The budgets every script evaluation runs under — the save/release test suite of a script template, templates_evaluate / POST /api/v1/templates/evaluate, and the SCRIPT node (7c). The pool is the record's §4.3 bulkhead: at most pool-size evaluations run at once, pool-queue bounds the evaluations ADMITTED in total (running plus waiting — an abandoned evaluation keeps its slot until its thread ends), and a submission that cannot be admitted or that waits its whole wall clock for a slot is the record §4.3's pool-refusal (§13.18, 7c).

Operator instruction: Helm operators (deploy/helm/) set these nine keys through the chart's script.js.* values instead of an env file (deployment.md §6.4) — all nine empty by default, an empty value rendering no env entry (the image's own isolate directory and the app's derived isolate memory stay in force), an isolateDir outside the image's launcher-path entries refused at render time, and an extraEnv double mapping of any of the nine refused like the DuckDB keys.

Removed keys (script-node-rename lane, owner ruling 2026-10-20): the nine budgets lived at datapipelines.transform.* before the TRANSFORM→SCRIPT node-type rename. There is no alias: a datapipelines.transform.* key binds nothing, and startup refuses BY NAME (§7, checkRemovedTransformKeys) telling the operator to rename it. This is the §3.17 rule again — a removed key is refused for being set at all, value notwithstanding.

3.29 Scheduler (#9)

The scheduler's knobs (Scheduler). Every instance serves the schedule REST routes; enabled decides whether THIS instance also dispatches. The library underneath (db-scheduler) reads its own db-scheduler.* prefix, and application.yml feeds it from these keys (§5) — the library's own defaults (ten threads, a 30-minute shutdown wait) never apply, and db-scheduler.* is not an operator surface. Each bound below is enforced when the keys bind: a value outside it refuses startup, naming the key.

Scheduled runs keep the instance's execution timeout (datapipelines.executor.execution-timeout-seconds) and ask for the maximum result TTL (datapipelines.result.ttl-max-seconds); there is no scheduler-specific budget in v1. One Spring Boot key is set with them: management.health.db-scheduler.enabled: false — the library's health indicator reports DOWN on an API-mode instance (it never starts there), and a dispatcher stall must not restart a pod; watch the scheduler's metrics instead (Observability §4).

3.30 Parameter engine (#194)

The parameter engine's limits (the design record §11). The save-time validator reads the collection, expression and option caps (every one enforced BEFORE the set's graph is built, and the collection caps before their members are even read), and the shared value validator reads max-regex-steps and max-input-length — the latter is the ENGINE's max_length default for a STRING or BINARY INPUT that declares none (a pipeline parameter keeps none, record P34). The evaluate keys (the timeouts, the selector bulkhead, the binds and the response budget) are read by the runtime. Each bound below is enforced at boot (§7) and again when the keys bind (ParametersProperties → ParametersConfig), naming the key.

No options cache in round one (selector-cache-ttl-seconds is deliberately absent — the record's §11: authority-aware keys and invalidation are its price, and it lands when a measured query count says it must).

Retention of the evaluation history (parameter_evaluations and their parameter_evaluation_queries, #376) has no key of its own — the dashboard refreshes' precedent: finished records are deleted by the execution-event retention step, on the same hourly tick and the same cutoff as datapipelines.executions.event-retention-days (§3.11, 7 days), one bounded batch per tick; a record left RUNNING past evaluate-timeout-seconds plus a fixed one-minute margin is closed INCOMPLETE on that tick — see Metadata DB §8.1/§8.5. A retention key of its own would be an additive change (the workspace spec's §11.5).

3.31 Web request limits

The request-body cap (pipeline-contract §13.21, REST API §4.2) applies to BOTH JSON surfaces — /api/v1 and /mcp. The servlet filter runs before authentication: an over-cap Content-Length is refused unread, while a body without that declaration is counted as downstream code consumes its wrapped stream and is refused on a read beyond the cap. The usual response is 413 request.body_too_large; the screenshot upload exception is stated in the table. The filter does not buffer the body. Jackson's request StreamReadConstraints — nesting depth 100, string length 4,000,000 characters, number length 1000 digits — are stated explicitly on the request mappers of both surfaces. A within-cap body that exceeds those parser limits is refused as a 400; routes parsing their own text body use a constrained copy of their domain mapper (pipeline-contract §13.21 records the scope). These parser limits are constants, not configuration keys.

The bound below is enforced at boot (§7, checkRequestLimits) and again when the key binds (RequestLimitsProperties).

Tomcat's swallow budget is the cap's counterpart on the connection: server.tomcat.max-swallow-size (default 2MB, §3.13) bounds how much of a REFUSED body the container drains before closing it, so an abandoned over-cap upload costs the connection, never the server.

3.32 Persistence batching (#266)

The batching writers in front of three stores (DAG Executor §10, Auth §10): the audit log (audit_log), the durable execution-event record (execution_events) and the one-hour Redis replay log. The audit log's writer ships switched off (audit.enabled, below): by default only the two execution-event stores are batched. Callers await a bounded durability attempt; an execution-event write that remains unresolved at its wait bound is reported as indeterminate, not claimed committed. The execution row itself (pipeline_executions: the RUNNING insert and the terminal UPDATE) is never batched. Each store gets its OWN writer and admission accounting. The byte counts are sink-provided item sizes, not measured heap, and each empty writer may admit one item larger than its byte bound; do not treat three times queue-max-bytes as an absolute process-memory cap. Every bound is validated when the keys bind (PersistenceProperties → BatchingConfig), naming the key.

What these keys never change: an audit row written inside the caller's transaction joins that transaction (it commits and rolls back with it; Auth §10); per-key and per-execution order; the retry dedup (the event record's ON CONFLICT (execution_id, event_id), the replay's first-copy-per-event-id); the loss window, which exists only for the non-awaited submit path — no production caller yet (Observability §7).

3.33 Visualizations and dashboards (#10)

The bounds of the two dashboard documents' strict readers (Dashboards §2, the dashboard implementation spec §3). Each collection bound is checked BEFORE the collection's members are read, so a document cannot make the read work in proportion to its own size; a document over one is refused visualization.validation.body_invalid / dashboard.validation.body_invalid with details.reason too_many (or too_large for the configuration) and details.config_key naming the key. The runtime's keys are in §3.34. The test-session TTL uses the flat session-ttl-minutes key — the shipped key grammar is exactly two levels (datapipelines.visualization.<key>); §3.33's row is its definition. Each bound below is enforced at boot (§7) and again when the keys bind (VisualizationProperties → VisualizationConfig), naming the key.

A renderer configuration also nests at most 32 levels and a literal parameter value at most 2 (a scalar, or a list of scalars) — constants, like the request mappers' Jackson constraints (§3.31), refused too_deep by a bounded walk that never uses the host stack.

3.34 The dashboard runtime (#10 L2)

The server side of a dashboard refresh (Dashboards §5, the implementation spec §9.4–§9.6): how many refreshes and executions may run at once, how many bytes a source and a refresh may hold in memory, and the deadlines. Every default is the number the owner confirmed on 2026-09-28. Each key is bound-checked at boot (§7) and again when the keys bind (DashboardRuntimeProperties → DashboardRuntimeConfig), naming the key; four RELATIONS are refused and one is warned (below the table).

The admission counters are JVM-LOCAL, like ExecutionSlots (the parameter-engine record's 050/R2 shape): two API instances each admit max-concurrent-refreshes-per-workspace refreshes of one workspace and max-concurrent-dashboard-executions-per-instance executions. A cluster-wide count is a later decision; nothing here is a per-user cap — a dashboard execution takes no per-user slot (D53), so a refresh never starves the viewer's own runs.

The five relations, each judged at boot naming both keys and values. Four are REFUSED: default-refresh-seconds ≤ max-refresh-seconds; max-bytes-per-source ≤ max-bytes-per-refresh; max-executions-per-refresh ≤ max-concurrent-dashboard-executions-per-instance; and max-executions-per-refresh ≤ the executor's max-concurrent-executions-per-instance (§3.2; the deprecated …-global alias's value when that is what runs) — the last two because a maximal refresh could otherwise never be admitted, a permanent 429. One is a WARNING: max-concurrent-dashboard-executions-per-instance ≤ the executor's ceiling (a higher dashboard cap is permitted and still limits overlapping refresh reservations; executor leases release earlier, per execution). A refresh that needs more distinct executions than the smaller of max-executions-per-refresh and the executor's ceiling is refused at once — not after max-wait-seconds — as 429 dashboard.refresh.saturated with details.reason = never_admissible, the count, the ceiling and its key (Pipeline contract §13.23); a genuinely full instance answers details.reason = busy. Save and promotion import both validate against this instance's max-executions-per-refresh, so only a dashboard saved or imported before a bound was lowered can reach that state; saved dashboards whose static all-source invocation upper bound exceeds the ceiling are named once at boot in a WARN event=config.dashboards_above_refresh_ceiling (workspace/name@version, each upper bound, the ceiling and its key). This is a conservative warning: equal evaluated bindings or a targeted refresh can need fewer executions. Review the dashboard and ceiling before raising the key or reducing sources. The earliest deadline wins: the refresh's own, an occurrence's timeout_seconds, and the source's own executor limits.

Shipped environment names: this group's placeholders omit the intermediate YAML group. Use the names below with the packaged application.yml; do not derive them by inserting ADMISSION, RESULTS or TIMEOUTS.

Retention of the dashboard_refreshes rows has no key of its own: they are deleted by the execution-event retention step, on the same hourly tick and the same cutoff as datapipelines.executions.event-retention-days (§3.11, 7 days) — see Metadata DB §8.

3.35 Release dependencies

The bounds a cascading release works inside (Versioning §5.3, the release-dependencies record §1.2, ruled §17 row 4), and the workspace mutation guard's lock wait (record §4.3). The closure of a release — every version the root pins, transitively, across the dependency sources registered by its caller — is enumerated with an explicit stack, memoised per exact version. The shipped cascading coordinator registers pipeline and template sources. Depth, version and edge limits are checked during that walk; wire size is checked when the plan request is read and when canonical bytes are produced. Exceeding those limits raises <family>.release.dependency_limit (422, details.limit naming the bound and details.maximum its value). max-blockers limits refusal detail, not closure traversal. Each key is range-checked when the keys bind (ReleaseDependencyProperties), refusing startup by key name. The mutation lock and blocker-detail bounds are also shared by other family lifecycle paths that receive ReleaseBounds.

4. Precedence

There are two separate resolution steps: the deployment loader constructs the process environment, then Spring resolves application properties.

For a normal bare-jar launch, the relevant Spring sources run from command-line --key=value, through SPRING_APPLICATION_JSON, JVM -Dkey=value, process environment, and then configuration files (profile files override the corresponding base file). Additional external configuration locations can also override packaged files. The product does not install a custom command-line parser: DatapipelinesApplication.main delegates to Spring Boot. Avoid setting the same key through several mechanisms.

The ${ENV:default} expressions in §5 are explicit lookups: a supplied environment value replaces that expression's fallback. They also explain why the documented underscore-separated environment names work even when they differ from Spring's relaxed-binding spelling. The nested posture expression accepts either DATAPIPELINES_POSTURE or the dotted property; §7 checks profile/posture alignment.

For app.sh, Compose reads deploy/env/defaults.env then deploy/secrets.env; the later file overrides the earlier file, and exported shell variables override Compose file interpolation. The --env, --posture and --demo options are explicit launcher overrides. A Compose env file is an interpolation source, not an instruction to forward every variable into the container: the service's environment mapping decides what reaches the JVM. For a bare jar or systemd, the loader must export the values itself; the jar does not read these .env files.

Runtime object overrides are separate from startup precedence. Examples: datasource/node/pipeline statement budgets (§3.2), pipeline staging settings (§3.3), datasource LAKE limits (§3.24), and a user's theme preference (§3.10). Use those rows for exact ordering and ceilings; no single global precedence rule applies to all of them.

5. Full application.yml Template

Complete — a deployment assembled from this block gets the framework wiring (§3.14) too. Omitting the management: block in particular re-serves the actuator on the application port, which is exactly the exposure the management port exists to prevent.

spring:
  application:
    name: datapipelines
  datasource:
    url: ${SPRING_DATASOURCE_URL}
    username: ${SPRING_DATASOURCE_USERNAME}
    password: ${SPRING_DATASOURCE_PASSWORD}
    hikari:
      maximum-pool-size: ${SPRING_DATASOURCE_HIKARI_MAXIMUM_POOL_SIZE:10}

  flyway:
    enabled: true
    locations: classpath:db/migration
    baseline-on-migrate: false

  profiles:
    active: ${DATAPIPELINES_POSTURE:${datapipelines.posture:}}

server:
  port: ${SERVER_PORT:8080}
  shutdown: graceful
  tomcat:
    use-relative-redirects: true
    max-swallow-size: ${SERVER_TOMCAT_MAX_SWALLOW_SIZE:2MB}

management:
  server:
    port: ${MANAGEMENT_SERVER_PORT:9090}
    address: ${MANAGEMENT_SERVER_ADDRESS:127.0.0.1}
  endpoints:
    web:
      exposure:
        include: "health"
  health:
    diskspace:
      enabled: false
    db-scheduler:
      enabled: false

datapipelines:
  redis:
    host: ${DATAPIPELINES_REDIS_HOST}
    port: ${DATAPIPELINES_REDIS_PORT:6379}
    password: ${DATAPIPELINES_REDIS_PASSWORD:}
    command-timeout: ${DATAPIPELINES_REDIS_COMMAND_TIMEOUT:2s}
    connect-timeout: ${DATAPIPELINES_REDIS_CONNECT_TIMEOUT:2s}

  jwt:
    secret: ${DATAPIPELINES_JWT_SECRET}

  db:
    key-provider: ${DATAPIPELINES_DB_KEY_PROVIDER:env}
    encryption-key: ${DATAPIPELINES_DB_ENCRYPTION_KEY}

  auth:
    base-url: ${DATAPIPELINES_AUTH_BASE_URL:}
    bootstrap-admin-email: ${DATAPIPELINES_AUTH_BOOTSTRAP_ADMIN_EMAIL:}
    oidc:
      provider-read-timeout-seconds: ${DATAPIPELINES_AUTH_OIDC_PROVIDER_READ_TIMEOUT_SECONDS:10}
      providers:
        - name: google
          client-id: ${GOOGLE_CLIENT_ID:}
          client-secret: ${GOOGLE_CLIENT_SECRET:}
          issuer-uri: https://accounts.google.com
          display-name: "Sign in with Google"
          picture-hosts: ${DATAPIPELINES_AUTH_GOOGLE_PICTURE_HOSTS:lh3.googleusercontent.com}
        - name: github
          protocol: oauth2
          client-id: ${GITHUB_CLIENT_ID:}
          client-secret: ${GITHUB_CLIENT_SECRET:}
          authorization-uri: https://github.com/login/oauth/authorize
          token-uri: https://github.com/login/oauth/access_token
          user-info-uri: https://api.github.com/user
          user-name-attribute: id
          scopes: [read:user, user:email]
          emails-uri: https://api.github.com/user/emails
          display-name: "Sign in with GitHub"
          picture-hosts: avatars.githubusercontent.com
    jwt:
      ttl-hours: ${DATAPIPELINES_AUTH_JWT_TTL_HOURS:8}
    allowlist:
      domains: ${DATAPIPELINES_AUTH_ALLOWLIST_DOMAINS:}
    api-keys:
      cache-ttl-seconds: ${DATAPIPELINES_AUTH_API_KEYS_CACHE_TTL_SECONDS:60}
    rate-limit:
      login-per-minute: ${DATAPIPELINES_AUTH_RATE_LIMIT_LOGIN_PER_MINUTE:10}
    trusted-proxies: ${DATAPIPELINES_AUTH_TRUSTED_PROXIES:}
    local:
      enabled: ${DATAPIPELINES_AUTH_LOCAL_ENABLED:false}
      bootstrap-password-hash: ${DATAPIPELINES_AUTH_LOCAL_BOOTSTRAP_PASSWORD_HASH:}
      bootstrap-password: ${DATAPIPELINES_AUTH_LOCAL_BOOTSTRAP_PASSWORD:}
      lockout:
        max-failures: ${DATAPIPELINES_AUTH_LOCAL_LOCKOUT_MAX_FAILURES:5}
        duration-minutes: ${DATAPIPELINES_AUTH_LOCAL_LOCKOUT_DURATION_MINUTES:15}
    cookie-secure: ${DATAPIPELINES_AUTH_COOKIE_SECURE:}
    allow-local-only: ${DATAPIPELINES_AUTH_ALLOW_LOCAL_ONLY:false}

  executor:
    max-parallel-nodes: ${DATAPIPELINES_EXECUTOR_MAX_PARALLEL_NODES:4}
    max-concurrent-executions-per-user: ${DATAPIPELINES_EXECUTOR_MAX_CONCURRENT_EXECUTIONS_PER_USER:10}
    max-concurrent-executions-per-instance: ${DATAPIPELINES_EXECUTOR_MAX_CONCURRENT_EXECUTIONS_PER_INSTANCE:100}
    node-query-timeout-seconds: ${DATAPIPELINES_EXECUTOR_NODE_QUERY_TIMEOUT_SECONDS:60}
    execution-timeout-seconds: ${DATAPIPELINES_EXECUTOR_EXECUTION_TIMEOUT_SECONDS:600}
    node-timeout-seconds: ${DATAPIPELINES_EXECUTOR_NODE_TIMEOUT_SECONDS:300}
    node-timeout-max-seconds: ${DATAPIPELINES_EXECUTOR_NODE_TIMEOUT_MAX_SECONDS:900}
    node-query-timeout-max-seconds: ${DATAPIPELINES_EXECUTOR_NODE_QUERY_TIMEOUT_MAX_SECONDS:900}
    node-query-timeout-seconds-by-dialect:
      lake: ${DATAPIPELINES_EXECUTOR_NODE_QUERY_TIMEOUT_SECONDS_BY_DIALECT_LAKE:180}
    cancel-grace-seconds: ${DATAPIPELINES_EXECUTOR_CANCEL_GRACE_SECONDS:5}
    source-fetch-size: ${DATAPIPELINES_EXECUTOR_SOURCE_FETCH_SIZE:1000}
    progress-write-interval-seconds: ${DATAPIPELINES_EXECUTOR_PROGRESS_WRITE_INTERVAL_SECONDS:5}
    progress-sample-interval-seconds: ${DATAPIPELINES_EXECUTOR_PROGRESS_SAMPLE_INTERVAL_SECONDS:1}
    heartbeat-seconds: ${DATAPIPELINES_EXECUTOR_HEARTBEAT_SECONDS:15}
    lifecycle-write-timeout-seconds: ${DATAPIPELINES_EXECUTOR_LIFECYCLE_WRITE_TIMEOUT_SECONDS:10}

  pipelines:
    max-composition-depth: ${DATAPIPELINES_PIPELINES_MAX_COMPOSITION_DEPTH:5}

  deployment:
    authoring-enabled: ${DATAPIPELINES_DEPLOYMENT_AUTHORING_ENABLED:true}
    promotion:
      server-key: ${DATAPIPELINES_DEPLOYMENT_PROMOTION_SERVER_KEY:}
      target:
        base-url: ${DATAPIPELINES_DEPLOYMENT_PROMOTION_TARGET_URL:}
        server-key: ${DATAPIPELINES_DEPLOYMENT_PROMOTION_TARGET_KEY:}
      inventory-cache-ttl-seconds: ${DATAPIPELINES_DEPLOYMENT_PROMOTION_INVENTORY_CACHE_TTL_SECONDS:60}

  workspaces:
    member-datasources-enabled: ${DATAPIPELINES_WORKSPACES_MEMBER_DATASOURCES_ENABLED:false}

  staging:
    h2:
      mode: ${DATAPIPELINES_STAGING_H2_MODE:PostgreSQL}
      max-memory-mb: ${DATAPIPELINES_STAGING_H2_MAX_MEMORY_MB:1024}
      insert-batch-size: ${DATAPIPELINES_STAGING_H2_INSERT_BATCH_SIZE:1000}
      result-batch-size: ${DATAPIPELINES_STAGING_H2_RESULT_BATCH_SIZE:10000}
      query-timeout-seconds: ${DATAPIPELINES_STAGING_H2_QUERY_TIMEOUT_SECONDS:60}
      max-connections: ${DATAPIPELINES_STAGING_H2_MAX_CONNECTIONS:4}

  result:
    ttl-default-seconds: ${DATAPIPELINES_RESULT_TTL_DEFAULT_SECONDS:300}
    ttl-min-seconds: ${DATAPIPELINES_RESULT_TTL_MIN_SECONDS:60}
    ttl-max-seconds: ${DATAPIPELINES_RESULT_TTL_MAX_SECONDS:3600}
    max-size-bytes: ${DATAPIPELINES_RESULT_MAX_SIZE_BYTES:104857600}
    page-size-rows: ${DATAPIPELINES_RESULT_PAGE_SIZE_ROWS:1000}
    page-max-rows: ${DATAPIPELINES_RESULT_PAGE_MAX_ROWS:100000}

  sse:
    heartbeat-interval-seconds: ${DATAPIPELINES_SSE_HEARTBEAT_INTERVAL_SECONDS:15}
    disconnect-grace-seconds: ${DATAPIPELINES_SSE_DISCONNECT_GRACE_SECONDS:30}
    max-streams-per-user: ${DATAPIPELINES_SSE_MAX_STREAMS_PER_USER:50}

  rate-limit:
    requests-per-second: ${DATAPIPELINES_RATE_LIMIT_REQUESTS_PER_SECOND:100}
    requests-per-minute: ${DATAPIPELINES_RATE_LIMIT_REQUESTS_PER_MINUTE:1000}

  idempotency:
    ttl-seconds: ${DATAPIPELINES_IDEMPOTENCY_TTL_SECONDS:86400}

  templates:
    cache-size: ${DATAPIPELINES_TEMPLATES_CACHE_SIZE:500}
    render-timeout-ms: ${DATAPIPELINES_TEMPLATES_RENDER_TIMEOUT_MS:5000}
    max-body-chars: ${DATAPIPELINES_TEMPLATES_MAX_BODY_CHARS:262144}

  ui:
    theme: ${DATAPIPELINES_UI_THEME:dark}
    actively-developed-days: ${DATAPIPELINES_UI_ACTIVELY_DEVELOPED_DAYS:14}

  executions:
    event-retention-days: ${DATAPIPELINES_EXECUTIONS_EVENT_RETENTION_DAYS:7}
    stale-timeout-minutes: ${DATAPIPELINES_EXECUTIONS_STALE_TIMEOUT_MINUTES:60}
    error-detail: ${DATAPIPELINES_EXECUTIONS_ERROR_DETAIL:full}

  audit:
    retention-days: ${DATAPIPELINES_AUDIT_RETENTION_DAYS:365}

  observability:
    tracing:
      enabled: ${DATAPIPELINES_OBSERVABILITY_TRACING_ENABLED:false}
    logging:
      format: ${DATAPIPELINES_OBSERVABILITY_LOGGING_FORMAT:json}

  bootstrap:
    datasources-file: ${DATAPIPELINES_BOOTSTRAP_DATASOURCES_FILE:}
    examples-file: ${DATAPIPELINES_BOOTSTRAP_EXAMPLES_FILE:}
    demo-api-key: ${DATAPIPELINES_DEMO_API_KEY:dpk_DEMOPUBLIC42.THISKEYISPUBLICBYDESIGNDEMOONLYREADONLY222222222}

  org:
    currency:
      name: ${DATAPIPELINES_ORG_CURRENCY_NAME:Dollar}
      symbol: ${DATAPIPELINES_ORG_CURRENCY_SYMBOL:$}
    fiscal-start-date: ${DATAPIPELINES_ORG_FISCAL_START_DATE:01-01}
    week-start: ${DATAPIPELINES_ORG_WEEK_START:monday}
    timezone: ${DATAPIPELINES_ORG_TIMEZONE:UTC}
  endpoints:
    timeout-default-seconds: ${DATAPIPELINES_ENDPOINTS_TIMEOUT_DEFAULT_SECONDS:30}
    timeout-min-seconds: ${DATAPIPELINES_ENDPOINTS_TIMEOUT_MIN_SECONDS:1}
    timeout-max-seconds: ${DATAPIPELINES_ENDPOINTS_TIMEOUT_MAX_SECONDS:300}
    key-request-budget:
      window-seconds: ${DATAPIPELINES_ENDPOINTS_KEY_REQUEST_BUDGET_WINDOW_SECONDS:60}
      max-requests: ${DATAPIPELINES_ENDPOINTS_KEY_REQUEST_BUDGET_MAX_REQUESTS:60}

  env: ${DATAPIPELINES_ENV:local}
  posture: ${DATAPIPELINES_POSTURE:}
  demo: ${DATAPIPELINES_DEMO:}

  duckdb:
    extension-directory: ${DATAPIPELINES_DUCKDB_EXTENSION_DIRECTORY:}
    memory-limit: ${DATAPIPELINES_DUCKDB_MEMORY_LIMIT:}
  datasources:
    retire-ceiling-seconds: ${DATAPIPELINES_DATASOURCES_RETIRE_CEILING_SECONDS:}
    file-roots: ${DATAPIPELINES_DATASOURCES_FILE_ROOTS:}
  mail:
    host: ${DATAPIPELINES_MAIL_HOST:}
    port: ${DATAPIPELINES_MAIL_PORT:587}
    username: ${DATAPIPELINES_MAIL_USERNAME:}
    password: ${DATAPIPELINES_MAIL_PASSWORD:}
    starttls: ${DATAPIPELINES_MAIL_STARTTLS:true}
    from: ${DATAPIPELINES_MAIL_FROM:}
    reply-to: ${DATAPIPELINES_MAIL_REPLY_TO:}
    ops-to: ${DATAPIPELINES_MAIL_OPS_TO:}
    message-stream: ${DATAPIPELINES_MAIL_MESSAGE_STREAM:}
  script:
    evaluate-timeout-seconds: ${DATAPIPELINES_SCRIPT_EVALUATE_TIMEOUT_SECONDS:10}
    suite-timeout-seconds: ${DATAPIPELINES_SCRIPT_SUITE_TIMEOUT_SECONDS:60}
    pool-size: ${DATAPIPELINES_SCRIPT_POOL_SIZE:4}
    pool-queue: ${DATAPIPELINES_SCRIPT_POOL_QUEUE:64}
    abandon-grace-seconds: ${DATAPIPELINES_SCRIPT_ABANDON_GRACE_SECONDS:30}
    max-input-rows: ${DATAPIPELINES_SCRIPT_MAX_INPUT_ROWS:100000}
    max-value-bytes: ${DATAPIPELINES_SCRIPT_MAX_VALUE_BYTES:1048576}
    max-string-bytes: ${DATAPIPELINES_SCRIPT_MAX_STRING_BYTES:1048576}
    max-depth: ${DATAPIPELINES_SCRIPT_MAX_DEPTH:100}
    js:
      isolate-dir: ${DATAPIPELINES_SCRIPT_JS_ISOLATE_DIR:}
      max-heap-mb: ${DATAPIPELINES_SCRIPT_JS_MAX_HEAP_MB:256}
      isolate-memory-mb: ${DATAPIPELINES_SCRIPT_JS_ISOLATE_MEMORY_MB:}
      max-statements: ${DATAPIPELINES_SCRIPT_JS_MAX_STATEMENTS:100000000}
      max-ast-depth: ${DATAPIPELINES_SCRIPT_JS_MAX_AST_DEPTH:1000}
      max-stack-frames: ${DATAPIPELINES_SCRIPT_JS_MAX_STACK_FRAMES:1000}
      max-output-bytes: ${DATAPIPELINES_SCRIPT_JS_MAX_OUTPUT_BYTES:1048576}
      max-result-mb: ${DATAPIPELINES_SCRIPT_JS_MAX_RESULT_MB:64}
      max-input-mb: ${DATAPIPELINES_SCRIPT_JS_MAX_INPUT_MB:64}
  scheduler:
    enabled: ${DATAPIPELINES_SCHEDULER_ENABLED:true}
    threads: ${DATAPIPELINES_SCHEDULER_THREADS:2}
    polling-interval-seconds: ${DATAPIPELINES_SCHEDULER_POLLING_INTERVAL_SECONDS:10}
    heartbeat-interval-seconds: ${DATAPIPELINES_SCHEDULER_HEARTBEAT_INTERVAL_SECONDS:30}
    tick-interval-seconds: ${DATAPIPELINES_SCHEDULER_TICK_INTERVAL_SECONDS:10}
    max-concurrent-runs: ${DATAPIPELINES_SCHEDULER_MAX_CONCURRENT_RUNS:4}
    lateness-seconds: ${DATAPIPELINES_SCHEDULER_LATENESS_SECONDS:600}
    catch-up-max-age-seconds: ${DATAPIPELINES_SCHEDULER_CATCH_UP_MAX_AGE_SECONDS:86400}
    shutdown-wait-seconds: ${DATAPIPELINES_SCHEDULER_SHUTDOWN_WAIT_SECONDS:5}
    min-interval-seconds: ${DATAPIPELINES_SCHEDULER_MIN_INTERVAL_SECONDS:300}
    max-schedules-per-workspace: ${DATAPIPELINES_SCHEDULER_MAX_SCHEDULES_PER_WORKSPACE:100}
    notifications:
      max-recipients: ${DATAPIPELINES_SCHEDULER_NOTIFICATIONS_MAX_RECIPIENTS:20}
      enabled: ${DATAPIPELINES_SCHEDULER_NOTIFICATIONS_ENABLED:false}
      recipient-domains: ${DATAPIPELINES_SCHEDULER_NOTIFICATIONS_RECIPIENT_DOMAINS:}
      max-attempts: ${DATAPIPELINES_SCHEDULER_NOTIFICATIONS_MAX_ATTEMPTS:3}
      retry-backoff-seconds: ${DATAPIPELINES_SCHEDULER_NOTIFICATIONS_RETRY_BACKOFF_SECONDS:30,120,600}
      batch-size: ${DATAPIPELINES_SCHEDULER_NOTIFICATIONS_BATCH_SIZE:20}
      backlog-max: ${DATAPIPELINES_SCHEDULER_NOTIFICATIONS_BACKLOG_MAX:1000}
  parameters:                      # §3.30 — the parameter engine (#194)
    max-parameters-per-set: ${DATAPIPELINES_PARAMETERS_MAX_PARAMETERS_PER_SET:64}
    max-options-per-selector: ${DATAPIPELINES_PARAMETERS_MAX_OPTIONS_PER_SELECTOR:200}
    max-multi-bind-values: ${DATAPIPELINES_PARAMETERS_MAX_MULTI_BIND_VALUES:1000}
    max-binds-per-statement: ${DATAPIPELINES_PARAMETERS_MAX_BINDS_PER_STATEMENT:2000}
    max-option-value-chars: ${DATAPIPELINES_PARAMETERS_MAX_OPTION_VALUE_CHARS:1024}
    max-option-label-chars: ${DATAPIPELINES_PARAMETERS_MAX_OPTION_LABEL_CHARS:256}
    max-input-length: ${DATAPIPELINES_PARAMETERS_MAX_INPUT_LENGTH:4096}
    max-expression-depth: ${DATAPIPELINES_PARAMETERS_MAX_EXPRESSION_DEPTH:16}
    max-expression-nodes: ${DATAPIPELINES_PARAMETERS_MAX_EXPRESSION_NODES:128}
    max-regex-steps: ${DATAPIPELINES_PARAMETERS_MAX_REGEX_STEPS:100000}
    evaluate-timeout-seconds: ${DATAPIPELINES_PARAMETERS_EVALUATE_TIMEOUT_SECONDS:30}
    selector-query-timeout-seconds: ${DATAPIPELINES_PARAMETERS_SELECTOR_QUERY_TIMEOUT_SECONDS:10}
    max-concurrent-selector-queries: ${DATAPIPELINES_PARAMETERS_MAX_CONCURRENT_SELECTOR_QUERIES:4}
    max-waiting-selector-queries: ${DATAPIPELINES_PARAMETERS_MAX_WAITING_SELECTOR_QUERIES:64}
    max-evaluate-response-bytes: ${DATAPIPELINES_PARAMETERS_MAX_EVALUATE_RESPONSE_BYTES:4194304}

  web:
    max-request-bytes: ${DATAPIPELINES_WEB_MAX_REQUEST_BYTES:2097152}

  persistence:
    enabled: ${DATAPIPELINES_PERSISTENCE_ENABLED:true}
    batch-max-events: ${DATAPIPELINES_PERSISTENCE_BATCH_MAX_EVENTS:200}
    batch-max-bytes: ${DATAPIPELINES_PERSISTENCE_BATCH_MAX_BYTES:1048576}
    linger-ms: ${DATAPIPELINES_PERSISTENCE_LINGER_MS:0}
    queue-max-events: ${DATAPIPELINES_PERSISTENCE_QUEUE_MAX_EVENTS:10000}
    queue-max-bytes: ${DATAPIPELINES_PERSISTENCE_QUEUE_MAX_BYTES:33554432}
    record-max-wait-ms: ${DATAPIPELINES_PERSISTENCE_RECORD_MAX_WAIT_MS:2000}
    writers: ${DATAPIPELINES_PERSISTENCE_WRITERS:4}
    shutdown-drain-ms: ${DATAPIPELINES_PERSISTENCE_SHUTDOWN_DRAIN_MS:10000}
    audit:
      enabled: ${DATAPIPELINES_PERSISTENCE_AUDIT_ENABLED:false}
  visualization:                   # §3.33 — the dashboard documents' bounds (#10)
    max-visualizations-per-dashboard: ${DATAPIPELINES_VISUALIZATION_MAX_VISUALIZATIONS_PER_DASHBOARD:50}
    max-cases-per-visualization: ${DATAPIPELINES_VISUALIZATION_MAX_CASES_PER_VISUALIZATION:20}
    max-fixture-rows-per-case: ${DATAPIPELINES_VISUALIZATION_MAX_FIXTURE_ROWS_PER_CASE:1000}
    max-config-bytes: ${DATAPIPELINES_VISUALIZATION_MAX_CONFIG_BYTES:262144}
    max-bindings-per-visualization: ${DATAPIPELINES_VISUALIZATION_MAX_BINDINGS_PER_VISUALIZATION:64}
    max-inputs-per-visualization: ${DATAPIPELINES_VISUALIZATION_MAX_INPUTS_PER_VISUALIZATION:8}
    max-columns-per-input: ${DATAPIPELINES_VISUALIZATION_MAX_COLUMNS_PER_INPUT:256}
    session-ttl-minutes: ${DATAPIPELINES_VISUALIZATION_SESSION_TTL_MINUTES:60}

  dashboards:                      # §3.34 — the dashboard runtime (#10 L2)
    admission:
      max-concurrent-refreshes-per-workspace: ${DATAPIPELINES_DASHBOARDS_MAX_CONCURRENT_REFRESHES_PER_WORKSPACE:4}
      max-executions-per-refresh: ${DATAPIPELINES_DASHBOARDS_MAX_EXECUTIONS_PER_REFRESH:16}
      max-concurrent-dashboard-executions-per-instance: ${DATAPIPELINES_DASHBOARDS_MAX_CONCURRENT_DASHBOARD_EXECUTIONS_PER_INSTANCE:40}
      max-wait-seconds: ${DATAPIPELINES_DASHBOARDS_MAX_WAIT_SECONDS:10}
    results:
      max-bytes-per-source: ${DATAPIPELINES_DASHBOARDS_MAX_BYTES_PER_SOURCE:4194304}
      max-bytes-per-refresh: ${DATAPIPELINES_DASHBOARDS_MAX_BYTES_PER_REFRESH:33554432}
    timeouts:
      default-refresh-seconds: ${DATAPIPELINES_DASHBOARDS_DEFAULT_REFRESH_SECONDS:600}
      max-refresh-seconds: ${DATAPIPELINES_DASHBOARDS_MAX_REFRESH_SECONDS:900}
      parameter-lock-seconds: ${DATAPIPELINES_DASHBOARDS_PARAMETER_LOCK_SECONDS:30}
      render-seconds: ${DATAPIPELINES_DASHBOARDS_RENDER_SECONDS:20}

  release:
    dependencies:
      max-depth: ${DATAPIPELINES_RELEASE_DEPENDENCIES_MAX_DEPTH:32}
      max-versions: ${DATAPIPELINES_RELEASE_DEPENDENCIES_MAX_VERSIONS:1024}
      max-edges: ${DATAPIPELINES_RELEASE_DEPENDENCIES_MAX_EDGES:4096}
      max-blockers: ${DATAPIPELINES_RELEASE_DEPENDENCIES_MAX_BLOCKERS:100}
      max-wire-bytes: ${DATAPIPELINES_RELEASE_DEPENDENCIES_MAX_WIRE_BYTES:1048576}
      lock-wait-seconds: ${DATAPIPELINES_RELEASE_DEPENDENCIES_LOCK_WAIT_SECONDS:5}

db-scheduler:
  threads: ${datapipelines.scheduler.threads}
  polling-interval: ${datapipelines.scheduler.polling-interval-seconds}s
  heartbeat-interval: ${datapipelines.scheduler.heartbeat-interval-seconds}s
  missed-heartbeats-limit: 6
  shutdown-max-wait: ${datapipelines.scheduler.shutdown-wait-seconds}s
  table-name: scheduled_tasks
  immediate-execution-enabled: true

Note: OIDC provider config is in the app's own YAML namespace (datapipelines.auth.oidc.providers), NOT in Spring Security's native spring.security.oauth2.client.* namespace. Our OidcConfig bean reads this list and builds ClientRegistration objects programmatically. See Auth spec §5.2.

Redis binding (#500): dag owns RedisClientProperties (@ConfigurationProperties("datapipelines.redis")) and RedisClientConfiguration, which builds the one LettuceConnectionFactory from that binding. Operators set the DATAPIPELINES_REDIS_* variables defined here. The factory retains enabled command expiry, Boot's 100 ms shutdown timeout and the shared, metrics-customized ClientResources; Boot still supplies StringRedisTemplate and Redis health. SSL bundles, client names and read-from settings have no house key and are not applied. Operators change nothing: the shipped deploy/ compose and env files already set only DATAPIPELINES_REDIS_*. A SPRING_DATA_REDIS_* variable set by hand outside them (a database index, SSL, a username — none ever a house key) is now ignored, not merged: this binding is the client's only authority.

6. Posture Profiles (application-development.yml / application-hardened.yml)

The POSTURE's non-secret defaults (§3.23). spring.profiles.active is derived from DATAPIPELINES_POSTURE in application.yml, so setting that one variable loads the right file under every loader — Compose, a bare java -jar, systemd, Kubernetes. Setting SPRING_PROFILES_ACTIVE yourself is unnecessary, and a value that disagrees with the posture refuses startup (§7).

These files carry POSTURE, never INFRASTRUCTURE. Before 075 this section documented application-dev.yml, which was really a LAPTOP file: localhost:5434, console logging, one developer's host ports. An org's dev environment on Kubernetes runs the development posture too and must not inherit any of that. The laptop's values are ordinary lines in deploy/env/defaults.env, which every loader reads (DEVELOPMENT.md §2/§4).

And they carry the posture's defaults ALONE (081). No env file may name DATAPIPELINES_DEPLOYMENT_AUTHORING_ENABLED or DATAPIPELINES_AUTH_COOKIE_SECURE: there is one tracked settings file, it is loaded for every posture, and an environment variable outranks a profile — so a value there would be one posture's answer imposed on both. 075 did exactly that and a hardened stack booted with authoring=on. deploy/compose.yml passes those two in Compose's valueless form (DATAPIPELINES_AUTH_COOKIE_SECURE:, nothing after the colon), the only form that leaves a variable unset rather than empty — an empty variable outranks a profile just as a set one does. An operator can still override either in deploy/secrets.env. PostureDefaultsSpecDriftTest and compose-env-audit.sh check 7 hold the two halves.

No literal secrets, in either file (2026-08-07 security review): both ship inside every production jar (src/main/resources), so a working literal there means one stray SPRING_PROFILES_ACTIVE on a production manifest runs real infrastructure on publicly-known keys — forgeable admin JWTs and decryptable datasource credentials. Every secret is a placeholder resolved from the deployment's own environment, in every posture.

# application-development.yml
datapipelines:
  deployment:
    authoring-enabled: ${DATAPIPELINES_DEPLOYMENT_AUTHORING_ENABLED:true}
  auth:
    cookie-secure: ${DATAPIPELINES_AUTH_COOKIE_SECURE:}

# application-hardened.yml
datapipelines:
  deployment:
    authoring-enabled: ${DATAPIPELINES_DEPLOYMENT_AUTHORING_ENABLED:false}
  auth:
    cookie-secure: ${DATAPIPELINES_AUTH_COOKIE_SECURE:true}

Everything else the posture means is a §7 REFUSAL, which no YAML file can carry: demo, a seeded bootstrap credential, loopback infrastructure and a missing OIDC provider are all refused under hardened. The full table is §3.23.

The laptop's own settings, for reference (DEVELOPMENT.md §2/§4) — ordinary lines in deploy/env/defaults.env, read FIRST like every other default, with no inversion of the secrets-last rule:

DATAPIPELINES_ENV=local
DATAPIPELINES_POSTURE=development
SPRING_DATASOURCE_URL=jdbc:postgresql://localhost:5434/datapipelines
SPRING_DATASOURCE_USERNAME=datapipelines
DATAPIPELINES_REDIS_HOST=localhost
DATAPIPELINES_REDIS_PORT=6381
DATAPIPELINES_OBSERVABILITY_LOGGING_FORMAT=console
DATAPIPELINES_AUTH_ALLOWLIST_DOMAINS=

The two passwords are absent on purpose: they are secrets, they live only in deploy/secrets.env, and deploy/compose.laptop-infra.yml starts its Postgres and Redis from those same two variables (each keeping its historical fixed dev value as the interpolation default). Before 081 that file hard-coded datapipelines and no Redis password at all, and a third env file had to be loaded after the secrets to undo them — the one inversion this design used to have. DATAPIPELINES_AUTH_BASE_URL is likewise absent: it is this deployment's own identity and lives in secrets.env, where the scaffold writes http://localhost:<port>.

Dev host ports (2026-08-12): the laptop's Postgres listens on host port 5434 and Redis on 6381 — not the universal defaults 5432/6379, which collide with other local stacks on developer machines (Postgres.app/brew default to 5432). The host mapping lives in deploy/compose.laptop-infra.yml; SPRING_DATASOURCE_URL and DATAPIPELINES_REDIS_PORT in deploy/env/defaults.env must stay in sync with it and with DEVELOPMENT.md §2/§4. In production, SPRING_DATASOURCE_URL and DATAPIPELINES_REDIS_* are operator-set and unaffected.

7. Config Validation

On startup, the app validates:

• All required keys present → fail-fast with a clear error if missing.
• DATAPIPELINES_JWT_SECRET ≥ 32 bytes decoded.
• DATAPIPELINES_DB_ENCRYPTION_KEY is exactly 32 bytes decoded.
• Key provider (§3.20): datapipelines.db.key-provider names a provider this build ships (a violation lists them and points at docs/key-providers.md), and that provider's own settings are present and well-formed. For env: every entry of datapipelines.db.encryption-keys has an integer version in 1..255 that is not 1, and a value that decodes to exactly 32 bytes; datapipelines.db.encryption-key-current is an integer naming a configured version (the violation lists which versions ARE configured). An unknown provider name short-circuits the rest — reporting env's key rules to an operator who asked for aws-kms is noise they cannot act on.
• DATAPIPELINES_UI_THEME matches a vendored theme directory.
• Actively developed window (§3.10, #598b): datapipelines.ui.actively-developed-days, when present, is an integer ≥ 1 — 0, a negative or a non-integer refuses startup naming the key and the value. An absent key is not a violation: the binder default (14) applies.
• At least one authentication method: a fully-configured provider or datapipelines.auth.local.enabled=true. Fully configured means non-empty client-id and client-secret plus, for protocol: oidc, issuer-uri, and for protocol: oauth2, authorization-uri, token-uri, user-info-uri, user-name-attribute and a non-empty scopes. A provider entry with an empty client-id is ignored with a WARN — it does not count, and it is not a violation on its own.
• Each provider entry against its protocol (#655, auth.md §5.1), every violation naming the provider and the key: protocol is oidc or oauth2; an oidc entry has an issuer-uri and none of the oauth2-only keys; an oauth2 entry has the five required keys, no issuer-uri and no openid scope, and every endpoint it names (emails-uri included) is https:// with a host — plain http:// only to localhost / 127.0.0.1. The provider list is read as the application binds it (the Binder), so a higher-precedence source that re-declares the list replaces it whole and never inherits a key from application.yml's entry at the same index. The hardened posture's "a provider is configured" (§3.23) uses the same per-protocol completeness.
• datapipelines.auth.local.bootstrap-password and datapipelines.auth.local.bootstrap-password-hash are never both set; either seed requires datapipelines.auth.local.enabled=true AND datapipelines.auth.bootstrap-admin-email — each violation names both keys.
• datapipelines.auth.oidc.provider-read-timeout-seconds is a positive integer no greater than 2147483 (§3.4; signed 32-bit milliseconds); zero would disable the HTTP timeout and is refused.
• datapipelines.auth.local.lockout.max-failures and datapipelines.auth.local.lockout.duration-minutes are positive integers.
• The deprecated executor alias datapipelines.executor.max-concurrent-executions-global (050/R2): set alone → its value runs and startup logs one WARN naming max-concurrent-executions-per-instance; set together with the new key and differing → startup REFUSES naming both keys.
• result.ttl-min-seconds ≤ result.ttl-default-seconds ≤ result.ttl-max-seconds.
• endpoints.timeout-min-seconds ≤ endpoints.timeout-default-seconds ≤ endpoints.timeout-max-seconds, and the minimum is ≥ 1 (§3.22). Unlike the result TTLs, an ABSENT key is not a violation here — the binder default applies — so only a present, out-of-order triple refuses startup.
• Demo API key (§3.18, #224): datapipelines.bootstrap.demo-api-key, when set to a value other than the shipped default or blank, must be a well-formed dpk_<id>.<secret> credential (auth §7.1: dpk_, 12 base32 [A-Z2-7] chars, a dot, a non-empty secret, ≤ 80 chars) — a malformed configured key would otherwise surface as a refused login instead of a refused boot. An operator-set value under the hardened posture is refused outright, the same treatment the demo flag gets (the shipped default itself does not refuse a hardened boot — "set" means the operator chose it). Blank is the kill switch, never a violation.
• Endpoint key budget (§3.22, #224): datapipelines.endpoints.key-request-budget.window-seconds ≥ 1 and max-requests ≥ 0 (0 = the budget is off). An absent key is not a violation — the binder default applies.
• datapipelines.workspaces.provisioning-mode and datapipelines.workspaces.open-join are NOT set — both keys were removed in v1.9 (§3.17, RBAC round 1) and a deployment still carrying either is refused by name. The value check and the open-join/mode agreement check they used to carry went with them.
• Every datapipelines.auth.trusted-proxies entry parses as a CIDR (a bare IP is a host CIDR); anything else refuses startup at the auth module's resolver construction — a typo'd range must not silently widen proxy trust.
• datapipelines.bootstrap.datasources-file is not set without datapipelines.auth.bootstrap-admin-email (§3.18) — the violation names both keys.
• (Dropped in v1.9.) datapipelines.bootstrap.examples-file used to be refused while provisioning-mode was anything but auto-per-user; the seeder now runs at demo's creation in every deployment (§3.18), so the rule has no case left and ConfigValidator does not carry it.
• No OIDC provider is named bootstrap, local, system or key. Those are the users.provider values the system writes for identities it creates itself (§6.1 bootstrap actor, §5A local accounts, Auth §4.5 system service account, Auth §4.7 an API or server key's own identity), and a provider's configured name is written to that column verbatim — an external provider under any of them would be indistinguishable from them, and for system and key that indistinguishability is the whole of the account's safety argument. The reservation is case-insensitive and applies to an entry with a blank client-id too.
• Environment name (§3.23): datapipelines.env matches [a-z0-9][a-z0-9_-]{0,31}. The deprecated datapipelines.deployment.name alias: set alone it still names the deployment and startup logs event=config.deployment_name_deprecated; set together with datapipelines.env and disagreeing, startup is REFUSED naming both keys.
• Posture (§3.23): datapipelines.posture is development or hardened. A named environment with no posture refuses to start, naming both variables — the sole exception is the environment named local, which resolves to development. Explicit beats guessed: the product cannot infer a stance from a name it is forbidden to branch on.
• Posture / profile alignment (§3.23): spring.profiles.active is derived from the posture, so setting SPRING_PROFILES_ACTIVE to a DIFFERENT posture refuses startup naming both — otherwise one posture's defaults would load while every posture rule below judged the other. The dev profile was renamed development in 075; a deployment still asking for dev is refused rather than silently getting no posture file at all.
• The hardened posture (§3.23), each refusal naming the variable and the posture: datapipelines.demo must be empty; no local bootstrap credential may be set (bootstrap-password or bootstrap-password-hash — local accounts themselves stay allowed); spring.datasource.url and datapipelines.redis.host must not be loopback (039's dev-profile guard, inverted and reused — "dev convenience must never touch production infrastructure" is the same fact as "a hardened deployment does not run against a laptop's database"); and at least one OIDC provider must be fully configured unless datapipelines.auth.allow-local-only=true, which is logged as event=config.auth_local_only.
• Organisation (§3.21): datapipelines.org.fiscal-start-date is MM-DD and a day the calendar has — 02-30 and 13-01 are refused, and a month name (SEP-15) is refused with a message naming the MM-DD form; datapipelines.org.week-start is monday or sunday; datapipelines.org.timezone is an IANA zone id (a fixed offset such as +02:00 is not one); datapipelines.org.currency.name and .symbol are non-blank. All four report together — every value is in every Context, so a wrong one is a wrong number in every report the deployment produces.
• Script (§3.28, 7b): datapipelines.script.evaluate-timeout-seconds ≤ suite-timeout-seconds; abandon-grace-seconds ≥ 1; pool-size ≥ 1 and pool-queue ≥ pool-size; max-input-rows, max-value-bytes, max-string-bytes and max-depth each ≥ 1 — every refusal naming the keys.
• Removed script keys (§3.28, the TRANSFORM→SCRIPT node-type rename): any set datapipelines.transform.* key refuses startup BY NAME — the nine budgets moved to datapipelines.script.* with no alias, so a stale key would bind nothing and the deployment would run on defaults silently. Presence, not value, is the signal (the §3.17 rule).
• Parameter engine (§3.30, #194): every datapipelines.parameters.* key is an integer within its row's bounds, and selector-query-timeout-seconds ≤ evaluate-timeout-seconds — one statement must fit inside its evaluate; every refusal names the key.
• Audit retention (§3.12, #310): datapipelines.audit.retention-days is an integer in 30..3650 — refused while the key binds, naming it (the retention job deletes every audit row older than it, so a floor protects the trail from a typo).
• Visualizations and dashboards (§3.33, #10): every datapipelines.visualization.* key is an integer within its row's bounds; every refusal names the key.
• Dashboard runtime (§3.34, #10 L2): every datapipelines.dashboards.* key is an integer within its row's bounds; default-refresh-seconds ≤ max-refresh-seconds, max-bytes-per-source ≤ max-bytes-per-refresh, max-executions-per-refresh ≤ max-concurrent-dashboard-executions-per-instance and max-executions-per-refresh ≤ datapipelines.executor.max-concurrent-executions-per-instance (#762); every refusal names the key(s). max-concurrent-dashboard-executions-per-instance above the executor's ceiling is a WARN (event=config.dashboard_executions_above_executor_ceiling), not a refusal. A separate boot check (AuthoringStartupCheck) WARNs once naming saved dashboards whose static all-source invocation upper bound exceeds the smaller ceiling (event=config.dashboards_above_refresh_ceiling).
• Request limits (§3.31, #279): datapipelines.web.max-request-bytes is an integer within its row's bounds (65536..67108864) — the cap bounds every JSON request body on /api/v1 and /mcp; a malformed or out-of-window value names the key and the window.
• Redis auth: when datapipelines.redis.password is empty (after trimming) and datapipelines.redis.host is not loopback — under development, log a structured WARN event=config.redis_no_password (production Redis holds materialized caller results — Deployment §9); under the hardened posture it is a REFUSAL naming the key and the host, the same treatment Postgres's password gets as a §2 required key (#189). The refusal replaces the warning; a hardened boot never logs both.
• Redis client bounds (§3.1, #488): datapipelines.redis.command-timeout and datapipelines.redis.connect-timeout, when present, parse as a duration above 0 and at most 60s — every refusal names the key and the value. Zero is the case the rule exists for: Lettuce reads a command timeout of 0 as no bound at all, so 0s would silently restore the outage stall #482 removed (RedisZeroCommandTimeoutIntegrationTest shows it against a stopped Redis); a blank value is refused too; 60s, Lettuce's own command default, is the ceiling. The validator judges the one RedisClientProperties binding from which RedisClientConfiguration builds Lettuce. TestRedisDefaultsTest pins the test factories to the shipped operator defaults.
• datapipelines.deployment.promotion.server-key set ⇒ WARN (091): the value is deprecated in favour of a server-kind API key and is removed next release. Presence only — the warning never carries the secret.
• datapipelines.deployment.promotion.target.base-url is not set without datapipelines.deployment.promotion.target.server-key (§3.19) — the violation names both keys. The target's pre-shared key is what authenticates the push, so a target without one would have every promotion refused at the far end, at the end of a UI action a human took. The reverse is not a violation: a server-key with no target is an ordinary receiver.
• Mail (§3.27, 137): datapipelines.mail.host and datapipelines.mail.from are set together or not at all — one without the other refuses startup naming the missing one; datapipelines.mail.ops-to without host is refused (a sink nobody can reach); mail on (both set) without datapipelines.auth.base-url is refused (the welcome mail's login URL); port is a port in 1..65535; from is an address (a bare address or Display Name <address>). Under the hardened posture, with mail on: starttls must be true, and username set requires password set — each refusal naming the variable and the posture. The SMTP password is carried as PRESENCE only; no violation ever prints it.
• The boot line (§3.23): event=config.posture env=<env> posture=<posture> authoring=<on|off> demo=<families> is logged once (the label's only consumer — no code branches on it, pinned by a guard test). When a promotion receiver key is configured AND datapipelines.deployment.authoring-enabled=true, log a structured WARN — a promotion receiver should not author (Versioning D7), though a one-box deployment may legitimately be both. And when authoring is DISABLED while draft pipeline/template versions still exist, startup FAILS naming them: someone authored on a receiver and version alignment may already be broken (Versioning §5.5/§9.3).

The validator's own test suite must assert that the documented laptop setup (deploy/env/defaults.env + deploy/secrets.env, §6) passes the production rules — so a broken local value gets fixed at the data, never by weakening the check.

Validation runs in @PostConstruct of a ConfigValidator bean. Failures stop startup with a clear log message listing every missing/invalid key.

8. Source trace

The following index records the bindings, wiring and effect-producing consumers inspected for this reference. Keys grouped in a row share that trace; §3 remains the explanation/default authority. Paths are relative to the repository. Framework settings are consumed by the pinned Spring Boot/Tomcat/Flyway/db-scheduler libraries rather than a product @ConfigurationProperties class.

9. Keeping this reference current

ConfigurationReferenceSpecDriftTest runs in the normal app test/build lifecycle. It independently flattens packaged YAML with Spring's loader and discovers compiled @ConfigurationProperties annotations (including constant prefixes), recursively walking constructor-bound fields and list/map members. It checks:

• Exact key/value equality between §5 and packaged base YAML, including complete environment placeholder expressions. A changed default, alias, nested fallback, added key or removed key fails.
• Scalar defaults in explanatory tables agree with the packaged fallbacks; derived and map defaults keep their domain-specific checks.
• Exact equality of each packaged posture file with its §6 block; a new posture file also requires a documented block.
• An explanatory table row for every packaged leaf and every discovered bound field, including fields omitted from packaged YAML. A prose mention alone does not count.
• Documentation for direct literal property reads and named key constants; legacy settings explicitly refused at startup remain documented as removals.
• Launcher case arguments and tracked sample-data defaults stay documented; the directly configurable launcher wait/port/project defaults match their shell fallbacks.
• Reverse coverage of application-key rows, against compiled/direct declarations, with explicit reserved exceptions for the unimplemented tracing controls. Shipped YAML alone cannot satisfy this reverse check.

The existing group-specific tests still check code defaults, enum bounds, validator rules and deployment mappings. scripts/compose-env-audit.sh checks the Compose/env-file forwarding contract; chart drift tests check the Helm mappings. Gradle's shared test convention declares repository files as task inputs, so a documentation-only edit re-runs the checks. DocsRenderTest, DocsLinkRewriteTest and scripts/docs-audit.sh check the packaged document and links.

Run the repository's normal checks (scripts/pregate.sh for a development change), or the unfiltered app/auth/web module checks when working on this reference. :modules:mcp-server:docsExport supplies the served manual input required by scripts/docs-audit.sh.

Limits: these checks verify declarations, defaults, mappings and references; they cannot prove that a prose description matches an arbitrary behavioral change. Method-level property factories, mutable-bean-only bindings and new input APIs require an inventory extension. The direct-read scan covers literal getProperty/getRequiredProperty calls and constants named as keys/properties, not arbitrary computed strings. Dynamic key-provider maps, datasource-file environment substitutions and external framework configuration remain source-review responsibilities. A new input mechanism must extend the inventory instead of bypassing it. When a consumer, validator or binding changes, re-trace the relevant §8 path and update its description in the same change. No live deployment, credential values or production configuration is read by these guards.

10. Launch and deployment inputs

These inputs configure the launcher, image or sample-data services. They are separate from the Spring application properties in §§2–3. The sources are app.sh, Dockerfile, deploy/compose.yml, deploy/compose.local-build.yml, deploy/env/defaults.env and the code paths named below. Changing a launcher environment variable affects the next invocation; changing an image build argument requires rebuilding the image.

10.1 Local launcher arguments

app.sh reads deploy/env/defaults.env first and deploy/secrets.env second and passes both to Compose. The --env, --posture and --demo options are stripped from the argument list before choosing the command, so they apply to the invocation as a whole. For these three values, an explicit option wins over a non-empty process environment value, which wins over values read from those files. The launcher's file lookup retains the last non-empty value; it is not a general dotenv parser. Use the supported forms below.

There is no --help command: no command or an unrecognized command prints usage and exits 2. The removed --demo-nyc and --demo-trade forms are explicitly refused with migration guidance.

10.2 Launcher environment and Compose inputs

The following launcher controls are read from the invoking process environment, not through the launcher's from_files helper. Set them in the shell running app.sh. Application properties already catalogued in §§2–3 continue to use their documented Compose mappings.

app.sh has no configurable builder-image or Gradle-cache input: its builder image and .gradle-docker cache path are assigned in the script. Its generated DP_GEN_* and DP_APP_URL variables are internal scaffold values, not supported operator settings.

10.3 Sample-data inputs

These defaults are the values in the tracked deploy/env/defaults.env, used by app.sh and the documented Compose command. They are not server property defaults. deploy/sample-data/load.sh consumes the artifact URLs/versions as positional arguments from Compose, verifies manifest checksums before restoring an engine, and records the loaded version. Deployment Appendix B gives the setup and reset procedures.

For direct use of deploy/sample-data/load.sh, the additional inputs are: required <base-url> <version> (falling back to SAMPLE_BASE_URL/SAMPLE_VERSION); --engines <comma-separated-engines> or --engines=... (default: all reachable supported engines); --family nyc|trade or --family=... (default SAMPLE_FAMILY, otherwise nyc); and SAMPLE_DIR (default /srv/sample, also used by place-lake.sh). The Postgres arm requires PGHOST, PGUSER and SAMPLE_PG_PASSWORD, uses PGDATABASE or postgres for its admin connection, and uses the database-client environment for credentials (PGPASSWORD in Compose). SAMPLE_PG_DB defaults to dp_sample_trips. The MySQL arm requires MYSQL_HOST, MYSQL_ROOT_PASSWORD and SAMPLE_MYSQL_PASSWORD; SAMPLE_MYSQL_DB defaults to dp_sample_weather for nyc or dp_sample_trade for trade. Database names use the same identifier validation as the demo login. Compose fixes these service addresses and SAMPLE_DIR; putting a different value in its env files does not override a fixed service field. Custom database names also require matching artifacts/bootstrap definitions; the input does not rewrite their SQL or JDBC URLs.

10.4 Process inputs outside Spring configuration

10.5 Build and Helm settings

modules/datasources/build.gradle.kts recognizes -Pmysql and -Poracle to include the optional drivers in the jar's runtime dependencies. Presence enables them, so -Pmysql=false still includes MySQL. modules/app/build.gradle.kts recognizes -Pdatapipelines.buildTime=<ISO-8601 instant> for the build timestamp and -Pdatapipelines.commit=<commit> for build-info commit metadata. Absent, their fields are omitted rather than derived. app.sh supplies the current UTC time when building. Deployment §3 and §3.5 remain the packaging and driver-license authority.

The Dockerfile's build arguments are TARGETARCH (BuildKit platform, only amd64 and arm64 accepted), DUCKDB_CORE_VERSION (DuckDB extension download/layout version), GRAALJS_VERSION (isolate artifact version), and GRAALJS_SHA256_LINUX_AMD64 / GRAALJS_SHA256_LINUX_AARCH64 (the expected isolate checksums). The exact version/checksum defaults are pinned in Dockerfile, deploy/duckdb-extensions.sha256 and the version catalog and checked by the existing image drift tests. These are coordinated dependency pins, not independent runtime tuning inputs. The build also preinstalls native resources using -Dloader.main=co.datapipelines.scripting.JavaScriptResourcesMain; this is a build utility selection, not the application's normal main class.

The Helm input reference already lives in the commented deploy/helm/datapipelines/values.yaml and Deployment §6.4. Keep chart defaults there: replicaCount, image, existingSecret, extraEnv, duckdb, script.js, service, management, resources, temporaryStorage, libVolume, autoscaling, podDisruptionBudget and terminationGracePeriodSeconds are consumed by the templates under deploy/helm/datapipelines/templates/. The chart requires an existing Secret; it does not generate credentials. Its dedicated runtime groups map to the application settings in this reference. The chart refuses duplicate dedicated env mappings, launcher-path overrides and the JVM option variables JAVA_TOOL_OPTIONS, JDK_JAVA_OPTIONS, _JAVA_OPTIONS in extraEnv; Secret contents passed by envFrom cannot be inspected at render time. Deployment §6.4 documents those boundaries and the existing scripts/helm-check.sh render checks.

Appendix A: Change Log

| 2026-10-25 | v1.67 | 442c (#442) | §3.29: serialized deployment-wide pending admission, including resend refusal at capacity; activation pointer for a mixed-version fleet. | | 2026-10-24 | v1.66 | 442b (#442) | §3.29 gains six scheduler notification keys — notifications.enabled (default false; true needs the mail transport and auth.base-url or startup refuses), recipient-domains (the allowlist: example.com / *.example.com; empty under development sends nothing, under hardened restricts nothing), max-attempts (3), retry-backoff-seconds (30,120,600), batch-size (20) and backlog-max (1000); max-recipients loses its "settings only" note. Compose and defaults.env carry them. | | 2026-10-24 | v1.65 | #688 (Helm portion) — renumbered at merge after #686’s v1.64 | §3.28: the nine JavaScript keys are exposed as the chart's script.js.* values (deployment.md §6.4) — empty renders no env entry, the isolate directory must be one of the image's launcher-path entries, extraEnv double mappings are refused. No key, default, route or permission changes. | | 2026-10-24 | v1.64 | #686 JWKS fetch joins the provider read bound | §3.4's provider-read-timeout-seconds row: the ID-token decoder's JWKS fetch is bounded by the same key through a JwtDecoderFactory bean; discovery stays separate. No key added, no default change. | | 2026-10-22 | v1.63 | #651 enabled-provider avatar hosts | §3.4's picture-hosts allowlist includes only providers with a non-blank client-id, using the same predicate as sign-in registration. Dormant entries are still validated for shape at startup but contribute no hosts. No key, default, route or permission changes. | | 2026-10-22 | v1.62 | 8b (#8) — renumbered at merge after #682's v1.61 | §3.28 gains the nine JavaScript keys, their measured defaults, derived isolate memory and launcher agreement; max-depth also bounds JavaScript stack frames. | | 2026-10-22 | v1.61 | #682, answers #683 | Configurable provider connect/read timeout, deploy mirrors and startup validation (§3.4, §5, §7); per-provider defaults drift-guarded. | | 2026-10-21 | v1.60 | 655 (#655) GitHub sign-in, answers #154 part 2 | §3.4 gains seven per-provider rows: protocol (default oidc), authorization-uri, token-uri, user-info-uri, user-name-attribute, scopes (all empty by default, required for oauth2) and emails-uri (unset, optional). §1's provider sentence names both protocols and the GitHub env pair; the §5 YAML gains the github entry as shipped; §7's authentication-method rule is per protocol and gains the entry-against-protocol rule (unknown protocol, missing key, issuer-uri on oauth2, oauth2 keys on oidc, openid scope, non-https endpoint — each naming the provider and the key). deploy/secrets.env.example gains empty GITHUB_CLIENT_ID / GITHUB_CLIENT_SECRET lines. | | 2026-10-21 | v1.59 | 636 (#636) the housekeeping statements' bounds | §3.11's "not a key" note names the four housekeeping statements #636 bounds — the dashboard refresh sweep's UPDATE at 10 s (REFRESH_SWEEP_STATEMENT_TIMEOUT_SECONDS), the dashboard refreshes' DELETE and the parameter evaluations' sweep UPDATE and batch DELETE at the retention tick's 25 s — and links metadata-db §8.4 and §8.5. No key added. | | 2026-10-20 | v1.58 | V50 (#640, script-node-rename lane, owner ruling 2026-10-20) | The TRANSFORM→SCRIPT node-type rename, §3.28's keys move with it: the nine script budgets now live at datapipelines.script.* (env DATAPIPELINES_SCRIPT_*) with no alias; §7 gains checkRemovedTransformKeys (CHECK_COUNT 35 → 36): any set datapipelines.transform.* key refuses startup BY NAME pointing at its replacement — presence, not value, is the signal (the §3.17 rule). ScriptConfigKeysSpecDriftTest (7b's, renamed) still pins the new keys in this document, application.yml and the validator. Renumbered at merge after #326's v1.57. | | 2026-10-20 | v1.57 | 326 (#326) the scheduled jobs' statement bounds | §3.11 gains a "not a key" note: the stale sweep's 10 s and the hourly retention's 25 s statement bounds are code constants beside their cadences, not configuration keys. No key is added, renamed or removed. | | 2026-10-20 | v1.56 | 599 merge follow-up (#599) the window counts on every lifecycle home | §3.10 row datapipelines.ui.actively-developed-days: the description names every lifecycle home (the four object families read the same window since #599), not the Pipelines home only. No key, default or validation change. | | 2026-10-20 | v1.55 | 598b (#598) the Pipelines home's activity chart | New §3.10 row datapipelines.ui.actively-developed-days (default 14, DATAPIPELINES_UI_ACTIVELY_DEVELOPED_DAYS): the window of the Pipelines home's "actively developed" count — a pipeline whose draft was last written within that many days (UI screens §4.25). §5's template gains the line; §7 gains the rule (UiRules.checkUiActivelyDevelopedDays, registered in ConfigValidator; CHECK_COUNT 34 → 35): a present value that is not an integer ≥ 1 refuses startup naming the key and the value. UiConfigKeysSpecDriftTest pins the datapipelines.ui.* keys in this document, application.yml and the validator. | | 2026-10-16 | v1.54 | 467 (#467) the over-cap screenshot refusal reaches the client | §3.13's server.tomcat.max-swallow-size row names the screenshot route's own bounded drain (its cap + 2 MiB, fixed; the key is unchanged and does not move it) and the application.yml comment says the same. No key, default or validation change | | 2026-10-15 | v1.53 | 500 (#500) | Redis has one authority: RedisClientProperties binds §3.1 and builds the Lettuce factory in dag; the framework bridge is removed from §3.14 and §5. The §7 validator judges that binding. Test registrations use operator keys with explicit passwords, and the defaults guard reads the operator block. Merge follow-up: the binding note states that operators change nothing and that a hand-set SPRING_DATA_REDIS_* variable is now ignored. | | 2026-10-09 | v1.52 | 488 (#488) | §3.1's two Redis timeout rows and a new §7 rule (RedisRules.checkRedisTimeoutBounds, check 34): datapipelines.redis.command-timeout / .connect-timeout must parse as a duration above 0 and at most 60s (the owner's ceiling: Lettuce's own default, never more); malformed, blank, zero, negative and over-ceiling values refuse startup naming the key and the value. Verified in Lettuce 6.6's source and by RedisZeroCommandTimeoutIntegrationTest: a zero command timeout is NO bound (a GET against a stopped Redis still pending at 3 s while the shipped 2 s client failed in 2,002 ms) and also a zero handshake deadline (a silent server fails the connect in one timer tick, ~100 ms). 0s on the connect timeout and negative values were already loud (Lettuce asserts); zero on the command timeout was the silent case. | | 2026-10-09 | v1.51 | 482 (#482) | New §3.1 rows: datapipelines.redis.command-timeout and .connect-timeout (both 2s) — no client timeout was configured anywhere, so Lettuce's 60 s per-command default made a Redis outage stall every rate-limited request on the order of a minute before the documented 429 rate_limit.unavailable (and the stopped-Redis suite case alone took ~242 s). §3.14's bridge row extended: spring.data.redis.timeout / .connect-timeout carry the SAME DATAPIPELINES_REDIS_COMMAND_TIMEOUT / DATAPIPELINES_REDIS_CONNECT_TIMEOUT placeholders; §5's template and the deploy mirrors updated. No consumer code changed — every Redis fault path already routes a timeout (a DataAccessException subclass) through its existing failure handling; TestRedis builds its factories with the same two bounds under a placeholder-parity test. | | 2026-10-03 | v1.50 | 442a (#442) | Saved schedule notification settings: recipients and five event choices, validated and shown by role; omitted PUT settings preserved; mail remains off pending part b. | | 2026-10-02 | v1.49 | S3 (#376) the evaluation history's retention | §3.30 states that the parameter-evaluation history has NO key of its own: the execution-event retention (§3.11) deletes finished records on its hourly tick, and the stale sweep's cutoff is evaluate-timeout-seconds plus a fixed one-minute margin. No key, default or bound added or changed (the owner's §11.5 ruling). | | 2026-10-02 | v1.48 | 140 (#140) the Helm chart maps the DuckDB operator keys | §3.25's operator instruction names the Helm route: the reference chart's duckdb.extensionDirectory and duckdb.memoryLimit values render these two env vars when non-empty, an empty value renders no entry (inheriting the image's bundled extension directory), and an extraEnv double mapping is refused at render time. No key, default or bound changed. | | 2026-10-02 | v1.47 | 167 (#167) the probe's timeout ceiling | §3.2: node-query-timeout-seconds and node-query-timeout-seconds-by-dialect.<dialect> are documented as ALSO the sql_probe timeout ceiling (was a static 30 s in code). No key, default or bound added or changed — the existing knob is the one truth. | | 2026-10-02 | v1.46 | 204 (#204 L4/L5) the in-process boundaries at the pool build — renumbered at merge after 375's v1.45 | §3.26's file-roots row: the file itself is resolved (a final-component symlink aimed outside every root, or a dangling one, is refused) and the rule is re-checked at every pool build (Datasources §4.2A). No key, default or bound changed. | | 2026-10-02 | v1.45 | S2 (#375) the observed parameter-set evaluation | §3.6: no new key. disconnect-grace-seconds names the observed evaluation beside the execution (and the dashboard refresh it already governed): a client gone past the grace aborts its evaluation (the owner's §11.7 ruling; REST API §21.5) — with the defaults the grace and parameters.evaluate-timeout-seconds are both 30, so the deadline usually ends it first. max-streams-per-user counts execution, refresh and observed-evaluation streams together (#375 D7). | | 2026-10-01 | v1.44 | L4a (#352) the visualization test sessions — renumbered at merge after L1c-c's v1.43 | §3.33 gains datapipelines.visualization.session-ttl-minutes (60, 1..1440): a test session's TTL — the run's expires_at, the preview capability's deadline and the ceiling on the submit-minted upload capability (the spec's §11.2 tests.session-ttl-minutes, flattened to the shipped two-level key grammar). In the §5 template, application.yml, deploy/compose.yml, deploy/env/defaults.env, deploy/secrets.env.example and ConfigValidator (§7). | | 2026-09-30 | v1.43 | L1c-c (#10) transfer limits and evidence — renumbered at merge after 337's v1.42 | §3.33's max-visualizations-per-dashboard row: the value is documented as ALSO the transfer families' aggregate ceiling (L1c-c) — it caps each whole export/import envelope array (visualizations bundle, templates closure) and each whole promotion batch arm (visualizations, dashboards), refused whole before any member binds. An additional ceiling on top of the per-document bound, not a consequence of it: two individually valid dashboards can together exceed it, and such a batch refuses with no partial writes and no automatic split. No key, default or bound changed; the production transfer bean now receives the operator's configured value (L1c-c's C1 — an override was silently ignored at the default before). | | 2026-09-30 | v1.42 | 337 (#337) | §3.15's logging.format row is now TRUE instead of aspirational: the key is bound before logging initialises (environment post-processor), the value set is closed (json | console), an unknown value refuses startup, and the row documents what each value produces and which deploy shape ships which. No new key, no YAML or deploy-mirror change — the declared key just works. | | 2026-09-29 | v1.41 | L2 (#10) the dashboard runtime | New §3.34 The dashboard runtime: datapipelines.dashboards.*, the ten admission / result / deadline numbers of the spec's §9.4–§9.6, each the owner-confirmed default (4 / 16 / 40 / 10 s; 4 MiB / 32 MiB; 600 / 900 / 30 / 20 s), with three boot-time relations, in the §5 template, application.yml, deploy/compose.yml, deploy/env/defaults.env, deploy/secrets.env.example and ConfigValidator (§7). The admission counters are stated JVM-LOCAL. §3.33 had already promised these keys. | | 2026-09-29 | v1.40 | 306/#311 (the execution row's order and bounds) — renumbered at merge: L1a took v1.39 | §3.2 gains datapipelines.executor.lifecycle-write-timeout-seconds (10): the bound on the two pipeline_executions lifecycle writes, read by two layers — a JDBC queryTimeout on exactly the RUNNING insert and the terminal UPDATE (ExecutionRepository; a database that answers slowly is cancelled at the statement) and the emitter's caller-side wait (WebEventEmitter; the layer that bounds a never-answering database, which a statement timeout cannot reach — measured). A past-bound write is stated, never fabricated: insert failure = the existing recorded = false path, terminal failure = the row left RUNNING for the stale sweep, counted (datapipelines.executions.lifecycle_write_failed). Mirrored in application.yml, defaults.env, secrets.env.example, compose.yml; WebPropertiesSpecDriftTest pins the default. | | 2026-09-29 | v1.39 | L1a (#10) the visualization module | New §3.33 Visualizations and dashboards: datapipelines.visualization.*, the seven collection bounds of the two documents' readers — max-visualizations-per-dashboard 50, max-cases-per-visualization 20, max-fixture-rows-per-case 1000, max-config-bytes 262144, max-bindings-per-visualization 64, max-inputs-per-visualization 8, max-columns-per-input 256 (proposed; confirmed at review) — in the §5 template, application.yml, deploy/compose.yml's pass-through, deploy/env/defaults.env and ConfigValidator (§7). The spec reserved §3.32 for the dashboard keys; 266 took it. | | 2026-09-29 | v1.38 | 310 (#310) audit-log retention — numbered after 266b's v1.37 | §3.12's datapipelines.audit.retention-days now BINDS (AuditProperties) and is ENFORCED: the retention sweep's last step deletes audit_log rows older than the database's NOW() minus the key, one cutoff for every event, in bounded batches (5,000 rows a statement, 50 statements or 2 s a tick). Before #310 the key was documented and shipped but read by nothing. New bounds 30–3650, refused at bind (§7 gains the rule); the default (365) is unchanged, and no shipped env file sets a value below the floor | | 2026-09-29 | v1.37 | 266 (#266) persistence batching, with its correction round 266b — the dashboard implementation spec reserved "§3.32" for its keys; this lane takes it and the spec renumbers | New §3.32 Persistence batching: the ten datapipelines.persistence.* keys of the batching writers in front of the audit log, the execution-event record and the replay log (enabled, audit.enabled, batch and queue bounds, linger, record-max-wait, writers, shutdown drain), enforced at binding (PersistenceProperties). audit.enabled ships false (the orchestrator's ruling on the measurement: audit rows never formed batches and the writer raised their tail), so by default only the execution-event stores are batched. batch-max-events is bounded at 7000 (the replay log's Lua script); writers also sizes the dp-event-persist pool, whose thread count was a code constant (4), and the row says what 1 serialises. §5's template gains the persistence: block appended after the whole datapipelines: tree. No existing key changed. | | 2026-09-28 | v1.36 | 298 (#291) request constraints on every body | §3.31's prose: the stated Jackson StreamReadConstraints hold on every route that parses its own text body as well (a request copy of its domain mapper, pipeline-contract §13.21), not only on the two surfaces' shared mappers. The v1.35 row, which had landed above this table's header, moved under it. No key changed. | | 2026-09-28 | v1.35 | 279 (#279) the request-body cap | New §3.31 Web — request limits: datapipelines.web.max-request-bytes (default 2 MiB, 65536..67108864), the platform-wide request-body cap refused 413 request.body_too_large on /api/v1 and /mcp before any handler or parser (pipeline-contract §13.21); §3.13 gains the server.tomcat.max-swallow-size row (default 2MB) that bounds draining a refused body; §5's template gains the web: block appended after the whole datapipelines: tree; §7 lists the rule (check 31). The stated Jackson StreamReadConstraints (nesting 100, string 4M chars, number 1000 digits) are recorded in §3.31's prose — they are constants of the request mappers, not operator keys. | | 2026-09-26 | v1.34 | 194b (#194) parameter engine lane B — numbered after origin/main's v1.33 (260) | New §3.30 Parameter engine: the fifteen datapipelines.parameters.* keys of the design record's §11 with their bounds (enforced at boot by ConfigValidator and at binding by ParametersProperties); §5's template gains the parameters: block as the last child of datapipelines:; §7 lists the rule. No options cache (the record's §11). | | 2026-09-26 | v1.33 | 260 the depth-accounting fix (#260) — numbered after origin/main's v1.32 (246) | §3.28's max-depth row re-stated: the depth bound is counted at every evaluate entry and exit by the engine itself (the library Timebox skipped isParallelCall frames, leaking a unit per item on set-level shapes until #260), nested evaluation and non-tail lambda recursion included; a tail-recursive lambda is trampolined by the library and the wall clock is its bound. No key, default or binding changed. | | 2026-09-26 | v1.32 | 246 (#246) the avatar proxy hardened — numbered after scheduler lane 1's v1.31 (#9, merging at dispatch) | §3.4's picture-hosts row says no port on both sides: an entry is a bare hostname (unchanged), and a stored picture URL naming a port is refused; it points at auth.md §11.1 for the two other #246 rules (raster types only, one 5 s budget for the whole fetch). No key, default or binding changed. | | 2026-09-26 | v1.31 | scheduler lane 1 (#9) — numbered after origin/main's v1.30 (197) | New §3.29 Scheduler: the eleven datapipelines.scheduler.* keys (dispatch on/off — API mode, threads, polling, heartbeat, tick, max concurrent runs, lateness, catch-up reach, shutdown wait, the two save guards) with their bounds, enforced at binding; management.health.db-scheduler.enabled: false. §5 template gains the scheduler: block and the top-level db-scheduler: wiring fed from it. Status caught up (it read v1.27 after v1.29's row; main's v1.30 row is 197's). | | 2026-09-25 | v1.30 | 197 (#197) the avatar proxy | §3.4 gains datapipelines.auth.oidc.providers[].picture-hosts (per-provider, default empty = the avatar proxy fetches nothing and avatars render initials): bare hostnames only — a scheme, port, path, userinfo or wildcard entry refuses startup, enforced where the allowlist is built (the trusted-proxies precedent). The demo google provider ships lh3.googleusercontent.com — measured, NOT derived from the issuer (accounts.google.com). Auth §11.1 carries the full key story; Deployment §6.2 the narrowed CSP row. | | 2026-09-24 | v1.29 | 224 (#224) | §3.18 gains datapipelines.bootstrap.demo-api-key (committed default, blank = off, changed = rotation) and §3.22 the datapipelines.endpoints.key-request-budget.* pair (window 60 s, 60 requests, 0 = off); §7 gains the demo-key shape/hardened check and the budget bounds check (checks 28 and 29). The demo key and its page section are rest-api §19's worked example. | | 2026-09-23 | v1.28 | 7b (#7) | New §3.28 Transform evaluation: the nine datapipelines.transform.* budgets (evaluate/suite timeouts, pool size/queue, abandon grace, the input/output caps, max depth) behind the transform test suite, templates_evaluate and the evaluate route; §7 gains the transform bounds check (check 27: evaluate ≤ suite, grace ≥ 1, pool-queue ≥ pool-size, the caps ≥ 1). §5 template block appended after mail (the whole datapipelines: tree, as every append). | | 2026-09-21 | v1.27 | 186 in-process datasource containment (#186) | §3.26 gains datapipelines.datasources.file-roots (default empty = no file-backed in-process datasource is registrable by anyone; each root must be an existing directory at boot). §3.17's member-datasources-enabled default flips to false, matching the shipped posture (deploy/env/defaults.env has shipped false since the workspaces round) — a documented default must match the shipped posture, and the drift guards now pin all three to the same value. §3.17's row also states the flag-independent rule: in-process engines (H2 mem:/file:, DuckDB, SQLite) are super-admin-only regardless. | | 2026-09-21 | v1.26 | 188 (#189) | §3.23's posture table and the §7 rule list gain the Redis-password row: passwordless Redis off loopback stays a WARN under development and is a REFUSAL under hardened (PostureRules.checkHardenedRedisPassword, check 26); the §3 Redis row says so. | | 2026-09-21 | v1.25 | 180 (#180, #182) | Doc-only, no key change. §3.18's examples-file row, its cross-key paragraph and the three §7 bullets (provisioning-mode value, open-join/mode agreement, examples-file/mode) described rules ConfigValidator dropped in v1.9 — the document contradicted its own §3.17 and v1.9 row. Each now states that the rule was dropped and why (the seeder runs at demo's creation in every deployment). | | 2026-09-19 | v1.24 | 172 (#172) | §3.22: the published-endpoint surface re-rooted to /api/<category>/<version>/<path…> (R-EP5); the datapipelines.endpoints.* keys and their bounds are unchanged. | | 2026-09-17 | v1.23 | 155 / #122 staging memory guard scope | §3.3: max-memory-mb re-described truthfully — the threshold is per-execution (the pipeline override changes the number one execution compares, never what is measured), but the MEASUREMENT is the whole JVM's used heap, sampled, so the guard is a shared circuit breaker, not an isolated budget, a reservation, or an OOM guarantee (Staging §8.2 says the same; one cross-link, not two copies). Sizing guidance rewritten from the demand side: worst-case heap demand stays max-memory-mb × max-concurrent-executions-per-instance per instance and -Xmx/container memory must cover it. No behaviour change; the 108 §C startup warn line is unchanged. | | 2026-09-17 | v1.22 | 153 operator memory-limit config (#136) | New §3.25 row: datapipelines.duckdb.memory-limit (DATAPIPELINES_DUCKDB_MEMORY_LIMIT, default empty). Deployment-wide default memory_limit for every LAKE engine build when a datasource sets none, restart-to-change, validated at startup with properties.dialect.memory_limit's own grammar. §3.24's memory_limit row gains the three-way precedence sentence (datasource property > this key > derived 25%). §3.25's title and intro widened to cover both operator keys. §5 template block appended after extension-directory (same block, not a new one). | | 2026-09-17 | v1.21 | Writable LAKE spill default (#133) | §3.24: unique engine-owned absolute path under java.io.tmpdir; writable disk requirement, explicit override and cleanup semantics. | | 2026-09-17 | v1.20 | 152 LAKE engine scope (#128) | §3.24 gains the resource-scope paragraph: a LAKE datasource pool owns ONE embedded DuckDB instance shared by all its pooled connections, so memory_limit / threads / temp_directory are one budget per datasource pool (not per connection, not multiplied by maximumPoolSize), the sizing rule is per LAKE datasource on the instance plus a draining generation's headroom, and the file cache survives Hikari's connection replacement but not a pool rebuild. No key added or changed. Status line re-synced to the changelog (it read v1.1 above rows that ended at v1.18) | | 2026-09-16 | v1.19 | 149 / #125 node_progress cadence | §3.2 gains datapipelines.executor.progress-sample-interval-seconds (1): the floor between two periodic node_progress samples of one operation; first entries into a state and the terminal sample are never throttled by it. Mirrored in application.yml, defaults.env, secrets.env.example, compose.yml; WebPropertiesSpecDriftTest pins the default. | | 2026-09-14 | v1.18 | 137 mail notices | New §3.27 Mail: datapipelines.mail.host / port (587) / username / password / starttls (true, required-not-opportunistic) / from / reply-to (empty = from) / ops-to (comma list) / message-stream (Postmark's optional header). No enabled key — mail is on exactly when host and from are both set. §5 template block appended after duckdb:; §7 gains the shape rules (host↔from, ops-to without host, mail on without auth.base-url, port range, from is an address) and the two hardened refusals (starttls must be true; a username needs a password) — checks 23 and 24 | | 2026-09-08 | v1.17 | 091 keys | §3.19: datapipelines.deployment.promotion.server-key is DEPRECATED in favour of a server-kind API key (Auth §7.7) — mintable by an admin on the API screen, expiring, revocable, rotatable without a restart, and visible in the key list. Both credentials are accepted for one release; the configured value is compared FIRST (so a deployment that has not migrated pays no database read) and its presence raises one WARN at boot. §7 gains that rule (check 23). Fail-closed is unchanged and now has two halves: no configured value AND no live server key ⇒ every push refused. | | 2026-09-07 | v1.16 | 089 dp-lake §D extension bundling | New §3.25 DuckDB extension directory (dp-lake): datapipelines.duckdb.extension-directory (DATAPIPELINES_DUCKDB_EXTENSION_DIRECTORY, default empty). Set — as the shipped image does via the Dockerfile, which now bundles httpfs/aws/iceberg/avro for DuckDB core v1.5.5 for the build's architecture (linux_amd64 or linux_arm64 via TARGETARCH, +108 MB uncompressed, ~39 MB at build) — lake connections SET extension_directory and emit bare LOADs, never an INSTALL, so a hardened dp-lake deployment needs no egress to extensions.duckdb.org. Unset keeps the INSTALL+LOAD behavior developer machines rely on. §5 template block appended after demo: | | 2026-09-07 | v1.15 | 089 dp-lake phases B–D | New §3.24 Lake datasource engine limits (dp-lake): the properties.dialect.memory_limit / threads / temp_directory datasource properties (datasource-row keys, NOT datapipelines.* env keys — validated per key at save), the default memory limit (25 % of the container's memory as the cgroup-aware JVM reports it, hard-capped at 4 GiB, floored at 64 MiB), and the always-on preserve_insertion_order = false. Phases B/C add no operator keys: per-table views and registry-backed introspection are behavior, not configuration | | 2026-09-06 | v1.14 | 081 one env file | §3.23, §4 and §6 rewritten to two files. The contract is deploy/env/defaults.env (tracked, every non-secret variable with its shipped value) then deploy/secrets.env (git-ignored, credentials and this deployment's own overrides), in that order under every loader; deploy/secrets.env.example is the template and names every variable. The five files 075 shipped under deploy/env/ are deleted, and with them the laptop's load-order inversion — deploy/compose.laptop-infra.yml now starts Postgres and Redis from SPRING_DATASOURCE_PASSWORD and DATAPIPELINES_REDIS_PASSWORD. §6: the posture's two default rows live ONLY in the profile ymls, and no env file may name them. compose-env-audit.sh gains the one-authority proof (each bound variable declared in exactly one tracked file) and the valueless-form check. | | 2026-09-05 | v1.13 | 075 environments and posture | New §3.23 Environment and posture: datapipelines.env (DATAPIPELINES_ENV, default local — the ORG's label, renamed from §3.19's datapipelines.deployment.name, which survives one release as a WARNing alias), datapipelines.posture (development | hardened, no default — only the env named local gets one for free) and datapipelines.demo (a FLAG: the sample-data families; refused under hardened), with the normative posture table. §3.4 gains cookie-secure (empty = derived from base-url's scheme; hardened ships true) and allow-local-only (the explicit no-OIDC acknowledgement). §6 is now the POSTURE profiles — application-dev.yml was really a laptop file and its infrastructure moved to deploy/env/laptop.env; application-development.yml and application-hardened.yml carry the posture defaults, and spring.profiles.active is derived from DATAPIPELINES_POSTURE. §7 gains four rules (env grammar + alias, posture required, posture/profile alignment, the hardened refusals) and loses the dev-profile guard, which those refusals subsume. §4 and §5 updated. Environments is the new operator page. | | 2026-09-05 | v1.12 | 074 published endpoints | New §3.22 Published endpoints: datapipelines.endpoints.timeout-default-seconds (30) / -min-seconds (1) / -max-seconds (300); one new §7 rule (min ≤ default ≤ max, min ≥ 1 — the four report together). Deliberately no page-rows-max: DP-Result-Page-Rows clamps to result.page-max-rows on both surfaces (R-EP4). §5 template block appended after org:. | | 2026-09-04 | v1.11 | 072 calculators | New §3.21 Organisation: datapipelines.org.currency.name / .symbol, fiscal-start-date (MM-DD), week-start, timezone — five keys that enter every execution Context as org_* (calculators design §0.1/§0.2, Pipeline Contract §7.2). §5 template block and one new §7 validation rule (all four org checks report together; a month name in fiscal-start-date is refused with a message naming MM-DD) | | 2026-09-04 | v1.10 | 068 key-provider seam | New §3.20 credential key provider: datapipelines.db.key-provider (default env, so no deployment needs a config edit), plus the env provider's optional encryption-keys rotation map and encryption-key-current. §2's encryption-key row now states that it is key version 1, forever. §5 template and §7 updated — one new validation rule: the provider name must be one this build ships, and that provider's own settings must be present and well-formed (an unknown name short-circuits the rest). | | 2026-09-02 | v1.9 | 051 auth/config sweep | Added §3.4 datapipelines.auth.trusted-proxies (CIDR list, default empty = header ignored; the login limiter and every auth source_ip resolve the client through it — R8/T46, deployment.md §6.2) with the §5 template line and two §7 rules (each entry must parse as a CIDR or startup is refused; enforced at the auth module's resolver construction). §3.17: open-join: true + closed provisioning now refused at startup, naming both keys (T45 — the self-join branch gates on open-join alone, so the pair would re-open the membership surface closed exists to keep admin-only); §7 gains the rule | | 2026-08-05 | v1.0 | initial draft | Complete configuration reference: 6 required keys + OIDC, ~30 optional keys, full application.yml template, dev profile, startup validation | | 2026-08-07 | v1.1 | consistency campaign | Authority + naming-derivation rules (§1); §3 tables and §5 YAML reconciled (unit-suffixed names win); added result.* (D9), sse.disconnect-grace-seconds (D7), idempotency, templates, audit, staging result-batch-size, login rate-limit keys; removed large-result-threshold-bytes and redis.ttl-seconds (superseded by result.*); rate limits per-user; encryption key required with no fallback; precedence section. See SPEC-REVIEW-2026-08 | | 2026-08-12 | v1.2 | dev infra ports | §6 dev profile now targets host ports 5434 (Postgres) / 6381 (Redis) instead of the colliding defaults 5432/6379; added the dev-host-ports note. Operator-facing keys and production defaults unchanged. | | 2026-08-17 | v1.3 | pipeline composition | Added §3.16 datapipelines.pipelines.max-composition-depth (default 5) — the depth guard for PIPELINE-node composition | | 2026-08-26 | v1.3.1 | workspaces (slice 019) | Added §3.17 Workspaces: datapipelines.workspaces.provisioning-mode (auto-per-user | self-serve | closed, default self-serve), open-join (default false), member-datasources-enabled (default true). (This row was written later than the section — v1.4's note promised the backfill, but the row itself never landed; recorded and landed 2026-08-29, 025 D3. Keys unchanged by the delay.) | | 2026-08-28 | v1.4 | sample data, slice A | Added §3.18 Bootstrap: datapipelines.bootstrap.datasources-file and datapipelines.bootstrap.examples-file (both unset = off), the cross-key rule pairing datasources-file with datapipelines.auth.bootstrap-admin-email, and the matching §7 validation bullet and §5 template block. Backfills the §3.17 Workspaces row this log was missing (added 2026-08-26 with slice 019, keys unchanged here) | | 2026-08-29 | v1.5 | local password auth | Added §3.4 datapipelines.auth.local.*: enabled, bootstrap-password-hash / bootstrap-password (first-admin seed only, forced first-login change), lockout.max-failures / lockout.duration-minutes — plus the §5 template block. The §7 "at least one OIDC provider" rule becomes "at least one authentication method"; a provider entry with an empty client-id is now ignored with a WARN instead of counting (the stock google entry binds empty when its env vars are unset, so a local-accounts-only deployment starts with zero providers). rate-limit.login-per-minute description widened to OIDC and local | | 2026-09-02 | v1.7 | 048 bootstrap seeding fixes | Two §7 rules added, no new keys: (a) the examples-file cross-key rule pairing it with datapipelines.workspaces.provisioning-mode = auto-per-user (021/F5 — the pair validated green while the seeder was structurally unreachable, on the shipped default); (b) bootstrap and local reserved as OIDC provider names (021/F8 — the users.provider placeholders were squatting in an operator-configurable namespace with nothing reserving them). §3.18 gains the matching cross-key paragraph | | 2026-09-02 | v1.8 | 050 multi-instance round 2 | §3.2: max-concurrent-executions-global renamed max-concurrent-executions-per-instance (the limit was always per JVM — the old name false at N replicas; 050/R2), old key kept as a one-release deprecated alias (alone → WARN naming the new key; both set and differing → §7 refusal); §3.2 heap note and §5 template updated to the per-instance multiplier; §7 gains the alias rule. §3.11: event-retention-days now bound and scheduled (the hourly retention job, M2's sibling) | | 2026-09-01 | v1.6 | 039 deployment role | Added §3.19 Deployment: datapipelines.deployment.name (label only — logged once at boot beside the authoring state; nothing branches on it, pinned by a guard test; deliberately not on /info) and datapipelines.deployment.authoring-enabled (default true; false turns the deployment into a promotion receiver whose authoring writes refuse with *.authoring.disabled; startup refuses if drafts exist while disabled). The reserved datapipelines.deployment.promotion.* sub-block is deliberately NOT declared here — it ships with promotion (Versioning §10.6's fenced sample), per this doc's shipped-keys-only rule. Matching §7 bullets (the one-sided receiver-also-authors WARN; the refuse-on-existing-drafts rule) and §5 template block | | 2026-09-10 | v1.9 | 112 RBAC round 1 | §3.17: datapipelines.workspaces.provisioning-mode and datapipelines.workspaces.open-join removed (D-R11 — capability moved onto the workspace membership, so who may create a workspace is a role, not a mode). Both are refused BY NAME at startup rather than ignored, and both are gone from the §5 template, application.yml and the shipped .env — a placeholder with a default is a set key to Spring's relaxed binding, so leaving them there would have refused the product's own boot. §7 loses the two rules that read them (the mode-value check and the open-join-under-closed check) and the examples-file cross-key rule (example seeding now runs once, when DemoWorkspaceSeeder creates demo); it gains one rule that names both removed keys. member-datasources-enabled survives, re-described: it gates whether a workspace ADMIN may register a datasource bound to their own workspace |