# Deployment & Packaging Specification

**Status:** v1.61
**Owner:** datapipelines.co core
**Depends on:** all other specs
**Last updated:** 2026-11-01

---

## 1. Purpose

This spec defines **how datapipelines.co is packaged, distributed, and deployed** in self-hosted environments. The product is open-source and self-hosted; this document covers the artifacts we produce and the deployment patterns we support.

This is **lighter than the core specs** — most operational concerns are the operator's responsibility (we ship the artifact, they run it). We document our artifacts, required infrastructure, configuration surface, and recommended deployment patterns.

---

## 2. Design Principles

1. **One artifact, many deployments.** A single Docker image runs in dev, staging, prod. Configuration via env vars + mounted files; no build-time decisions.
2. **Stateless app, externalized state.** The app has no local persistent state. Metadata DB (Postgres) and Redis (result store, idempotency keys, cancellation flags, post-completion event log) are external. H2 staging is per-execution in-memory.
3. **Container-first.** Primary distribution is a Docker image. JVM-only deployments (no container) are supported but secondary.
4. **k8s-native but not k8s-required.** Kubernetes is the recommended target. Docker Compose works for small / single-node deployments. Bare JVM works for development.
5. **Configurable without forking.** Every deployment-relevant knob is an env var or config-file value. Operators do not edit source code to deploy.
6. **Open-source distribution.** The image and JARs are published to public registries / Maven Central. No proprietary distribution channel.

---

## 3. Build Artifacts

### 3.1 Docker image

Published to `ghcr.io/datapipelines/datapipelines:{version}` and `docker.io/datapipelines/datapipelines:{version}` (mirror).

- **Base image**: `eclipse-temurin:21-jre` (the Ubuntu glibc JRE 21 image, chosen over a smaller alpine base because argon2-jvm loads its native library through JNA, which ships glibc binaries).
- **Layers**: single stage: the boot jar is built on the host (`./gradlew :modules:app:bootJar`) and copied in; the image adds the DuckDB extension bundle, the GraalJS isolate with its pre-installed resource cache, and the third-party notices file.
- **Bundled DuckDB extensions (dp-lake)**: `httpfs`, `aws`, `iceberg` and `avro` for DuckDB core v1.5.5, pre-populated under `/opt/duckdb/extensions/v1.5.5/<platform>/` (`<platform>` = `linux_amd64` or `linux_arm64`, following the build's `TARGETARCH`) and activated by the image's `ENV DATAPIPELINES_DUCKDB_EXTENSION_DIRECTORY` — a LAKE datasource `LOAD`s them from disk and never `INSTALL`s at runtime, so no egress to `extensions.duckdb.org` is needed ([Configuration §3.25](/docs/configuration.md)). Adds ~108 MB uncompressed (~39 MB downloaded at build). The eight decompressed binaries (four per platform) are pinned in `deploy/duckdb-extensions.sha256`; the extension build layer runs `sha256sum -c --ignore-missing --strict` before `USER` and refuses a mismatch. `/app/THIRD-PARTY-NOTICES` includes the extensions at their footer commits, their copyrights and licences, and their pinned linked closure: direct and transitive vcpkg ports, the DuckDB core `third_party` components, and code bundled inside ports (the committed closure files and re-pin procedure are described below).
- **Size target**: < 250 MB compressed.
- **User**: non-root (`datapipelines`, a system account created with `useradd --system`; no fixed UID — a chart or compose `runAsUser` must not assume 1000).
- **Entrypoint**: `["java", "-Dpolyglot.engine.userResourceCache=/opt/graaljs/resources", "-jar", "/app/datapipelines.jar"]` (exec form). The image defines no JVM-flags hook: a JVM option variable (`JAVA_TOOL_OPTIONS` and friends) would reach the JVM without one, which is why the shipped deploy files refuse those names (§6.4, §9).

To re-pin on a `DUCKDB_CORE_VERSION` bump, update **three inputs together**: the
Dockerfile core ARG, `deploy/duckdb-extensions.sha256` (eight sums plus the four
`# <name> commit=<hash>` annotations), and the commit-bound licence inventory in
`buildSrc/src/main/kotlin/DuckDbExtensionLicenses.kt`. In a temporary directory,
for each of `linux_amd64` and `linux_arm64` and each of `httpfs`, `aws`, `iceberg`,
`avro`, fetch `https://extensions.duckdb.org/v<core>/<platform>/<name>.duckdb_extension.gz`,
decompress with `gunzip`, and run `sha256sum` on the **decompressed** file. Write
sorted `sha256sum -c` lines relative to `/opt/duckdb/extensions`; inspect each
binary's final 512 bytes (`tail -c 512 <file> | strings`) for its extension
repository commit, core and platform. Both platforms must report the same commit
per extension. Refresh each extension's linked closure,
`deploy/duckdb-extension-ports/<name>.tsv`, at those commits and the core version, as
DEVELOPMENT.md's third-party notices section describes: the vcpkg port closure, direct
and transitive, plus the DuckDB core `third_party` code and port-vendored code that
both binaries' symbols witness. Review every new component's `LICENSE`, update its
licence in `DuckDbExtensionLicenses.kt` and vendor any new SPDX text from the snapshot
named in the allowlist. Run `:modules:app:generateThirdPartyNotices`, the buildSrc licence
fixtures and `DockerfileDuckDbExtensionPinTest`, rebuild the jar, and prove one
bad sum refuses the image build before building the restored pins successfully.

**`-Duser.timezone=UTC` is normative, not a suggestion.** The type system assumes a UTC JVM: ingest normalizes timezone-aware source values to UTC and treats naive source timestamps as already-UTC ([Type System §8.4](/docs/type-system.md#84-timestamp-timezone-normalization)), and every internal `TIMESTAMPTZ` is stored and rendered in UTC. A container whose JVM default zone is anything else silently shifts rendered timestamps. No image entrypoint sets it: every deployment — container, bare JVM or JAR (§3.2, §6.5) — MUST pass `-Duser.timezone=UTC` itself.

Heap and container-memory sizing: §6.6.

**Native thread stacks.** The image sets `GLIBC_TUNABLES=glibc.pthread.stack_cache_size=0` before its build-time
and application JVMs start (#685). This disables reuse of native thread-stack mappings when the JVM hosts the
script isolate. An operator-supplied `GLIBC_TUNABLES` replaces the image value: retain
`glibc.pthread.stack_cache_size=0` when adding other colon-separated tunables. `DockerfileGraalJsPinTest` pins
the declaration and its position before the JVM launches.

### 3.2 JAR distribution

- Published to Maven Central: `co.datapipelines:datapipelines-app:{version}`.
- Executable Spring Boot fat JAR (`./gradlew bootJar`).
- Runnable directly: `java -jar datapipelines-app.jar`.

### 3.3 Module artifacts (for embedding)

- Each module published to Maven Central: `co.datapipelines:typesystem`, `co.datapipelines:dag`, etc.
- Allows third parties to embed parts of datapipelines.co in their own products (e.g., use our type system, or our MCP server in a different application).
- Marked "experimental API" until v1.1 stabilizes module boundaries.

### 3.4 Source distribution

- GitHub release per version, with signed tag.
- Source tarball attached to the release.
- Reproducible builds (Gradle lockfile committed, exact JDK pinned via toolchain).

### 3.5 JDBC driver matrix (what ships in the image)

Which drivers are present is a **packaging** property of the artifact, not a configuration one. The licensing rationale and the driver-class lookup live in [Datasources §10](/docs/datasources.md#10-jdbc-driver-packaging); this table is the operator's view of the published image.

| Dialect | In the published image? | How to get it otherwise |
|---|---|---|
| `POSTGRES` | Yes (bundled) | — |
| `MSSQL` | Yes (bundled) | — |
| `H2` | Yes (bundled — also the staging engine) | — |
| `DUCKDB` | Yes (bundled) | — |
| `SQLITE` | Yes (bundled) | — |
| `LAKE` | Yes (bundled — the same `duckdb_jdbc` jar as `DUCKDB`) | — |
| `ORACLE` | **No** (OTN license) | Rebuild with `./gradlew -Poracle bootJar`, **or** drop `ojdbc11.jar` into `lib/` |
| `MYSQL` | **No** (GPL-2.0 + FOSS exception, redistribution unverified) | Rebuild with `./gradlew -Pmysql bootJar`, **or** drop `mysql-connector-j.jar` into `lib/` |

**The `lib/` drop-in.** Spring Boot's `PropertiesLauncher` adds every jar in `loader.path` to the application classpath ([Module Structure §5.4.1](/docs/module-structure.md)). For the container, mount the DRIVER jar read-only into `/app/lib/`: the image ships that directory empty, `root:root` mode `0555`, and sets `ENV LOADER_PATH=/app/lib,/opt/graaljs` itself: the drop-in directory, then the directory holding the image's JavaScript isolate jar, as one list because a `-Dloader.path` would override the variable, and each entry absolute because a relative entry resolves against the working directory and the image sets no `WORKDIR`. Neither `deploy/compose.yml` nor the Helm chart sets `LOADER_PATH`. For bare JVM, place it in a `lib/` directory beside `datapipelines-app.jar` and start with `-Dloader.path=lib` (relative to the working directory) or an absolute path. This is the no-rebuild path — the operator accepts the driver's license by supplying it.

Registering a datasource whose dialect has no driver on the classpath fails at save time with `datasource.driver_not_loaded` ([Datasources §9](/docs/datasources.md#9-validation-rules)) — a deployment/packaging error, not a bad payload. The same payload succeeds after the driver is supplied.

---

## 4. Required Infrastructure

### 4.1 Postgres (metadata DB)

- **Version**: 14+ recommended.
- **Purpose**: persistent storage for pipelines, templates, datasources, executions, audit log.
- **Provisioning**: operator-managed. We support connecting to an existing Postgres instance.
- **Database size**: small (a typical deployment has < 1 GB metadata).

Connection keys (URL, credentials, Hikari pool size) are defined in [Configuration §2 / §3.13](/docs/configuration.md#2-required-configuration).

### 4.2 Redis (result store and coordination)

- **Version**: 6+ recommended.
- **Provisioning**: operator-managed.
- **Purpose** — four distinct workloads share one store:
  1. **Caller results.** Every completed execution's caller result is materialized to Redis and read through the cursor ([REST API §7](/docs/rest-api.md#7-result-delivery)). There is no inline-vs-large split.
  2. **Idempotency keys** (`Idempotency-Key` records, retained per `datapipelines.idempotency.ttl-seconds`).
  3. **Cancellation flags** (`dp:cancel:{execution_id}`) — how `DELETE /executions/{id}` reaches the executing instance ([DAG Executor §8.3.1](/docs/dag-executor.md#831-the-registry)).
  4. **Post-completion event log**, 1 hour, backing SSE replay ([REST API §10.3](/docs/rest-api.md#103-replay-sse-stream)).

Connection keys are defined in [Configuration §2 / §3.1](/docs/configuration.md#2-required-configuration); TTL and size limits in [Configuration §3.5](/docs/configuration.md#35-results).

#### 4.2.1 Required Redis configuration

**`maxmemory-policy noeviction`.** This is not tuning advice — it is a correctness requirement. Any LRU/LFU eviction policy lets Redis silently discard keys under memory pressure, and every key class above is load-bearing:

- an evicted **result** turns a completed execution into a spurious "expired" 404 before its TTL;
- an evicted **idempotency key** lets a client retry execute the pipeline a second time;
- an evicted **cancellation flag** makes `DELETE /executions/{id}` a no-op, leaving the execution running;
- an evicted **event-log entry** silently truncates SSE replay.

Every key the app writes carries an explicit TTL, so `noeviction` does not leak: the store drains on its own schedule. Under genuine memory exhaustion the app fails loudly (`result.storage_unavailable` on the write path) rather than corrupting semantics quietly.

**Where the password lives (#189, #196).** In the reference compose files the Redis password has one home inside the container: the service's **environment** (`REDISCLI_AUTH`). The healthcheck's `redis-cli` reads it from there, and the server reads its `requirepass` from the same variable through **stdin** — `redis-server -` takes its configuration from standard input, and a `sh -c` wrapper feeds it a heredoc, then `exec`s so `redis-server` stays PID 1. Nothing carries the value on a command line (`docker inspect … {{.Config.Cmd}}` and `ps` inside the container show the variable's NAME only), and nothing writes it to a file. `--maxmemory` and `--maxmemory-policy noeviction` stay on argv; options after `-` are read after stdin and win. The wrapper is `sh`, so the service declares `user: redis` — the image's entrypoint drops privileges only when argv[0] is `redis-server`. Operators: the next `app.sh --start` (or `docker compose up -d`) after this change **recreates** the Redis container; what that loses is exactly what any Redis restart loses (§7: unexpired results, idempotency records, cancellation flags, the event log — all volatile by design).

#### 4.2.2 Sizing

Budget for the peak sum of all four workloads:

```
peak ≈ (datapipelines.result.max-size-bytes × concurrent recent executions)   ← dominant term
     + idempotency keys (small, × datapipelines.idempotency.ttl-seconds window)
     + 1h of post-completion event logs (small, proportional to nodes × executions)
     + cancellation flags (negligible)
```

The result term dominates and is the only one worth arithmetic. Results live for their TTL (`datapipelines.result.ttl-default-seconds`, clamped between ttl-min/ttl-max — [Configuration §3.5](/docs/configuration.md#35-results)), so "concurrent recent executions" means executions completed within one TTL window, not executions running right now. Worst case with defaults (100 MB cap, 300 s TTL) is dominated by how many callers actually pull 100 MB results; the practical lever is lowering `result.max-size-bytes` and pointing bulk workloads at `output.target: datasource` instead — the [explicit NOT-goal](/docs/rest-api.md#7-result-delivery) of result delivery is bulk data transfer.

512 MB is a reasonable starting point for small deployments; raise it before raising `result.max-size-bytes`.

### 4.3 Network egress

The app must reach:
- Configured datasources (PG, Oracle, MSSQL, MySQL, DuckDB files, SQLite files).
- The metadata Postgres.
- Redis.
- (Optional) OTLP collector, Sentry, KMS — if those features are configured.

The app **does not** require inbound network access beyond the HTTP/MCP ports it serves.

---

## 5. Configuration

**[configuration.md](/docs/configuration.md) is the single authority for every configuration key** — YAML path, env var name, default, and description. This spec deliberately does **not** restate key names or defaults: a duplicated default is a default that drifts. What follows is only the operator's startup checklist and the file-mounting mechanics.

Use the exact environment placeholders in [Configuration §5](/docs/configuration.md#5-full-applicationyml-template). Most names use upper snake case, but provider, bootstrap, promotion and dashboard settings have explicit exceptions; deriving a name from the YAML path can leave a setting unchanged.

**Before any of this, two variables:** `DATAPIPELINES_ENV` is your organisation's name for this deployment and `DATAPIPELINES_POSTURE` is `development` or `hardened`. A named environment with no posture does not start. [Environments](/docs/environments.md) is the page for that decision; [Configuration §3.23](/docs/configuration.md#323-environment-and-posture) is the key reference.

### 5.1 What the app requires to start

The app **fail-fasts** on startup if any of the following is missing. Full definitions: [Configuration §2](/docs/configuration.md#2-required-configuration).

1. Metadata Postgres: `spring.datasource.url`, `spring.datasource.username`, `spring.datasource.password`.
2. Redis: `datapipelines.redis.host`.
3. `datapipelines.jwt.secret` — internal JWT signing secret.
4. `datapipelines.db.encryption-key` — AES-256 master key for datasource credentials. **There is no fallback source**: no KMS lookup, no auto-generated key file. Lose it and every stored datasource credential is unrecoverable, so it belongs in a secret manager and in the operator's backup plan. (KMS sourcing is a [ROADMAP §2](/docs/roadmap.md#2-v11-candidates) item.)
5. **At least one authentication method**: a fully-configured OIDC provider under `datapipelines.auth.oidc.providers` (each with `client-id`, `client-secret`, and `issuer-uri`), **or** local password accounts (`datapipelines.auth.local.enabled=true`, [Auth §5A](/docs/auth.md#5a-local-password-accounts-optional)). The first admin comes from `datapipelines.auth.bootstrap-admin-email` either way ([Auth §4.4](/docs/auth.md#44-bootstrap-admin)). A provider entry whose `client-id` is empty is ignored with a WARN, not counted. The client-id/secret env var names are chosen by the deployment (`GOOGLE_CLIENT_ID`, `OKTA_CLIENT_ID`, …) — they are the one deliberate exception to the derivation rule above.

   **The operator's first-admin story with local accounts** (no IdP): set `datapipelines.auth.bootstrap-admin-email` to the admin address, enable `datapipelines.auth.local.enabled`, and seed the one-time credential — preferably as a pre-computed hash from `./gradlew :modules:auth:hashPassword` into `datapipelines.auth.local.bootstrap-password-hash` (the plaintext form exists for demos). Hand the password to the admin out-of-band; the app forces a change at first login ([Auth §5A.2](/docs/auth.md#5a2-seeding-the-first-admin)), and a deployment still running the seeded credential announces itself with a startup WARN. Every account after the first is created by an admin on the user-administration screen — there is no self-registration and no self-service email reset: a forgotten password means an admin resets it there, which issues a new one-time credential under the same forced-change rule. **With outbound mail configured** (`DATAPIPELINES_MAIL_HOST` + `DATAPIPELINES_MAIL_FROM` — any SMTP server, Postmark's included; the nine variables are [Configuration §3.27](/docs/configuration.md#327-mail) and `deploy/secrets.env.example` names them) that credential is **emailed to the user** and sys-ops (`DATAPIPELINES_MAIL_OPS_TO`) hears about every new user, local or first social login ([Auth §5A.8](/docs/auth.md#5a8-mail-the-welcome-mail-and-the-new-user-notice)); the admin screen then shows where the password went, not what it was. Mail is on exactly when those two are set — there is no flag — and it then also requires `DATAPIPELINES_AUTH_BASE_URL` (the login URL in the mail).

Everything else has a default and is optional. One optional key adds a startup requirement of its own: `datapipelines.scheduler.notifications.enabled=true` (scheduler notification mail, off by default — [Scheduler §2.3](/docs/scheduler.md#23-notification-settings-442), #442b) refuses to start, naming the keys, unless the shared mail transport (`datapipelines.mail.host` and `.from`, [Configuration §3.27](/docs/configuration.md#327-mail)) and `datapipelines.auth.base-url` (the link every notice carries) are set; under the `development` posture it also sends nothing until `datapipelines.scheduler.notifications.recipient-domains` lists a domain.

### 5.2 Everything else

Optional keys — executor concurrency, staging memory, result TTLs and caps, SSE heartbeat and disconnect grace, rate limits, idempotency TTL, template cache, UI theme, retention windows, observability — are cataloged with their defaults in [Configuration §3](/docs/configuration.md#3-optional-configuration-with-defaults). Resolution precedence (env > profile YAML > base YAML, plus the two per-entity runtime overrides) is [Configuration §4](/docs/configuration.md#4-precedence).

The keys an operator most often changes at deploy time are `datapipelines.result.max-size-bytes` (Redis sizing, §4.2.2), `datapipelines.executor.max-concurrent-executions-per-instance` and `datapipelines.staging.h2.max-memory-mb` (heap sizing, §6.6), and `datapipelines.executor.execution-timeout-seconds` (the wall clock that bounds any single execution).

#### Log output format — turning JSON on (#337)

`DATAPIPELINES_OBSERVABILITY_LOGGING_FORMAT` decides what stdout carries ([Configuration §3.15](/docs/configuration.md#315-observability), [Observability §3.1](/docs/observability.md#31-format)): `json` — one structured JSON object per line, the machine-parsed format a log collector wants — or `console`, the human-readable development format. The value is resolved before logging initialises; the set is closed, and a mistyped value **refuses startup** with the offending value named rather than silently falling back. Redaction ([Observability §9.2](/docs/observability.md#92-redaction--normative)) runs under both values — there is no format that bypasses it.

What each deploy path resolves **today**, with nothing set beyond what the repo ships:

| Path | Resolves | Because |
|---|---|---|
| `app.sh` deploy, the VPS included | `console` | `deploy/env/defaults.env` sets it; unchanged until the operator moves it. Turn JSON on by setting `DATAPIPELINES_OBSERVABILITY_LOGGING_FORMAT=json` in `deploy/secrets.env` (or the environment the compose stack reads) and redeploying. |
| Raw `docker compose` run | `json` | `deploy/compose.yml` passes `${DATAPIPELINES_OBSERVABILITY_LOGGING_FORMAT:-json}` through. |
| Bare jar (`java -jar`) | `json` | The shipped `application.yml` default. |

A container deployment that turns JSON on changes nothing else: the container runtime already collects stdout, and the JSON line carries `correlation_id`/`execution_id` at the root — the fields an operator quotes in an incident ([Observability §3.3](/docs/observability.md#33-correlation-id-propagation)). Log **levels** stay Spring Boot's own `logging.level.*` ([Observability §9.1](/docs/observability.md#91-configuration-keys)).

### 5.3 Full config file

Every setting is an environment variable, and `deploy/env/defaults.env` lists all the non-secret ones with their shipped values (`deploy/secrets.env.example` names the rest) — [Environments](/docs/environments.md) is the operator's page for what to set and where. Operators who prefer a YAML file can mount `application.yml` at `/etc/datapipelines/application.yml` (configurable via `SPRING_CONFIG_ADDITIONAL_LOCATION`); this is the supported route for **more than one OIDC provider**, whose nested list is awkward as variables. Every key is expressible either as YAML or as its derived env var; the OIDC provider list is a nested structure and is normally supplied as YAML with `${...}` placeholders for the secrets. A complete annotated template is in [Configuration §5](/docs/configuration.md#5-full-applicationyml-template).

---

## 6. Deployment Patterns

### 6.1 Single-instance (dev / small team)

One instance handles everything — UI, API, MCP, pipeline execution. Docker Compose or bare JVM.

Suitable for: dev, evaluation, small teams (< 20 users).

### 6.2 Multi-instance horizontal scaling (production)

The application is **stateless** for all CRUD operations, UI, MCP, and auth. Multiple instances run behind a load balancer and serve requests interchangeably.

```
                    ┌─────────────┐
                    │ Load Balancer│
                    └──┬───┬───┬──┘
                       │   │   │
              ┌────────┘   │   └────────┐
              ↓            ↓            ↓
        ┌──────────┐ ┌──────────┐ ┌──────────┐
        │Instance A│ │Instance B│ │Instance C│   ← stateless web/API/MCP/UI
        └────┬─────┘ └────┬─────┘ └────┬─────┘
             │            │            │
             └────────────┴────────────┘
                      │     │
              ┌───────┘     └────────┐
              ↓                      ↓
        ┌──────────┐           ┌──────────┐
        │ Postgres │           │  Redis   │
        │ (shared) │           │ (shared) │
        └──────────┘           └──────────┘
```

**No sticky sessions required.** Each request is independent.

**Client addresses behind the load balancer (`datapipelines.auth.trusted-proxies`).**
Every instance sees the LB's address as `remoteAddr`, so anything keyed or recorded on
it — the per-IP login rate limiter, every audit `source_ip` — collapses to one
deployment-wide value: without configuration, the login budget (default 10/min) is a
single shared bucket any client can exhaust for everyone. Set the key to the CIDRs of
proxies you control (a bare IP is a host CIDR):

```yaml
datapipelines:
  auth:
    trusted-proxies: 10.0.0.0/8   # the LB's network
```

The resolution is spoof-safe: when the direct peer is NOT in the list, the
`X-Forwarded-For` header is ignored entirely — an untrusted peer cannot forge its way
past the limiter by setting it. When the peer IS trusted, the client is the rightmost
header entry that is not itself trusted. Each entry must parse as a CIDR or startup is
refused ([Configuration §3.4/§7](/docs/configuration.md#34-auth)) — a typo'd range must not
silently widen proxy trust. The shipped default is EMPTY: a deployment with no proxy
in front behaves exactly as before, and the header stays ignored.

#### What the edge sets, and what it must not touch (188, #188)

The app states its own security posture on **every** response — HTML, JSON, SSE and the
`/mcp` refusal alike — and pins it with tests (`SecurityHeadersTest`, the wire-level case
in `AuthHttpBoundaryTest`, `ApplicationSmokeTest`). What it sends:

| Header | Value | Why |
|---|---|---|
| `Content-Security-Policy` | `default-src 'self'; script-src 'self'; style-src 'self' 'sha256-pgvDUBa4IjFA2yuSJ2cqcyxmNYJMborsd0ORcRv9vw8='; img-src 'self' data:; font-src 'self'; connect-src 'self'; frame-ancestors 'self'; base-uri 'self'; form-action 'self'; object-src 'none'` | Enforced, no `'unsafe-inline'`, no nonce, and — since #195 — no `'unsafe-eval'` on ANY route: every script and stylesheet is a file the app serves. `img-src` names no external host (#197): the OIDC profile picture is fetched server-side by `GET /avatar` from an operator-allowlisted host (`datapipelines.auth.oidc.providers[*].picture-hosts`, [Configuration §3.4](/docs/configuration.md#34-auth)), and the page's `<img>` points at the proxy — the provider's picture URL never reaches the browser. **The one exception 188 shipped** (`GET /pipelines/{id}/editor` carried `script-src 'self' 'unsafe-eval'` for Alpine's standard build, owner ruling 2026-09-21) **is retired (#195)**: the editor's page runs Alpine's CSP build (the same 3.14.1, same path — the vendor manifest records the npm tarball), its expressions are pure property paths, and the editor-only policy is deleted; `SecurityHeadersTest` pins the directive's absence on the editor route, and the browser suite's zero-violation collector is the live proof the page needs no eval. `style-src` carries one SHA-256 hash — the exact `<style>` element Cytoscape injects on the editor page at init (`'sha256-pgvDUBa4IjFA2yuSJ2cqcyxmNYJMborsd0ORcRv9vw8='`, printed in full here; it is DERIVED, not typed: `SecurityHeadersTest` computes it from the vendored sheet `SecurityHeaders.CYTOSCAPE_STYLESHEET` carries verbatim, so a Cytoscape bump that changes the sheet goes red rather than silently unblocked); a hash names that sheet and nothing else, so on routes that never load Cytoscape it permits a style element that never occurs, which is what keeps ONE policy for every policed route. `/sitemap.xml` carries no policy (an XML data document; the browser's own viewer is the only style it ever has). |
| `X-Frame-Options` | `SAMEORIGIN` | The product's intent — embedded dashboards on its own origin are on the roadmap; nothing is meant for a foreign frame. `frame-ancestors 'self'` says the same in CSP terms. |
| `X-Content-Type-Options` | `nosniff` | |
| `Referrer-Policy` | `strict-origin-when-cross-origin` | |
| `Permissions-Policy` | `camera=(), microphone=(), geolocation=()` | The product uses none of these. |
| `X-XSS-Protection` | `0` | The legacy filter off, as every current guide recommends; the CSP is the control. |
| `Cache-Control` | `no-cache, no-store, max-age=0, must-revalidate` on authenticated responses; the public site and docs set their own public windows | |

**The app deliberately sends no `Strict-Transport-Security`.** It cannot know it is
behind TLS — it sees plain HTTP from the edge — and an HSTS header on a plain-HTTP
deployment locks a browser out of it. The edge (the reverse proxy / load balancer that
terminates TLS: Caddy, nginx, an ALB) owns it. What the product EXPECTS of any edge in
front of it:

- **HSTS, one year, subdomains included, no `preload`:**
  `Strict-Transport-Security: max-age=31536000; includeSubDomains`. Preload is a
  registry submission the operator makes deliberately for their own domain, not a header
  the product should imply on their behalf.
- **Pass the app's headers through untouched.** An edge MUST NOT weaken
  `Content-Security-Policy` (no rewriting to add `'unsafe-inline'`, no stripping) and
  MUST NOT relax `X-Frame-Options` beyond `SAMEORIGIN` or drop `frame-ancestors`. An
  edge that sets its own `Referrer-Policy` should set the same value, not a looser one.
- **TLS termination and redirect:** HTTP → HTTPS redirect at the edge; the app is reached
  over the private network only. The app never needs to know the public scheme for its own
  redirects: every `Location` it sends is RELATIVE (`/dashboard`, `/login?error=…` —
  `server.tomcat.use-relative-redirects`, #207) and resolves against the origin the browser
  used, so no `X-Forwarded-Proto` handling is required for redirects, and an edge that
  forwards it changes nothing. (Before #207 the container built absolute `http://` Locations
  behind the edge, and the browser refused to follow them: `form-action 'self'` for plain
  forms, mixed content for htmx requests.)
- Forward the client address (`X-Forwarded-For`) and declare the edge in
  `datapipelines.auth.trusted-proxies` — the paragraph above.

The reference edge for datapipelines.co lives in its own repository by design; this
section is the contract it implements, not its configuration.

#### What's stateless (any instance serves any request)

- REST API (all CRUD, list, detail, execute endpoints)
- MCP server
- UI (all screens — Thymeleaf server-rendered)
- Auth (JWT sessions, API key validation — both backed by Postgres)
- Template rendering (reads from Postgres, per-instance LRU cache)
- Pipeline validation (reads from Postgres)

#### What's instance-local (tied to the instance that started it)

- **In-flight pipeline executions.** When `POST /pipelines/{id}/execute` hits Instance A, the execution runs on Instance A's JVM (coroutines + per-execution H2 staging). The SSE stream is a direct connection from Instance A to the client.
- **Implication:** if the SSE connection drops mid-execution, there is **no reconnection or resumption path**. Instance A starts a grace timer (`datapipelines.sse.disconnect-grace-seconds`, default 30) and, if the execution has not reached a terminal event by the time it elapses, cancels it — the execution ends `ABORTED` ([REST API §6.8](/docs/rest-api.md#68-client-disconnect)). A client that loses its stream should assume the abort and re-execute. This is deliberate: an execution nobody is waiting for must not keep holding source-database connections and staging memory.
- **Cross-instance cancel works anyway.** `DELETE /api/v1/executions/{id}` may land on Instance B, which has never heard of the execution. Instance B writes a Redis cancellation flag; Instance A honors it on its next heartbeat tick or node boundary — worst-case latency ≈ one heartbeat interval ([REST API §10.4](/docs/rest-api.md#104-cancel-execution), [DAG Executor §8.3.1](/docs/dag-executor.md#831-the-registry)). No sticky sessions are needed for cancellation to be reliable.
- **Completed executions are not instance-local at all.** Results live in Redis for their TTL and are readable from any instance via the cursor ([REST API §7](/docs/rest-api.md#7-result-delivery)); execution metadata is in Postgres; the 1-hour event log is in Redis. Only the *running* execution is pinned to one JVM.
- **Instance crash:** loses only that instance's in-flight executions (H2 staging is in-memory and non-recoverable). Their rows are swept to `ABORTED` by the stale-execution sweep (marked `pipeline.execution.instance_lost` once they are older than `datapipelines.executions.stale-timeout-minutes`; every replica runs the idempotent sweep, so no surviving-instance coordination is needed). Completed results and history are unaffected; the lost work must be re-executed.

#### Multi-instance checklist

| Requirement | How |
|---|---|
| Shared Postgres | All instances connect to the same external Postgres. |
| Shared Redis | All instances connect to the same Redis (results, idempotency keys, cancellation flags, event log, datasource pool invalidation) with `maxmemory-policy noeviction` — §4.2.1. A per-instance Redis breaks cross-instance cancel, result reads and pool invalidation. |
| Identical image | Same Docker image on every instance — design system CSS, JS, templates baked in. |
| DB migrations | Flyway uses Postgres advisory locks — concurrent startup is safe (one runs, others wait). |
| SSE heartbeat | Server sends `: heartbeat` comments every 15s to prevent LB idle-timeout kills. See [REST API §6.6](/docs/rest-api.md#66-heartbeat-keepalive). |
| LB idle timeout | Configure to ≥ 120s (or rely on heartbeat). |
| Health checks | `/health` and `/ready` work independently per instance. |
| Size per-instance limits by replica count | The execution-slot ceiling `datapipelines.executor.max-concurrent-executions-per-instance` (default 100) is **per instance** (050/R2): N replicas admit **N × the setting** in total against the source databases, and the tempdb heap multiplier of [Configuration §3.2](/docs/configuration.md#32-executor) applies per box the same way. Raise replicas with that multiplication in mind, not just the per-instance number. |

### 6.2A Vendored browser assets — provenance and the rebuild recipe (#10 L3a)

Every third-party file under `static/vendor/` is recorded in
`static/vendor/design-system/vendor-manifest.json` — per file: version, sha256, source, licence — and
pinned by the audit tests (`VendoredAlpineAuditTest`, `VendoredHtmxAuditTest`,
`VendoredPlotlyAuditTest`, …): a served byte that stops matching its recorded hash is a red build,
and so is a manifest entry nobody serves. The manifest also carries the FIRST-PARTY client runtime and
its adapters (`js/datapipelines-dashboard*.js`, the same sha256 discipline — a hand edit must move the
manifest in the same commit).
`generateVendoredAssetNotices` (`:modules:web:generateVendoredAssetNotices`) generates the offline
licence inventory and full texts served at `/vendor/THIRD-PARTY-NOTICES.txt` and copied beside the website export assets.

**Plotly 4.1.1** ships as two custom bundles built with the package's own procedure (its
`CUSTOM_BUNDLE.md`), from the pinned tag, plus source patches (`patches/0001-colorbar-style-reset-without-attribute.patch`:
the colorbar's fill reset wrote `.attr('style', '')`, an inline-style attribute the enforced CSP refuses, and now removes
the attribute — #525). The second patch, `patches/0002-heatmap-pixelated-hint-via-cssom.patch`, sets the default heatmap
image’s `image-rendering` fallbacks through the CSSOM in their original order, ending in `pixelated` (#531).
The build is path-independent since `patches/0003-inline-css-id-independent-of-clone-directory.patch`
(#571): the inline-CSS plugin hashed the clone's absolute path (#530); the in-tree plugin hashes the
root-relative CSS path under the historical directory as a constant salt, keeping the committed bytes unchanged.
The salt is historical; the next Plotly upgrade may keep or drop it, since the pins move then anyway.
Patch `patches/0004-drop-unreachable-maplibre-css-require.patch` removes the maplibre-gl stylesheet
require from the registry: neither custom trace list includes a `map` trace, so the stylesheet is
unreachable; drop this patch when a `map` trace joins a trace list.
To rebuild any bundle — a trace added, a security fix adopted — run exactly this and commit the
OUTPUTS with their new hashes. The #531 rebuild used Node 24.21.0 + npm 11.19.0 + `npm ci`
(the earlier #525 build also matched Node 22.23.3 + `npm ci` and Node 24.21.0 + `npm i`).

```sh
cd "$(mktemp -d)"
git clone --depth 1 --branch v4.1.1 https://github.com/plotly/plotly.js.git plotly.js
cd plotly.js
# the numbered vendor patches, applied in order; a new patch is a new file here
git apply <repo>/modules/web/src/main/resources/static/vendor/plotly/patches/*.patch
npm ci --no-audit --no-fund
# the 2D bundle (the default): scatter, bar, pie, histogram, box, heatmap
npm run custom-bundle -- --traces scatter,bar,pie,histogram,box,heatmap --strict --out 2d
# the 3D bundle (dashboards with a 3D trace): the six above + scatter3d, surface, mesh3d
npm run custom-bundle -- --traces scatter,bar,pie,histogram,box,heatmap,scatter3d,surface,mesh3d --strict --out 3d
cp dist/plotly-2d.min.js dist/plotly-3d.min.js  <repo>/modules/web/src/main/resources/static/vendor/plotly/
```

A patch is `git format-patch -1` against the tagged source with preceding patches applied, so `git apply` takes the
numbered series on a pristine clone; never hand-edit the
minified text. Run `scripts/verify-plotly-bundles.sh` (exit 0) before copying a rebuild in; the lander runs it
on any diff under `static/vendor/plotly/` or to the manifest's plotly entry. The verifier rebuilds in a fresh
temporary directory and compares both bundles: exit 0 means a match, 1 a byte mismatch, 2 a failed rebuild.
It is not part of the gate, pregate or CI. Remove the manual recipe's temporary clone after use.

`plotly.css` is the npm package's own `dist/plotly.css` — the release build's strict-CSP sheet, the
same rules the bundle would inject at runtime. It is part of the CSP DESIGN-AROUND ([Dashboards
§6.6](/docs/dashboards.md#66-the-states-notifications-and-the-csp)): each host pre-places the empty
`<template id="plotly.js-style-global" class="no-inline-styles">` marker Plotly checks (the runtime's
fallback creates the same template for a host without one), the page loads this file instead, and `style-src` stays `'self'` plus Cytoscape's one hash — no policy change, no
`'unsafe-hashes'`, no `'unsafe-eval'`. Sizes (4.1.1 with patches 0001–0004, #525, #531, #571 and #570): 2D 1,170,988 B (406,683 gz), 3D
1,748,030 B (576,983 gz), plotly.css 7,467 B; gzip uses zlib deflate level 9, gzip framing, mtime 0
(no filename), as reproduced by the JDK's `GZIPOutputStream` with `Deflater.BEST_COMPRESSION`; the npm tarball's sha1
`f797e8f2d54603fd9d11b08aae8eb31147200b67` verifies the source against the tag. Node 22+ per
Plotly's own build matrix (the manifest's `build` field records the exact versions). After ANY rebuild: update the manifest's `sha256` map, the `sizes`, and the
trace lists if they changed — `VendoredPlotlyAuditTest` guards all three raw byte counts, both gzip counts
and their pinned method, as well as hashes and trace lists, and fails the build on a mismatch.

### 6.3 Docker Compose (dev / evaluation)

Reference compose file provided in `deploy/compose.yml`. Single instance + Postgres + Redis.

Compose is **one loader of the environment-variable contract**, not the contract itself ([Environments §3](/docs/environments.md#3-the-variable-contract)): the same settings drive a bare JAR, systemd, Kubernetes, ECS and Nomad. It reads TWO env files, in this order: `deploy/env/defaults.env`, which is tracked and carries every non-secret default, then `deploy/secrets.env`, which is git-ignored and holds every credential plus this deployment's own overrides (`DATAPIPELINES_POSTURE=hardened`, `DATAPIPELINES_ENV=prod`, the real database URL, the OIDC client).

```bash
docker compose -f deploy/compose.yml \
  --env-file deploy/env/defaults.env \
  --env-file deploy/secrets.env up -d
```

The app service passes **every** `DATAPIPELINES_*` variable the app binds, each with
the same default `application.yml` ships — so a key that works against a host-run app
(by exporting the variable) reaches the container too, and leaving it unset yields the
shipped default. `scripts/compose-env-audit.sh` diffs the compose block against
`application.yml`'s placeholders — and the two tracked env files — and fails on a missing
pass-through, a diverged default, a variable declared in both tracked files or in neither,
or a posture-dependent key passed with a default instead of in the valueless form. It runs
on **every `./gradlew build`**, not when someone remembers to run it.
It also inventories every app-container environment name, refuses JVM code-loading names,
and requires a reasoned allowlist for names outside `DATAPIPELINES_*`, rejecting stale entries.

### 6.3A Promotion: a receiver deployment (055)

A second deployment that receives released content from the first ([Versioning §10](/docs/versioning.md#10-promotion-ui-driven-separate-use-case)). Both sides are the same image in different postures — there is no separate build.

**On the RECEIVER (e.g. uat):**

```bash
DATAPIPELINES_ENV=uat
DATAPIPELINES_POSTURE=hardened                            # authoring defaults OFF here
DATAPIPELINES_DEPLOYMENT_PROMOTION_SERVER_KEY=<the shared secret>
```

**On the SENDER (e.g. dev):**

```bash
DATAPIPELINES_ENV=dev
DATAPIPELINES_POSTURE=development                         # this is where content is built
DATAPIPELINES_DEPLOYMENT_PROMOTION_TARGET_URL=https://uat.example.com
DATAPIPELINES_DEPLOYMENT_PROMOTION_TARGET_KEY=<the same shared secret>
```

`DATAPIPELINES_ENV` is **your** name for each deployment and nothing branches on it; the receiver records the sender's label as the `source_env` of what it received. `DATAPIPELINES_POSTURE` is what actually changes behaviour — `hardened` defaults `authoring-enabled` to `false`, which is the receiver's whole configuration. Set `DATAPIPELINES_DEPLOYMENT_AUTHORING_ENABLED` explicitly if this one box must do both ([Environments §1](/docs/environments.md#1-why-two-variables)).

Generate the secret the way every other one here is generated — `openssl rand -base64 32` — and set the identical value on both sides. It is a **bearer credential**: it belongs in `deploy/secrets.env` with the rest of the secrets, never inline in a compose file or a chart's values.

What each setting buys, and what goes wrong without it:

| Setting | Consequence if wrong |
|---|---|
| `posture: hardened` (or an explicit `authoring-enabled: false`) on the receiver | With it `true`, promotion into it is **refused** (`pipeline.promotion.target_is_authoring`) — deliberately, because drafts belong in the authoring environment. Startup also REFUSES on a receiver that already holds drafts, naming them ([Configuration §7](/docs/configuration.md#7-config-validation)) |
| A promotion credential on the receiver | **Nothing configured means promotion is refused, always.** Fail closed. Since 091 the credential is a `server`-kind API key an `admin` mints on the receiver's API screen ([Auth §7.7](/docs/auth.md#77-key-kinds-and-published-endpoint-bindings)); the older `server-key` config value still works for one release and WARNs at boot |
| `target.base-url` + `target.server-key` on the sender | A base-url without a key **refuses startup**, naming both. A key that does not match the receiver's gets `401 auth.promotion.key_invalid` at push time |
| Matching workspace NAMES on both | Promotion addresses workspaces by name (names are a global namespace; ids are not). A workspace that does not exist on the receiver is `404 workspace.not_found` — create it there first |
| The datasources the promoted pipelines reference, registered on the receiver | Pre-validated before anything is pushed; a missing one fails the whole batch with `pipeline.promotion.missing_datasources` and leaves the receiver untouched. **Register them with the receiver's own credentials** — a pipeline body never carries environment-specific connection details |
| The RECEIVER's request-body cap, `datapipelines.web.max-request-bytes` (default 2 MiB, at most 64 MiB — [Configuration §3.31](/docs/configuration.md#331-web-request-limits)) | A promotion is ONE push body (`POST /api/v1/promotion/push`); over the receiver's cap it is refused `413 request.body_too_large` (`details.limit_bytes`) before anything applies — atomic, nothing lands — and the sender reports the receiver's refusal verbatim. Measured 2026-09-28 (#291): the product's whole shipped example content — 40 templates and pipelines across the nyc, trade and lake families — is 73 KiB, 3.6% of the default; its largest document is 5.6 KiB. A single template may hold up to 256K characters (`datapipelines.templates.max-body-chars`), so a batch of several near-maximum templates can pass 2 MiB: raise the RECEIVER's key (the sender's cap is not involved) rather than splitting the promotion. |

**Minting the receiver's key** (091; the page is `/api-keys` since 179): on the RECEIVER, sign in as a super admin → the avatar menu's **API keys** → New API key → kind **Server** → name it after the sender (`dev → uat`) → copy the plaintext, shown once. Paste it into the SENDER's `DATAPIPELINES_DEPLOYMENT_PROMOTION_TARGET_KEY`. **Rotation:** mint a second `server` key on the receiver, set it in the sender configuration, restart the sender to bind the new credential, then revoke the first key. The receiver does not need a restart for stored-key issuance or revocation. (With the deprecated config value instead: set the new value on both sides and restart both.) The stored credential authenticates as its key identity rather than a human account; offboarding a human does not revoke that promotion credential.

**Direction is one-way by construction.** The receiver validates; it never calls the sender. A compromised receiver cannot reach back into the authoring environment.

**Running two stacks on one machine** (a rehearsal, or CI): `APP_COMPOSE_PROJECT` gives `./app.sh` a second isolated compose project with its own image tag, volumes and containers, and `APP_HOST_PORT` moves its published port off the first one's 8080 (the container port never moves). Both are lane knobs, not app config — that is why neither carries the `DATAPIPELINES_` prefix.

```bash
./app.sh --start                                             # dev, the sender, on :8080
APP_COMPOSE_PROJECT=uat APP_HOST_PORT=8081 ./app.sh --start  # uat, the receiver, on :8081
```

The sender reaches the receiver from INSIDE its container, so `target.base-url` is not the `localhost` URL your browser uses: on Docker Desktop it is `http://host.docker.internal:8081`, and on Linux it is the host's address on the docker bridge. Two compose projects are two networks; a service name from one does not resolve in the other.

### 6.4 Kubernetes (recommended for production)

Reference Helm chart in `deploy/helm/`. Includes:
- `Deployment` (N+ replicas, behind a `Service`; `terminationGracePeriodSeconds: 60` — #251, #277, #266: the `preStop` sleep (5 s) plus the scheduler's admission wait (up to 15 s) plus the drain's flush (up to 20 s) plus Tomcat's graceful shutdown (10 s) plus the persistence drain (up to 10 s) must complete before the pod is killed; the shipped `deploy/compose.yml` sets the matching `stop_grace_period: 60s` — §8.3.2 has the arithmetic).
- Externalized Postgres (managed recommended).
- Externalized Redis (managed recommended).
- `HorizontalPodAutoscaler` (scales on CPU).
- `PodDisruptionBudget` (availability during node drains).
- Management port exposure, all of it OFF by default (the render is unchanged
  until you opt in — [Configuration §3.14](/docs/configuration.md#314-framework-wiring-keys)):
  `management.expose` adds the management `containerPort` (from `management.port`, default 9090)
  and pins both `MANAGEMENT_SERVER_*` env entries — including the address opt-open `0.0.0.0` —
  and a headless `Service` (`clusterIP: None`: the scraper targets pod IPs, not a cluster VIP);
  `management.networkPolicy.enabled` adds the chart's one ingress `NetworkPolicy` for the app
  pods — the scraper peers on the management port AND your declared peers on the app port
  (below); `management.serviceMonitor.enabled` adds a `ServiceMonitor` for the headless
  `Service`. Any `extraEnv.MANAGEMENT_SERVER_ADDRESS` / `_PORT` double mapping is refused
  (Kubernetes honours the last duplicate env name silently).

**The management `NetworkPolicy` is a complete ingress policy for the app pods** (#712). A
NetworkPolicy that selects a pod isolates that pod for **every** ingress, not only for the ports
its rules name, and the allow rules of all policies selecting the pod are unioned
([Kubernetes: Network Policies](https://kubernetes.io/docs/concepts/services-networking/network-policies/#the-two-sorts-of-pod-isolation)).
A policy that admitted only the scraper would therefore cut the app port off — so the chart's
policy, rendered only with both `management.expose` and `management.networkPolicy.enabled`,
carries two rules, and the render refuses to enable it without either:

1. **Scraper → management port:** one peer per non-empty scraper selector
   (`scraperNamespaceSelector`, `scraperPodSelector`). The two are separate peers, OR'd: a pod in a
   matching namespace OR a matching pod in the app's own namespace. (One peer carrying both a
   `namespaceSelector` and a `podSelector` would be their AND — the Kubernetes rule.) Both empty
   is refused: an empty `from:` admits every source.
2. **Declared peers → app port (8080):** `appIngress.from`, a list of `NetworkPolicyPeer` objects
   (`namespaceSelector` / `podSelector` / `ipBlock`) copied verbatim — typically your ingress
   controller's namespace. Instead of a list, `appIngress.allowAll: true` is the explicit
   acknowledgement that every source may reach the app port; it renders the rule without a
   `from` and prints the acknowledgement as a comment in the rendered policy. Neither set is
   refused; both set is refused as a contradiction. An `ipBlock` of `0.0.0.0/0` is allow-all
   by another name.

Every selector is a full `LabelSelector` object (`matchLabels` / `matchExpressions`), copied
verbatim. A bare label map is refused at render time: it is not a selector, the API schema
rejects it, and an apply path that drops unknown fields would leave an empty selector, which
selects every namespace.

```yaml
management:
  expose: true
  networkPolicy:
    enabled: true
    scraperNamespaceSelector:
      matchLabels:
        kubernetes.io/metadata.name: monitoring
    appIngress:
      from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: ingress-nginx
```

**With policies you already own.** Policies are additive. If your own policies already select
these pods, they keep every allowance they grant, and the chart's policy adds exactly its
declared peers to that union. It never narrows your policy, and it widens it by precisely
what you declare. To leave an existing stricter app-port policy unwidened, declare no peer it
does not already admit. `allowAll: true` widens the app port to every source. Egress is not
touched (`policyTypes: [Ingress]`): §9's egress policy stays yours. With
`management.networkPolicy.enabled` off (the default) the chart renders no policy at all.

The pod keeps its root filesystem read-only and mounts a disk-backed `emptyDir` at `/tmp`.
This supports JVM native libraries and the default LAKE spill directories. Set
`temporaryStorage.sizeLimit` to bound the volume (empty by default), and set ephemeral-storage
requests/limits through `resources` for the workload. Do not use a memory-backed volume for
large LAKE spills: that would consume the memory budget spilling is intended to relieve.
Each pod has its own volume; Kubernetes removes it with the pod. Size it for concurrent
queries and see [Configuration §3.24](/docs/configuration.md#324-lake-datasource-engine-limits-dp-lake)
for explicit paths and cleanup after a process crash. Docker Compose uses the image's writable
`/tmp`; custom read-only deployments must mount a writable temporary directory too.

JavaScript (#8 lane 8b) uses `/opt/graaljs/js-isolate.jar` in the image, selected for the image
platform and checksum-verified at build. `DATAPIPELINES_SCRIPT_JS_ISOLATE_DIR=/opt/graaljs`
must name one directory in the launcher's effective `loader.path` system property or, when
unset, `LOADER_PATH` environment list (the image: `/app/lib,/opt/graaljs`). Startup refuses a
configured directory outside that list or without `js-isolate.jar`; the check never adds a
class-path entry. A bare jar without the isolate starts with JavaScript disabled, said once
at INFO; a JavaScript save/evaluate refuses `script.js.unavailable`, while JSONata works.

The nine JavaScript runtime keys (configuration.md §3.28) are named chart values under
`script.js`: `isolateDir`, `maxHeapMb`, `isolateMemoryMb`, `maxStatements`, `maxAstDepth`,
`maxStackFrames`, `maxOutputBytes`, `maxResultMb`, `maxInputMb` (env
`DATAPIPELINES_SCRIPT_JS_*`, the mechanical camelCase map). All nine default to empty, and an
empty or null value renders NO env entry — the image's own isolate directory and the app's derived
isolate memory stay in force, exactly like the DuckDB pair below; a rendered `""` would override
the image's `ENV` and fail the startup agreement above. `script.js.isolateDir` is not an
arbitrary path: the chart refuses, at render time, any value outside the image's launcher-path
entries (`/app/lib`, `/opt/graaljs`) — the same entries the startup rule admits — and an
`extraEnv` double mapping of any of the nine fails `helm template` like the DuckDB keys do. The
remaining eight keys are operator sizing: the chart renders them verbatim and does not validate
numbers — explicitly supplied zero values render as quoted "0", and an out-of-bound value fails the pod's startup, by name (configuration.md §3.28, §7).

The image pre-installs native resources in the root-owned, read-only
`/opt/graaljs/resources` cache and passes
`-Dpolyglot.engine.userResourceCache=/opt/graaljs/resources`. This is why the read-only root
remains supported: no language resources are installed at runtime. A Docker tmpfs at `/tmp`
must permit native-library mapping (for example `--tmpfs /tmp:rw,exec`); `noexec` fails in JNA
before JavaScript starts. Non-Docker deployments
must pre-install resources for their platform using `JavaScriptResourcesMain` and set the
same cache property before starting with a read-only root.

One Engine per JVM shares its isolate heap across fresh per-evaluation Contexts (8a's measured
D1 arm; author Sources are not cached). The default isolate ceiling is
`pool-size × js.max-heap-mb + 256` MiB (1280 at 4 × 256), outside `-Xmx`; an explicit ceiling
must exceed all pool slots' heaps. A tenant's hard isolate-heap trip can cancel another
tenant's evaluation once; subsequent evaluations rebuild the Engine. Include both heaps in
container sizing; each Context's heap limit is soft.


The two DuckDB operator keys (configuration.md §3.25) are named chart values:
`duckdb.extensionDirectory` (env `DATAPIPELINES_DUCKDB_EXTENSION_DIRECTORY`) and
`duckdb.memoryLimit` (env `DATAPIPELINES_DUCKDB_MEMORY_LIMIT`). Both default to empty, and an
empty value renders NO env entry — the image's own bundled extension directory stays in force, so
LAKE datasources keep their zero-egress `LOAD` behavior; a rendered `""` would override the
image's `ENV` and push every LAKE pool back to `INSTALL`+`LOAD`, which needs egress and a
writable home the read-only root filesystem does not provide. Every other non-secret env key goes
through `extraEnv`; an `extraEnv` key that double-maps either DuckDB key fails `helm template`,
because Kubernetes does not reject duplicate `env` names and would silently honor the last one.
CI renders the chart on every push and pull request: the `gate` job runs `scripts/helm-check.sh`,
a pinned `helm` (version and checksum pin bumped together per DEVELOPMENT.md §10.2) running
`helm lint --strict` plus eighty-nine verdicts — defaults render no `env:` block or driver volume, both
DuckDB values render exactly their two quoted entries, all nine `script.js` values render exactly their
nine quoted entries (each sizing key alone also renders its explicit zero; null renders no entry), each `extraEnv` double mapping
(the two DuckDB keys, the nine JavaScript keys and the two `MANAGEMENT_SERVER_*` keys) is refused
with the template's own message, a missing `existingSecret`, `extraEnv.LOADER_PATH`, the three JVM option variables and a
`script.js.isolateDir` outside the image's launcher-path entries are refused,
an enabled driver volume renders its read-only mount (with additional PVC/source checks),
and the management block renders nothing by default (no 9090, no `MANAGEMENT_*` env entry, no
`NetworkPolicy`, no `ServiceMonitor`) while `management.expose` renders the four objects — the
management `containerPort`, both `MANAGEMENT_SERVER_*` entries (`0.0.0.0` opt-open), the headless
`Service` and the `NetworkPolicy`, whose parsed rules must admit the scraper peers on the
management port AND the declared peers (or, under `allowAll`, every source) on the app port —
with `management.serviceMonitor.enabled` adding exactly one `ServiceMonitor`, and each of the
policy's refusals (both scraper selectors empty, no app-port declaration, `allowAll` with a
`from`, a bare label map, an unknown or empty peer, a quoted `allowAll`) exercised with its own
message. Every successful render is also validated `-strict` against the Kubernetes 1.35.0 API
schema (and the CRDs-catalog schema for the `ServiceMonitor`) by a pinned `kubeconform`
(DEVELOPMENT.md §10.2), with every object accounted for. It exits 0 when every case passes,
1 when a chart check fails, and 2 on a tooling failure (a download, a checksum-pin mismatch, a
schema fetch, or a missing python3/PyYAML — never a chart verdict).

The `lib/` driver drop-in (§3.5) is opt-in: set `libVolume.enabled: true` and supply exactly one Kubernetes volume source in `libVolume.source` (for example, `{configMap: {name: drivers}}`); the chart mounts it at `/app/lib` with `readOnly: true`, and a `persistentVolumeClaim` source must itself set `readOnly: true`. Populate the source only through a trusted image, init container or CI-written ConfigMap, never through the app (§9). The default is disabled; an `extraEnv` key naming the launcher path in any spelling the launcher reads (`LOADER_PATH`, `loader_path`, `loader.path`) is refused at render time, and so are `JAVA_TOOL_OPTIONS`, `JDK_JAVA_OPTIONS` and `_JAVA_OPTIONS` (the JVM reads `-D` options from them, and a system property would win over the image's value), so the image's launcher path stays in force against `extraEnv`. Keys inside the `existingSecret` reach the container through `envFrom` and cannot be checked at render time: the Secret holds the §5.1 startup contract and never the launcher path or a JVM option variable.

No sticky session affinity needed. Standard `ClusterIP` service with round-robin or random load balancing.

### 6.5 Bare JVM

For development:
```
GLIBC_TUNABLES="${GLIBC_TUNABLES:+${GLIBC_TUNABLES}:}glibc.pthread.stack_cache_size=0" \
java -Duser.timezone=UTC -jar datapipelines-app.jar \
  --spring.datasource.url=jdbc:postgresql://localhost:5434/datapipelines \
  --datapipelines.redis.host=localhost \
  --datapipelines.redis.port=6381
```

`-Duser.timezone=UTC` is required here too (§3.1): every deployment — container, bare JVM or JAR — must pass the flag explicitly.

On glibc, also set the native thread-stack policy from §3.1 before starting Java; setting an environment value
inside an already-running JVM is too late. Keep other required tunables and replace any conflicting stack-cache
entry with `glibc.pthread.stack_cache_size=0`.

Not recommended for production.

### 6.6 Resource sizing

Two numbers matter: JVM heap, and the container memory limit that must contain it.

**Heap.** The dominant consumer is H2 staging — each executing pipeline holds up to `datapipelines.staging.h2.max-memory-mb` (default 1024 MB), or its own `settings.tempdb.config.max_memory_mb` override:

```
heap ≥ (staging max-memory-mb × concurrent executions on THIS instance) + ~512 MB baseline
```

The multiplier is the **per-instance** execution ceiling `max-concurrent-executions-per-instance` (050/R2): every execution runs on exactly one instance, and this instance can legitimately hold the full setting's worth at an unbalanced moment. The default 100 × 1024 MB is the worst case ONE box must absorb — size the container for it (with N replicas behind a round-robin LB the typical share is `setting / N`, but size for the ceiling, not the share; cap the exposure by lowering the setting — or `max-memory-mb` — per box, or scale on smaller per-instance limits). The ~512 MB baseline covers Spring context, Hikari pools, the template cache, and SSE buffers.

Worked example — 4 concurrent executions per instance at the 1024 MB default: `4 × 1024 + 512 ≈ 4.6 GB` heap.

**Container memory limit.**

```
container limit ≈ heap × 1.5
```

The 0.5 covers what the heap number does not: metaspace, code cache, thread stacks, and — significant here — JDBC direct/native buffers, which scale with result-set width and fetch size. A limit set equal to `-Xmx` gets the container OOM-killed by the kernel rather than getting a clean `OutOfMemoryError`, which is strictly worse to diagnose. For the example above: ~7 GB limit.

**JVM flags.** Either `-XX:MaxRAMPercentage=65` (heap tracks the container limit — preferred for k8s, where limits change without an image rebuild) or an explicit `-Xmx`. Do not set both. Always `-Duser.timezone=UTC` (§3.1).

**Servlet threads (MCP blocking calls).** An MCP `pipelines_execute` call ([MCP Server §6.2.3](/docs/mcp-server.md#623-pipelines_execute)) is a **single blocking HTTP request** that holds one servlet thread until the execution reaches a terminal state or `datapipelines.executor.execution-timeout-seconds` (default 600) elapses. Per-user concurrency is bounded (`max-concurrent-executions-per-user`, default 10) but the default Tomcat pool is 200 threads, so on the order of ~20 concurrent long-running MCP callers can saturate it and starve REST/UI traffic on the same instance for minutes. Size `server.tomcat.threads.max` at or above the expected count of simultaneously-blocking MCP executions plus normal REST concurrency, or isolate `/mcp` on its own connector/instance. This is in addition to raising proxy/LB idle timeouts above `execution-timeout-seconds` (MCP Server §6.2.3).

**CPU.** Execution is coroutine-based and largely I/O-bound on source databases; 2 vCPU per instance is a reasonable floor. Scale out on `max-concurrent-executions-per-instance` pressure, not CPU.

### 6.7 Marketing site & in-product docs

Since v1.4 the app serves the marketing site and the documentation itself — the site and the product are ONE deployment (owner decision 2026-08-31). There is no separate static deploy to keep available, and the docs shipped in the jar always match the version running them.

- **`GET /`** — the marketing site (public). Template `templates/site/index.html`, assets under `static/site/**`, referencing the app's vendored design system at `/vendor/design-system/**` (the retired `website/` directory carried a second vendored copy — the app copy is now the single sync target of `scripts/sync-design-system.sh`). The only dynamic fact (the MCP tool count) is a compile-time constant baked at render time; public routes touch no database.
- **`GET /dashboard`** — the signed-in dashboard, moved off `/`. There is no auto-redirect: signed-in users hitting `/` get the marketing page.
- **`GET /docs`, `GET /docs/{slug}`** — the packaged spec set (`docs/*.md` minus the contributor/research exclusions, packaged by `processResources` in `modules/web/build.gradle.kts`). **Public since 073**: the viewer renders Markdown out of the jar and reaches no principal, no workspace and no datastore, and the identical content is already public in the AGPL repository — so exposing it moves ~25 pages of documentation into this domain's search index instead of GitHub's. Anonymous readers get the public site chrome; signed-in readers keep the application chrome.
- **`GET /mcp-server-for-sql-databases`, `/mcp-server/{engine}`, `/add-mcp-server-to-claude-code`, `/ai-data-pipeline`, `/text-to-sql-agent`, `/compare/{airflow,dbt}`, `/federated-query`** — the intent-cluster pages (073), same shape as `/`: GET-only, anonymous, constant content, no database. See [UI Screens §4.15](/docs/ui-screens.md#415-marketing-site-public).
- **`GET /robots.txt`, `GET /sitemap.xml`** — crawler surfaces (073). `robots.txt` is a static file; `sitemap.xml` is generated from the page registry and the packaged doc slugs, with no `lastmod` (145 §7 — a build timestamp on every URL read as "every page changed" on every deploy).
- **`GET /llms.txt`, `GET /llms-full.txt`, `GET /docs/{slug}.md`** — the agent surface (173): the llmstxt.org index and full text generated from the same two registries as the sitemap, and every packaged doc as raw Markdown with its links rewritten. Same shape as the sitemap — GET-only, anonymous, no database — and in the allowlist with their reasons ([Auth §8.3](/docs/auth.md#83-public-endpoints-no-auth-required)). See [UI Screens §4.15a](/docs/ui-screens.md#415a-crawler-surfaces-073).
- **Public-surface defence is cache headers, not a rate limiter.** `/` is `Cache-Control: public, max-age=300`; `/site/**` is public with a 1-hour TTL plus `Last-Modified` revalidation. The login rate limiter is deliberately NOT applied here (033/D1): the content is constant between deploys, so a shared-cache TTL costs nothing per request — and the T46 remoteAddr-keying concern is closed regardless: the limiter now resolves the CLIENT address through `datapipelines.auth.trusted-proxies` (§6.2), so pointing it at `/` would no longer create an LB-address-wide bucket.
- **Allowlist.** `/` and `/site/**` join the `permitAll` list in `SecurityConfig` with their reasons inline; 073 added the seven cluster-page matchers (enumerated, not globbed), `/docs` + `/docs/*`, `/robots.txt` and `/sitemap.xml`, each with its own comment. Nothing else was widened, and `anyRequest().authenticated()` still closes the list.

#### Search Console and the sitemap (operator note, 073)

Indexing is not automatic and is not a deploy step — it is a one-time verification plus an occasional check.

1. **Verify the domain** in [Google Search Console](https://search.google.com/search-console) (and Bing Webmaster Tools, if you care about it). Use the **DNS TXT** method: it verifies the whole domain including subdomains, and it survives redeploys — an HTML-file method would need the file added to `static/`, which is a code change for a DNS problem. Do this **before** any launch announcement, so the first crawl after the first inbound links is already attributed.
2. **Submit `https://<your-host>/sitemap.xml` once.** It regenerates on every request from the page registry and the packaged docs, so a page added in a later release is in it the moment that release is deployed — there is nothing to resubmit. `/llms.txt` needs no submission anywhere: an agent or an AI crawler fetches it by convention at the root, and it is generated from the same registries, so it is current the moment the sitemap is.
3. **`robots.txt` allows everything**, and deliberately lists no private paths: it is a public file, so naming internal routes in it advertises them. What keeps the private surface private is `SecurityConfig`'s `anyRequest().authenticated()`, not this file.
4. **A self-hosted deployment emits `datapipelines.co` canonicals.** That is deliberate — `rel="canonical"` names the one address that should be indexed, and a private instance asking Google to index it is the failure mode the tag exists to prevent. If you WANT your own deployment indexed under your own domain, change `SITE_ORIGIN` in `modules/web/.../ui/site/SitePages.kt` and rebuild.
5. **What to look at afterwards.** Coverage (are the doc pages indexed?), Performance filtered to `/mcp-server/` (are the engine pages taking their measured queries?), and any Core Web Vitals note. Nothing here needs a schedule; a look a month after launch and a look a quarter later is the whole practice.

**S3/CloudFront cold fallback (kept, drop if it rots unused).** If the app is down but marketing must stay up, the same template renders to static files with facts baked:

```bash
./gradlew :modules:web:websiteExport          # → modules/web/build/website-export/
aws s3 sync modules/web/build/website-export/ s3://YOUR-BUCKET/ --delete
aws cloudfront create-invalidation --distribution-id YOUR_DIST_ID --paths "/*"
```

The export renders `/`-rooted links (`/site/...`, `/vendor/design-system/...`), which the bucket layout mirrors exactly — upload the CONTENTS of `website-export/`, not the folder. S3 static website hosting (or CloudFront with an origin access control) serves `index.html` as the index document. Nothing fingerprinted: keep TTLs short for `index.html` and longer for `vendor/**` (those files change only when the design system is re-vendored). This is an emergency procedure, not a second primary deploy — the app is the primary. The exported pages need nothing the app's `Content-Security-Policy` (§6.2) forbids — no inline script, handler or style; their only `<script>` blocks are JSON-LD data — so the bucket's edge (CloudFront response-headers policy) can and should set the same CSP verbatim; static hosting sends no headers of its own (188).

---

## 7. Persistence & Backup

### 7.1 Metadata DB

- Backup: operator's standard Postgres backup practice (`pg_dump`, managed snapshots, etc.).
- Restore tested regularly by operator (runbook provided).
- **Critical data**: pipelines, templates, datasources (with credentials), audit log.
- **Disposable data**: executions table (can be truncated; only used for history).

### 7.2 Redis

- **Volatile by design.** Every key is TTL-bounded: results, idempotency records, cancellation flags, the 1-hour event log (§4.2).
- **No backup needed.** A Redis restart loses unexpired results (clients re-execute), unexpired idempotency records (a retry may execute a second time), and pending cancellation flags. All are recoverable by re-running; none is a durable record. The durable records — pipelines, templates, datasources, execution history and `execution_events` — are all in Postgres.
- Persistence (RDB/AOF) is **not required** and not recommended; it buys nothing that a re-run does not.
- **`maxmemory-policy noeviction` is required regardless** (§4.2.1). "Volatile" means TTL-expiring, not evictable — eviction breaks correctness in a way that restart does not, because it happens silently during normal operation.

### 7.3 H2 staging

- **In-memory only**. Per-request. No persistence. No backup. No recovery.

### 7.4 Restore drill

Operators should run a quarterly restore drill: restore metadata DB from backup to a sandbox instance, verify pipelines still execute. Documented in the runbook (future).

---

## 8. Upgrades

### 8.1 Versioning

- Semantic versioning: `MAJOR.MINOR.PATCH`.
- **PATCH**: bug fixes only. Safe to upgrade in place.
- **MINOR**: new features, backward-compatible. Schema migrations applied automatically on app startup.
- **MAJOR**: breaking changes. Migration guide published.

### 8.2 Database migrations

- **Flyway** for metadata DB migrations.
- Migrations are versioned SQL files in `app/src/main/resources/db/migration/`.
- Applied automatically on app startup.
- Forward-only; no rollback (restore from backup if needed).

### 8.3 Upgrade procedure (recommended)

1. Review release notes for breaking changes.
2. Backup metadata DB.
3. **Pre-deploy check for builds after 2026-09-19 (R-EP5, #274):** a database holding a
   `published_endpoints` row saved under the old two-segment grammar used to refuse the boot.
   Since V41 the migration retires such rows (disabled, `retired_reason = 'pre-R-EP5 path'`,
   never deleted) and the repository skips them regardless, so the upgrade is safe either way;
   run this to know beforehand what you will see flagged in the API console afterwards:

   ```sql
   SELECT path_pattern FROM published_endpoints
    WHERE array_length(string_to_array(trim(both '/' from path_pattern), '/'), 1) < 3;
   ```

   Zero rows = nothing for V41 to retire (the two-segment shape of the incident; a row malformed
   another way is mapped legacy by the new code, never served, and needs no SQL — on a build OLDER
   than #274 it broke the boot just the same). Rows listed = they come back disabled and flagged after
   the upgrade; unpublishing one is the fix. An operator in a hurry on a build OLDER than V41
   can delete the rows by hand — the demo seeder republishes the demo endpoint at its current
   path on the next boot.
4. **Pre-deploy check for builds after 2026-09-26 (#194, #268):** two behaviours that stored
   pipeline bodies can meet for the first time. (a) A `DECIMAL`/`BIGDECIMAL` parameter's declared
   `scale` now binds its values, so a stored `default` with more places than the scale is refused
   when it applies (`pipeline.execution.parameter_constraint_violation`). (b) A body saved before
   #194 could carry a `constraints` block or a `cardinality` that `Parameter` then ignored; those
   are read now, and one save refuses today is answered
   `409 pipeline.execution.parameter_declaration_invalid` where it is used (a supplied value or an
   applied default), never a 500. Run both against the metadata DB to know beforehand (read-only;
   since #298 the first also LISTS a stored `scale` that is not an integer — `"2.0"`, `"x"` — rather
   than stopping on the cast):

   ```sql
   -- pre-deploy #194: stored DECIMAL/BIGDECIMAL defaults with more places than their scale
   SELECT p.name AS pipeline, v.version, v.status, e.key AS parameter,
          e.value->>'default' AS default_value, e.value->>'scale' AS scale
     FROM pipeline_versions v
     JOIN pipelines p ON p.id = v.pipeline_id
    CROSS JOIN LATERAL jsonb_each(CASE WHEN jsonb_typeof(v.body_json->'parameters') = 'object'
                                       THEN v.body_json->'parameters' ELSE '{}'::jsonb END) e
    WHERE e.value->>'type' IN ('DECIMAL', 'BIGDECIMAL')
      AND e.value ? 'default' AND e.value ? 'scale'
      AND CASE WHEN (e.value->>'scale') ~ '^-?[0-9]{1,9}$'
               THEN length(split_part(trim(trailing '0' from (e.value->>'default')), '.', 2)) > (e.value->>'scale')::int
               ELSE TRUE  -- a scale that is not an integer is itself a defect: listed, never a cast error
          END
    ORDER BY 1, 2, 4;
   ```

   ```sql
   -- pre-deploy #268: stored parameters declaring constraints or a cardinality
   SELECT p.name AS pipeline, v.version, v.status, e.key AS parameter,
          e.value->'constraints' AS constraints, e.value->>'cardinality' AS cardinality, v.created_at
     FROM pipeline_versions v
     JOIN pipelines p ON p.id = v.pipeline_id
    CROSS JOIN LATERAL jsonb_each(CASE WHEN jsonb_typeof(v.body_json->'parameters') = 'object'
                                       THEN v.body_json->'parameters' ELSE '{}'::jsonb END) e
    WHERE e.value ? 'constraints' OR e.value ? 'cardinality'
    ORDER BY 1, 2, 4;
   ```

   Zero rows from each = nothing stored meets the new behaviour. Rows from the first = defaults to
   correct (a new draft with the default at the declared scale). Rows from the second whose
   version was written BEFORE the #194 build was deployed were never checked at save: open the
   pipeline and re-save it — the save names what is wrong (a `cardinality` other than `SINGLE`
   is refused outright; a pipeline parameter is `SINGLE`). Rows written after it passed the save
   check and need nothing. Both scans read every stored version, drafts and discarded ones
   included; neither covers a schedule's stored inputs. `PreDeployScanQueriesTest` runs the two
   blocks above, verbatim, against the shipped schema.
5. Signal shutdown and let the instance drain (§8.3.1 — this is automatic, not a manual step).
6. Start the new version (migrations apply on startup).
7. Verify `/health` returns UP.
8. Restore traffic.

For k8s: rolling update via `kubectl rollout`. Each terminated pod flips its readiness and cancels its in-flight executions on the way out (§8.3.1); the `preStop` and `terminationGracePeriodSeconds` settings in §8.3.2 keep that orderly.

**Scheduler notification kinds (V51).**

1. Keep `datapipelines.scheduler.notifications.enabled=false` (the default) throughout the first rollout containing V51; with mail off this is the ordinary rolling update.
2. Enable notifications only after **every instance** runs the new image. A new sender writes `notification_*` trail kinds which an older image cannot read: its strict `TrailKind.fromWire` throws when `ScheduleRunRepository` reads that run's detail. Widening the database CHECK is not reader compatibility. Do not enable during the mixed-version fleet.

#### 8.3.1 Graceful shutdown mechanism

Shutdown is a defined sequence, not "stop the process and hope". On `SIGTERM` the instance, in this order:

1. **Fails readiness first.** `ReadinessState.REFUSING_TRAFFIC` is published before anything is cancelled, so `/ready` starts returning 503 while the process is still fully up and the load balancer / k8s Service bleeds traffic off. The order is the contract: flipping after the drain would keep fresh work flowing into an instance that is already cancelling it. `/health` keeps reporting UP, so nothing kills the pod for being unhealthy mid-drain.
2. **Cancels every in-flight execution** through the ordinary cancellation path ([DAG Executor §8.3](/docs/dag-executor.md#83-cancellation)): `Statement.cancel()` on every registered statement first — which is what actually stops the query on the source database, so the drain is more than a status update — then the root `Job` cancellation, `execution_aborted` (`reason: "shutdown"`) emitted to connected streams, tempdb dropped and connections released in `finally`, and the row written `ABORTED`. The drain **cancels; it does not wait for executions to finish** — an execution in flight at SIGTERM ends `ABORTED`, not `SUCCESS`.
3. **Waits for the flush, bounded** (20 seconds). An execution leaves the live count only after its `ABORTED` status and events are written, so the wait means the bookkeeping reached Postgres and Redis before exit; the bound means a wedged execution meets the kubelet's deadline rather than hanging shutdown forever.
4. **Exits.** The web server's own graceful shutdown (`server.shutdown: graceful`) runs after the drain, so SSE clients are still connected while their terminal events arrive, and in-flight ordinary requests complete rather than being cut off. Then the batching writers drain (#266, bounded by `shutdown-drain-ms`), and last the scheduled jobs' scheduler stops (#316): no tick starts once shutdown began, a tick already running is awaited while the metadata pool is still open, then `shutdown.scheduled_jobs_stopped` is logged.

**The accepted loss, stated plainly:** an execution running when an instance stops is **cancelled, not preserved**. It ends `ABORTED` with `reason: "shutdown"`, is visible as such in execution history, and its client must re-execute. There is no execution hand-off to another instance and no resumption — in-memory H2 staging makes migration impossible, and pretending otherwise would be worse than the honest abort. Bounded, visible loss beats a silent hang.

One residual race, honestly: a request that reaches the instance *between* the readiness flip and the web server stopping can still launch an execution. The drain re-cancels on every flush tick, so such an execution is cancelled within ~100ms of starting — but the right answer is that traffic should have stopped at the readiness flip, which is what the `preStop` in §8.3.2 exists to give the endpoints controller time to do.

#### 8.3.2 Kubernetes pod lifecycle

```yaml
terminationGracePeriodSeconds: 60      # preStop 5 + admission gate ≤ 15 + drain flush 20 + Tomcat 10 + persistence drain 10
lifecycle:
  preStop:
    exec:
      command: ["sleep", "5"]
```

- **`terminationGracePeriodSeconds: 60`** — the sum of the sequence at its configured maximum (#277, #266): the `preStop` sleep (5 s) + the scheduler's admission gate, which waits up to `datapipelines.scheduler.shutdown-wait-seconds` (default 5, at most 15) for launches in progress + the drain's bounded flush (20 s, §8.3.1 step 3 — the drain cancels rather than waits, so the flush is its only clock) + 10 s for Tomcat's graceful phase (Spring bounds that phase itself at `spring.lifecycle.timeout-per-shutdown-phase`, default 30 s; a request still running after the 10 s is cut by the kill, which costs that request and none of the bookkeeping the drain already wrote). `deploy/compose.yml` has no `preStop` and uses the same 50 s (15 s left for Tomcat). `ShutdownGraceArithmeticTest` derives the sum from the code's own bounds and fails a smaller value. Anything shorter risks the kubelet `SIGKILL`ing mid-flush: executions die without their `finally` blocks, so no `execution_aborted` event and no status update — rows left `RUNNING` until the stale-execution sweep marks them `ABORTED` (§6.2). The persistence drain (#266, `datapipelines.persistence.shutdown-drain-ms`, 10 s by default) follows Tomcat's phase and is in the sum since the 316 review; the scheduled jobs' scheduler (#316) stops after every drain and is NOT — a tick in flight is awaited up to `spring.lifecycle.timeout-per-shutdown-phase` (30 s) plus 2 s, and a kill during that wait costs the hung tick (redone next tick) and bean destruction, nothing the drains own.
- **`preStop: sleep 5`** closes the standard k8s race: endpoint removal and `SIGTERM` are concurrent, so without it the pod can flip readiness microseconds before the Service stops routing to it. Five seconds of overlap is enough for Endpoints propagation — and it is what makes step 1's readiness flip actually take traffic off the pod before step 2 cancels.
- A `PodDisruptionBudget` (§6.4) keeps node drains from taking every replica's drain at once.

### 8.4 Rollback

**Scheduler notifications (V51): older-image boundary.** Before downgrading to an image older than V51, disable notifications on every instance and let the current sender settle the backlog: `SELECT count(*) FROM schedule_notification_deliveries WHERE state = 'pending'` must be zero. Disabling stops new enqueue; the current sender settles existing rows as failed/disabled, so this drain is not a promise to send them. Already recorded kinds remain: `SELECT DISTINCT run_id FROM schedule_run_events WHERE kind LIKE 'notification\_%'` identifies runs whose detail is unreadable by the older image until it is upgraded again. The reader stays strict ([Enums](/docs/enums.md) additive-only wire convention); this feature promises no rolling reader compatibility, and an additive CHECK alone does not make an old artifact safe. Do not delete historical trail rows to hide the boundary.

- App rollback: redeploy previous image. Schema migrations are forward-only — if a migration was applied, you can't run the old app version against the new schema.
- **For MINOR upgrades**: schema changes are additive; rollback usually works.
- **For MAJOR upgrades**: take a DB backup before upgrade; restore it to roll back.

---

## 9. Security Hardening Checklist (Deployment)

- [ ] TLS termination at load balancer / proxy (let it handle cert renewal), with HSTS set THERE (one year, no preload) — the app sends none, by design (§6.2 "What the edge sets").
- [ ] The edge passes the app's security headers through untouched — the app's `Content-Security-Policy` (enforced; no `'unsafe-inline'`), `X-Frame-Options: SAMEORIGIN`, `Referrer-Policy`, `Permissions-Policy` — and never weakens one (§6.2).
- [ ] Metadata DB password rotated and not in source control.
- [ ] `DATAPIPELINES_JWT_SECRET` is high-entropy (≥ 32 bytes random).
- [ ] `DATAPIPELINES_DB_ENCRYPTION_KEY` is high-entropy (32 bytes random) and stored in a secret manager, not a plaintext env file.
- [ ] Redis password set if Redis is networked (`requirepass` on the server, `datapipelines.redis.password` on every app instance — they must match). Under the `hardened` posture an empty password with a non-loopback Redis host is REFUSED at boot ([Configuration §3.23](/docs/configuration.md#323-environment-and-posture), #189); `development` warns.
- [ ] No credential on a command line inside a container: the reference compose files give `redis-cli` its password through `REDISCLI_AUTH` in the redis service's environment, the Redis **server** its `requirepass` from that same variable through stdin (`redis-server -`, §4.2.1 — #196 took the last argv credential off the reference stack), and `mysqladmin` through `MYSQL_PWD` — an argv password is readable in `ps` by every process in the container. Check it: `docker inspect <redis> --format '{{.Config.Cmd}}'` and `docker exec <redis> ps -o args` name the variable, never its value.
- [ ] Redis `maxmemory-policy noeviction` (§4.2.1) — correctness, not tuning.
- [ ] OIDC client secrets from a secret manager, not a plaintext env file; at least one provider configured (§5.1).
- [ ] NetworkPolicy restricts app's egress.
- [ ] Service account / IAM role has least privilege.
- [ ] Container runs as non-root user (enforced in Dockerfile).
- [ ] Container filesystem read-only except for configured volume mounts.
- [ ] Resource limits set (CPU + memory) per deployment.
- [ ] Audit log retained per compliance policy.
- [ ] No `/actuator/*` path reachable on the application port; `/actuator/prometheus` on the management port (`management.server.port`), cluster-internal only ([Observability §4.2](/docs/observability.md#42-exposure)). The management port is never published to a host or load balancer; `MANAGEMENT_SERVER_ADDRESS` defaults to loopback — setting it to `0.0.0.0` (required for k8s scraping) demands an accompanying NetworkPolicy that admits only the scraper to the management port. Because that policy selects the app pods, it isolates their app port too, so it must also carry the app-port rule you declare. The reference chart implements this check as the `management` block: `management.expose` is the opt-open switch (it pins `0.0.0.0` and renders the port and headless `Service`), and `management.networkPolicy.enabled` renders the policy. That policy carries the scraper rule on the management port plus the app-port rule from `appIngress.from`, or every source under the explicit `appIngress.allowAll: true`. The render refuses both scraper selectors empty, no app-port declaration, and a bare label map in place of a `LabelSelector`. Everything stays OFF by default ([§6.4](#64-kubernetes-recommended-for-production)).
- [ ] Internet-exposed deployments may prefer to omit `-Pdatapipelines.commit` at build time — a public `/info` commit hash maps the instance to exact source revisions.
- [ ] The `lib/` driver drop-in mount is **read-only** in the container, populated at image build or by a trusted init container, never writable by the app user; `LOADER_PATH`, if set, comes from the image — never inherited from the deployment environment (a writable `lib/` is code-execution-by-file-drop).
- [ ] Production Redis requires `requirepass` (or ACLs) **and** TLS, or is confined to a private network segment with a NetworkPolicy — it holds fully materialized caller results for up to an hour (D9). The app logs a structured WARN at startup when the Redis password is empty and the host is not loopback under `development`, and refuses to start under `hardened` ([Configuration §7](/docs/configuration.md#7-config-validation)).

---

## 10. Distribution License

- **Code**: AGPL-3.0 (see [LICENSE](https://github.com/msabiransari/datapipelines/blob/main/LICENSE); contributions under the [CLA](https://github.com/msabiransari/datapipelines/blob/main/CLA.md)).
- **Third-party runtime notices**: `:modules:app:generateThirdPartyNotices` reads the
  resolved `productionRuntimeClasspath` POMs (walking parents for inherited licences),
  plus Spring Boot's packaged jar-mode tools and both Linux GraalJS isolate POMs.
  The isolate versions come from the Dockerfile's `GRAALJS_VERSION` ARG. The generated
  file lists artifact coordinates, POM licence names, SPDX identifiers and vendored
  licence texts; unknown, missing or stale licence mappings fail the build.
- **Distribution locations**: the boot jar contains `META-INF/THIRD-PARTY-NOTICES`;
  the image also contains `/app/THIRD-PARTY-NOTICES`, readable by the application user.
  `LICENSE` at the repository root covers the product's own code. The generated
  notices are build outputs, not a hand-maintained root `NOTICE` file.
- **Optional drivers**: `-Pmysql` and `-Poracle` include their drivers and corresponding
  notices only in those builds. MySQL declares GPL-2.0-only with the Universal FOSS
  exception; Oracle's FUTC is identified as `LicenseRef-Oracle-FUTC` (no SPDX-listed
  identifier). Operators remain responsible for their use and redistribution terms,
  including drivers added through `lib/` after the build.
- **Coverage boundary**: these notices cover Maven runtime components and the isolate,
  not the base operating system, separately downloaded DuckDB extensions
  ([#692](https://github.com/msabiransari/datapipelines/issues/692)), or vendored browser
  assets ([#693](https://github.com/msabiransari/datapipelines/issues/693)).
  See [Development §10.2](https://github.com/msabiransari/datapipelines/blob/main/DEVELOPMENT.md#102-quality-tooling) for the generation and update procedure.

---

## 11. What's Out of Scope for v1

- **Managed / SaaS deployment**: future commercial offering.
- **Multi-tenant isolation**: single-tenant v1.
- **High-availability Postgres in our chart**: operator provides managed PG.
- **Backup automation**: operator responsibility; we provide runbook.
- **Air-gapped deployment**: should work (no phoning-home telemetry) but not explicitly tested in v1.
- **Federated deployments** (independent installations sharing state or federating queries across each other): not supported. Multi-instance horizontal scaling (§6.2) is a different thing and *is* supported.
- **Execution hand-off / resumption**: an execution is pinned to the instance that started it and is aborted rather than migrated (§6.2, §8.3.1).

---

## Appendix A: Reference compose.yml Sketch

This sketch **boots** — it satisfies every §5.1 startup requirement. Removing any of the marked items produces a container that exits during context startup, not one that runs degraded. `deploy/compose.yml` is the real file; every secret below comes from `deploy/secrets.env` under **the same name the app binds**, so the same file is sourceable by a bare `java -jar`, a systemd `EnvironmentFile` or a Kubernetes `Secret` ([Environments §4](/docs/environments.md#4-loaders)).

```yaml
services:
  datapipelines:
    image: ghcr.io/datapipelines/datapipelines:1.0.0
    ports:
      - "8080:8080"
    environment:
      SPRING_DATASOURCE_URL: jdbc:postgresql://postgres:5432/datapipelines
      SPRING_DATASOURCE_USERNAME: datapipelines
      SPRING_DATASOURCE_PASSWORD: ${SPRING_DATASOURCE_PASSWORD}

      DATAPIPELINES_REDIS_HOST: redis
      # REQUIRED whenever redis runs with --requirepass. Must be the SAME value
      # the redis service below is started with, or every Redis call fails at runtime.
      DATAPIPELINES_REDIS_PASSWORD: ${DATAPIPELINES_REDIS_PASSWORD}

      # REQUIRED, no fallback. Generate once: openssl rand -base64 32
      DATAPIPELINES_JWT_SECRET: ${DATAPIPELINES_JWT_SECRET}
      # REQUIRED, no fallback. Exactly 32 bytes base64: openssl rand -base64 32
      # Losing this makes every stored datasource credential unrecoverable.
      DATAPIPELINES_DB_ENCRYPTION_KEY: ${DATAPIPELINES_DB_ENCRYPTION_KEY}

      # REQUIRED: at least ONE authentication method (ConfigValidator §7) — an OIDC
      # provider configured below, or local accounts (auth.md §5A: set
      # DATAPIPELINES_AUTH_LOCAL_ENABLED=true plus a one-time seed for
      # DATAPIPELINES_AUTH_BOOTSTRAP_ADMIN_EMAIL, which names the first admin either way).
      # These names are referenced by the providers list in application.yml below;
      # Google is only an example — Microsoft/Okta/Keycloak/any OIDC IdP works.
      GOOGLE_CLIENT_ID: ${GOOGLE_CLIENT_ID}
      GOOGLE_CLIENT_SECRET: ${GOOGLE_CLIENT_SECRET}
    volumes:
      # The OIDC provider list is a nested structure — supply it as YAML.
      - ./application.yml:/etc/datapipelines/application.yml:ro
    depends_on:
      - postgres
      - redis
    restart: unless-stopped

  postgres:
    image: postgres:16-alpine
    environment:
      POSTGRES_DB: datapipelines
      POSTGRES_USER: datapipelines
      POSTGRES_PASSWORD: ${SPRING_DATASOURCE_PASSWORD}
    volumes:
      - postgres-data:/var/lib/postgresql/data
    restart: unless-stopped

  redis:
    image: redis:7-alpine
    # noeviction is REQUIRED (§4.2.1): eviction silently destroys results,
    # idempotency keys, and cancellation flags. Same password as the app above,
    # in the environment only: the server reads it from stdin, redis-cli from
    # REDISCLI_AUTH — never on argv (§4.2.1, #196). `$$` keeps the expansion in
    # the shell; `user: redis` because the wrapper is sh, not redis-server.
    user: redis
    environment:
      REDISCLI_AUTH: ${DATAPIPELINES_REDIS_PASSWORD}
    command:
      - sh
      - -c
      - |
        exec redis-server - --maxmemory 512mb --maxmemory-policy noeviction <<EOF
        requirepass $$REDISCLI_AUTH
        EOF
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 10s
      timeout: 5s
      retries: 5
    restart: unless-stopped

volumes:
  postgres-data:
```

The mounted `application.yml` needs only the provider list (everything else has a default or is set above):

```yaml
datapipelines:
  auth:
    oidc:
      providers:
        - name: google
          client-id: ${GOOGLE_CLIENT_ID}
          client-secret: ${GOOGLE_CLIENT_SECRET}
          issuer-uri: https://accounts.google.com
          display-name: "Sign in with Google"
```

Format and further provider examples: [Auth §5.1](/docs/auth.md#51-provider-configuration-generic) and [§11.1](/docs/auth.md#111-oidc-provider-configuration). Full key template: [Configuration §5](/docs/configuration.md#5-full-applicationyml-template).

---

## Appendix B: Demo Quickstart — the Published Sample Data

The sample data is **published, versioned artifact sets** (one per family), not
something the app downloads: loading it is a deployment step (sample-data design
D5). Any deployment that pulls the same version gets the same databases, byte for
byte in content. The build scripts that produce them live at
[`scripts/sample-data/`](https://github.com/msabiransari/datapipelines/blob/main/scripts/sample-data/README.md) (nyc family),
[`scripts/sample-data-trade/`](https://github.com/msabiransari/datapipelines/blob/main/scripts/sample-data-trade/README.md) (trade
family) and [`scripts/sample-data-lake/`](https://github.com/msabiransari/datapipelines/blob/main/scripts/sample-data-lake/README.md)
(**dp-lake**); everything below is the consuming side.

Three independent families, each behind its own compose profile and app.sh flag —
an engineer spins up exactly the data they need:

| Family | Flag / profile | What you get |
|---|---|---|
| **nyc** (mobility) | `--demo nyc` / profile `demo-nyc` | NYC TLC yellow-taxi trips on Postgres (~4.9M sampled rows, plus rollups), NOAA weather on MySQL, TLC reference on SQLite — 3 datasources + 6 example pipelines |
| **trade** (trade/v4) | `--demo trade` / profile `demo-trade` | US Census monthly imports/exports at HS-6 grain on **DuckDB** (2.4M rows), UN Comtrade mirror statistics on MySQL, Federal Reserve H.10 exchange rates on SQLite — 3 datasources + 3 example pipelines |
| **lake** (dp-lake) | `--demo lake` / profile `demo-lake` | NYC TLC high-volume for-hire trips as Parquet and Iceberg on S3, **read in place** — 1 `LAKE` datasource (`sample-lake`) whose registry is seeded from the published `manifest.json`, + 1 four-engine example pipeline. No loader, nothing downloaded at start |

Any combination is fine — the app's bootstrap keys are comma-separated lists built
from the active families, and the MySQL service is shared.

The public marketing site's [`/demo-data` page](https://datapipelines.co/demo-data)
renders these published manifests as documentation — every table, row count, source
and licence per family, generated from the same manifest files the loader verifies
(vendored at `modules/web/src/main/resources/site/demo/`, pinned to the versions
above). It is the page to hand anyone who asks what the demo contains.

### The demo API: every seeded pipeline is a published endpoint (#224)

Seeding a demo family now also publishes each seeded pipeline as a **`GET`
endpoint** under `/demo/…` and binds them all to ONE public `api_caller` key —
`demo-public-key`, minted at boot from `datapipelines.bootstrap.demo-api-key`
(default `dpk_DEMOPUBLIC42.…`, configuration.md §3.18). The key is **public by
design**: the same value renders on the `/demo-data` page's "Call it as an API"
section, and an outsider can call any demo endpoint with it and no account
(rest-api.md §19.8 is the worked example, including the per-key request budget —
60 requests per 60 s per key, `429 rate_limit.exceeded` — and the lake family's
gate: the lake endpoint pays S3 egress per call, so it is published only while a
budget stands behind it). Blank the setting in `deploy/secrets.env` and the next
boot retracts the public API — no key, no demo endpoints, no page section; change
the value and the next boot rotates the key. Nothing changes for keys real users
mint on the `/api-keys` page.

The **lake** family is different in kind from the other two and the difference is
worth stating before the commands: it has a compose profile and a `--demo` flag
like its siblings, but **no loader service**, because nothing is loaded. What
`--profile demo-lake` turns on is the datasource registration and the manifest
import; the data itself stays in the bucket and is read at query time — see
"dp-lake — the family with no loader" below.

### One command

The base URLs and the pinned versions live in **`deploy/env/defaults.env`**, which is
tracked — a data change is a new version directory and a commit, so what a
deployment loads is visible in the repository. From a checkout that builds its own
image, the whole thing is:

```bash
./app.sh --start --demo nyc,trade,lake
```

For the raw compose path, set the family ON markers your invocation wants and pass
the same env files `app.sh` does — secrets last:

```bash
SAMPLE_NYC_ON=1 SAMPLE_TRADE_ON=1 SAMPLE_LAKE_ON=1 docker compose \
  -f deploy/compose.yml -f deploy/compose.local-build.yml \
  --env-file deploy/env/defaults.env \
  --env-file deploy/secrets.env \
  --profile demo-nyc --profile demo-trade --profile demo-lake up -d --wait
```

`--wait` returns when the app is healthy, and by then the artifacts have been
downloaded, every checksum verified, the databases restored, the SELECT-only demo
login created, and the families' datasources registered.

The `nyc` and `trade` families also build the jar with `-Pmysql` (see the driver
note below); `lake` alone does not need it.
`./app.sh --stop` and `--status` take the same `--demo` list, so the demo services
are not left running invisibly. **`./app.sh --clean [--yes]` is the clean slate:** it
stops and removes the project's containers, then deletes the METADATA volume
(`<project>-postgres-data` — pipelines, templates, executions, users, API keys,
workspaces, datasource rows) by explicit name, and keeps `<project>-mysql-data` (demo
source data) and `<project>-sample-data` (the downloaded artifacts). It asks for the
project name back on a terminal, or needs `--yes` without one; the next `--start`
migrates a fresh database and re-seeds the demo. Sign-in users are re-created on their
next login; API keys must be minted again. **Demo is a flag, not an environment**
([Environments §5](/docs/environments.md#5-demo)): `DATAPIPELINES_DEMO` is the source of
truth, `--demo` merely sets it, and the `hardened` posture refuses it at boot. The
per-family `--demo-nyc` / `--demo-trade` switches are gone; the families are
`nyc`, `trade` and `lake`, and an unknown name is refused by name.

**MySQL driver.** MySQL Connector/J is GPL with a FOSS exception and is *not* in
the default build (§3.5, [Datasources §10.2](/docs/datasources.md#102-strategy)). The
`sample-weather` and `sample-trade-world` datasources are MYSQL, and bootstrap
registration fail-fasts startup with `datasource.driver_not_loaded` without it.
Build with the driver:

```bash
./gradlew -Pmysql :modules:app:bootJar && docker build -t datapipelines:local .
```

or drop the jar into `lib/`. `./app.sh --start --demo nyc[,trade[,lake]]` does the
`-Pmysql` build for you.

### Point an agent at it — three steps

1. **Log in** with the account `./app.sh --start` prints. The demo needs **no OIDC
   client at all** ([Auth §5A](/docs/auth.md#5a-local-password-accounts-optional)): local
   password accounts are enabled and the first admin gets a GENERATED one-time
   password, written to `deploy/secrets.env` at first scaffold and forced to change
   at first sign-in. `app.sh` reads the seeded account back **out of the database**
   after the stack is healthy and prints the login that actually exists — the seed
   fires once, at row creation, so a `DATAPIPELINES_AUTH_BOOTSTRAP_ADMIN_EMAIL`
   changed later names an account that was never created ([Environments
   §8](/docs/environments.md#8-first-login)). The first login lands you in the shipped
   `demo` workspace — workspaces are created by super admins, not provisioned per
   user ([Auth §4.2](/docs/auth.md#42-user-provisioning) step 4, D-R11) — which is where
   the example pipelines are seeded.
2. **Create your MCP key** after signing in: open the [Keys page
   (UI Screens §4.19)](/docs/ui-screens.md#419-api-keys-keys-v2--the-one-creation-path-for-every-kind)
   at `/api-keys`, choose **New API key**, select kind `mcp`, and choose a role the
   creator may give under the subset rule. The demo's bootstrap super admin may
   choose any offered MCP role; authors, promoters and workspace admins may also
   create MCP keys, but viewers cannot. Copy the plaintext from the one-time reveal.
   To rotate a key, revoke it on the Keys page and create another. REST clients can
   create one with `POST /api/v1/auth/api-keys` ([Auth §7.4](/docs/auth.md#74-issuance),
   [REST API §16.1](/docs/rest-api.md#161-api-keys-creation-is-on-the-keys-page--keys-v2)).
3. **Give the agent the MCP endpoint** `http://localhost:8080/mcp` and that key.
   It can list the seeded pipelines, read the three sample datasources' schemas,
   and execute `nyc/mobility/revenue_by_borough` or `nyc/mobility/rainy_vs_dry_ridership` immediately —
   see [MCP Server](/docs/mcp-server.md).

### What the demo profile turns on

The profiles add a `mysql` service and the families' one-shot loaders, and point
the app at the files they place on a read-only volume. The scaffolded
`deploy/secrets.env` enables **local password accounts** so the demo needs no OIDC
client, and names the account the credential lands on; the password itself is a
secret and is GENERATED into that same file — never a constant in a tracked file, on
an app that binds every interface ([Auth §5A.2](/docs/auth.md#5a2-seeding-the-first-admin)).
There is now exactly ONE place the credential is written and ONE place it is read
back from, which is why `app.sh` can print a login that works. It also sets the §7
demo posture: `member-datasources-enabled=false` (an
open datasource form on a public server is an SSRF and port-scan primitive —
demo users get the seeded datasources only). Without `--profile demo` none of
it exists: both `datapipelines.bootstrap.*` keys are paths and empty means
off, so the non-demo stack is configured exactly as it was.

The demo datasources are protected in three independent layers: workspace-scoped
access, the `is_readonly` flag on the datasource row
([Datasources §5.7](/docs/datasources.md#57-readonly-datasources-flag-semantics-and-enforcement-layers)),
and a database login granted `SELECT` and nothing else — created by the loader,
never assumed. The SQLite entry additionally opens its file with the driver's
read-only mode ([Datasources §8A.4](/docs/datasources.md#8a4-sqlite-read-only-open-mode)).
On a public demo, add the §9 hardening item that belongs with this posture:
egress-restrict the app container's network, so a datasource anyone can reach
cannot become a path to anywhere else.

### The loader

`deploy/sample-data/load.sh <base-url> <version>` is what the one-shot services
run. Its contract, in order: download every artifact the manifest names, verify
**every** checksum, and only then touch an engine — a corrupted download can
never leave a half-loaded database behind. Each engine then gets a
`_sample_meta(version)` marker, written **last**, so a re-run of a loaded
deployment skips it and a failed engine is retried cleanly on the next start.

It runs as two services because no pinned image carries both a Postgres and a
MySQL client, and installing one at container start would put an unpinned package
fetch in the one place that has to be reproducible.

### Demo Postgres sizing (T200, 109 §E)

The `postgres` service that the **nyc** family restores into carries the demo's
biggest table — `trips`, ~804 MB, ~4.9M sampled rows plus rollups — against the
Postgres image's compiled defaults (`shared_buffers=128MB`, `random_page_cost=4.0`
tuned for spinning disks). Under load, a simple year-2024 aggregate therefore
ran in ~21 s: nearly every page came from the OS cache, but the planner kept
choosing plans as if each random page cost a seek. The compose file sets:

```
command: postgres -c shared_buffers=512MB -c effective_cache_size=1536MB
                -c work_mem=32MB -c random_page_cost=1.1
```

with the container's memory reservation raised to 1 GB to match. The numbers are
shaped by the data: `shared_buffers` ≈ half the restored set, `effective_cache_size`
≈ shared_buffers plus the OS page cache the box actually has, `work_mem` enough for
the demo's hash joins and sorts, and `random_page_cost=1.1` because the volume sits
on SSD. **This is demo sizing, not production advice** — the operator's own
datasource is their own DBA's, and the app never tunes a database it did not bring.

Measured on a quiet box (109 §E), year-2024 airport-count query
(`pu_location_id IN (1, 132, 138)`, three runs cold→warm): compiled defaults
**42.7 s → 23.9 s → 8.6 s** (buffers read=43,871 — nearly every page from disk);
sized flags **20.5 s → 9.6 s → 8.1 s** (buffers hit=46,795, read=0 — the working set
fits the 512 MB). The win is the cold/cache-pressured run, ~2.1×; warm steady-state
is unchanged, as expected when the OS cache already holds the table. A composite
index `(pu_location_id, pickup_date)` on the artifact itself (the datasource is
readonly to agents, so the artifact is the only place an index can come from) takes
the same query to **~0.95 s cold / 0.09 s warm** (index-only scan, 196 buffer hits)
— see the 109 handback for the artifact decision.

### dp-lake — the family with no loader

`sample-data/lake/<version>/` is different in kind from the two above and the
difference is the point: **it is read in place**. Nothing is downloaded at demo
start and nothing is restored into an engine. The `sample-data-lake` one-shot
service runs `deploy/sample-data/place-lake.sh` to place the repository's bootstrap
and example files on the shared volume; it does not download the dataset.
The query engine reads the Parquet objects over HTTPS and fetches only
the ones a query's predicates name.

What is published (build runbook:
[`scripts/sample-data-lake/README.md`](https://github.com/msabiransari/datapipelines/blob/main/scripts/sample-data-lake/README.md)):

| Object set | Shape | Read when |
|---|---|---|
| `hvfhv_zone_day`, `hvfhs_companies` | one Parquet file each, single-digit MB | every run of the showcase pipeline |
| `hvfhv_trips_sample` | Parquet, one file per month, ~500 MB total | a trip-level question; one month is ~21 MB |
| `hvfhv_trips` | Parquet partitioned by `pickup_date`, ~7 GB over ~730 objects | **only the day partitions a query names** |
| `hvfhv_trips_iceberg` | an Apache Iceberg table, ~500 MB | the Iceberg read path |
| `manifest.json` | — | the registry seed: `tables[]` carries name, format, path, partition column, row count |

**Egress is the operating cost to watch, and it is bounded by the query, not by
the artifact.** A demo session running the shipped pipeline at its default
one-month window reads a few MB. The exposure is a visitor who writes
`SELECT * FROM hvfhv_trips` with no date predicate — a ~7 GB full scan at S3
egress prices, per run. Set a bucket request-rate or budget alarm before the lake
demo is announced.

Because there is no loader, the consuming side of this family is a datasource
registration — shipped in round 089 as the `lake` demo family: the `sample-lake`
LAKE datasource is registered create-if-absent from
`deploy/sample-data/bootstrap-datasources-lake.yml`, which also imports the
published `manifest.json`'s `tables[]` into the dp-catalog registry
([Datasources §8A.1](/docs/datasources.md#8a1-file-shape), §8C). The showcase pipeline
ships in its own examples file (`scripts/sample-data/content/examples-lake.json`)
and the seeder loads it exactly when `sample-lake` is registered (the
`requires_datasources` gate — [environments.md §5](/docs/environments.md#5-demo)).
It needs egress to S3 (or a configured mirror) whenever a lake query runs.

### Resetting an engine volume desyncs the demo login

The loader creates the `dp_demo_ro` login with the passwords from the env files,
and bootstrap registration is **create-if-absent — it never updates an existing
datasource row** (§8A). So if you delete an engine's data volume
(`docker volume rm dp-mysql-data`) or rotate a `SAMPLE_*` password, the
freshly created login no longer matches the credential the app stored at the
original bootstrap, and that datasource fails validation in the UI while the
engine itself is fine. The repair is to re-run registration for the stale entry
only:

```bash
docker exec dp-postgres-1 sh -c \
  'psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -c "delete from datasources where name='"'"'sample-weather'"'"';"'
docker restart dp-datapipelines-1   # bootstrap re-registers with the current env password
```

(2026-08-30: hit live — a recreated MySQL volume left `sample-weather` failing
UI validation until its row was re-registered this way.)

### Re-publishing is a new version

Version directories are immutable (design §4). Any change to the data — a wider
window, a different sample, a corrected lookup, even a typo — is published as
`v2` under a new prefix; `v1` is never edited. Consumers pin the version
(`SAMPLE_VERSION`), and nothing references a `latest` alias. This is what makes
"the same version means the same data" a fact rather than a hope, and it is why
the loader refuses a manifest whose `version` does not match the directory it
came from.

### Publish confirmation & release rehearsal — the drift guards

Two guards bracket every publish and every release rehearsal. Both exist
because of T70 (2026-09-02): the published `v1` `examples.json` still carried
the `${}` interpolations 042 had already migrated out of the repo copy, the
demo 500ed on every fresh first login for two days, and nothing compared the
published bytes with the repo's.

1. **Before confirming an upload** (and in every release rehearsal), run the
   published-drift guard for the version the demo pins:

   ```bash
   ./scripts/sample-data/check-published.sh v2                  # nyc (mobility), the default family
   ./scripts/sample-data/check-published.sh --family trade v2   # the trade family
   ./scripts/sample-data-lake/check-published.sh --family lake v1   # dp-lake (a different contract, below)
   ```

   It fetches the published manifest and `examples.json`, and fails unless the
   published copy, the published manifest's declared checksum, and the family's
   repo `content/examples.json` all agree. `--family` selects all three of the
   repo copy, the base-URL variable (`SAMPLE_BASE_URL` for nyc,
   `SAMPLE_TRADE_BASE_URL` for trade) and the default published prefix. Against
   an unpublished version it fails on the manifest fetch — which is the upload
   gate itself. Set the family's base-URL variable to check a mirror or a
   locally staged build (`file://…/scripts/sample-data/work`, or the local-serve
   recipe in `app.sh`). Network by nature, so it is a rehearsal step, never part
   of `./gradlew build`.
   The **lake** family's guard is a separate script because it checks a different
   thing. That family has no `examples.json` — its content lives with the nyc
   family — so what can drift is not content but the objects themselves. It
   fetches the published manifest, refuses a manifest whose `version` disagrees
   with its directory, **fails if any provenance row still carries
   `license_verified: null`**, samples one object per table and compares SHA-256,
   and fetches the Iceberg metadata file to confirm every location it records
   sits inside the published prefix — an Iceberg table that still names a build
   machine's directory is unreadable, and nothing about an object listing shows
   it. Folding two unrelated contracts behind one `--family` flag would make
   "check-published passed" mean two different things.
2. **The repo copy is validated in `build`** — the templates module's
   `SampleDataExamplesContentTest` runs every shipped template and pipeline
   through the app's own save-time validators (049 C1), so content the seeder
   would refuse cannot merge. `verify.sh` step 5 remains structural only; the
   two checks are different jobs and each says which it does.

### Licence gate — before serving this data publicly

Every `provenance` row in `manifest.json` ships `license_verified: null`. The
build verifies no licence and claims none; the licence strings are research
claims carried with their evidence links in each family's `sources.lock`
(`scripts/sample-data/`, `scripts/sample-data-trade/`,
`scripts/sample-data-lake/`).

**Publishing with any `license_verified` still null blocks go-live** (design §8).
Verify each source's current terms, record the date, and swap — do not ship — any
dataset that fails. This applies to datapipelines.co; a self-hosted evaluation
loading the artifacts for its own use is a different question, and one for the
operator.

---

## Appendix C: Change Log

| Date | Version | Author | Change |
|---|---|---|---|
| 2026-11-01 | v1.61 | #685 merge follow-up | **§3.1 / §6.5:** the image disables glibc's native thread-stack cache before its JVMs start; overrides retain that entry, and bare JVM launches set it before Java starts. The Dockerfile drift guard pins the declaration and launch order. |
| 2026-10-28 | v1.60 | #747 | **§3.1:** the DuckDB re-pin recipe refreshes each extension's linked closure, `deploy/duckdb-extension-ports/<name>.tsv` (the vcpkg port closure, direct and transitive, plus the DuckDB core `third_party` code and port-vendored code both binaries' symbols witness), before reviewing every new component's licence; the generator refuses a closure bound to another commit, a closure entry without a reviewed licence and a reviewed component missing from the closure. |
| 2026-10-27 | v1.59 | #743 — renumbered at merge after #692’s v1.58 | **§3.1:** the four image bullets state what the Dockerfile builds — base image `eclipse-temurin:21-jre` (Ubuntu glibc; argon2-jvm loads its native library through JNA), a single-stage build (the boot jar is built on the host and copied in; the image adds the DuckDB extension bundle, the GraalJS isolate + resource cache and the third-party notices file), the `useradd --system` account with no fixed UID, and the exec-form entrypoint naming the GraalJS resource cache and `/app/datapipelines.jar`; the timezone paragraph no longer claims the entrypoint bakes the UTC flag in — every deployment passes `-Duser.timezone=UTC` itself. `DeploymentImageBulletsSpecDriftTest` pins the bullets to the Dockerfile (one `FROM` tag, the `ENTRYPOINT` array verbatim, the `USER`/`useradd --system` account) and refuses the retired claims — the base tag, the builder-stage build, the builder-image pin, the JVM-flags variable, the fixed UID, the wrong jar path and the baked-in-UTC claim — described here rather than quoted, so the retired-words grep over this file stays empty. The Dockerfile's healthcheck comment now names the Ubuntu base, not alpine. |
| 2026-10-26 | v1.58 | #692 | §3.1: checksum-pin all eight decompressed DuckDB extension binaries, include commit-bound extension and direct vcpkg component notices, and document the three-input re-pin procedure. |
| 2026-10-26 | v1.57 | #693 — renumbered at merge after #570’s v1.56 vendored browser-asset notices | **§6.2A:** the offline `:modules:web:generateVendoredAssetNotices` task renders the vendor manifest and upstream licence files to `/vendor/THIRD-PARTY-NOTICES.txt`, packaged in the web jar and website export. Missing coverage, unknown or mismatched licences and stale allowances fail the build. |
| 2026-10-26 | v1.56 | #570 unused MapLibre CSS | **§6.2A:** patch 0004 drops the unreachable maplibre-gl stylesheet require from the non-map custom bundles; both bundles rebuilt and hashes and sizes re-pinned, with the rule to drop the patch when a `map` trace joins a trace list. |
| 2026-10-25 | v1.55 | #702 | **Appendix B step 2:** replaced the top-bar chip, login mint and `GET /api/v1/auth/api-keys/mine` flow retired by keys v2 (#233, Auth v3.5) with the Keys page / `POST /api/v1/auth/api-keys`, the role chosen under the subset rule and the one-time reveal. The v1.28 row remains as history. Prose only. |
| 2026-10-25 | v1.54 | #712 — renumbered at merge after #442’s v1.52/v1.53 | **§6.4:** the management port's chart exposure (configuration.md §3.14) — a `management` values block, everything OFF by default (byte-identical default render): `management.expose` adds the management `containerPort` (from `management.port`), pins both `MANAGEMENT_SERVER_*` env entries (address `0.0.0.0` — the opt-open switch) and a headless `Service` (`clusterIP: None`); `management.networkPolicy.enabled` adds the chart's one ingress `NetworkPolicy` for the app pods, which carries two rules: the scraper peers on the management port, and the app-port rule from `appIngress.from` (`NetworkPolicyPeer`s, verbatim) or every source under the explicit `appIngress.allowAll: true`. Every selector is a full `LabelSelector`. The render refuses both scraper selectors empty, no app-port declaration, `allowAll` with a `from`, a bare label map, an unknown or empty peer and a quoted `allowAll`; the isolation and union semantics, and how the policy combines with operator-owned ones, are stated. `management.serviceMonitor.enabled` adds a `ServiceMonitor`. Any `extraEnv.MANAGEMENT_SERVER_*` double mapping is refused (Kubernetes honours the last duplicate env name silently). Chart 0.2.0 → 0.2.1. `helm-check.sh` grows to lint plus eighty-nine verdicts (`(h)`–`(h5h)`), with a pinned `kubeconform` validating every successful render against the Kubernetes 1.35.0 schema and a parsed NetworkPolicy contract. **§9:** the management-port sentence names the app-port rule. `HelmManagementValuesSpecDriftTest` pins the values, the expose guards, the refusals, the three templates' flag lines, the shared app port, the `matchLabels` example and the configuration.md §3.14 row. |
| 2026-10-25 | v1.53 | 442c (#442) | §8.3: keep scheduler mail off until every instance understands V51 notification trail kinds. §8.4: disable, drain and account for old readers refusing historical notification kinds before downgrade. |
| 2026-10-24 | v1.52 | 442b (#442) | §5.1: the one optional key with a startup requirement of its own — `datapipelines.scheduler.notifications.enabled=true` needs the mail transport and `auth.base-url` or startup refuses; under `development` nothing sends until the recipient-domain allowlist names a domain. |
| 2026-10-24 | v1.51 | #688 (Helm portion) — renumbered at merge after #677’s v1.50 | **§6.4:** the nine JavaScript runtime keys (configuration.md §3.28) are named chart values under `script.js`, all empty by default — an empty value renders no env entry, inheriting the image's own isolate directory and the app's derived isolate memory. `script.js.isolateDir` is refused at render time outside the image's launcher-path entries (`/app/lib`, `/opt/graaljs`), and an `extraEnv` double mapping of any of the nine is refused like the DuckDB keys. Number validation stays the app's (startup, by name). Chart 0.2.0; `helm-check.sh` grows to lint plus ten template cases (all-nine render, the single-key or-clause proof, the nine double-mapping refusals, the isolateDir refusal and its two admitted entries); `HelmScriptJsValuesSpecDriftTest` pins doc, chart, values and the Dockerfile's launcher-path list together. |
| 2026-10-24 | v1.50 | #677 third-party notices | §10 documents generated Maven/isolate notices in the jar and image, the unresolved/stale licence guards, optional-driver terms and the remaining distribution boundaries. |
| 2026-10-22 | v1.49 | 8b (#8) | Documents the isolate directory/loader agreement, pre-installed resources on read-only roots, shared-isolate failure mode and memory sizing. |
| 2026-10-21 | v1.48 | #8 lane 8a the JavaScript isolate (owner ruling 2026-10-21) | **§3.5** the `lib/` drop-in paragraph quotes the image's value as it now is, `ENV LOADER_PATH=/app/lib,/opt/graaljs`: the drop-in directory, then `/opt/graaljs`, where the image fetches its platform's SHA-256-pinned GraalJS isolate jar. One list in one `ENV` because `PropertiesLauncher` reads a `-Dloader.path` before the variable — an ENTRYPOINT `-Dloader.path=/opt/graaljs` would have silently dropped the drop-in (measured in lane 8a). The mount point, its mode and the never-from-the-environment rule are unchanged. `ImageLoaderPathSpecDriftTest` pins the new value. Renumbered at merge after #666's v1.47. The engine's operator-facing configuration and the rest of the image's JavaScript story are lane 8b's. |
| JavaScript isolate | `js-isolate.jar` is in `datapipelines.script.js.isolate-dir`, which is one loader-path entry; the native resource cache is pre-installed and read-only; startup logs one isolate-loaded line |
| 2026-10-21 | v1.47 | #666 Helm driver drop-in | **§6.4:** opt-in `libVolume.enabled` and one operator-supplied `libVolume.source` mount trusted drivers read-only at `/app/lib`; PVC sources must also be read-only. `extraEnv.LOADER_PATH` is refused with the image-owned-path message, in every spelling the launcher reads (`loader_path`, `loader.path`: Spring Boot's `SystemPropertyUtils` tries the key as given, dots to underscores, then upper-cased). `JAVA_TOOL_OPTIONS`, `JDK_JAVA_OPTIONS` and `_JAVA_OPTIONS` in `extraEnv` are refused too, as the compose audit's class 8 refuses them (#679). Chart 0.1.2; helm-check covers the default, refusal, ConfigMap mount and PVC/source boundaries; the deploy-file drift guard admits exactly the template refusal and requires it once. |
| 2026-10-21 | v1.46 | #657 the image's driver drop-in | **§3.5** the `lib/` drop-in paragraph now matches the image: the Dockerfile creates `/app/lib` empty, `root:root` mode `0555`, before `USER`, and sets `ENV LOADER_PATH=/app/lib` — until now it did neither, so a mounted driver was never on the classpath. The value is absolute because `PropertiesLauncher` resolves a relative `loader.path` against `loader.home` (default `${user.dir}`) and the image has no `WORKDIR`: `LOADER_PATH=lib` adds `/lib`. The operator mounts the DRIVER jar read-only at `/app/lib/`; neither compose nor the chart sets `LOADER_PATH` (§9). **§6.4** states that the chart does not support the drop-in today (no volume knob; `extraEnv` does not refuse `LOADER_PATH`). `ImageLoaderPathSpecDriftTest` pins the Dockerfile, this paragraph and Module Structure §5.4.1 to one value and refuses a deploy file that names `LOADER_PATH`. |
| 2026-10-19 | v1.45 | #571 path-independent Plotly rebuilds | **§6.2A:** patch 0003 salts root-relative inline-CSS paths with the historical build directory, preserving every committed byte from any clone directory. The recipe uses a temporary directory; `scripts/verify-plotly-bundles.sh` rebuilds and compares both bundles for the lander on Plotly vendor or manifest-entry diffs, with distinct match, mismatch and rebuild-failure exits. |
| 2026-10-16 | v1.44 | #538 audit manifest sizes | **§6.2A:** the vendored Plotly audit now compares all three raw byte lengths and both gzip counts with the manifest, and pins zlib deflate level 9 with gzip framing and mtime 0 in each bundle's size string. Size-only and method-only drift fail the build. |
| 2026-10-15 | v1.43 | #531 the default heatmap keeps crisp cells under the CSP | **§6.2A:** a second numbered Plotly source patch sets the heatmap’s `image-rendering` fallbacks through the CSSOM instead of a style attribute; both custom bundles are rebuilt at the fixed path and their hashes and sizes move together in the manifest. The unminified comparison contains only the heatmap import and write changes; minified identifier allocation also changes. |
| 2026-10-14 | v1.42 | #525 the Plotly recipe carries a vendor patch and a fixed build directory | **§6.2:** the Plotly rebuild gains `git apply` of `static/vendor/plotly/patches/*.patch` (the colorbar fill reset removes its `style` attribute instead of writing an empty one — two source lines, +4 bytes per bundle) and `npm ci`; the clone directory is fixed at `/tmp/dp-plotly-4.1.1/plotly.js` because the minified output depends on the clone's absolute path (#530; the 2026-09-30 build did not record its directory and is not reproducible, so both bundles were rebuilt, not patched in place). Sizes and hashes are in the vendor manifest; `VendoredPlotlyAuditTest` pins the bytes. #525 merge follow-up (2026-10-15): §6.2 names the marker each host pre-places as the empty `template` (#456) that the runtime's fallback also creates (#526), not a `style` element; the manifest's plotly `note` says the same. |
| 2026-10-03 | v1.41 | 411 (#411) CI renders the reference chart | §6.4: the chart is rendered by CI — the `gate` job runs `scripts/helm-check.sh`, a pinned `helm` (v4.3.0; the release's checksum file SHA-256-pinned in-repo, bumped with the version in one commit) running `helm lint --strict` plus four `helm template` cases: defaults render no `env:` block, both `duckdb.*` values render exactly their two quoted entries, each `extraEnv` double mapping is refused with the template's own message, and a missing `existingSecret` is refused. Exit contract: 0 all cases green, 1 a chart check failed, 2 tooling. Also corrects this section's HPA bullet — the chart scales on CPU only (`hpa.yaml` carries one `cpu` metric; there is no memory metric). No chart change. |
| 2026-10-02 | v1.40 | 140 (#140) the Helm chart maps the DuckDB operator keys | §6.4: the reference chart gains a `duckdb` values block — `duckdb.extensionDirectory` and `duckdb.memoryLimit` (configuration.md §3.25's two env vars) — each rendered as an env entry ONLY when non-empty: an empty default inherits the image's bundled extension directory, and a rendered `""` would override the image's `ENV` and break zero-egress LAKE under the read-only root filesystem. `extraEnv` stays the escape hatch for every other non-secret key; an `extraEnv` double mapping of either DuckDB key is refused at render time. Chart version 0.1.0 → 0.1.1. `HelmDuckDbValuesSpecDriftTest` pins doc = chart values = env entries. |
| 2026-09-30 | v1.39 | L3a (#10) the dashboards client runtime — renumbered at merge after 337's v1.38 | **New §6.2A Vendored browser assets** — where the vendor manifest and its audit tests live, and the Plotly 4.1.1 provenance: the two custom bundles' exact rebuild recipe (clone the pinned tag, `npm run custom-bundle` per trace list, commit the outputs), the sizes and the npm tarball sha1, and the plotly.css design-around that keeps `style-src` at `'self'` plus Cytoscape's one hash. No code, route or policy change. |
| 2026-09-30 | v1.38 | 337 (#337) structured local logging | **§5.2 gains "Log output format — turning JSON on"**: what each deploy path resolves today (`app.sh`/VPS `console` from `defaults.env`, raw compose and bare jar `json`), the operator recipe for turning JSON on (set the env in `secrets.env`, redeploy), the startup refusal on an unknown value, and the statement that redaction runs under both values. No deploy file changed — the declared env var just works now. |
| 2026-09-29 | v1.37 | the 316 merge's review (#316, #266) | §6's Deployment bullet and §8.3.2: the grace is **60 s** (chart and compose) — the sum had left out 266's persistence drain (`shutdown-drain-ms`, 10 s), so at the configured maxima the runtime could kill mid-drain; §8.3.1 step 4 names the persistence drain and the scheduled jobs' scheduler stop (#316), which sits after every drain and outside the sum. `ShutdownGraceArithmeticTest` carries the new term (red at 50 s: compose needed 55, Helm 60). |
| 2026-09-28 | v1.36 | 301 (#305) | §6.2's CSP row prints the style-src hash LITERALLY again — 195's rewrite had left a `'<hash>'` placeholder where the value stood: `'sha256-pgvDUBa4IjFA2yuSJ2cqcyxmNYJMborsd0ORcRv9vw8='`, the SHA-256 of the one `<style>` element Cytoscape injects on the editor page. The value is derived, not typed: `SecurityHeadersTest` computes it from `SecurityHeaders.CYTOSCAPE_STYLESHEET` (the vendored sheet verbatim), and `ApplicationSmokeTest` asserts the header the way an operator reads it, literal included — a Cytoscape bump that changes the sheet goes red rather than documented wrong. No code, route or policy change. |
| 2026-09-28 | v1.35 | 195 (#195) the editor's eval exemption retired — renumbered at merge after 298's v1.33/v1.34 | §6.2's CSP row rewritten: the route-scoped `'unsafe-eval'` exception 188 carried on `GET /pipelines/{id}/editor` is GONE — the editor runs Alpine's CSP build (`@alpinejs/csp` 3.14.1, same vendored path, the vendor manifest records the npm tarball's sha256) and its expressions are pure property paths. One policy covers every policed route; `style-src` carries Cytoscape's one sheet hash on every route (a hash admits exactly that sheet, so no route's posture is weakened). `SecurityHeaders.kt` lost `CSP_POLICY_EDITOR`/`isEditorRoute`; `SecurityHeadersTest` pins the directive's absence on the editor route (falsified by re-adding it), and the browser suite's zero-violation collector is the live proof. No route, permission or role changed. |
| 2026-09-28 | v1.34 | 298 (#291 item 3) the promotion batch and the body cap | §6.3A's settings table gains the receiver's `datapipelines.web.max-request-bytes`: a promotion is one push body, refused atomically with `413 request.body_too_large` past the RECEIVER's cap. Measured over the shipped example content (all three demo families, 40 documents): 73 KiB, 3.6% of the 2 MiB default — a split is not warranted; a batch of several near-256K-character templates is the case that needs the knob raised on the receiver. |
| 2026-09-28 | v1.33 | 298 (#298) the #194 scan survives a malformed scale | §8.3 step 4's first scan guards its `::int` cast (`CASE WHEN scale ~ '^-?[0-9]{1,9}$'`): a stored `scale` that is not an integer (`"2.0"`, `"x"`, or past `int`) is LISTED — a malformed declaration is what the operator runs the scan to find — where the bare cast stopped the whole read-only scan with an error. `PreDeployScanQueriesTest` runs the block verbatim over a fixture that now carries both. |
| 2026-09-28 | v1.32 | 286 (#277) the stop grace's arithmetic | §6's Deployment bullet and §8.3.2: the chart's `terminationGracePeriodSeconds` and compose's `stop_grace_period` are **50 s**, written as the sum of the shutdown sequence at its configured maximum — preStop 5 + the scheduler's admission gate ≤ 15 + the drain flush 20 + Tomcat 10. At 40 s (#251) the three scheduler phases alone reached 40 under Helm and left Tomcat nothing; §8.3.2's snippet and bullet still said 30. `ShutdownGraceArithmeticTest` pins both files and every value this document states. |
| 2026-09-28 | v1.31 | 286 (#268, #194) pre-deploy scans | §8.3 gains step 4, the pre-deploy check for builds after 2026-09-26: 194a's over-scale-default scan (it lived in that lane's handback as an untested sketch — now made robust to a body with no `parameters` object) and #268's scan for stored parameters declaring `constraints` or a `cardinality`, each tagged and run verbatim by `PreDeployScanQueriesTest` against the shipped schema; the later steps renumber. The Status line read v1.28 behind the v1.30 row; it now reads v1.31. |
| 2026-09-24 | v1.30 | 224 (#224) demo API | New Appendix B subsection "The demo API": seeding a demo family publishes every seeded pipeline under `/demo/…` and binds them to one public `api_caller` key minted from `DATAPIPELINES_DEMO_API_KEY` (blank = off, changed = rotation); the per-key request budget and the lake family's budget gate are the same act (rest-api.md §19.8). |

| 2026-09-22 | v1.28 | key wording | Appendix B step 2 says what the product does since R3: the MCP key is created at sign-in and copied from the top bar; nothing is "minted" by hand and there is no create endpoint. `app.sh`'s demo message says the same. |
| 2026-09-22 | v1.27 | #207 relative redirects | §8 edge contract: every app `Location` is RELATIVE (`server.tomcat.use-relative-redirects=true`), so the public scheme never leaks into a redirect and the edge needs no `X-Forwarded-Proto` handling for them. Behind a TLS-terminating edge the absolute `http://` Locations Tomcat built by default were refused by the browser (CSP `form-action` on plain forms, mixed content on htmx requests) after the server had already acted — logout, workspace switch and member removal on datapipelines.co, 2026-09-22. |
| 2026-09-21 | v1.26 | 199 (#196) the Redis password leaves argv | §4.2.1 gains "Where the password lives": the reference redis services (`deploy/compose.yml`, `deploy/compose.laptop-infra.yml`, Appendix A) start `redis-server -` — configuration from stdin, `requirepass` fed by a heredoc from `REDISCLI_AUTH`, `exec` so the server stays PID 1, `user: redis` because the wrapper is `sh` — so no argv, and no file, carries the value; `--maxmemory`/`noeviction` stay on argv. §9's argv line closes the "tracked separately" remainder and names the two commands that check it. Operators: the next start recreates the Redis container (§7 says what a restart loses). Numbered on base db12e043; the merger renumbers if another lane took v1.26. |
| 2026-09-21 | v1.25 | 188 (#188) the app states its headers | §6.2 gains "What the edge sets, and what it must not touch": the header table the app sends on every response (an enforced CSP with no `'unsafe-inline'`, `X-Frame-Options: SAMEORIGIN`, `Referrer-Policy`, `Permissions-Policy`; the editor route's `'unsafe-eval'` exception, #195), why the app sends no HSTS, and the four things the product expects of an edge (HSTS one year without preload, headers passed through unweakened, TLS + redirect, the forwarded address). §9: two checklist lines for the same. |
| 2026-09-21 | v1.24 | 188 (#189) Redis credentials | §9: the hardened posture refuses a passwordless non-loopback Redis; the reference compose files pass the healthcheck password as `REDISCLI_AUTH` (environment), never `redis-cli -a` (argv); the server's `--requirepass` argv named as the remaining exposure. |
| 2026-09-19 | v1.23 | 173 (#173) the agent surface before indexing | §6.7: `/llms.txt`, `/llms-full.txt` and `/docs/{slug}.md` listed beside the crawler surfaces; the Search Console note says llms.txt needs no submission; the sitemap bullet corrected — `lastmod` has been absent since 145, not read from `build-info.properties`. |
| 2026-09-17 | v1.22 | Writable temporary storage (#133) | §6.4: disk-backed `/tmp` emptyDir with optional size limit while preserving the read-only root filesystem; LAKE disk sizing and cleanup requirements. |
| 2026-09-14 | v1.21 | 137 mail notices | §5.1: the first-admin story no longer says "no SMTP" — with `DATAPIPELINES_MAIL_HOST` + `_FROM` set the one-time credential is emailed to the user and sys-ops (`DATAPIPELINES_MAIL_OPS_TO`) is told about every new user ([Auth §5A.8](/docs/auth.md#5a8-mail-the-welcome-mail-and-the-new-user-notice)); the variables are catalogued in [Configuration §3.27](/docs/configuration.md#327-mail) and named in `deploy/secrets.env.example`, per this doc's no-restated-keys rule. |
| 2026-09-10 | v1.20 | 109 §E mobility v7 | **The nyc family pins `SAMPLE_VERSION=v7`** — a new immutable version directory carrying ONE change over v6: the `trips` table gains the composite index `(pu_location_id, pickup_date)` (`idx_trips_pu_location_pickup_date`, DDL beside the two singles), measured ~90× on the year-2024 airport shape (8.1 s sized seq scan → 0.09 s index-only). The other three artifacts are byte-identical v6 copies and every table content-checksum is unchanged — the v2 precedent's restore-and-redump shape. The v7 set (pg-trips.dump 117,733,060 bytes, sha256 `02627d0846…`, plus manifest) is handed to the owner in `handbacks/109-artifacts/mobility-v7/` for publishing; a fresh `--demo nyc` start needs v7 published to load. |
| 2026-09-09 | v1.19 | 109 §E demo Postgres sizing | **New "Demo Postgres sizing" subsection (Appendix B)**: the nyc family's `trips` table (~804 MB) meets sized-for-the-data flags — `shared_buffers=512MB`, `effective_cache_size=1536MB`, `work_mem=32MB`, `random_page_cost=1.1` (SSD) — with the container memory reservation raised to 1 GB and the "demo sizing, not production advice — the operator's own datasource is their own DBA's" boundary stated. The compose change itself is `cfb65c9c`; this section is the rationale the compose comment points at. |
| 2026-09-08 | v1.18 | 093 announcement-day truth pass | **Appendix B's demo-families preamble corrected against the merged 089 product.** It said dp-lake was *published but not yet consumable* with "no compose profile, no loader and no `--demo` flag" while `app.sh` already accepts `--demo lake` and adds `--profile demo-lake` (app.sh lines 144/209, `SAMPLE_LAKE_ON`) — the paragraph was written at 088 and 089 shipped the consuming side. The families table gains a **lake** row; "Two independent families" becomes three; the nyc row's "2 example pipelines" becomes **6** (`scripts/sample-data/content/examples.json` holds six); the one-command and raw-compose recipes gain the third family; the `-Pmysql` sentence now says which families need it. The "family with no loader" section below is unchanged — it was already correct — and the distinction it draws (a profile and a flag, but nothing to load) is now stated in the preamble too, which is where the contradiction was read. |
| 2026-09-07 | v1.16 | 089 dp-lake §D extension bundling | **§3.1 gains the bundled DuckDB extensions bullet**: the image now downloads `httpfs`/`aws`/`iceberg`/`avro` for DuckDB core v1.5.5 at build (+108 MB uncompressed, ~39 MB downloaded), gunzips them into `/opt/duckdb/extensions/v1.5.5/<linux_amd64|linux_arm64>/` (the build's `TARGETARCH`), and exports `DATAPIPELINES_DUCKDB_EXTENSION_DIRECTORY=/opt/duckdb/extensions` — a LAKE datasource then LOADs from disk and never INSTALLs, so a hardened dp-lake deployment needs no egress to `extensions.duckdb.org` (the 089 §7.3 spike's CAN verdict, verified on this exact base image with `--network none`). `deploy/compose.yml` defaults the variable to the same path (declared in `compose-env-audit.sh`'s new `IMAGE_DEFAULT_VARS`, the first compose default that mirrors the image rather than application.yml). The operator key itself is [Configuration §3.25](/docs/configuration.md). |
| 2026-09-07 | v1.15 | 088 dp-lake data | **Appendix B gains a third sample-data family, `dp-lake`** — NYC TLC High Volume FHV (Uber/Lyft/Via/Juno) trips published at `s3://datapipelines-co/sample-data/lake/<version>/` as Parquet partitioned by `pickup_date`, a 1-in-16 sample, a zone/day pre-aggregate and an Apache Iceberg copy of the sample. New section "**dp-lake — the family with no loader**": it is **read in place**, so there is no compose profile, no loader service and no `--demo` flag — the datasource that reads it (dialect `lake`, `sample-lake`) is round 089, and the showcase pipeline ships in its own `scripts/sample-data/content/examples-lake.json` that no deployment loads yet. The drift-guards section gains `scripts/sample-data-lake/check-published.sh --family lake <version>` and says why it is a separate script (no `examples.json`; it checks object hashes, the licence gate and the Iceberg metadata's recorded locations). Egress is stated as the operating cost: bounded by a query's date predicate, unbounded for an unfiltered scan of the ~7 GB table. No application code, no deployment change. |
| 2026-09-07 | v1.15 | dp-lake v1 published (088) | `s3://datapipelines-co/sample-data/lake/v1/` — 855 objects, 7.57 GB: `hvfhv_trips` (471,851,707 NYC rideshare trips, day-partitioned Parquet), `hvfhv_trips_sample` (1-in-16), `hvfhv_zone_day`, `hvfhs_companies`, `hvfhv_trips_iceberg` (the sample as an Iceberg table). Read in place by the dp-lake datasource (round 089) — no loader, no download at demo start; licence verified 2026-09-07 (NYC Open Data, FHV disclaimer quoted in the manifest). `check-published.sh v1` in the lake family is the proof. |
| 2026-09-07 | v1.14 | mobility v6 (082) | The demo briefing gains a CALCULATOR node (`fiscal_quarter` → `run_fiscal_quarter`, bound as the first caller column) so the feature is visible out of the box; data files unchanged; baselines re-keyed. Pin → `SAMPLE_VERSION=v6` in `deploy/env/defaults.env`; `check-published.sh v6` byte-identical. |
| 2026-09-06 | v1.13 | mobility v5 + trade v4 (077 mandatory folders) | Both artifact sets republish with every template id under a folder (`nyc/mobility/…`, `nyc/reference/…`, `nyc/weather/…`, `trade/…`) — 13 of 25 demo templates were flat and are refused by 077's rule at seed time. Data files unchanged in content. Pins → `SAMPLE_VERSION=v5`, `SAMPLE_TRADE_VERSION=v4`; `check-published.sh v5` / `--family trade v4` byte-identical on publish. |
| 2026-09-06 | v1.13 | 081 one env file | The `deploy/` layout is **two env files**: tracked `deploy/env/defaults.env` (every non-secret variable with the value this deployment ships) then git-ignored `deploy/secrets.env`, in that order under every loader, with `deploy/secrets.env.example` as the template that names every variable. The five files 075 put under `deploy/env/` (`laptop.env`, `demo.env`, `example.env`, `posture/development.env`, `posture/hardened.env`) are deleted; §6.2, §6.3A and Appendix B's raw-compose recipe use the two-file list. `deploy/compose.laptop-infra.yml` starts its Postgres and Redis from `SPRING_DATASOURCE_PASSWORD` / `DATAPIPELINES_REDIS_PASSWORD`, removing the laptop's load-order inversion. `./app.sh --scaffold` writes `deploy/secrets.env` from the template and stops, so the admin address can be set before the one-time seed; `--start` waits until the app is actually healthy (up to `HEALTH_WAIT_SECONDS`, default 360) instead of reporting the HEALTHCHECK deadline as a failure, and `--status` prints the seeded login too. |
| 2026-09-05 | v1.12 | 075 environments and posture | **[Environments](/docs/environments.md) joins the spec set** — the operator page for `DATAPIPELINES_ENV` (the org's label) and `DATAPIPELINES_POSTURE` (`development` \| `hardened`), with the normative posture table, the environment-variable contract, and loader recipes for Compose, a bare JAR, systemd, Kubernetes, ECS and Nomad. The `deploy/` layout is renamed with no shims: `docker-compose.yml` → **`compose.yml`**, `docker-compose.local.yml` → **`compose.local-build.yml`**, `docker-compose.dev.yml` → **`compose.laptop-infra.yml`**; `deploy/application.yml` is **folded away** (every line was already in the image's own `application.yml` behind the same placeholders) and orgs wanting a yml use `SPRING_CONFIG_ADDITIONAL_LOCATION`; `deploy/.env`/`.env.demo`/`.env.example` and the root `.env.example` are replaced by tracked `deploy/env/**` (posture, demo, laptop, and `example.env` — every variable with its shipped default) plus git-ignored `deploy/secrets.env`. Secrets carry **the app's own variable names** (`DATAPIPELINES_JWT_SECRET`, `SPRING_DATASOURCE_PASSWORD`, …) instead of compose-only renames, so one file feeds every loader. §6.3A's promotion example is now env+posture. Demo is a **flag**: `DATAPIPELINES_DEMO=nyc,trade` (`./app.sh --demo nyc,trade`) replaces `--demo-nyc`/`--demo-trade`, with the versions in tracked `deploy/env/demo.env`, and `hardened` refuses it. `scripts/compose-env-audit.sh` now runs on every `./gradlew build` (it had drifted: 074's three `endpoints` keys shipped with no compose pass-through) and also checks `deploy/env/example.env`. Compose volume names derive from `COMPOSE_PROJECT_NAME`, so `-p <lane>` alone scopes a second copy's data. |
| 2026-09-05 | v1.11 | mobility v4 + trade v3 (067 pipeline folders) | Both artifact sets republish with `examples.json` carrying the folder convention (`nyc/…`, `trade/…`); data files unchanged in content (every pinned table checksum re-derived identical; the DuckDB file's bytes differ as DuckDB files are not byte-deterministic). Pins move to `SAMPLE_VERSION=v4`, `SAMPLE_TRADE_VERSION=v3`. Until an operator moves the pin, a fresh demo seeds the old flat names — expected, version pins are operator config. |
| 2026-09-04 | v1.10 | trade/v2 — Binance out, Federal Reserve H.10 in | The trade family republishes as **trade/v2**. The Binance market slice is **removed entirely** (its Vision terms are CC BY-NC-SA with an explicit no-hosting-of-derivative-feeds clause and a separate enterprise licence for commercial use — owner ruling 2026-09-04); the SQLite artifact is now `fx_rates.db`, built from the **Federal Reserve H.10** daily noon buying rates and their G.5 monthly averages for the five reconciled partners' currencies (a US Government work — no copyright), and the datasource is renamed `sample-market` → **`sample-fx`**. A third example pipeline, `imports_in_partner_currency`, restates US import value in the partner's own money across three engines (DuckDB facts → SQLite rates → H2 join). Two licence conditions that were never in the tree now ship with the data: the **verbatim Census notice** ("This product uses the Census Bureau Data API but is not endorsed or certified by the Census Bureau.") in the family README, the `sample-trade-us` datasource description and the manifest's Census provenance entry, and the **UN Comtrade citation** with the under-100,000-record note. `check-published.sh` learns `--family trade`; `SAMPLE_TRADE_VERSION` defaults to `v2` in `app.sh`, `deploy/.env.example` and the compose services. **An existing `deploy/.env.demo` pinning `SAMPLE_TRADE_VERSION=v1` is not rewritten by `app.sh` (version pins are operator config) — edit it, or the loader fetches a version whose market object no longer exists.** |
| 2026-09-04 | v1.10 | mobility v3 (070 showcase pipelines) | **Mobility artifact v3 published** — v2's data files byte-identical, `examples.json` now 17 templates / 6 pipelines (the 070 showcase set, baselined by `check-baselines.sh`); `SAMPLE_VERSION` default → v3. Operators with an existing `.env.demo` set `SAMPLE_VERSION=v3` by hand — `app.sh` never overwrites present keys. |
| 2026-09-04 | v1.9 | two-family demo split | **The demo splits into two independent sample-data families** — `demo-nyc` (mobility) and `demo-trade` (trade/v1), each with its own compose profile, loader services, bootstrap datasources file and examples file; the app's bootstrap keys are now comma-separated lists and the seeder runs all templates before any pipelines. app.sh gains `--demo-nyc`/`--demo-trade` (the old `--demo` dies with a hint); the ON markers (`SAMPLE_NYC_ON`/`SAMPLE_TRADE_ON`) compose the active-family lists. The trade family adds the demo's fourth engine (DUCKDB, `sample-trade-us` — driver ships in core, read-only via the probed `properties.jdbc.access_mode: READ_ONLY`, datasources.md §8A.5). Appendix B rewritten for the two flags; the compose project is renamed `deploy`→`dp` (containers/volumes carry the dp- prefix; the metadata-DB volume was migrated, dp-postgres-data). |
| 2026-09-02 | v1.7 | 051 auth/config sweep | §6.2 gains `datapipelines.auth.trusted-proxies` (R8/T46): behind the LB the login limiter and every audit `source_ip` must resolve the client through the trusted-proxy list — empty default keeps bare deployments on `remoteAddr`, header ignored. §6.7's homepage note updated (T46 closed; no-limiter decision stands on its own grounds). §6.3 documents the compose env contract with its `scripts/compose-env-audit.sh` guard (T32). Appendix B tells the truth about the demo password (T47/T73): on a clean checkout the scaffolded GENERATED password wins over the `demo-admin` seed — `grep DATAPIPELINES_AUTH_LOCAL_BOOTSTRAP_PASSWORD deploy/.env`. app.sh's `up --wait` timeout now reports "still starting" and exits 0 when the app container is running (T75; a cold JVM can outrun the HEALTHCHECK window — 243s in the 2026-09-02 rehearsal) |
| 2026-09-02 | v1.9 | multi-instance round 2 (050) | **§6.2 checklist gains the per-instance-limits row** (R2/M4/M7): `max-concurrent-executions-per-instance` is per instance and N replicas admit N × it — the multiplication stated once, plainly; the Redis row names the datasource pool-invalidation channel. §6.6 heap paragraph rewritten around the per-instance multiplier (the old text read the key as a "cluster-wide ceiling" — false at N > 1, and the exact trap the rename closed); §5.2 deploy-time keys updated to the renamed key. |
| 2026-09-02 | v1.8 | demo artifact v2 + guards (049) | **Appendix B gains "Publish confirmation & release rehearsal — the drift guards"**: `scripts/sample-data/check-published.sh` (published `examples.json` vs repo vs manifest, 049 C2) is now a named pre-upload / rehearsal step, and the repo copy's semantic validation is pinned in `build` (`SampleDataExamplesContentTest`, 049 C1) — the two guards T70 (published-v1 drift → first-login 500 ×2 days) proved missing. A v2 artifact is staged from unchanged pins with the licence gate re-stamped 2026-09-02; this row's commit is the held version bump — it moved the demo pin (`SAMPLE_VERSION=v2`) and every current-version citation, and merges only after the owner's upload is confirmed live (`check-published.sh v2` against the bucket). |
| 2026-09-04 | v1.8 | SEO (073) | §6.7 rewritten for the widened public surface: `/docs` and `/docs/{slug}` are **public** (the viewer renders packaged Markdown and reaches no principal, workspace or datastore; the same content is public on GitHub), the intent-cluster pages join `/`, and `/robots.txt` + a generated `/sitemap.xml` ship. The allowlist bullet records exactly what was added. New operator note: **Search Console and the sitemap** — verify the domain by DNS TXT before the launch announcement, submit the sitemap once (it regenerates per request, so later releases need no resubmission), and why a self-hosted deployment emits `datapipelines.co` canonicals unless `SITE_ORIGIN` is changed. |
| 2026-09-01 | v1.7 | multi-instance readiness (036) | **The stale-execution sweep now ships (ARCH-AUDIT M2), and the §6.2 crash bullet is true again.** Every replica runs the idempotent sweep on a one-minute cadence (the project's first `@Scheduled`); `RUNNING` rows older than `datapipelines.executions.stale-timeout-minutes` are marked `ABORTED` with `pipeline.execution.instance_lost`. §8.3.2's SIGKILL-mid-flush consequence restored to match. Side effect: `DELETE /executions/{id}` on a crashed instance's stale row stops being a silent no-op — once swept, the row is terminal and the cancel is refused `pipeline.execution.not_running` instead of returning 204. |
| 2026-09-01 | v1.6 | multi-instance readiness (036) | **The drain now ships (ARCH-AUDIT M1), and §8.3.1/§8.3.2 describe it.** On SIGTERM: readiness flips to REFUSING_TRAFFIC first, every local execution is cancelled through the ordinary path (`Statement.cancel()` first, `execution_aborted` with `reason: "shutdown"`, row `ABORTED`), the flush is awaited with a 20s bound, and `server.shutdown: graceful` lets in-flight requests end. The drain CANCELS rather than draining-to-completion — the v1.2 text's "drain up to `execution-timeout-seconds`" behavior is deliberately not what shipped; `terminationGracePeriodSeconds: 30` (not 630) and `preStop: sleep 5` are the matching pod settings, and the Helm chart carries both. The §6.2 crash bullet is still honest (the sweep lands next). |
| 2026-09-01 | v1.5 | multi-instance readiness (036) | **Honesty fix (ARCH-AUDIT M11): the doc promised three things the code does not do.** §8.3.1/§8.3.2 rewritten as shutdown behavior *as shipped* — no drain, no `server.shutdown: graceful`, no readiness flip, no sweep; the old drain-to-timeout/`cancelAll(shutdown)` sequence and the `terminationGracePeriodSeconds: 630` + `preStop` guidance described unimplemented code. §6.2 instance-crash bullet no longer claims rows are "swept to `ABORTED`" (the sweep exists but has no caller). §5.2's "shutdown grace" gloss on `execution-timeout-seconds` removed. §6.4's `deploy/helm/` reference made real: a minimal chart (Deployment, Service, optional HPA/PDB) now ships. The drain and sweep claims return with the code that implements them. |
| 2026-08-31 | v1.4 | website + docs in-app (033) | New §6.7: the app serves the marketing site (`/`, public) and the packaged spec set (`/docs`, session-only); the dashboard moved to `/dashboard`; the standalone `website/` static deploy retires to the `websiteExport` cold-fallback procedure; public surface defended by cache headers, NOT the login rate limiter (OPEN-ITEMS T46). Header version corrected (v1.3's entry had not bumped it). |
| 2026-08-05 | v1.0 draft | initial draft | Initial deployment spec sketch — Docker image, infra requirements, configuration, deployment patterns, upgrade/rollback, security checklist |
| 2026-08-05 | v1.1 | horizontal scaling | Added multi-instance horizontal scaling section. Application is stateless for all CRUD/UI/MCP/auth. In-flight executions are instance-local (acceptable for short-running pipelines). No sticky sessions required. Added multi-instance checklist. Added LB idle-timeout + SSE heartbeat note. |
| 2026-08-07 | v1.2 | consistency campaign | Applied [SPEC-REVIEW-2026-08 §2.15](https://github.com/msabiransari/datapipelines/blob/main/docs/SPEC-REVIEW-2026-08.md#215-deploymentmd): §5 env-var tables replaced by the startup-requirements list + pointer to configuration.md; the inline-vs-claim-check threshold key (superseded by the D9 result keys) and every other key configuration.md does not define were deleted [D8]; §4.2 rewritten as the result store with required `maxmemory-policy noeviction` and a sizing model [D9]; §8.3.1/§8.3.2 graceful-shutdown mechanism (readiness fail → drain to `execution-timeout-seconds` → `cancelAll(shutdown)` → exit) with k8s `preStop` + `terminationGracePeriodSeconds`, accepted loss stated [D7]; §6.2 instance-local story updated to cancel-on-disconnect + cross-instance cancel via Redis flag [D7]; Appendix A compose made bootable (OIDC provider env vars, Redis password wired to `requirepass`, noeviction, mounted provider YAML); new §3.5 JDBC driver matrix (bundled vs `-Poracle`/`-Pmysql` vs `lib/` drop-in); new §6.6 resource sizing (heap, container limit, `-XX:MaxRAMPercentage`); `-Duser.timezone=UTC` made normative in the image and bare-JVM entrypoints ([Type System §8.4](/docs/type-system.md#84-timestamp-timezone-normalization)); §6.2 diagram residue and §11 malformed bullet fixed |
| 2026-08-29 | v1.3 | local password auth | §5.1 item 5 becomes "at least one authentication method": OIDC provider OR local accounts (auth.md §5A), with the operator's first-admin story for the no-IdP case (hash-seeded one-time credential, forced first-login change, admin resets — no SMTP, no self-registration). Appendix A compose comment and Appendix B quickstart updated: the demo now logs in with a local account (`demo-admin@demo.local` / `demo-admin`, one-time) and needs no OIDC client. |
| 2026-09-07 | v1.8 | 087 connector seams | §3.5 driver matrix gains **`LAKE`** — bundled, and the same `duckdb_jdbc` jar as `DUCKDB`. It is a distinct dialect rather than a mode because the two need opposite §5.6 postures ([Datasources §4.1](/docs/datasources.md#41-dialect-catalog)); nothing about packaging changes, since there is no second driver to ship. |
| 2026-09-08 | v1.17 | 089 dp-lake consuming side shipped | Appendix B's dp-lake section: the "that is round 089 / until it lands, the published objects are inert" paragraph rewritten as shipped — the `lake` demo family registers `sample-lake` from `deploy/sample-data/bootstrap-datasources-lake.yml` (which imports the published manifest's `tables[]` into the dp-catalog registry), and the seeder loads `examples-lake.json` exactly when `sample-lake` is present (the `requires_datasources` gate, environments.md §5). |