Deployment & Packaging Specification

Status: v1.61 Owner: datapipelines.co core Depends on: all other specs Last updated: 2026-11-01

1. Purpose

This spec defines how datapipelines.co is packaged, distributed, and deployed in self-hosted environments. The product is open-source and self-hosted; this document covers the artifacts we produce and the deployment patterns we support.

This is lighter than the core specs — most operational concerns are the operator's responsibility (we ship the artifact, they run it). We document our artifacts, required infrastructure, configuration surface, and recommended deployment patterns.

2. Design Principles

1. One artifact, many deployments. A single Docker image runs in dev, staging, prod. Configuration via env vars + mounted files; no build-time decisions.
2. Stateless app, externalized state. The app has no local persistent state. Metadata DB (Postgres) and Redis (result store, idempotency keys, cancellation flags, post-completion event log) are external. H2 staging is per-execution in-memory.
3. Container-first. Primary distribution is a Docker image. JVM-only deployments (no container) are supported but secondary.
4. k8s-native but not k8s-required. Kubernetes is the recommended target. Docker Compose works for small / single-node deployments. Bare JVM works for development.
5. Configurable without forking. Every deployment-relevant knob is an env var or config-file value. Operators do not edit source code to deploy.
6. Open-source distribution. The image and JARs are published to public registries / Maven Central. No proprietary distribution channel.

3. Build Artifacts

3.1 Docker image

Published to ghcr.io/datapipelines/datapipelines:{version} and docker.io/datapipelines/datapipelines:{version} (mirror).

• Base image: eclipse-temurin:21-jre (the Ubuntu glibc JRE 21 image, chosen over a smaller alpine base because argon2-jvm loads its native library through JNA, which ships glibc binaries).
• Layers: single stage: the boot jar is built on the host (./gradlew :modules:app:bootJar) and copied in; the image adds the DuckDB extension bundle, the GraalJS isolate with its pre-installed resource cache, and the third-party notices file.
• Bundled DuckDB extensions (dp-lake): httpfs, aws, iceberg and avro for DuckDB core v1.5.5, pre-populated under /opt/duckdb/extensions/v1.5.5/<platform>/ (<platform> = linux_amd64 or linux_arm64, following the build's TARGETARCH) and activated by the image's ENV DATAPIPELINES_DUCKDB_EXTENSION_DIRECTORY — a LAKE datasource LOADs them from disk and never INSTALLs at runtime, so no egress to extensions.duckdb.org is needed (Configuration §3.25). Adds ~108 MB uncompressed (~39 MB downloaded at build). The eight decompressed binaries (four per platform) are pinned in deploy/duckdb-extensions.sha256; the extension build layer runs sha256sum -c --ignore-missing --strict before USER and refuses a mismatch. /app/THIRD-PARTY-NOTICES includes the extensions at their footer commits, their copyrights and licences, and their pinned linked closure: direct and transitive vcpkg ports, the DuckDB core third_party components, and code bundled inside ports (the committed closure files and re-pin procedure are described below).
• Size target: < 250 MB compressed.
• User: non-root (datapipelines, a system account created with useradd --system; no fixed UID — a chart or compose runAsUser must not assume 1000).
• Entrypoint: ["java", "-Dpolyglot.engine.userResourceCache=/opt/graaljs/resources", "-jar", "/app/datapipelines.jar"] (exec form). The image defines no JVM-flags hook: a JVM option variable (JAVA_TOOL_OPTIONS and friends) would reach the JVM without one, which is why the shipped deploy files refuse those names (§6.4, §9).

To re-pin on a DUCKDB_CORE_VERSION bump, update three inputs together: the Dockerfile core ARG, deploy/duckdb-extensions.sha256 (eight sums plus the four # <name> commit=<hash> annotations), and the commit-bound licence inventory in buildSrc/src/main/kotlin/DuckDbExtensionLicenses.kt. In a temporary directory, for each of linux_amd64 and linux_arm64 and each of httpfs, aws, iceberg, avro, fetch https://extensions.duckdb.org/v<core>/<platform>/<name>.duckdb_extension.gz, decompress with gunzip, and run sha256sum on the decompressed file. Write sorted sha256sum -c lines relative to /opt/duckdb/extensions; inspect each binary's final 512 bytes (tail -c 512 <file> | strings) for its extension repository commit, core and platform. Both platforms must report the same commit per extension. Refresh each extension's linked closure, deploy/duckdb-extension-ports/<name>.tsv, at those commits and the core version, as DEVELOPMENT.md's third-party notices section describes: the vcpkg port closure, direct and transitive, plus the DuckDB core third_party code and port-vendored code that both binaries' symbols witness. Review every new component's LICENSE, update its licence in DuckDbExtensionLicenses.kt and vendor any new SPDX text from the snapshot named in the allowlist. Run :modules:app:generateThirdPartyNotices, the buildSrc licence fixtures and DockerfileDuckDbExtensionPinTest, rebuild the jar, and prove one bad sum refuses the image build before building the restored pins successfully.

-Duser.timezone=UTC is normative, not a suggestion. The type system assumes a UTC JVM: ingest normalizes timezone-aware source values to UTC and treats naive source timestamps as already-UTC (Type System §8.4), and every internal TIMESTAMPTZ is stored and rendered in UTC. A container whose JVM default zone is anything else silently shifts rendered timestamps. No image entrypoint sets it: every deployment — container, bare JVM or JAR (§3.2, §6.5) — MUST pass -Duser.timezone=UTC itself.

Heap and container-memory sizing: §6.6.

Native thread stacks. The image sets GLIBC_TUNABLES=glibc.pthread.stack_cache_size=0 before its build-time and application JVMs start (#685). This disables reuse of native thread-stack mappings when the JVM hosts the script isolate. An operator-supplied GLIBC_TUNABLES replaces the image value: retain glibc.pthread.stack_cache_size=0 when adding other colon-separated tunables. DockerfileGraalJsPinTest pins the declaration and its position before the JVM launches.

3.2 JAR distribution

• Published to Maven Central: co.datapipelines:datapipelines-app:{version}.
• Executable Spring Boot fat JAR (./gradlew bootJar).
• Runnable directly: java -jar datapipelines-app.jar.

3.3 Module artifacts (for embedding)

• Each module published to Maven Central: co.datapipelines:typesystem, co.datapipelines:dag, etc.
• Allows third parties to embed parts of datapipelines.co in their own products (e.g., use our type system, or our MCP server in a different application).
• Marked "experimental API" until v1.1 stabilizes module boundaries.

3.4 Source distribution

• GitHub release per version, with signed tag.
• Source tarball attached to the release.
• Reproducible builds (Gradle lockfile committed, exact JDK pinned via toolchain).

3.5 JDBC driver matrix (what ships in the image)

Which drivers are present is a packaging property of the artifact, not a configuration one. The licensing rationale and the driver-class lookup live in Datasources §10; this table is the operator's view of the published image.

The lib/ drop-in. Spring Boot's PropertiesLauncher adds every jar in loader.path to the application classpath (Module Structure §5.4.1). For the container, mount the DRIVER jar read-only into /app/lib/: the image ships that directory empty, root:root mode 0555, and sets ENV LOADER_PATH=/app/lib,/opt/graaljs itself: the drop-in directory, then the directory holding the image's JavaScript isolate jar, as one list because a -Dloader.path would override the variable, and each entry absolute because a relative entry resolves against the working directory and the image sets no WORKDIR. Neither deploy/compose.yml nor the Helm chart sets LOADER_PATH. For bare JVM, place it in a lib/ directory beside datapipelines-app.jar and start with -Dloader.path=lib (relative to the working directory) or an absolute path. This is the no-rebuild path — the operator accepts the driver's license by supplying it.

Registering a datasource whose dialect has no driver on the classpath fails at save time with datasource.driver_not_loaded (Datasources §9) — a deployment/packaging error, not a bad payload. The same payload succeeds after the driver is supplied.

4. Required Infrastructure

4.1 Postgres (metadata DB)

• Version: 14+ recommended.
• Purpose: persistent storage for pipelines, templates, datasources, executions, audit log.
• Provisioning: operator-managed. We support connecting to an existing Postgres instance.
• Database size: small (a typical deployment has < 1 GB metadata).

Connection keys (URL, credentials, Hikari pool size) are defined in Configuration §2 / §3.13.

4.2 Redis (result store and coordination)

• Version: 6+ recommended.
• Provisioning: operator-managed.
• Purpose — four distinct workloads share one store:
  1. Caller results. Every completed execution's caller result is materialized to Redis and read through the cursor (REST API §7). There is no inline-vs-large split.
  2. Idempotency keys (Idempotency-Key records, retained per datapipelines.idempotency.ttl-seconds).
  3. Cancellation flags (dp:cancel:{execution_id}) — how DELETE /executions/{id} reaches the executing instance (DAG Executor §8.3.1).
  4. Post-completion event log, 1 hour, backing SSE replay (REST API §10.3).

Connection keys are defined in Configuration §2 / §3.1; TTL and size limits in Configuration §3.5.

4.2.1 Required Redis configuration

maxmemory-policy noeviction. This is not tuning advice — it is a correctness requirement. Any LRU/LFU eviction policy lets Redis silently discard keys under memory pressure, and every key class above is load-bearing:

• an evicted result turns a completed execution into a spurious "expired" 404 before its TTL;
• an evicted idempotency key lets a client retry execute the pipeline a second time;
• an evicted cancellation flag makes DELETE /executions/{id} a no-op, leaving the execution running;
• an evicted event-log entry silently truncates SSE replay.

Every key the app writes carries an explicit TTL, so noeviction does not leak: the store drains on its own schedule. Under genuine memory exhaustion the app fails loudly (result.storage_unavailable on the write path) rather than corrupting semantics quietly.

Where the password lives (#189, #196). In the reference compose files the Redis password has one home inside the container: the service's environment (REDISCLI_AUTH). The healthcheck's redis-cli reads it from there, and the server reads its requirepass from the same variable through stdin — redis-server - takes its configuration from standard input, and a sh -c wrapper feeds it a heredoc, then execs so redis-server stays PID 1. Nothing carries the value on a command line (docker inspect … {{.Config.Cmd}} and ps inside the container show the variable's NAME only), and nothing writes it to a file. --maxmemory and --maxmemory-policy noeviction stay on argv; options after - are read after stdin and win. The wrapper is sh, so the service declares user: redis — the image's entrypoint drops privileges only when argv[0] is redis-server. Operators: the next app.sh --start (or docker compose up -d) after this change recreates the Redis container; what that loses is exactly what any Redis restart loses (§7: unexpired results, idempotency records, cancellation flags, the event log — all volatile by design).

4.2.2 Sizing

Budget for the peak sum of all four workloads:

peak ≈ (datapipelines.result.max-size-bytes × concurrent recent executions)   ← dominant term
     + idempotency keys (small, × datapipelines.idempotency.ttl-seconds window)
     + 1h of post-completion event logs (small, proportional to nodes × executions)
     + cancellation flags (negligible)

The result term dominates and is the only one worth arithmetic. Results live for their TTL (datapipelines.result.ttl-default-seconds, clamped between ttl-min/ttl-max — Configuration §3.5), so "concurrent recent executions" means executions completed within one TTL window, not executions running right now. Worst case with defaults (100 MB cap, 300 s TTL) is dominated by how many callers actually pull 100 MB results; the practical lever is lowering result.max-size-bytes and pointing bulk workloads at output.target: datasource instead — the explicit NOT-goal of result delivery is bulk data transfer.

512 MB is a reasonable starting point for small deployments; raise it before raising result.max-size-bytes.

4.3 Network egress

The app must reach:

• Configured datasources (PG, Oracle, MSSQL, MySQL, DuckDB files, SQLite files).
• The metadata Postgres.
• Redis.
• (Optional) OTLP collector, Sentry, KMS — if those features are configured.

The app does not require inbound network access beyond the HTTP/MCP ports it serves.

5. Configuration

configuration.md is the single authority for every configuration key — YAML path, env var name, default, and description. This spec deliberately does not restate key names or defaults: a duplicated default is a default that drifts. What follows is only the operator's startup checklist and the file-mounting mechanics.

Use the exact environment placeholders in Configuration §5. Most names use upper snake case, but provider, bootstrap, promotion and dashboard settings have explicit exceptions; deriving a name from the YAML path can leave a setting unchanged.

Before any of this, two variables: DATAPIPELINES_ENV is your organisation's name for this deployment and DATAPIPELINES_POSTURE is development or hardened. A named environment with no posture does not start. Environments is the page for that decision; Configuration §3.23 is the key reference.

5.1 What the app requires to start

The app fail-fasts on startup if any of the following is missing. Full definitions: Configuration §2.

1. Metadata Postgres: spring.datasource.url, spring.datasource.username, spring.datasource.password.

2. Redis: datapipelines.redis.host.

3. datapipelines.jwt.secret — internal JWT signing secret.

4. datapipelines.db.encryption-key — AES-256 master key for datasource credentials. There is no fallback source: no KMS lookup, no auto-generated key file. Lose it and every stored datasource credential is unrecoverable, so it belongs in a secret manager and in the operator's backup plan. (KMS sourcing is a ROADMAP §2 item.)

5. At least one authentication method: a fully-configured OIDC provider under datapipelines.auth.oidc.providers (each with client-id, client-secret, and issuer-uri), or local password accounts (datapipelines.auth.local.enabled=true, Auth §5A). The first admin comes from datapipelines.auth.bootstrap-admin-email either way (Auth §4.4). A provider entry whose client-id is empty is ignored with a WARN, not counted. The client-id/secret env var names are chosen by the deployment (GOOGLE_CLIENT_ID, OKTA_CLIENT_ID, …) — they are the one deliberate exception to the derivation rule above.

   The operator's first-admin story with local accounts (no IdP): set datapipelines.auth.bootstrap-admin-email to the admin address, enable datapipelines.auth.local.enabled, and seed the one-time credential — preferably as a pre-computed hash from ./gradlew :modules:auth:hashPassword into datapipelines.auth.local.bootstrap-password-hash (the plaintext form exists for demos). Hand the password to the admin out-of-band; the app forces a change at first login (Auth §5A.2), and a deployment still running the seeded credential announces itself with a startup WARN. Every account after the first is created by an admin on the user-administration screen — there is no self-registration and no self-service email reset: a forgotten password means an admin resets it there, which issues a new one-time credential under the same forced-change rule. With outbound mail configured (DATAPIPELINES_MAIL_HOST + DATAPIPELINES_MAIL_FROM — any SMTP server, Postmark's included; the nine variables are Configuration §3.27 and deploy/secrets.env.example names them) that credential is emailed to the user and sys-ops (DATAPIPELINES_MAIL_OPS_TO) hears about every new user, local or first social login (Auth §5A.8); the admin screen then shows where the password went, not what it was. Mail is on exactly when those two are set — there is no flag — and it then also requires DATAPIPELINES_AUTH_BASE_URL (the login URL in the mail).

Everything else has a default and is optional. One optional key adds a startup requirement of its own: datapipelines.scheduler.notifications.enabled=true (scheduler notification mail, off by default — Scheduler §2.3, #442b) refuses to start, naming the keys, unless the shared mail transport (datapipelines.mail.host and .from, Configuration §3.27) and datapipelines.auth.base-url (the link every notice carries) are set; under the development posture it also sends nothing until datapipelines.scheduler.notifications.recipient-domains lists a domain.

5.2 Everything else

Optional keys — executor concurrency, staging memory, result TTLs and caps, SSE heartbeat and disconnect grace, rate limits, idempotency TTL, template cache, UI theme, retention windows, observability — are cataloged with their defaults in Configuration §3. Resolution precedence (env > profile YAML > base YAML, plus the two per-entity runtime overrides) is Configuration §4.

The keys an operator most often changes at deploy time are datapipelines.result.max-size-bytes (Redis sizing, §4.2.2), datapipelines.executor.max-concurrent-executions-per-instance and datapipelines.staging.h2.max-memory-mb (heap sizing, §6.6), and datapipelines.executor.execution-timeout-seconds (the wall clock that bounds any single execution).

Log output format — turning JSON on (#337)

DATAPIPELINES_OBSERVABILITY_LOGGING_FORMAT decides what stdout carries (Configuration §3.15, Observability §3.1): json — one structured JSON object per line, the machine-parsed format a log collector wants — or console, the human-readable development format. The value is resolved before logging initialises; the set is closed, and a mistyped value refuses startup with the offending value named rather than silently falling back. Redaction (Observability §9.2) runs under both values — there is no format that bypasses it.

What each deploy path resolves today, with nothing set beyond what the repo ships:

A container deployment that turns JSON on changes nothing else: the container runtime already collects stdout, and the JSON line carries correlation_id/execution_id at the root — the fields an operator quotes in an incident (Observability §3.3). Log levels stay Spring Boot's own logging.level.* (Observability §9.1).

5.3 Full config file

Every setting is an environment variable, and deploy/env/defaults.env lists all the non-secret ones with their shipped values (deploy/secrets.env.example names the rest) — Environments is the operator's page for what to set and where. Operators who prefer a YAML file can mount application.yml at /etc/datapipelines/application.yml (configurable via SPRING_CONFIG_ADDITIONAL_LOCATION); this is the supported route for more than one OIDC provider, whose nested list is awkward as variables. Every key is expressible either as YAML or as its derived env var; the OIDC provider list is a nested structure and is normally supplied as YAML with ${...} placeholders for the secrets. A complete annotated template is in Configuration §5.

6. Deployment Patterns

6.1 Single-instance (dev / small team)

One instance handles everything — UI, API, MCP, pipeline execution. Docker Compose or bare JVM.

Suitable for: dev, evaluation, small teams (< 20 users).

6.2 Multi-instance horizontal scaling (production)

The application is stateless for all CRUD operations, UI, MCP, and auth. Multiple instances run behind a load balancer and serve requests interchangeably.

                    ┌─────────────┐
                    │ Load Balancer│
                    └──┬───┬───┬──┘
                       │   │   │
              ┌────────┘   │   └────────┐
              ↓            ↓            ↓
        ┌──────────┐ ┌──────────┐ ┌──────────┐
        │Instance A│ │Instance B│ │Instance C│   ← stateless web/API/MCP/UI
        └────┬─────┘ └────┬─────┘ └────┬─────┘
             │            │            │
             └────────────┴────────────┘
                      │     │
              ┌───────┘     └────────┐
              ↓                      ↓
        ┌──────────┐           ┌──────────┐
        │ Postgres │           │  Redis   │
        │ (shared) │           │ (shared) │
        └──────────┘           └──────────┘

No sticky sessions required. Each request is independent.

Client addresses behind the load balancer (datapipelines.auth.trusted-proxies). Every instance sees the LB's address as remoteAddr, so anything keyed or recorded on it — the per-IP login rate limiter, every audit source_ip — collapses to one deployment-wide value: without configuration, the login budget (default 10/min) is a single shared bucket any client can exhaust for everyone. Set the key to the CIDRs of proxies you control (a bare IP is a host CIDR):

datapipelines:
  auth:
    trusted-proxies: 10.0.0.0/8   # the LB's network

The resolution is spoof-safe: when the direct peer is NOT in the list, the X-Forwarded-For header is ignored entirely — an untrusted peer cannot forge its way past the limiter by setting it. When the peer IS trusted, the client is the rightmost header entry that is not itself trusted. Each entry must parse as a CIDR or startup is refused (Configuration §3.4/§7) — a typo'd range must not silently widen proxy trust. The shipped default is EMPTY: a deployment with no proxy in front behaves exactly as before, and the header stays ignored.

What the edge sets, and what it must not touch (188, #188)

The app states its own security posture on every response — HTML, JSON, SSE and the /mcp refusal alike — and pins it with tests (SecurityHeadersTest, the wire-level case in AuthHttpBoundaryTest, ApplicationSmokeTest). What it sends:

The app deliberately sends no Strict-Transport-Security. It cannot know it is behind TLS — it sees plain HTTP from the edge — and an HSTS header on a plain-HTTP deployment locks a browser out of it. The edge (the reverse proxy / load balancer that terminates TLS: Caddy, nginx, an ALB) owns it. What the product EXPECTS of any edge in front of it:

• HSTS, one year, subdomains included, no preload: Strict-Transport-Security: max-age=31536000; includeSubDomains. Preload is a registry submission the operator makes deliberately for their own domain, not a header the product should imply on their behalf.
• Pass the app's headers through untouched. An edge MUST NOT weaken Content-Security-Policy (no rewriting to add 'unsafe-inline', no stripping) and MUST NOT relax X-Frame-Options beyond SAMEORIGIN or drop frame-ancestors. An edge that sets its own Referrer-Policy should set the same value, not a looser one.
• TLS termination and redirect: HTTP → HTTPS redirect at the edge; the app is reached over the private network only. The app never needs to know the public scheme for its own redirects: every Location it sends is RELATIVE (/dashboard, /login?error=… — server.tomcat.use-relative-redirects, #207) and resolves against the origin the browser used, so no X-Forwarded-Proto handling is required for redirects, and an edge that forwards it changes nothing. (Before #207 the container built absolute http:// Locations behind the edge, and the browser refused to follow them: form-action 'self' for plain forms, mixed content for htmx requests.)
• Forward the client address (X-Forwarded-For) and declare the edge in datapipelines.auth.trusted-proxies — the paragraph above.

The reference edge for datapipelines.co lives in its own repository by design; this section is the contract it implements, not its configuration.

What's stateless (any instance serves any request)

• REST API (all CRUD, list, detail, execute endpoints)
• MCP server
• UI (all screens — Thymeleaf server-rendered)
• Auth (JWT sessions, API key validation — both backed by Postgres)
• Template rendering (reads from Postgres, per-instance LRU cache)
• Pipeline validation (reads from Postgres)

What's instance-local (tied to the instance that started it)

• In-flight pipeline executions. When POST /pipelines/{id}/execute hits Instance A, the execution runs on Instance A's JVM (coroutines + per-execution H2 staging). The SSE stream is a direct connection from Instance A to the client.
• Implication: if the SSE connection drops mid-execution, there is no reconnection or resumption path. Instance A starts a grace timer (datapipelines.sse.disconnect-grace-seconds, default 30) and, if the execution has not reached a terminal event by the time it elapses, cancels it — the execution ends ABORTED (REST API §6.8). A client that loses its stream should assume the abort and re-execute. This is deliberate: an execution nobody is waiting for must not keep holding source-database connections and staging memory.
• Cross-instance cancel works anyway. DELETE /api/v1/executions/{id} may land on Instance B, which has never heard of the execution. Instance B writes a Redis cancellation flag; Instance A honors it on its next heartbeat tick or node boundary — worst-case latency ≈ one heartbeat interval (REST API §10.4, DAG Executor §8.3.1). No sticky sessions are needed for cancellation to be reliable.
• Completed executions are not instance-local at all. Results live in Redis for their TTL and are readable from any instance via the cursor (REST API §7); execution metadata is in Postgres; the 1-hour event log is in Redis. Only the running execution is pinned to one JVM.
• Instance crash: loses only that instance's in-flight executions (H2 staging is in-memory and non-recoverable). Their rows are swept to ABORTED by the stale-execution sweep (marked pipeline.execution.instance_lost once they are older than datapipelines.executions.stale-timeout-minutes; every replica runs the idempotent sweep, so no surviving-instance coordination is needed). Completed results and history are unaffected; the lost work must be re-executed.

Multi-instance checklist

6.2A Vendored browser assets — provenance and the rebuild recipe (#10 L3a)

Every third-party file under static/vendor/ is recorded in static/vendor/design-system/vendor-manifest.json — per file: version, sha256, source, licence — and pinned by the audit tests (VendoredAlpineAuditTest, VendoredHtmxAuditTest, VendoredPlotlyAuditTest, …): a served byte that stops matching its recorded hash is a red build, and so is a manifest entry nobody serves. The manifest also carries the FIRST-PARTY client runtime and its adapters (js/datapipelines-dashboard*.js, the same sha256 discipline — a hand edit must move the manifest in the same commit). generateVendoredAssetNotices (:modules:web:generateVendoredAssetNotices) generates the offline licence inventory and full texts served at /vendor/THIRD-PARTY-NOTICES.txt and copied beside the website export assets.

Plotly 4.1.1 ships as two custom bundles built with the package's own procedure (its CUSTOM_BUNDLE.md), from the pinned tag, plus source patches (patches/0001-colorbar-style-reset-without-attribute.patch: the colorbar's fill reset wrote .attr('style', ''), an inline-style attribute the enforced CSP refuses, and now removes the attribute — #525). The second patch, patches/0002-heatmap-pixelated-hint-via-cssom.patch, sets the default heatmap image’s image-rendering fallbacks through the CSSOM in their original order, ending in pixelated (#531). The build is path-independent since patches/0003-inline-css-id-independent-of-clone-directory.patch (#571): the inline-CSS plugin hashed the clone's absolute path (#530); the in-tree plugin hashes the root-relative CSS path under the historical directory as a constant salt, keeping the committed bytes unchanged. The salt is historical; the next Plotly upgrade may keep or drop it, since the pins move then anyway. Patch patches/0004-drop-unreachable-maplibre-css-require.patch removes the maplibre-gl stylesheet require from the registry: neither custom trace list includes a map trace, so the stylesheet is unreachable; drop this patch when a map trace joins a trace list. To rebuild any bundle — a trace added, a security fix adopted — run exactly this and commit the OUTPUTS with their new hashes. The #531 rebuild used Node 24.21.0 + npm 11.19.0 + npm ci (the earlier #525 build also matched Node 22.23.3 + npm ci and Node 24.21.0 + npm i).

cd "$(mktemp -d)"
git clone --depth 1 --branch v4.1.1 https://github.com/plotly/plotly.js.git plotly.js
cd plotly.js
# the numbered vendor patches, applied in order; a new patch is a new file here
git apply <repo>/modules/web/src/main/resources/static/vendor/plotly/patches/*.patch
npm ci --no-audit --no-fund
# the 2D bundle (the default): scatter, bar, pie, histogram, box, heatmap
npm run custom-bundle -- --traces scatter,bar,pie,histogram,box,heatmap --strict --out 2d
# the 3D bundle (dashboards with a 3D trace): the six above + scatter3d, surface, mesh3d
npm run custom-bundle -- --traces scatter,bar,pie,histogram,box,heatmap,scatter3d,surface,mesh3d --strict --out 3d
cp dist/plotly-2d.min.js dist/plotly-3d.min.js  <repo>/modules/web/src/main/resources/static/vendor/plotly/

A patch is git format-patch -1 against the tagged source with preceding patches applied, so git apply takes the numbered series on a pristine clone; never hand-edit the minified text. Run scripts/verify-plotly-bundles.sh (exit 0) before copying a rebuild in; the lander runs it on any diff under static/vendor/plotly/ or to the manifest's plotly entry. The verifier rebuilds in a fresh temporary directory and compares both bundles: exit 0 means a match, 1 a byte mismatch, 2 a failed rebuild. It is not part of the gate, pregate or CI. Remove the manual recipe's temporary clone after use.

plotly.css is the npm package's own dist/plotly.css — the release build's strict-CSP sheet, the same rules the bundle would inject at runtime. It is part of the CSP DESIGN-AROUND (Dashboards §6.6): each host pre-places the empty <template id="plotly.js-style-global" class="no-inline-styles"> marker Plotly checks (the runtime's fallback creates the same template for a host without one), the page loads this file instead, and style-src stays 'self' plus Cytoscape's one hash — no policy change, no 'unsafe-hashes', no 'unsafe-eval'. Sizes (4.1.1 with patches 0001–0004, #525, #531, #571 and #570): 2D 1,170,988 B (406,683 gz), 3D 1,748,030 B (576,983 gz), plotly.css 7,467 B; gzip uses zlib deflate level 9, gzip framing, mtime 0 (no filename), as reproduced by the JDK's GZIPOutputStream with Deflater.BEST_COMPRESSION; the npm tarball's sha1 f797e8f2d54603fd9d11b08aae8eb31147200b67 verifies the source against the tag. Node 22+ per Plotly's own build matrix (the manifest's build field records the exact versions). After ANY rebuild: update the manifest's sha256 map, the sizes, and the trace lists if they changed — VendoredPlotlyAuditTest guards all three raw byte counts, both gzip counts and their pinned method, as well as hashes and trace lists, and fails the build on a mismatch.

6.3 Docker Compose (dev / evaluation)

Reference compose file provided in deploy/compose.yml. Single instance + Postgres + Redis.

Compose is one loader of the environment-variable contract, not the contract itself (Environments §3): the same settings drive a bare JAR, systemd, Kubernetes, ECS and Nomad. It reads TWO env files, in this order: deploy/env/defaults.env, which is tracked and carries every non-secret default, then deploy/secrets.env, which is git-ignored and holds every credential plus this deployment's own overrides (DATAPIPELINES_POSTURE=hardened, DATAPIPELINES_ENV=prod, the real database URL, the OIDC client).

docker compose -f deploy/compose.yml \
  --env-file deploy/env/defaults.env \
  --env-file deploy/secrets.env up -d

The app service passes every DATAPIPELINES_* variable the app binds, each with the same default application.yml ships — so a key that works against a host-run app (by exporting the variable) reaches the container too, and leaving it unset yields the shipped default. scripts/compose-env-audit.sh diffs the compose block against application.yml's placeholders — and the two tracked env files — and fails on a missing pass-through, a diverged default, a variable declared in both tracked files or in neither, or a posture-dependent key passed with a default instead of in the valueless form. It runs on every ./gradlew build, not when someone remembers to run it. It also inventories every app-container environment name, refuses JVM code-loading names, and requires a reasoned allowlist for names outside DATAPIPELINES_*, rejecting stale entries.

6.3A Promotion: a receiver deployment (055)

A second deployment that receives released content from the first (Versioning §10). Both sides are the same image in different postures — there is no separate build.

On the RECEIVER (e.g. uat):

DATAPIPELINES_ENV=uat
DATAPIPELINES_POSTURE=hardened                            # authoring defaults OFF here
DATAPIPELINES_DEPLOYMENT_PROMOTION_SERVER_KEY=<the shared secret>

On the SENDER (e.g. dev):

DATAPIPELINES_ENV=dev
DATAPIPELINES_POSTURE=development                         # this is where content is built
DATAPIPELINES_DEPLOYMENT_PROMOTION_TARGET_URL=https://uat.example.com
DATAPIPELINES_DEPLOYMENT_PROMOTION_TARGET_KEY=<the same shared secret>

DATAPIPELINES_ENV is your name for each deployment and nothing branches on it; the receiver records the sender's label as the source_env of what it received. DATAPIPELINES_POSTURE is what actually changes behaviour — hardened defaults authoring-enabled to false, which is the receiver's whole configuration. Set DATAPIPELINES_DEPLOYMENT_AUTHORING_ENABLED explicitly if this one box must do both (Environments §1).

Generate the secret the way every other one here is generated — openssl rand -base64 32 — and set the identical value on both sides. It is a bearer credential: it belongs in deploy/secrets.env with the rest of the secrets, never inline in a compose file or a chart's values.

What each setting buys, and what goes wrong without it:

Minting the receiver's key (091; the page is /api-keys since 179): on the RECEIVER, sign in as a super admin → the avatar menu's API keys → New API key → kind Server → name it after the sender (dev → uat) → copy the plaintext, shown once. Paste it into the SENDER's DATAPIPELINES_DEPLOYMENT_PROMOTION_TARGET_KEY. Rotation: mint a second server key on the receiver, set it in the sender configuration, restart the sender to bind the new credential, then revoke the first key. The receiver does not need a restart for stored-key issuance or revocation. (With the deprecated config value instead: set the new value on both sides and restart both.) The stored credential authenticates as its key identity rather than a human account; offboarding a human does not revoke that promotion credential.

Direction is one-way by construction. The receiver validates; it never calls the sender. A compromised receiver cannot reach back into the authoring environment.

Running two stacks on one machine (a rehearsal, or CI): APP_COMPOSE_PROJECT gives ./app.sh a second isolated compose project with its own image tag, volumes and containers, and APP_HOST_PORT moves its published port off the first one's 8080 (the container port never moves). Both are lane knobs, not app config — that is why neither carries the DATAPIPELINES_ prefix.

./app.sh --start                                             # dev, the sender, on :8080
APP_COMPOSE_PROJECT=uat APP_HOST_PORT=8081 ./app.sh --start  # uat, the receiver, on :8081

The sender reaches the receiver from INSIDE its container, so target.base-url is not the localhost URL your browser uses: on Docker Desktop it is http://host.docker.internal:8081, and on Linux it is the host's address on the docker bridge. Two compose projects are two networks; a service name from one does not resolve in the other.

6.4 Kubernetes (recommended for production)

Reference Helm chart in deploy/helm/. Includes:

• Deployment (N+ replicas, behind a Service; terminationGracePeriodSeconds: 60 — #251, #277, #266: the preStop sleep (5 s) plus the scheduler's admission wait (up to 15 s) plus the drain's flush (up to 20 s) plus Tomcat's graceful shutdown (10 s) plus the persistence drain (up to 10 s) must complete before the pod is killed; the shipped deploy/compose.yml sets the matching stop_grace_period: 60s — §8.3.2 has the arithmetic).
• Externalized Postgres (managed recommended).
• Externalized Redis (managed recommended).
• HorizontalPodAutoscaler (scales on CPU).
• PodDisruptionBudget (availability during node drains).
• Management port exposure, all of it OFF by default (the render is unchanged until you opt in — Configuration §3.14): management.expose adds the management containerPort (from management.port, default 9090) and pins both MANAGEMENT_SERVER_* env entries — including the address opt-open 0.0.0.0 — and a headless Service (clusterIP: None: the scraper targets pod IPs, not a cluster VIP); management.networkPolicy.enabled adds the chart's one ingress NetworkPolicy for the app pods — the scraper peers on the management port AND your declared peers on the app port (below); management.serviceMonitor.enabled adds a ServiceMonitor for the headless Service. Any extraEnv.MANAGEMENT_SERVER_ADDRESS / _PORT double mapping is refused (Kubernetes honours the last duplicate env name silently).

The management NetworkPolicy is a complete ingress policy for the app pods (#712). A NetworkPolicy that selects a pod isolates that pod for every ingress, not only for the ports its rules name, and the allow rules of all policies selecting the pod are unioned (Kubernetes: Network Policies). A policy that admitted only the scraper would therefore cut the app port off — so the chart's policy, rendered only with both management.expose and management.networkPolicy.enabled, carries two rules, and the render refuses to enable it without either:

1. Scraper → management port: one peer per non-empty scraper selector (scraperNamespaceSelector, scraperPodSelector). The two are separate peers, OR'd: a pod in a matching namespace OR a matching pod in the app's own namespace. (One peer carrying both a namespaceSelector and a podSelector would be their AND — the Kubernetes rule.) Both empty is refused: an empty from: admits every source.
2. Declared peers → app port (8080): appIngress.from, a list of NetworkPolicyPeer objects (namespaceSelector / podSelector / ipBlock) copied verbatim — typically your ingress controller's namespace. Instead of a list, appIngress.allowAll: true is the explicit acknowledgement that every source may reach the app port; it renders the rule without a from and prints the acknowledgement as a comment in the rendered policy. Neither set is refused; both set is refused as a contradiction. An ipBlock of 0.0.0.0/0 is allow-all by another name.

Every selector is a full LabelSelector object (matchLabels / matchExpressions), copied verbatim. A bare label map is refused at render time: it is not a selector, the API schema rejects it, and an apply path that drops unknown fields would leave an empty selector, which selects every namespace.

management:
  expose: true
  networkPolicy:
    enabled: true
    scraperNamespaceSelector:
      matchLabels:
        kubernetes.io/metadata.name: monitoring
    appIngress:
      from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: ingress-nginx

With policies you already own. Policies are additive. If your own policies already select these pods, they keep every allowance they grant, and the chart's policy adds exactly its declared peers to that union. It never narrows your policy, and it widens it by precisely what you declare. To leave an existing stricter app-port policy unwidened, declare no peer it does not already admit. allowAll: true widens the app port to every source. Egress is not touched (policyTypes: [Ingress]): §9's egress policy stays yours. With management.networkPolicy.enabled off (the default) the chart renders no policy at all.

The pod keeps its root filesystem read-only and mounts a disk-backed emptyDir at /tmp. This supports JVM native libraries and the default LAKE spill directories. Set temporaryStorage.sizeLimit to bound the volume (empty by default), and set ephemeral-storage requests/limits through resources for the workload. Do not use a memory-backed volume for large LAKE spills: that would consume the memory budget spilling is intended to relieve. Each pod has its own volume; Kubernetes removes it with the pod. Size it for concurrent queries and see Configuration §3.24 for explicit paths and cleanup after a process crash. Docker Compose uses the image's writable /tmp; custom read-only deployments must mount a writable temporary directory too.

JavaScript (#8 lane 8b) uses /opt/graaljs/js-isolate.jar in the image, selected for the image platform and checksum-verified at build. DATAPIPELINES_SCRIPT_JS_ISOLATE_DIR=/opt/graaljs must name one directory in the launcher's effective loader.path system property or, when unset, LOADER_PATH environment list (the image: /app/lib,/opt/graaljs). Startup refuses a configured directory outside that list or without js-isolate.jar; the check never adds a class-path entry. A bare jar without the isolate starts with JavaScript disabled, said once at INFO; a JavaScript save/evaluate refuses script.js.unavailable, while JSONata works.

The nine JavaScript runtime keys (configuration.md §3.28) are named chart values under script.js: isolateDir, maxHeapMb, isolateMemoryMb, maxStatements, maxAstDepth, maxStackFrames, maxOutputBytes, maxResultMb, maxInputMb (env DATAPIPELINES_SCRIPT_JS_*, the mechanical camelCase map). All nine default to empty, and an empty or null value renders NO env entry — the image's own isolate directory and the app's derived isolate memory stay in force, exactly like the DuckDB pair below; a rendered "" would override the image's ENV and fail the startup agreement above. script.js.isolateDir is not an arbitrary path: the chart refuses, at render time, any value outside the image's launcher-path entries (/app/lib, /opt/graaljs) — the same entries the startup rule admits — and an extraEnv double mapping of any of the nine fails helm template like the DuckDB keys do. The remaining eight keys are operator sizing: the chart renders them verbatim and does not validate numbers — explicitly supplied zero values render as quoted "0", and an out-of-bound value fails the pod's startup, by name (configuration.md §3.28, §7).

The image pre-installs native resources in the root-owned, read-only /opt/graaljs/resources cache and passes -Dpolyglot.engine.userResourceCache=/opt/graaljs/resources. This is why the read-only root remains supported: no language resources are installed at runtime. A Docker tmpfs at /tmp must permit native-library mapping (for example --tmpfs /tmp:rw,exec); noexec fails in JNA before JavaScript starts. Non-Docker deployments must pre-install resources for their platform using JavaScriptResourcesMain and set the same cache property before starting with a read-only root.

One Engine per JVM shares its isolate heap across fresh per-evaluation Contexts (8a's measured D1 arm; author Sources are not cached). The default isolate ceiling is pool-size × js.max-heap-mb + 256 MiB (1280 at 4 × 256), outside -Xmx; an explicit ceiling must exceed all pool slots' heaps. A tenant's hard isolate-heap trip can cancel another tenant's evaluation once; subsequent evaluations rebuild the Engine. Include both heaps in container sizing; each Context's heap limit is soft.

The two DuckDB operator keys (configuration.md §3.25) are named chart values: duckdb.extensionDirectory (env DATAPIPELINES_DUCKDB_EXTENSION_DIRECTORY) and duckdb.memoryLimit (env DATAPIPELINES_DUCKDB_MEMORY_LIMIT). Both default to empty, and an empty value renders NO env entry — the image's own bundled extension directory stays in force, so LAKE datasources keep their zero-egress LOAD behavior; a rendered "" would override the image's ENV and push every LAKE pool back to INSTALL+LOAD, which needs egress and a writable home the read-only root filesystem does not provide. Every other non-secret env key goes through extraEnv; an extraEnv key that double-maps either DuckDB key fails helm template, because Kubernetes does not reject duplicate env names and would silently honor the last one. CI renders the chart on every push and pull request: the gate job runs scripts/helm-check.sh, a pinned helm (version and checksum pin bumped together per DEVELOPMENT.md §10.2) running helm lint --strict plus eighty-nine verdicts — defaults render no env: block or driver volume, both DuckDB values render exactly their two quoted entries, all nine script.js values render exactly their nine quoted entries (each sizing key alone also renders its explicit zero; null renders no entry), each extraEnv double mapping (the two DuckDB keys, the nine JavaScript keys and the two MANAGEMENT_SERVER_* keys) is refused with the template's own message, a missing existingSecret, extraEnv.LOADER_PATH, the three JVM option variables and a script.js.isolateDir outside the image's launcher-path entries are refused, an enabled driver volume renders its read-only mount (with additional PVC/source checks), and the management block renders nothing by default (no 9090, no MANAGEMENT_* env entry, no NetworkPolicy, no ServiceMonitor) while management.expose renders the four objects — the management containerPort, both MANAGEMENT_SERVER_* entries (0.0.0.0 opt-open), the headless Service and the NetworkPolicy, whose parsed rules must admit the scraper peers on the management port AND the declared peers (or, under allowAll, every source) on the app port — with management.serviceMonitor.enabled adding exactly one ServiceMonitor, and each of the policy's refusals (both scraper selectors empty, no app-port declaration, allowAll with a from, a bare label map, an unknown or empty peer, a quoted allowAll) exercised with its own message. Every successful render is also validated -strict against the Kubernetes 1.35.0 API schema (and the CRDs-catalog schema for the ServiceMonitor) by a pinned kubeconform (DEVELOPMENT.md §10.2), with every object accounted for. It exits 0 when every case passes, 1 when a chart check fails, and 2 on a tooling failure (a download, a checksum-pin mismatch, a schema fetch, or a missing python3/PyYAML — never a chart verdict).

The lib/ driver drop-in (§3.5) is opt-in: set libVolume.enabled: true and supply exactly one Kubernetes volume source in libVolume.source (for example, {configMap: {name: drivers}}); the chart mounts it at /app/lib with readOnly: true, and a persistentVolumeClaim source must itself set readOnly: true. Populate the source only through a trusted image, init container or CI-written ConfigMap, never through the app (§9). The default is disabled; an extraEnv key naming the launcher path in any spelling the launcher reads (LOADER_PATH, loader_path, loader.path) is refused at render time, and so are JAVA_TOOL_OPTIONS, JDK_JAVA_OPTIONS and _JAVA_OPTIONS (the JVM reads -D options from them, and a system property would win over the image's value), so the image's launcher path stays in force against extraEnv. Keys inside the existingSecret reach the container through envFrom and cannot be checked at render time: the Secret holds the §5.1 startup contract and never the launcher path or a JVM option variable.

No sticky session affinity needed. Standard ClusterIP service with round-robin or random load balancing.

6.5 Bare JVM

For development:

GLIBC_TUNABLES="${GLIBC_TUNABLES:+${GLIBC_TUNABLES}:}glibc.pthread.stack_cache_size=0" \
java -Duser.timezone=UTC -jar datapipelines-app.jar \
  --spring.datasource.url=jdbc:postgresql://localhost:5434/datapipelines \
  --datapipelines.redis.host=localhost \
  --datapipelines.redis.port=6381

-Duser.timezone=UTC is required here too (§3.1): every deployment — container, bare JVM or JAR — must pass the flag explicitly.

On glibc, also set the native thread-stack policy from §3.1 before starting Java; setting an environment value inside an already-running JVM is too late. Keep other required tunables and replace any conflicting stack-cache entry with glibc.pthread.stack_cache_size=0.

Not recommended for production.

6.6 Resource sizing

Two numbers matter: JVM heap, and the container memory limit that must contain it.

Heap. The dominant consumer is H2 staging — each executing pipeline holds up to datapipelines.staging.h2.max-memory-mb (default 1024 MB), or its own settings.tempdb.config.max_memory_mb override:

heap ≥ (staging max-memory-mb × concurrent executions on THIS instance) + ~512 MB baseline

The multiplier is the per-instance execution ceiling max-concurrent-executions-per-instance (050/R2): every execution runs on exactly one instance, and this instance can legitimately hold the full setting's worth at an unbalanced moment. The default 100 × 1024 MB is the worst case ONE box must absorb — size the container for it (with N replicas behind a round-robin LB the typical share is setting / N, but size for the ceiling, not the share; cap the exposure by lowering the setting — or max-memory-mb — per box, or scale on smaller per-instance limits). The ~512 MB baseline covers Spring context, Hikari pools, the template cache, and SSE buffers.

Worked example — 4 concurrent executions per instance at the 1024 MB default: 4 × 1024 + 512 ≈ 4.6 GB heap.

Container memory limit.

container limit ≈ heap × 1.5

The 0.5 covers what the heap number does not: metaspace, code cache, thread stacks, and — significant here — JDBC direct/native buffers, which scale with result-set width and fetch size. A limit set equal to -Xmx gets the container OOM-killed by the kernel rather than getting a clean OutOfMemoryError, which is strictly worse to diagnose. For the example above: ~7 GB limit.

JVM flags. Either -XX:MaxRAMPercentage=65 (heap tracks the container limit — preferred for k8s, where limits change without an image rebuild) or an explicit -Xmx. Do not set both. Always -Duser.timezone=UTC (§3.1).

Servlet threads (MCP blocking calls). An MCP pipelines_execute call (MCP Server §6.2.3) is a single blocking HTTP request that holds one servlet thread until the execution reaches a terminal state or datapipelines.executor.execution-timeout-seconds (default 600) elapses. Per-user concurrency is bounded (max-concurrent-executions-per-user, default 10) but the default Tomcat pool is 200 threads, so on the order of ~20 concurrent long-running MCP callers can saturate it and starve REST/UI traffic on the same instance for minutes. Size server.tomcat.threads.max at or above the expected count of simultaneously-blocking MCP executions plus normal REST concurrency, or isolate /mcp on its own connector/instance. This is in addition to raising proxy/LB idle timeouts above execution-timeout-seconds (MCP Server §6.2.3).

CPU. Execution is coroutine-based and largely I/O-bound on source databases; 2 vCPU per instance is a reasonable floor. Scale out on max-concurrent-executions-per-instance pressure, not CPU.

6.7 Marketing site & in-product docs

Since v1.4 the app serves the marketing site and the documentation itself — the site and the product are ONE deployment (owner decision 2026-08-31). There is no separate static deploy to keep available, and the docs shipped in the jar always match the version running them.

• GET / — the marketing site (public). Template templates/site/index.html, assets under static/site/**, referencing the app's vendored design system at /vendor/design-system/** (the retired website/ directory carried a second vendored copy — the app copy is now the single sync target of scripts/sync-design-system.sh). The only dynamic fact (the MCP tool count) is a compile-time constant baked at render time; public routes touch no database.
• GET /dashboard — the signed-in dashboard, moved off /. There is no auto-redirect: signed-in users hitting / get the marketing page.
• GET /docs, GET /docs/{slug} — the packaged spec set (docs/*.md minus the contributor/research exclusions, packaged by processResources in modules/web/build.gradle.kts). Public since 073: the viewer renders Markdown out of the jar and reaches no principal, no workspace and no datastore, and the identical content is already public in the AGPL repository — so exposing it moves ~25 pages of documentation into this domain's search index instead of GitHub's. Anonymous readers get the public site chrome; signed-in readers keep the application chrome.
• GET /mcp-server-for-sql-databases, /mcp-server/{engine}, /add-mcp-server-to-claude-code, /ai-data-pipeline, /text-to-sql-agent, /compare/{airflow,dbt}, /federated-query — the intent-cluster pages (073), same shape as /: GET-only, anonymous, constant content, no database. See UI Screens §4.15.
• GET /robots.txt, GET /sitemap.xml — crawler surfaces (073). robots.txt is a static file; sitemap.xml is generated from the page registry and the packaged doc slugs, with no lastmod (145 §7 — a build timestamp on every URL read as "every page changed" on every deploy).
• GET /llms.txt, GET /llms-full.txt, GET /docs/{slug}.md — the agent surface (173): the llmstxt.org index and full text generated from the same two registries as the sitemap, and every packaged doc as raw Markdown with its links rewritten. Same shape as the sitemap — GET-only, anonymous, no database — and in the allowlist with their reasons (Auth §8.3). See UI Screens §4.15a.
• Public-surface defence is cache headers, not a rate limiter. / is Cache-Control: public, max-age=300; /site/** is public with a 1-hour TTL plus Last-Modified revalidation. The login rate limiter is deliberately NOT applied here (033/D1): the content is constant between deploys, so a shared-cache TTL costs nothing per request — and the T46 remoteAddr-keying concern is closed regardless: the limiter now resolves the CLIENT address through datapipelines.auth.trusted-proxies (§6.2), so pointing it at / would no longer create an LB-address-wide bucket.
• Allowlist. / and /site/** join the permitAll list in SecurityConfig with their reasons inline; 073 added the seven cluster-page matchers (enumerated, not globbed), /docs + /docs/*, /robots.txt and /sitemap.xml, each with its own comment. Nothing else was widened, and anyRequest().authenticated() still closes the list.

Search Console and the sitemap (operator note, 073)

Indexing is not automatic and is not a deploy step — it is a one-time verification plus an occasional check.

1. Verify the domain in Google Search Console (and Bing Webmaster Tools, if you care about it). Use the DNS TXT method: it verifies the whole domain including subdomains, and it survives redeploys — an HTML-file method would need the file added to static/, which is a code change for a DNS problem. Do this before any launch announcement, so the first crawl after the first inbound links is already attributed.
2. Submit https://<your-host>/sitemap.xml once. It regenerates on every request from the page registry and the packaged docs, so a page added in a later release is in it the moment that release is deployed — there is nothing to resubmit. /llms.txt needs no submission anywhere: an agent or an AI crawler fetches it by convention at the root, and it is generated from the same registries, so it is current the moment the sitemap is.
3. robots.txt allows everything, and deliberately lists no private paths: it is a public file, so naming internal routes in it advertises them. What keeps the private surface private is SecurityConfig's anyRequest().authenticated(), not this file.
4. A self-hosted deployment emits datapipelines.co canonicals. That is deliberate — rel="canonical" names the one address that should be indexed, and a private instance asking Google to index it is the failure mode the tag exists to prevent. If you WANT your own deployment indexed under your own domain, change SITE_ORIGIN in modules/web/.../ui/site/SitePages.kt and rebuild.
5. What to look at afterwards. Coverage (are the doc pages indexed?), Performance filtered to /mcp-server/ (are the engine pages taking their measured queries?), and any Core Web Vitals note. Nothing here needs a schedule; a look a month after launch and a look a quarter later is the whole practice.

S3/CloudFront cold fallback (kept, drop if it rots unused). If the app is down but marketing must stay up, the same template renders to static files with facts baked:

./gradlew :modules:web:websiteExport          # → modules/web/build/website-export/
aws s3 sync modules/web/build/website-export/ s3://YOUR-BUCKET/ --delete
aws cloudfront create-invalidation --distribution-id YOUR_DIST_ID --paths "/*"

The export renders /-rooted links (/site/..., /vendor/design-system/...), which the bucket layout mirrors exactly — upload the CONTENTS of website-export/, not the folder. S3 static website hosting (or CloudFront with an origin access control) serves index.html as the index document. Nothing fingerprinted: keep TTLs short for index.html and longer for vendor/** (those files change only when the design system is re-vendored). This is an emergency procedure, not a second primary deploy — the app is the primary. The exported pages need nothing the app's Content-Security-Policy (§6.2) forbids — no inline script, handler or style; their only <script> blocks are JSON-LD data — so the bucket's edge (CloudFront response-headers policy) can and should set the same CSP verbatim; static hosting sends no headers of its own (188).

7. Persistence & Backup

7.1 Metadata DB

• Backup: operator's standard Postgres backup practice (pg_dump, managed snapshots, etc.).
• Restore tested regularly by operator (runbook provided).
• Critical data: pipelines, templates, datasources (with credentials), audit log.
• Disposable data: executions table (can be truncated; only used for history).

7.2 Redis

• Volatile by design. Every key is TTL-bounded: results, idempotency records, cancellation flags, the 1-hour event log (§4.2).
• No backup needed. A Redis restart loses unexpired results (clients re-execute), unexpired idempotency records (a retry may execute a second time), and pending cancellation flags. All are recoverable by re-running; none is a durable record. The durable records — pipelines, templates, datasources, execution history and execution_events — are all in Postgres.
• Persistence (RDB/AOF) is not required and not recommended; it buys nothing that a re-run does not.
• maxmemory-policy noeviction is required regardless (§4.2.1). "Volatile" means TTL-expiring, not evictable — eviction breaks correctness in a way that restart does not, because it happens silently during normal operation.

7.3 H2 staging

• In-memory only. Per-request. No persistence. No backup. No recovery.

7.4 Restore drill

Operators should run a quarterly restore drill: restore metadata DB from backup to a sandbox instance, verify pipelines still execute. Documented in the runbook (future).

8. Upgrades

8.1 Versioning

• Semantic versioning: MAJOR.MINOR.PATCH.
• PATCH: bug fixes only. Safe to upgrade in place.
• MINOR: new features, backward-compatible. Schema migrations applied automatically on app startup.
• MAJOR: breaking changes. Migration guide published.

8.2 Database migrations

• Flyway for metadata DB migrations.
• Migrations are versioned SQL files in app/src/main/resources/db/migration/.
• Applied automatically on app startup.
• Forward-only; no rollback (restore from backup if needed).

8.3 Upgrade procedure (recommended)

1. Review release notes for breaking changes.

2. Backup metadata DB.

3. Pre-deploy check for builds after 2026-09-19 (R-EP5, #274): a database holding a published_endpoints row saved under the old two-segment grammar used to refuse the boot. Since V41 the migration retires such rows (disabled, retired_reason = 'pre-R-EP5 path', never deleted) and the repository skips them regardless, so the upgrade is safe either way; run this to know beforehand what you will see flagged in the API console afterwards:

   SELECT path_pattern FROM published_endpoints
    WHERE array_length(string_to_array(trim(both '/' from path_pattern), '/'), 1) < 3;
   

   Zero rows = nothing for V41 to retire (the two-segment shape of the incident; a row malformed another way is mapped legacy by the new code, never served, and needs no SQL — on a build OLDER than #274 it broke the boot just the same). Rows listed = they come back disabled and flagged after the upgrade; unpublishing one is the fix. An operator in a hurry on a build OLDER than V41 can delete the rows by hand — the demo seeder republishes the demo endpoint at its current path on the next boot.

4. Pre-deploy check for builds after 2026-09-26 (#194, #268): two behaviours that stored pipeline bodies can meet for the first time. (a) A DECIMAL/BIGDECIMAL parameter's declared scale now binds its values, so a stored default with more places than the scale is refused when it applies (pipeline.execution.parameter_constraint_violation). (b) A body saved before #194 could carry a constraints block or a cardinality that Parameter then ignored; those are read now, and one save refuses today is answered 409 pipeline.execution.parameter_declaration_invalid where it is used (a supplied value or an applied default), never a 500. Run both against the metadata DB to know beforehand (read-only; since #298 the first also LISTS a stored scale that is not an integer — "2.0", "x" — rather than stopping on the cast):

   -- pre-deploy #194: stored DECIMAL/BIGDECIMAL defaults with more places than their scale
   SELECT p.name AS pipeline, v.version, v.status, e.key AS parameter,
          e.value->>'default' AS default_value, e.value->>'scale' AS scale
     FROM pipeline_versions v
     JOIN pipelines p ON p.id = v.pipeline_id
    CROSS JOIN LATERAL jsonb_each(CASE WHEN jsonb_typeof(v.body_json->'parameters') = 'object'
                                       THEN v.body_json->'parameters' ELSE '{}'::jsonb END) e
    WHERE e.value->>'type' IN ('DECIMAL', 'BIGDECIMAL')
      AND e.value ? 'default' AND e.value ? 'scale'
      AND CASE WHEN (e.value->>'scale') ~ '^-?[0-9]{1,9}$'
               THEN length(split_part(trim(trailing '0' from (e.value->>'default')), '.', 2)) > (e.value->>'scale')::int
               ELSE TRUE  -- a scale that is not an integer is itself a defect: listed, never a cast error
          END
    ORDER BY 1, 2, 4;
   

   -- pre-deploy #268: stored parameters declaring constraints or a cardinality
   SELECT p.name AS pipeline, v.version, v.status, e.key AS parameter,
          e.value->'constraints' AS constraints, e.value->>'cardinality' AS cardinality, v.created_at
     FROM pipeline_versions v
     JOIN pipelines p ON p.id = v.pipeline_id
    CROSS JOIN LATERAL jsonb_each(CASE WHEN jsonb_typeof(v.body_json->'parameters') = 'object'
                                       THEN v.body_json->'parameters' ELSE '{}'::jsonb END) e
    WHERE e.value ? 'constraints' OR e.value ? 'cardinality'
    ORDER BY 1, 2, 4;
   

   Zero rows from each = nothing stored meets the new behaviour. Rows from the first = defaults to correct (a new draft with the default at the declared scale). Rows from the second whose version was written BEFORE the #194 build was deployed were never checked at save: open the pipeline and re-save it — the save names what is wrong (a cardinality other than SINGLE is refused outright; a pipeline parameter is SINGLE). Rows written after it passed the save check and need nothing. Both scans read every stored version, drafts and discarded ones included; neither covers a schedule's stored inputs. PreDeployScanQueriesTest runs the two blocks above, verbatim, against the shipped schema.

5. Signal shutdown and let the instance drain (§8.3.1 — this is automatic, not a manual step).

6. Start the new version (migrations apply on startup).

7. Verify /health returns UP.

8. Restore traffic.

For k8s: rolling update via kubectl rollout. Each terminated pod flips its readiness and cancels its in-flight executions on the way out (§8.3.1); the preStop and terminationGracePeriodSeconds settings in §8.3.2 keep that orderly.

Scheduler notification kinds (V51).

1. Keep datapipelines.scheduler.notifications.enabled=false (the default) throughout the first rollout containing V51; with mail off this is the ordinary rolling update.
2. Enable notifications only after every instance runs the new image. A new sender writes notification_* trail kinds which an older image cannot read: its strict TrailKind.fromWire throws when ScheduleRunRepository reads that run's detail. Widening the database CHECK is not reader compatibility. Do not enable during the mixed-version fleet.

8.3.1 Graceful shutdown mechanism

Shutdown is a defined sequence, not "stop the process and hope". On SIGTERM the instance, in this order:

1. Fails readiness first. ReadinessState.REFUSING_TRAFFIC is published before anything is cancelled, so /ready starts returning 503 while the process is still fully up and the load balancer / k8s Service bleeds traffic off. The order is the contract: flipping after the drain would keep fresh work flowing into an instance that is already cancelling it. /health keeps reporting UP, so nothing kills the pod for being unhealthy mid-drain.
2. Cancels every in-flight execution through the ordinary cancellation path (DAG Executor §8.3): Statement.cancel() on every registered statement first — which is what actually stops the query on the source database, so the drain is more than a status update — then the root Job cancellation, execution_aborted (reason: "shutdown") emitted to connected streams, tempdb dropped and connections released in finally, and the row written ABORTED. The drain cancels; it does not wait for executions to finish — an execution in flight at SIGTERM ends ABORTED, not SUCCESS.
3. Waits for the flush, bounded (20 seconds). An execution leaves the live count only after its ABORTED status and events are written, so the wait means the bookkeeping reached Postgres and Redis before exit; the bound means a wedged execution meets the kubelet's deadline rather than hanging shutdown forever.
4. Exits. The web server's own graceful shutdown (server.shutdown: graceful) runs after the drain, so SSE clients are still connected while their terminal events arrive, and in-flight ordinary requests complete rather than being cut off. Then the batching writers drain (#266, bounded by shutdown-drain-ms), and last the scheduled jobs' scheduler stops (#316): no tick starts once shutdown began, a tick already running is awaited while the metadata pool is still open, then shutdown.scheduled_jobs_stopped is logged.

The accepted loss, stated plainly: an execution running when an instance stops is cancelled, not preserved. It ends ABORTED with reason: "shutdown", is visible as such in execution history, and its client must re-execute. There is no execution hand-off to another instance and no resumption — in-memory H2 staging makes migration impossible, and pretending otherwise would be worse than the honest abort. Bounded, visible loss beats a silent hang.

One residual race, honestly: a request that reaches the instance between the readiness flip and the web server stopping can still launch an execution. The drain re-cancels on every flush tick, so such an execution is cancelled within ~100ms of starting — but the right answer is that traffic should have stopped at the readiness flip, which is what the preStop in §8.3.2 exists to give the endpoints controller time to do.

8.3.2 Kubernetes pod lifecycle

terminationGracePeriodSeconds: 60      # preStop 5 + admission gate ≤ 15 + drain flush 20 + Tomcat 10 + persistence drain 10
lifecycle:
  preStop:
    exec:
      command: ["sleep", "5"]

• terminationGracePeriodSeconds: 60 — the sum of the sequence at its configured maximum (#277, #266): the preStop sleep (5 s) + the scheduler's admission gate, which waits up to datapipelines.scheduler.shutdown-wait-seconds (default 5, at most 15) for launches in progress + the drain's bounded flush (20 s, §8.3.1 step 3 — the drain cancels rather than waits, so the flush is its only clock) + 10 s for Tomcat's graceful phase (Spring bounds that phase itself at spring.lifecycle.timeout-per-shutdown-phase, default 30 s; a request still running after the 10 s is cut by the kill, which costs that request and none of the bookkeeping the drain already wrote). deploy/compose.yml has no preStop and uses the same 50 s (15 s left for Tomcat). ShutdownGraceArithmeticTest derives the sum from the code's own bounds and fails a smaller value. Anything shorter risks the kubelet SIGKILLing mid-flush: executions die without their finally blocks, so no execution_aborted event and no status update — rows left RUNNING until the stale-execution sweep marks them ABORTED (§6.2). The persistence drain (#266, datapipelines.persistence.shutdown-drain-ms, 10 s by default) follows Tomcat's phase and is in the sum since the 316 review; the scheduled jobs' scheduler (#316) stops after every drain and is NOT — a tick in flight is awaited up to spring.lifecycle.timeout-per-shutdown-phase (30 s) plus 2 s, and a kill during that wait costs the hung tick (redone next tick) and bean destruction, nothing the drains own.
• preStop: sleep 5 closes the standard k8s race: endpoint removal and SIGTERM are concurrent, so without it the pod can flip readiness microseconds before the Service stops routing to it. Five seconds of overlap is enough for Endpoints propagation — and it is what makes step 1's readiness flip actually take traffic off the pod before step 2 cancels.
• A PodDisruptionBudget (§6.4) keeps node drains from taking every replica's drain at once.

8.4 Rollback

Scheduler notifications (V51): older-image boundary. Before downgrading to an image older than V51, disable notifications on every instance and let the current sender settle the backlog: SELECT count(*) FROM schedule_notification_deliveries WHERE state = 'pending' must be zero. Disabling stops new enqueue; the current sender settles existing rows as failed/disabled, so this drain is not a promise to send them. Already recorded kinds remain: SELECT DISTINCT run_id FROM schedule_run_events WHERE kind LIKE 'notification\_%' identifies runs whose detail is unreadable by the older image until it is upgraded again. The reader stays strict (Enums additive-only wire convention); this feature promises no rolling reader compatibility, and an additive CHECK alone does not make an old artifact safe. Do not delete historical trail rows to hide the boundary.

• App rollback: redeploy previous image. Schema migrations are forward-only — if a migration was applied, you can't run the old app version against the new schema.
• For MINOR upgrades: schema changes are additive; rollback usually works.
• For MAJOR upgrades: take a DB backup before upgrade; restore it to roll back.

9. Security Hardening Checklist (Deployment)

• [ ] TLS termination at load balancer / proxy (let it handle cert renewal), with HSTS set THERE (one year, no preload) — the app sends none, by design (§6.2 "What the edge sets").
• [ ] The edge passes the app's security headers through untouched — the app's Content-Security-Policy (enforced; no 'unsafe-inline'), X-Frame-Options: SAMEORIGIN, Referrer-Policy, Permissions-Policy — and never weakens one (§6.2).
• [ ] Metadata DB password rotated and not in source control.
• [ ] DATAPIPELINES_JWT_SECRET is high-entropy (≥ 32 bytes random).
• [ ] DATAPIPELINES_DB_ENCRYPTION_KEY is high-entropy (32 bytes random) and stored in a secret manager, not a plaintext env file.
• [ ] Redis password set if Redis is networked (requirepass on the server, datapipelines.redis.password on every app instance — they must match). Under the hardened posture an empty password with a non-loopback Redis host is REFUSED at boot (Configuration §3.23, #189); development warns.
• [ ] No credential on a command line inside a container: the reference compose files give redis-cli its password through REDISCLI_AUTH in the redis service's environment, the Redis server its requirepass from that same variable through stdin (redis-server -, §4.2.1 — #196 took the last argv credential off the reference stack), and mysqladmin through MYSQL_PWD — an argv password is readable in ps by every process in the container. Check it: docker inspect <redis> --format '{{.Config.Cmd}}' and docker exec <redis> ps -o args name the variable, never its value.
• [ ] Redis maxmemory-policy noeviction (§4.2.1) — correctness, not tuning.
• [ ] OIDC client secrets from a secret manager, not a plaintext env file; at least one provider configured (§5.1).
• [ ] NetworkPolicy restricts app's egress.
• [ ] Service account / IAM role has least privilege.
• [ ] Container runs as non-root user (enforced in Dockerfile).
• [ ] Container filesystem read-only except for configured volume mounts.
• [ ] Resource limits set (CPU + memory) per deployment.
• [ ] Audit log retained per compliance policy.
• [ ] No /actuator/* path reachable on the application port; /actuator/prometheus on the management port (management.server.port), cluster-internal only (Observability §4.2). The management port is never published to a host or load balancer; MANAGEMENT_SERVER_ADDRESS defaults to loopback — setting it to 0.0.0.0 (required for k8s scraping) demands an accompanying NetworkPolicy that admits only the scraper to the management port. Because that policy selects the app pods, it isolates their app port too, so it must also carry the app-port rule you declare. The reference chart implements this check as the management block: management.expose is the opt-open switch (it pins 0.0.0.0 and renders the port and headless Service), and management.networkPolicy.enabled renders the policy. That policy carries the scraper rule on the management port plus the app-port rule from appIngress.from, or every source under the explicit appIngress.allowAll: true. The render refuses both scraper selectors empty, no app-port declaration, and a bare label map in place of a LabelSelector. Everything stays OFF by default (§6.4).
• [ ] Internet-exposed deployments may prefer to omit -Pdatapipelines.commit at build time — a public /info commit hash maps the instance to exact source revisions.
• [ ] The lib/ driver drop-in mount is read-only in the container, populated at image build or by a trusted init container, never writable by the app user; LOADER_PATH, if set, comes from the image — never inherited from the deployment environment (a writable lib/ is code-execution-by-file-drop).
• [ ] Production Redis requires requirepass (or ACLs) and TLS, or is confined to a private network segment with a NetworkPolicy — it holds fully materialized caller results for up to an hour (D9). The app logs a structured WARN at startup when the Redis password is empty and the host is not loopback under development, and refuses to start under hardened (Configuration §7).

10. Distribution License

• Code: AGPL-3.0 (see LICENSE; contributions under the CLA).
• Third-party runtime notices: :modules:app:generateThirdPartyNotices reads the resolved productionRuntimeClasspath POMs (walking parents for inherited licences), plus Spring Boot's packaged jar-mode tools and both Linux GraalJS isolate POMs. The isolate versions come from the Dockerfile's GRAALJS_VERSION ARG. The generated file lists artifact coordinates, POM licence names, SPDX identifiers and vendored licence texts; unknown, missing or stale licence mappings fail the build.
• Distribution locations: the boot jar contains META-INF/THIRD-PARTY-NOTICES; the image also contains /app/THIRD-PARTY-NOTICES, readable by the application user. LICENSE at the repository root covers the product's own code. The generated notices are build outputs, not a hand-maintained root NOTICE file.
• Optional drivers: -Pmysql and -Poracle include their drivers and corresponding notices only in those builds. MySQL declares GPL-2.0-only with the Universal FOSS exception; Oracle's FUTC is identified as LicenseRef-Oracle-FUTC (no SPDX-listed identifier). Operators remain responsible for their use and redistribution terms, including drivers added through lib/ after the build.
• Coverage boundary: these notices cover Maven runtime components and the isolate, not the base operating system, separately downloaded DuckDB extensions (#692), or vendored browser assets (#693). See Development §10.2 for the generation and update procedure.

11. What's Out of Scope for v1

• Managed / SaaS deployment: future commercial offering.
• Multi-tenant isolation: single-tenant v1.
• High-availability Postgres in our chart: operator provides managed PG.
• Backup automation: operator responsibility; we provide runbook.
• Air-gapped deployment: should work (no phoning-home telemetry) but not explicitly tested in v1.
• Federated deployments (independent installations sharing state or federating queries across each other): not supported. Multi-instance horizontal scaling (§6.2) is a different thing and is supported.
• Execution hand-off / resumption: an execution is pinned to the instance that started it and is aborted rather than migrated (§6.2, §8.3.1).

Appendix A: Reference compose.yml Sketch

This sketch boots — it satisfies every §5.1 startup requirement. Removing any of the marked items produces a container that exits during context startup, not one that runs degraded. deploy/compose.yml is the real file; every secret below comes from deploy/secrets.env under the same name the app binds, so the same file is sourceable by a bare java -jar, a systemd EnvironmentFile or a Kubernetes Secret (Environments §4).

services:
  datapipelines:
    image: ghcr.io/datapipelines/datapipelines:1.0.0
    ports:
      - "8080:8080"
    environment:
      SPRING_DATASOURCE_URL: jdbc:postgresql://postgres:5432/datapipelines
      SPRING_DATASOURCE_USERNAME: datapipelines
      SPRING_DATASOURCE_PASSWORD: ${SPRING_DATASOURCE_PASSWORD}

      DATAPIPELINES_REDIS_HOST: redis
      # REQUIRED whenever redis runs with --requirepass. Must be the SAME value
      # the redis service below is started with, or every Redis call fails at runtime.
      DATAPIPELINES_REDIS_PASSWORD: ${DATAPIPELINES_REDIS_PASSWORD}

      # REQUIRED, no fallback. Generate once: openssl rand -base64 32
      DATAPIPELINES_JWT_SECRET: ${DATAPIPELINES_JWT_SECRET}
      # REQUIRED, no fallback. Exactly 32 bytes base64: openssl rand -base64 32
      # Losing this makes every stored datasource credential unrecoverable.
      DATAPIPELINES_DB_ENCRYPTION_KEY: ${DATAPIPELINES_DB_ENCRYPTION_KEY}

      # REQUIRED: at least ONE authentication method (ConfigValidator §7) — an OIDC
      # provider configured below, or local accounts (auth.md §5A: set
      # DATAPIPELINES_AUTH_LOCAL_ENABLED=true plus a one-time seed for
      # DATAPIPELINES_AUTH_BOOTSTRAP_ADMIN_EMAIL, which names the first admin either way).
      # These names are referenced by the providers list in application.yml below;
      # Google is only an example — Microsoft/Okta/Keycloak/any OIDC IdP works.
      GOOGLE_CLIENT_ID: ${GOOGLE_CLIENT_ID}
      GOOGLE_CLIENT_SECRET: ${GOOGLE_CLIENT_SECRET}
    volumes:
      # The OIDC provider list is a nested structure — supply it as YAML.
      - ./application.yml:/etc/datapipelines/application.yml:ro
    depends_on:
      - postgres
      - redis
    restart: unless-stopped

  postgres:
    image: postgres:16-alpine
    environment:
      POSTGRES_DB: datapipelines
      POSTGRES_USER: datapipelines
      POSTGRES_PASSWORD: ${SPRING_DATASOURCE_PASSWORD}
    volumes:
      - postgres-data:/var/lib/postgresql/data
    restart: unless-stopped

  redis:
    image: redis:7-alpine
    # noeviction is REQUIRED (§4.2.1): eviction silently destroys results,
    # idempotency keys, and cancellation flags. Same password as the app above,
    # in the environment only: the server reads it from stdin, redis-cli from
    # REDISCLI_AUTH — never on argv (§4.2.1, #196). `$$` keeps the expansion in
    # the shell; `user: redis` because the wrapper is sh, not redis-server.
    user: redis
    environment:
      REDISCLI_AUTH: ${DATAPIPELINES_REDIS_PASSWORD}
    command:
      - sh
      - -c
      - |
        exec redis-server - --maxmemory 512mb --maxmemory-policy noeviction <<EOF
        requirepass $$REDISCLI_AUTH
        EOF
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 10s
      timeout: 5s
      retries: 5
    restart: unless-stopped

volumes:
  postgres-data:

The mounted application.yml needs only the provider list (everything else has a default or is set above):

datapipelines:
  auth:
    oidc:
      providers:
        - name: google
          client-id: ${GOOGLE_CLIENT_ID}
          client-secret: ${GOOGLE_CLIENT_SECRET}
          issuer-uri: https://accounts.google.com
          display-name: "Sign in with Google"

Format and further provider examples: Auth §5.1 and §11.1. Full key template: Configuration §5.

Appendix B: Demo Quickstart — the Published Sample Data

The sample data is published, versioned artifact sets (one per family), not something the app downloads: loading it is a deployment step (sample-data design D5). Any deployment that pulls the same version gets the same databases, byte for byte in content. The build scripts that produce them live at scripts/sample-data/ (nyc family), scripts/sample-data-trade/ (trade family) and scripts/sample-data-lake/ (dp-lake); everything below is the consuming side.

Three independent families, each behind its own compose profile and app.sh flag — an engineer spins up exactly the data they need:

Any combination is fine — the app's bootstrap keys are comma-separated lists built from the active families, and the MySQL service is shared.

The public marketing site's /demo-data page renders these published manifests as documentation — every table, row count, source and licence per family, generated from the same manifest files the loader verifies (vendored at modules/web/src/main/resources/site/demo/, pinned to the versions above). It is the page to hand anyone who asks what the demo contains.

The demo API: every seeded pipeline is a published endpoint (#224)

Seeding a demo family now also publishes each seeded pipeline as a GET endpoint under /demo/… and binds them all to ONE public api_caller key — demo-public-key, minted at boot from datapipelines.bootstrap.demo-api-key (default dpk_DEMOPUBLIC42.…, configuration.md §3.18). The key is public by design: the same value renders on the /demo-data page's "Call it as an API" section, and an outsider can call any demo endpoint with it and no account (rest-api.md §19.8 is the worked example, including the per-key request budget — 60 requests per 60 s per key, 429 rate_limit.exceeded — and the lake family's gate: the lake endpoint pays S3 egress per call, so it is published only while a budget stands behind it). Blank the setting in deploy/secrets.env and the next boot retracts the public API — no key, no demo endpoints, no page section; change the value and the next boot rotates the key. Nothing changes for keys real users mint on the /api-keys page.

The lake family is different in kind from the other two and the difference is worth stating before the commands: it has a compose profile and a --demo flag like its siblings, but no loader service, because nothing is loaded. What --profile demo-lake turns on is the datasource registration and the manifest import; the data itself stays in the bucket and is read at query time — see "dp-lake — the family with no loader" below.

One command

The base URLs and the pinned versions live in deploy/env/defaults.env, which is tracked — a data change is a new version directory and a commit, so what a deployment loads is visible in the repository. From a checkout that builds its own image, the whole thing is:

./app.sh --start --demo nyc,trade,lake

For the raw compose path, set the family ON markers your invocation wants and pass the same env files app.sh does — secrets last:

SAMPLE_NYC_ON=1 SAMPLE_TRADE_ON=1 SAMPLE_LAKE_ON=1 docker compose \
  -f deploy/compose.yml -f deploy/compose.local-build.yml \
  --env-file deploy/env/defaults.env \
  --env-file deploy/secrets.env \
  --profile demo-nyc --profile demo-trade --profile demo-lake up -d --wait

--wait returns when the app is healthy, and by then the artifacts have been downloaded, every checksum verified, the databases restored, the SELECT-only demo login created, and the families' datasources registered.

The nyc and trade families also build the jar with -Pmysql (see the driver note below); lake alone does not need it. ./app.sh --stop and --status take the same --demo list, so the demo services are not left running invisibly. ./app.sh --clean [--yes] is the clean slate: it stops and removes the project's containers, then deletes the METADATA volume (<project>-postgres-data — pipelines, templates, executions, users, API keys, workspaces, datasource rows) by explicit name, and keeps <project>-mysql-data (demo source data) and <project>-sample-data (the downloaded artifacts). It asks for the project name back on a terminal, or needs --yes without one; the next --start migrates a fresh database and re-seeds the demo. Sign-in users are re-created on their next login; API keys must be minted again. Demo is a flag, not an environment (Environments §5): DATAPIPELINES_DEMO is the source of truth, --demo merely sets it, and the hardened posture refuses it at boot. The per-family --demo-nyc / --demo-trade switches are gone; the families are nyc, trade and lake, and an unknown name is refused by name.

MySQL driver. MySQL Connector/J is GPL with a FOSS exception and is not in the default build (§3.5, Datasources §10.2). The sample-weather and sample-trade-world datasources are MYSQL, and bootstrap registration fail-fasts startup with datasource.driver_not_loaded without it. Build with the driver:

./gradlew -Pmysql :modules:app:bootJar && docker build -t datapipelines:local .

or drop the jar into lib/. ./app.sh --start --demo nyc[,trade[,lake]] does the -Pmysql build for you.

Point an agent at it — three steps

1. Log in with the account ./app.sh --start prints. The demo needs no OIDC client at all (Auth §5A): local password accounts are enabled and the first admin gets a GENERATED one-time password, written to deploy/secrets.env at first scaffold and forced to change at first sign-in. app.sh reads the seeded account back out of the database after the stack is healthy and prints the login that actually exists — the seed fires once, at row creation, so a DATAPIPELINES_AUTH_BOOTSTRAP_ADMIN_EMAIL changed later names an account that was never created (Environments §8). The first login lands you in the shipped demo workspace — workspaces are created by super admins, not provisioned per user (Auth §4.2 step 4, D-R11) — which is where the example pipelines are seeded.
2. Create your MCP key after signing in: open the Keys page (UI Screens §4.19) at /api-keys, choose New API key, select kind mcp, and choose a role the creator may give under the subset rule. The demo's bootstrap super admin may choose any offered MCP role; authors, promoters and workspace admins may also create MCP keys, but viewers cannot. Copy the plaintext from the one-time reveal. To rotate a key, revoke it on the Keys page and create another. REST clients can create one with POST /api/v1/auth/api-keys (Auth §7.4, REST API §16.1).
3. Give the agent the MCP endpoint http://localhost:8080/mcp and that key. It can list the seeded pipelines, read the three sample datasources' schemas, and execute nyc/mobility/revenue_by_borough or nyc/mobility/rainy_vs_dry_ridership immediately — see MCP Server.

What the demo profile turns on

The profiles add a mysql service and the families' one-shot loaders, and point the app at the files they place on a read-only volume. The scaffolded deploy/secrets.env enables local password accounts so the demo needs no OIDC client, and names the account the credential lands on; the password itself is a secret and is GENERATED into that same file — never a constant in a tracked file, on an app that binds every interface (Auth §5A.2). There is now exactly ONE place the credential is written and ONE place it is read back from, which is why app.sh can print a login that works. It also sets the §7 demo posture: member-datasources-enabled=false (an open datasource form on a public server is an SSRF and port-scan primitive — demo users get the seeded datasources only). Without --profile demo none of it exists: both datapipelines.bootstrap.* keys are paths and empty means off, so the non-demo stack is configured exactly as it was.

The demo datasources are protected in three independent layers: workspace-scoped access, the is_readonly flag on the datasource row (Datasources §5.7), and a database login granted SELECT and nothing else — created by the loader, never assumed. The SQLite entry additionally opens its file with the driver's read-only mode (Datasources §8A.4). On a public demo, add the §9 hardening item that belongs with this posture: egress-restrict the app container's network, so a datasource anyone can reach cannot become a path to anywhere else.

The loader

deploy/sample-data/load.sh <base-url> <version> is what the one-shot services run. Its contract, in order: download every artifact the manifest names, verify every checksum, and only then touch an engine — a corrupted download can never leave a half-loaded database behind. Each engine then gets a _sample_meta(version) marker, written last, so a re-run of a loaded deployment skips it and a failed engine is retried cleanly on the next start.

It runs as two services because no pinned image carries both a Postgres and a MySQL client, and installing one at container start would put an unpinned package fetch in the one place that has to be reproducible.

Demo Postgres sizing (T200, 109 §E)

The postgres service that the nyc family restores into carries the demo's biggest table — trips, ~804 MB, ~4.9M sampled rows plus rollups — against the Postgres image's compiled defaults (shared_buffers=128MB, random_page_cost=4.0 tuned for spinning disks). Under load, a simple year-2024 aggregate therefore ran in ~21 s: nearly every page came from the OS cache, but the planner kept choosing plans as if each random page cost a seek. The compose file sets:

command: postgres -c shared_buffers=512MB -c effective_cache_size=1536MB
                -c work_mem=32MB -c random_page_cost=1.1

with the container's memory reservation raised to 1 GB to match. The numbers are shaped by the data: shared_buffers ≈ half the restored set, effective_cache_size ≈ shared_buffers plus the OS page cache the box actually has, work_mem enough for the demo's hash joins and sorts, and random_page_cost=1.1 because the volume sits on SSD. This is demo sizing, not production advice — the operator's own datasource is their own DBA's, and the app never tunes a database it did not bring.

Measured on a quiet box (109 §E), year-2024 airport-count query (pu_location_id IN (1, 132, 138), three runs cold→warm): compiled defaults 42.7 s → 23.9 s → 8.6 s (buffers read=43,871 — nearly every page from disk); sized flags 20.5 s → 9.6 s → 8.1 s (buffers hit=46,795, read=0 — the working set fits the 512 MB). The win is the cold/cache-pressured run, ~2.1×; warm steady-state is unchanged, as expected when the OS cache already holds the table. A composite index (pu_location_id, pickup_date) on the artifact itself (the datasource is readonly to agents, so the artifact is the only place an index can come from) takes the same query to ~0.95 s cold / 0.09 s warm (index-only scan, 196 buffer hits) — see the 109 handback for the artifact decision.

dp-lake — the family with no loader

sample-data/lake/<version>/ is different in kind from the two above and the difference is the point: it is read in place. Nothing is downloaded at demo start and nothing is restored into an engine. The sample-data-lake one-shot service runs deploy/sample-data/place-lake.sh to place the repository's bootstrap and example files on the shared volume; it does not download the dataset. The query engine reads the Parquet objects over HTTPS and fetches only the ones a query's predicates name.

What is published (build runbook: scripts/sample-data-lake/README.md):

Egress is the operating cost to watch, and it is bounded by the query, not by the artifact. A demo session running the shipped pipeline at its default one-month window reads a few MB. The exposure is a visitor who writes SELECT * FROM hvfhv_trips with no date predicate — a ~7 GB full scan at S3 egress prices, per run. Set a bucket request-rate or budget alarm before the lake demo is announced.

Because there is no loader, the consuming side of this family is a datasource registration — shipped in round 089 as the lake demo family: the sample-lake LAKE datasource is registered create-if-absent from deploy/sample-data/bootstrap-datasources-lake.yml, which also imports the published manifest.json's tables[] into the dp-catalog registry (Datasources §8A.1, §8C). The showcase pipeline ships in its own examples file (scripts/sample-data/content/examples-lake.json) and the seeder loads it exactly when sample-lake is registered (the requires_datasources gate — environments.md §5). It needs egress to S3 (or a configured mirror) whenever a lake query runs.

Resetting an engine volume desyncs the demo login

The loader creates the dp_demo_ro login with the passwords from the env files, and bootstrap registration is create-if-absent — it never updates an existing datasource row (§8A). So if you delete an engine's data volume (docker volume rm dp-mysql-data) or rotate a SAMPLE_* password, the freshly created login no longer matches the credential the app stored at the original bootstrap, and that datasource fails validation in the UI while the engine itself is fine. The repair is to re-run registration for the stale entry only:

docker exec dp-postgres-1 sh -c \
  'psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -c "delete from datasources where name='"'"'sample-weather'"'"';"'
docker restart dp-datapipelines-1   # bootstrap re-registers with the current env password

(2026-08-30: hit live — a recreated MySQL volume left sample-weather failing UI validation until its row was re-registered this way.)

Re-publishing is a new version

Version directories are immutable (design §4). Any change to the data — a wider window, a different sample, a corrected lookup, even a typo — is published as v2 under a new prefix; v1 is never edited. Consumers pin the version (SAMPLE_VERSION), and nothing references a latest alias. This is what makes "the same version means the same data" a fact rather than a hope, and it is why the loader refuses a manifest whose version does not match the directory it came from.

Publish confirmation & release rehearsal — the drift guards

Two guards bracket every publish and every release rehearsal. Both exist because of T70 (2026-09-02): the published v1 examples.json still carried the ${} interpolations 042 had already migrated out of the repo copy, the demo 500ed on every fresh first login for two days, and nothing compared the published bytes with the repo's.

1. Before confirming an upload (and in every release rehearsal), run the published-drift guard for the version the demo pins:

   ./scripts/sample-data/check-published.sh v2                  # nyc (mobility), the default family
   ./scripts/sample-data/check-published.sh --family trade v2   # the trade family
   ./scripts/sample-data-lake/check-published.sh --family lake v1   # dp-lake (a different contract, below)
   

   It fetches the published manifest and examples.json, and fails unless the published copy, the published manifest's declared checksum, and the family's repo content/examples.json all agree. --family selects all three of the repo copy, the base-URL variable (SAMPLE_BASE_URL for nyc, SAMPLE_TRADE_BASE_URL for trade) and the default published prefix. Against an unpublished version it fails on the manifest fetch — which is the upload gate itself. Set the family's base-URL variable to check a mirror or a locally staged build (file://…/scripts/sample-data/work, or the local-serve recipe in app.sh). Network by nature, so it is a rehearsal step, never part of ./gradlew build. The lake family's guard is a separate script because it checks a different thing. That family has no examples.json — its content lives with the nyc family — so what can drift is not content but the objects themselves. It fetches the published manifest, refuses a manifest whose version disagrees with its directory, fails if any provenance row still carries license_verified: null, samples one object per table and compares SHA-256, and fetches the Iceberg metadata file to confirm every location it records sits inside the published prefix — an Iceberg table that still names a build machine's directory is unreadable, and nothing about an object listing shows it. Folding two unrelated contracts behind one --family flag would make "check-published passed" mean two different things.

2. The repo copy is validated in build — the templates module's SampleDataExamplesContentTest runs every shipped template and pipeline through the app's own save-time validators (049 C1), so content the seeder would refuse cannot merge. verify.sh step 5 remains structural only; the two checks are different jobs and each says which it does.

Licence gate — before serving this data publicly

Every provenance row in manifest.json ships license_verified: null. The build verifies no licence and claims none; the licence strings are research claims carried with their evidence links in each family's sources.lock (scripts/sample-data/, scripts/sample-data-trade/, scripts/sample-data-lake/).

Publishing with any license_verified still null blocks go-live (design §8). Verify each source's current terms, record the date, and swap — do not ship — any dataset that fails. This applies to datapipelines.co; a self-hosted evaluation loading the artifacts for its own use is a different question, and one for the operator.

Appendix C: Change Log

| 2026-09-22 | v1.28 | key wording | Appendix B step 2 says what the product does since R3: the MCP key is created at sign-in and copied from the top bar; nothing is "minted" by hand and there is no create endpoint. app.sh's demo message says the same. | | 2026-09-22 | v1.27 | #207 relative redirects | §8 edge contract: every app Location is RELATIVE (server.tomcat.use-relative-redirects=true), so the public scheme never leaks into a redirect and the edge needs no X-Forwarded-Proto handling for them. Behind a TLS-terminating edge the absolute http:// Locations Tomcat built by default were refused by the browser (CSP form-action on plain forms, mixed content on htmx requests) after the server had already acted — logout, workspace switch and member removal on datapipelines.co, 2026-09-22. | | 2026-09-21 | v1.26 | 199 (#196) the Redis password leaves argv | §4.2.1 gains "Where the password lives": the reference redis services (deploy/compose.yml, deploy/compose.laptop-infra.yml, Appendix A) start redis-server - — configuration from stdin, requirepass fed by a heredoc from REDISCLI_AUTH, exec so the server stays PID 1, user: redis because the wrapper is sh — so no argv, and no file, carries the value; --maxmemory/noeviction stay on argv. §9's argv line closes the "tracked separately" remainder and names the two commands that check it. Operators: the next start recreates the Redis container (§7 says what a restart loses). Numbered on base db12e043; the merger renumbers if another lane took v1.26. | | 2026-09-21 | v1.25 | 188 (#188) the app states its headers | §6.2 gains "What the edge sets, and what it must not touch": the header table the app sends on every response (an enforced CSP with no 'unsafe-inline', X-Frame-Options: SAMEORIGIN, Referrer-Policy, Permissions-Policy; the editor route's 'unsafe-eval' exception, #195), why the app sends no HSTS, and the four things the product expects of an edge (HSTS one year without preload, headers passed through unweakened, TLS + redirect, the forwarded address). §9: two checklist lines for the same. | | 2026-09-21 | v1.24 | 188 (#189) Redis credentials | §9: the hardened posture refuses a passwordless non-loopback Redis; the reference compose files pass the healthcheck password as REDISCLI_AUTH (environment), never redis-cli -a (argv); the server's --requirepass argv named as the remaining exposure. | | 2026-09-19 | v1.23 | 173 (#173) the agent surface before indexing | §6.7: /llms.txt, /llms-full.txt and /docs/{slug}.md listed beside the crawler surfaces; the Search Console note says llms.txt needs no submission; the sitemap bullet corrected — lastmod has been absent since 145, not read from build-info.properties. | | 2026-09-17 | v1.22 | Writable temporary storage (#133) | §6.4: disk-backed /tmp emptyDir with optional size limit while preserving the read-only root filesystem; LAKE disk sizing and cleanup requirements. | | 2026-09-14 | v1.21 | 137 mail notices | §5.1: the first-admin story no longer says "no SMTP" — with DATAPIPELINES_MAIL_HOST + _FROM set the one-time credential is emailed to the user and sys-ops (DATAPIPELINES_MAIL_OPS_TO) is told about every new user (Auth §5A.8); the variables are catalogued in Configuration §3.27 and named in deploy/secrets.env.example, per this doc's no-restated-keys rule. | | 2026-09-10 | v1.20 | 109 §E mobility v7 | The nyc family pins SAMPLE_VERSION=v7 — a new immutable version directory carrying ONE change over v6: the trips table gains the composite index (pu_location_id, pickup_date) (idx_trips_pu_location_pickup_date, DDL beside the two singles), measured ~90× on the year-2024 airport shape (8.1 s sized seq scan → 0.09 s index-only). The other three artifacts are byte-identical v6 copies and every table content-checksum is unchanged — the v2 precedent's restore-and-redump shape. The v7 set (pg-trips.dump 117,733,060 bytes, sha256 02627d0846…, plus manifest) is handed to the owner in handbacks/109-artifacts/mobility-v7/ for publishing; a fresh --demo nyc start needs v7 published to load. | | 2026-09-09 | v1.19 | 109 §E demo Postgres sizing | New "Demo Postgres sizing" subsection (Appendix B): the nyc family's trips table (~804 MB) meets sized-for-the-data flags — shared_buffers=512MB, effective_cache_size=1536MB, work_mem=32MB, random_page_cost=1.1 (SSD) — with the container memory reservation raised to 1 GB and the "demo sizing, not production advice — the operator's own datasource is their own DBA's" boundary stated. The compose change itself is cfb65c9c; this section is the rationale the compose comment points at. | | 2026-09-08 | v1.18 | 093 announcement-day truth pass | Appendix B's demo-families preamble corrected against the merged 089 product. It said dp-lake was published but not yet consumable with "no compose profile, no loader and no --demo flag" while app.sh already accepts --demo lake and adds --profile demo-lake (app.sh lines 144/209, SAMPLE_LAKE_ON) — the paragraph was written at 088 and 089 shipped the consuming side. The families table gains a lake row; "Two independent families" becomes three; the nyc row's "2 example pipelines" becomes 6 (scripts/sample-data/content/examples.json holds six); the one-command and raw-compose recipes gain the third family; the -Pmysql sentence now says which families need it. The "family with no loader" section below is unchanged — it was already correct — and the distinction it draws (a profile and a flag, but nothing to load) is now stated in the preamble too, which is where the contradiction was read. | | 2026-09-07 | v1.16 | 089 dp-lake §D extension bundling | §3.1 gains the bundled DuckDB extensions bullet: the image now downloads httpfs/aws/iceberg/avro for DuckDB core v1.5.5 at build (+108 MB uncompressed, ~39 MB downloaded), gunzips them into /opt/duckdb/extensions/v1.5.5/<linux_amd64|linux_arm64>/ (the build's TARGETARCH), and exports DATAPIPELINES_DUCKDB_EXTENSION_DIRECTORY=/opt/duckdb/extensions — a LAKE datasource then LOADs from disk and never INSTALLs, so a hardened dp-lake deployment needs no egress to extensions.duckdb.org (the 089 §7.3 spike's CAN verdict, verified on this exact base image with --network none). deploy/compose.yml defaults the variable to the same path (declared in compose-env-audit.sh's new IMAGE_DEFAULT_VARS, the first compose default that mirrors the image rather than application.yml). The operator key itself is Configuration §3.25. | | 2026-09-07 | v1.15 | 088 dp-lake data | Appendix B gains a third sample-data family, dp-lake — NYC TLC High Volume FHV (Uber/Lyft/Via/Juno) trips published at s3://datapipelines-co/sample-data/lake/<version>/ as Parquet partitioned by pickup_date, a 1-in-16 sample, a zone/day pre-aggregate and an Apache Iceberg copy of the sample. New section "dp-lake — the family with no loader": it is read in place, so there is no compose profile, no loader service and no --demo flag — the datasource that reads it (dialect lake, sample-lake) is round 089, and the showcase pipeline ships in its own scripts/sample-data/content/examples-lake.json that no deployment loads yet. The drift-guards section gains scripts/sample-data-lake/check-published.sh --family lake <version> and says why it is a separate script (no examples.json; it checks object hashes, the licence gate and the Iceberg metadata's recorded locations). Egress is stated as the operating cost: bounded by a query's date predicate, unbounded for an unfiltered scan of the ~7 GB table. No application code, no deployment change. | | 2026-09-07 | v1.15 | dp-lake v1 published (088) | s3://datapipelines-co/sample-data/lake/v1/ — 855 objects, 7.57 GB: hvfhv_trips (471,851,707 NYC rideshare trips, day-partitioned Parquet), hvfhv_trips_sample (1-in-16), hvfhv_zone_day, hvfhs_companies, hvfhv_trips_iceberg (the sample as an Iceberg table). Read in place by the dp-lake datasource (round 089) — no loader, no download at demo start; licence verified 2026-09-07 (NYC Open Data, FHV disclaimer quoted in the manifest). check-published.sh v1 in the lake family is the proof. | | 2026-09-07 | v1.14 | mobility v6 (082) | The demo briefing gains a CALCULATOR node (fiscal_quarter → run_fiscal_quarter, bound as the first caller column) so the feature is visible out of the box; data files unchanged; baselines re-keyed. Pin → SAMPLE_VERSION=v6 in deploy/env/defaults.env; check-published.sh v6 byte-identical. | | 2026-09-06 | v1.13 | mobility v5 + trade v4 (077 mandatory folders) | Both artifact sets republish with every template id under a folder (nyc/mobility/…, nyc/reference/…, nyc/weather/…, trade/…) — 13 of 25 demo templates were flat and are refused by 077's rule at seed time. Data files unchanged in content. Pins → SAMPLE_VERSION=v5, SAMPLE_TRADE_VERSION=v4; check-published.sh v5 / --family trade v4 byte-identical on publish. | | 2026-09-06 | v1.13 | 081 one env file | The deploy/ layout is two env files: tracked deploy/env/defaults.env (every non-secret variable with the value this deployment ships) then git-ignored deploy/secrets.env, in that order under every loader, with deploy/secrets.env.example as the template that names every variable. The five files 075 put under deploy/env/ (laptop.env, demo.env, example.env, posture/development.env, posture/hardened.env) are deleted; §6.2, §6.3A and Appendix B's raw-compose recipe use the two-file list. deploy/compose.laptop-infra.yml starts its Postgres and Redis from SPRING_DATASOURCE_PASSWORD / DATAPIPELINES_REDIS_PASSWORD, removing the laptop's load-order inversion. ./app.sh --scaffold writes deploy/secrets.env from the template and stops, so the admin address can be set before the one-time seed; --start waits until the app is actually healthy (up to HEALTH_WAIT_SECONDS, default 360) instead of reporting the HEALTHCHECK deadline as a failure, and --status prints the seeded login too. | | 2026-09-05 | v1.12 | 075 environments and posture | Environments joins the spec set — the operator page for DATAPIPELINES_ENV (the org's label) and DATAPIPELINES_POSTURE (development | hardened), with the normative posture table, the environment-variable contract, and loader recipes for Compose, a bare JAR, systemd, Kubernetes, ECS and Nomad. The deploy/ layout is renamed with no shims: docker-compose.yml → compose.yml, docker-compose.local.yml → compose.local-build.yml, docker-compose.dev.yml → compose.laptop-infra.yml; deploy/application.yml is folded away (every line was already in the image's own application.yml behind the same placeholders) and orgs wanting a yml use SPRING_CONFIG_ADDITIONAL_LOCATION; deploy/.env/.env.demo/.env.example and the root .env.example are replaced by tracked deploy/env/** (posture, demo, laptop, and example.env — every variable with its shipped default) plus git-ignored deploy/secrets.env. Secrets carry the app's own variable names (DATAPIPELINES_JWT_SECRET, SPRING_DATASOURCE_PASSWORD, …) instead of compose-only renames, so one file feeds every loader. §6.3A's promotion example is now env+posture. Demo is a flag: DATAPIPELINES_DEMO=nyc,trade (./app.sh --demo nyc,trade) replaces --demo-nyc/--demo-trade, with the versions in tracked deploy/env/demo.env, and hardened refuses it. scripts/compose-env-audit.sh now runs on every ./gradlew build (it had drifted: 074's three endpoints keys shipped with no compose pass-through) and also checks deploy/env/example.env. Compose volume names derive from COMPOSE_PROJECT_NAME, so -p <lane> alone scopes a second copy's data. | | 2026-09-05 | v1.11 | mobility v4 + trade v3 (067 pipeline folders) | Both artifact sets republish with examples.json carrying the folder convention (nyc/…, trade/…); data files unchanged in content (every pinned table checksum re-derived identical; the DuckDB file's bytes differ as DuckDB files are not byte-deterministic). Pins move to SAMPLE_VERSION=v4, SAMPLE_TRADE_VERSION=v3. Until an operator moves the pin, a fresh demo seeds the old flat names — expected, version pins are operator config. | | 2026-09-04 | v1.10 | trade/v2 — Binance out, Federal Reserve H.10 in | The trade family republishes as trade/v2. The Binance market slice is removed entirely (its Vision terms are CC BY-NC-SA with an explicit no-hosting-of-derivative-feeds clause and a separate enterprise licence for commercial use — owner ruling 2026-09-04); the SQLite artifact is now fx_rates.db, built from the Federal Reserve H.10 daily noon buying rates and their G.5 monthly averages for the five reconciled partners' currencies (a US Government work — no copyright), and the datasource is renamed sample-market → sample-fx. A third example pipeline, imports_in_partner_currency, restates US import value in the partner's own money across three engines (DuckDB facts → SQLite rates → H2 join). Two licence conditions that were never in the tree now ship with the data: the verbatim Census notice ("This product uses the Census Bureau Data API but is not endorsed or certified by the Census Bureau.") in the family README, the sample-trade-us datasource description and the manifest's Census provenance entry, and the UN Comtrade citation with the under-100,000-record note. check-published.sh learns --family trade; SAMPLE_TRADE_VERSION defaults to v2 in app.sh, deploy/.env.example and the compose services. An existing deploy/.env.demo pinning SAMPLE_TRADE_VERSION=v1 is not rewritten by app.sh (version pins are operator config) — edit it, or the loader fetches a version whose market object no longer exists. | | 2026-09-04 | v1.10 | mobility v3 (070 showcase pipelines) | Mobility artifact v3 published — v2's data files byte-identical, examples.json now 17 templates / 6 pipelines (the 070 showcase set, baselined by check-baselines.sh); SAMPLE_VERSION default → v3. Operators with an existing .env.demo set SAMPLE_VERSION=v3 by hand — app.sh never overwrites present keys. | | 2026-09-04 | v1.9 | two-family demo split | The demo splits into two independent sample-data families — demo-nyc (mobility) and demo-trade (trade/v1), each with its own compose profile, loader services, bootstrap datasources file and examples file; the app's bootstrap keys are now comma-separated lists and the seeder runs all templates before any pipelines. app.sh gains --demo-nyc/--demo-trade (the old --demo dies with a hint); the ON markers (SAMPLE_NYC_ON/SAMPLE_TRADE_ON) compose the active-family lists. The trade family adds the demo's fourth engine (DUCKDB, sample-trade-us — driver ships in core, read-only via the probed properties.jdbc.access_mode: READ_ONLY, datasources.md §8A.5). Appendix B rewritten for the two flags; the compose project is renamed deploy→dp (containers/volumes carry the dp- prefix; the metadata-DB volume was migrated, dp-postgres-data). | | 2026-09-02 | v1.7 | 051 auth/config sweep | §6.2 gains datapipelines.auth.trusted-proxies (R8/T46): behind the LB the login limiter and every audit source_ip must resolve the client through the trusted-proxy list — empty default keeps bare deployments on remoteAddr, header ignored. §6.7's homepage note updated (T46 closed; no-limiter decision stands on its own grounds). §6.3 documents the compose env contract with its scripts/compose-env-audit.sh guard (T32). Appendix B tells the truth about the demo password (T47/T73): on a clean checkout the scaffolded GENERATED password wins over the demo-admin seed — grep DATAPIPELINES_AUTH_LOCAL_BOOTSTRAP_PASSWORD deploy/.env. app.sh's up --wait timeout now reports "still starting" and exits 0 when the app container is running (T75; a cold JVM can outrun the HEALTHCHECK window — 243s in the 2026-09-02 rehearsal) | | 2026-09-02 | v1.9 | multi-instance round 2 (050) | §6.2 checklist gains the per-instance-limits row (R2/M4/M7): max-concurrent-executions-per-instance is per instance and N replicas admit N × it — the multiplication stated once, plainly; the Redis row names the datasource pool-invalidation channel. §6.6 heap paragraph rewritten around the per-instance multiplier (the old text read the key as a "cluster-wide ceiling" — false at N > 1, and the exact trap the rename closed); §5.2 deploy-time keys updated to the renamed key. | | 2026-09-02 | v1.8 | demo artifact v2 + guards (049) | Appendix B gains "Publish confirmation & release rehearsal — the drift guards": scripts/sample-data/check-published.sh (published examples.json vs repo vs manifest, 049 C2) is now a named pre-upload / rehearsal step, and the repo copy's semantic validation is pinned in build (SampleDataExamplesContentTest, 049 C1) — the two guards T70 (published-v1 drift → first-login 500 ×2 days) proved missing. A v2 artifact is staged from unchanged pins with the licence gate re-stamped 2026-09-02; this row's commit is the held version bump — it moved the demo pin (SAMPLE_VERSION=v2) and every current-version citation, and merges only after the owner's upload is confirmed live (check-published.sh v2 against the bucket). | | 2026-09-04 | v1.8 | SEO (073) | §6.7 rewritten for the widened public surface: /docs and /docs/{slug} are public (the viewer renders packaged Markdown and reaches no principal, workspace or datastore; the same content is public on GitHub), the intent-cluster pages join /, and /robots.txt + a generated /sitemap.xml ship. The allowlist bullet records exactly what was added. New operator note: Search Console and the sitemap — verify the domain by DNS TXT before the launch announcement, submit the sitemap once (it regenerates per request, so later releases need no resubmission), and why a self-hosted deployment emits datapipelines.co canonicals unless SITE_ORIGIN is changed. | | 2026-09-01 | v1.7 | multi-instance readiness (036) | The stale-execution sweep now ships (ARCH-AUDIT M2), and the §6.2 crash bullet is true again. Every replica runs the idempotent sweep on a one-minute cadence (the project's first @Scheduled); RUNNING rows older than datapipelines.executions.stale-timeout-minutes are marked ABORTED with pipeline.execution.instance_lost. §8.3.2's SIGKILL-mid-flush consequence restored to match. Side effect: DELETE /executions/{id} on a crashed instance's stale row stops being a silent no-op — once swept, the row is terminal and the cancel is refused pipeline.execution.not_running instead of returning 204. | | 2026-09-01 | v1.6 | multi-instance readiness (036) | The drain now ships (ARCH-AUDIT M1), and §8.3.1/§8.3.2 describe it. On SIGTERM: readiness flips to REFUSING_TRAFFIC first, every local execution is cancelled through the ordinary path (Statement.cancel() first, execution_aborted with reason: "shutdown", row ABORTED), the flush is awaited with a 20s bound, and server.shutdown: graceful lets in-flight requests end. The drain CANCELS rather than draining-to-completion — the v1.2 text's "drain up to execution-timeout-seconds" behavior is deliberately not what shipped; terminationGracePeriodSeconds: 30 (not 630) and preStop: sleep 5 are the matching pod settings, and the Helm chart carries both. The §6.2 crash bullet is still honest (the sweep lands next). | | 2026-09-01 | v1.5 | multi-instance readiness (036) | Honesty fix (ARCH-AUDIT M11): the doc promised three things the code does not do. §8.3.1/§8.3.2 rewritten as shutdown behavior as shipped — no drain, no server.shutdown: graceful, no readiness flip, no sweep; the old drain-to-timeout/cancelAll(shutdown) sequence and the terminationGracePeriodSeconds: 630 + preStop guidance described unimplemented code. §6.2 instance-crash bullet no longer claims rows are "swept to ABORTED" (the sweep exists but has no caller). §5.2's "shutdown grace" gloss on execution-timeout-seconds removed. §6.4's deploy/helm/ reference made real: a minimal chart (Deployment, Service, optional HPA/PDB) now ships. The drain and sweep claims return with the code that implements them. | | 2026-08-31 | v1.4 | website + docs in-app (033) | New §6.7: the app serves the marketing site (/, public) and the packaged spec set (/docs, session-only); the dashboard moved to /dashboard; the standalone website/ static deploy retires to the websiteExport cold-fallback procedure; public surface defended by cache headers, NOT the login rate limiter (OPEN-ITEMS T46). Header version corrected (v1.3's entry had not bumped it). | | 2026-08-05 | v1.0 draft | initial draft | Initial deployment spec sketch — Docker image, infra requirements, configuration, deployment patterns, upgrade/rollback, security checklist | | 2026-08-05 | v1.1 | horizontal scaling | Added multi-instance horizontal scaling section. Application is stateless for all CRUD/UI/MCP/auth. In-flight executions are instance-local (acceptable for short-running pipelines). No sticky sessions required. Added multi-instance checklist. Added LB idle-timeout + SSE heartbeat note. | | 2026-08-07 | v1.2 | consistency campaign | Applied SPEC-REVIEW-2026-08 §2.15: §5 env-var tables replaced by the startup-requirements list + pointer to configuration.md; the inline-vs-claim-check threshold key (superseded by the D9 result keys) and every other key configuration.md does not define were deleted [D8]; §4.2 rewritten as the result store with required maxmemory-policy noeviction and a sizing model [D9]; §8.3.1/§8.3.2 graceful-shutdown mechanism (readiness fail → drain to execution-timeout-seconds → cancelAll(shutdown) → exit) with k8s preStop + terminationGracePeriodSeconds, accepted loss stated [D7]; §6.2 instance-local story updated to cancel-on-disconnect + cross-instance cancel via Redis flag [D7]; Appendix A compose made bootable (OIDC provider env vars, Redis password wired to requirepass, noeviction, mounted provider YAML); new §3.5 JDBC driver matrix (bundled vs -Poracle/-Pmysql vs lib/ drop-in); new §6.6 resource sizing (heap, container limit, -XX:MaxRAMPercentage); -Duser.timezone=UTC made normative in the image and bare-JVM entrypoints (Type System §8.4); §6.2 diagram residue and §11 malformed bullet fixed | | 2026-08-29 | v1.3 | local password auth | §5.1 item 5 becomes "at least one authentication method": OIDC provider OR local accounts (auth.md §5A), with the operator's first-admin story for the no-IdP case (hash-seeded one-time credential, forced first-login change, admin resets — no SMTP, no self-registration). Appendix A compose comment and Appendix B quickstart updated: the demo now logs in with a local account (demo-admin@demo.local / demo-admin, one-time) and needs no OIDC client. | | 2026-09-07 | v1.8 | 087 connector seams | §3.5 driver matrix gains LAKE — bundled, and the same duckdb_jdbc jar as DUCKDB. It is a distinct dialect rather than a mode because the two need opposite §5.6 postures (Datasources §4.1); nothing about packaging changes, since there is no second driver to ship. | | 2026-09-08 | v1.17 | 089 dp-lake consuming side shipped | Appendix B's dp-lake section: the "that is round 089 / until it lands, the published objects are inert" paragraph rewritten as shipped — the lake demo family registers sample-lake from deploy/sample-data/bootstrap-datasources-lake.yml (which imports the published manifest's tables[] into the dp-catalog registry), and the seeder loads examples-lake.json exactly when sample-lake is present (the requires_datasources gate, environments.md §5). |