Deployment & Packaging Specification

Status: v1.22 Owner: datapipelines.co core Depends on: all other specs Last updated: 2026-09-17

1. Purpose

This spec defines how datapipelines.co is packaged, distributed, and deployed in self-hosted environments. The product is open-source and self-hosted; this document covers the artifacts we produce and the deployment patterns we support.

This is lighter than the core specs — most operational concerns are the operator's responsibility (we ship the artifact, they run it). We document our artifacts, required infrastructure, configuration surface, and recommended deployment patterns.

2. Design Principles

1. One artifact, many deployments. A single Docker image runs in dev, staging, prod. Configuration via env vars + mounted files; no build-time decisions.
2. Stateless app, externalized state. The app has no local persistent state. Metadata DB (Postgres) and Redis (result store, idempotency keys, cancellation flags, post-completion event log) are external. H2 staging is per-execution in-memory.
3. Container-first. Primary distribution is a Docker image. JVM-only deployments (no container) are supported but secondary.
4. k8s-native but not k8s-required. Kubernetes is the recommended target. Docker Compose works for small / single-node deployments. Bare JVM works for development.
5. Configurable without forking. Every deployment-relevant knob is an env var or config-file value. Operators do not edit source code to deploy.
6. Open-source distribution. The image and JARs are published to public registries / Maven Central. No proprietary distribution channel.

3. Build Artifacts

3.1 Docker image

Published to ghcr.io/datapipelines/datapipelines:{version} and docker.io/datapipelines/datapipelines:{version} (mirror).

• Base image: eclipse-temurin:21-jre-jammy (LTS JDK 21, Ubuntu Jammy).
• Layers: multi-stage build (gradle:8.7-jdk21 builder → eclipse-temurin:21-jre-jammy runtime).
• Bundled DuckDB extensions (dp-lake): httpfs, aws, iceberg and avro for DuckDB core v1.5.5, pre-populated under /opt/duckdb/extensions/v1.5.5/<platform>/ (<platform> = linux_amd64 or linux_arm64, following the build's TARGETARCH) and activated by the image's ENV DATAPIPELINES_DUCKDB_EXTENSION_DIRECTORY — a LAKE datasource LOADs them from disk and never INSTALLs at runtime, so no egress to extensions.duckdb.org is needed (Configuration §3.25). Adds ~108 MB uncompressed (~39 MB downloaded at build).
• Size target: < 250 MB compressed.
• User: non-root (datapipelines user, UID 1000).
• Entrypoint: java $JAVA_OPTS -Duser.timezone=UTC -jar /app/app.jar.

-Duser.timezone=UTC is normative, not a suggestion. The type system assumes a UTC JVM: ingest normalizes timezone-aware source values to UTC and treats naive source timestamps as already-UTC (Type System §8.4), and every internal TIMESTAMPTZ is stored and rendered in UTC. A container whose JVM default zone is anything else silently shifts rendered timestamps. The flag is baked into the image entrypoint; bare-JVM and JAR deployments (§3.2, §6.5) MUST pass it themselves.

Heap and container-memory sizing: §6.6.

3.2 JAR distribution

• Published to Maven Central: co.datapipelines:datapipelines-app:{version}.
• Executable Spring Boot fat JAR (./gradlew bootJar).
• Runnable directly: java -jar datapipelines-app.jar.

3.3 Module artifacts (for embedding)

• Each module published to Maven Central: co.datapipelines:typesystem, co.datapipelines:dag, etc.
• Allows third parties to embed parts of datapipelines.co in their own products (e.g., use our type system, or our MCP server in a different application).
• Marked "experimental API" until v1.1 stabilizes module boundaries.

3.4 Source distribution

• GitHub release per version, with signed tag.
• Source tarball attached to the release.
• Reproducible builds (Gradle lockfile committed, exact JDK pinned via toolchain).

3.5 JDBC driver matrix (what ships in the image)

Which drivers are present is a packaging property of the artifact, not a configuration one. The licensing rationale and the driver-class lookup live in Datasources §10; this table is the operator's view of the published image.

The lib/ drop-in. Spring Boot's loader adds lib/ (relative to the JAR's working directory) to the application classpath. For the container, mount the JAR at /app/lib/; for bare JVM, place it beside datapipelines-app.jar. This is the no-rebuild path — the operator accepts the driver's license by supplying it.

Registering a datasource whose dialect has no driver on the classpath fails at save time with datasource.driver_not_loaded (Datasources §9) — a deployment/packaging error, not a bad payload. The same payload succeeds after the driver is supplied.

4. Required Infrastructure

4.1 Postgres (metadata DB)

• Version: 14+ recommended.
• Purpose: persistent storage for pipelines, templates, datasources, executions, audit log.
• Provisioning: operator-managed. We support connecting to an existing Postgres instance.
• Database size: small (a typical deployment has < 1 GB metadata).

Connection keys (URL, credentials, Hikari pool size) are defined in Configuration §2 / §3.13.

4.2 Redis (result store and coordination)

• Version: 6+ recommended.
• Provisioning: operator-managed.
• Purpose — four distinct workloads share one store:
  1. Caller results. Every completed execution's caller result is materialized to Redis and read through the cursor (REST API §7). There is no inline-vs-large split.
  2. Idempotency keys (Idempotency-Key records, retained per datapipelines.idempotency.ttl-seconds).
  3. Cancellation flags (dp:cancel:{execution_id}) — how DELETE /executions/{id} reaches the executing instance (DAG Executor §8.3.1).
  4. Post-completion event log, 1 hour, backing SSE replay (REST API §10.3).

Connection keys are defined in Configuration §2 / §3.1; TTL and size limits in Configuration §3.5.

4.2.1 Required Redis configuration

maxmemory-policy noeviction. This is not tuning advice — it is a correctness requirement. Any LRU/LFU eviction policy lets Redis silently discard keys under memory pressure, and every key class above is load-bearing:

• an evicted result turns a completed execution into a spurious "expired" 404 before its TTL;
• an evicted idempotency key lets a client retry execute the pipeline a second time;
• an evicted cancellation flag makes DELETE /executions/{id} a no-op, leaving the execution running;
• an evicted event-log entry silently truncates SSE replay.

Every key the app writes carries an explicit TTL, so noeviction does not leak: the store drains on its own schedule. Under genuine memory exhaustion the app fails loudly (result.storage_unavailable on the write path) rather than corrupting semantics quietly.

4.2.2 Sizing

Budget for the peak sum of all four workloads:

peak ≈ (datapipelines.result.max-size-bytes × concurrent recent executions)   ← dominant term
     + idempotency keys (small, × datapipelines.idempotency.ttl-seconds window)
     + 1h of post-completion event logs (small, proportional to nodes × executions)
     + cancellation flags (negligible)

The result term dominates and is the only one worth arithmetic. Results live for their TTL (datapipelines.result.ttl-default-seconds, clamped between ttl-min/ttl-max — Configuration §3.5), so "concurrent recent executions" means executions completed within one TTL window, not executions running right now. Worst case with defaults (100 MB cap, 300 s TTL) is dominated by how many callers actually pull 100 MB results; the practical lever is lowering result.max-size-bytes and pointing bulk workloads at output.target: datasource instead — the explicit NOT-goal of result delivery is bulk data transfer.

512 MB is a reasonable starting point for small deployments; raise it before raising result.max-size-bytes.

4.3 Network egress

The app must reach:

• Configured datasources (PG, Oracle, MSSQL, MySQL, DuckDB files, SQLite files).
• The metadata Postgres.
• Redis.
• (Optional) OTLP collector, Sentry, KMS — if those features are configured.

The app does not require inbound network access beyond the HTTP/MCP ports it serves.

5. Configuration

configuration.md is the single authority for every configuration key — YAML path, env var name, default, and description. This spec deliberately does not restate key names or defaults: a duplicated default is a default that drifts. What follows is only the operator's startup checklist and the file-mounting mechanics.

Env var names are derivable, never memorized: datapipelines. → DATAPIPELINES_, remaining YAML path upper-snake-cased (Configuration §1).

Before any of this, two variables: DATAPIPELINES_ENV is your organisation's name for this deployment and DATAPIPELINES_POSTURE is development or hardened. A named environment with no posture does not start. Environments is the page for that decision; Configuration §3.23 is the key reference.

5.1 What the app requires to start

The app fail-fasts on startup if any of the following is missing. Full definitions: Configuration §2.

1. Metadata Postgres: spring.datasource.url, spring.datasource.username, spring.datasource.password.

2. Redis: datapipelines.redis.host.

3. datapipelines.jwt.secret — internal JWT signing secret.

4. datapipelines.db.encryption-key — AES-256 master key for datasource credentials. There is no fallback source: no KMS lookup, no auto-generated key file. Lose it and every stored datasource credential is unrecoverable, so it belongs in a secret manager and in the operator's backup plan. (KMS sourcing is a ROADMAP §2 item.)

5. At least one authentication method: a fully-configured OIDC provider under datapipelines.auth.oidc.providers (each with client-id, client-secret, and issuer-uri), or local password accounts (datapipelines.auth.local.enabled=true, Auth §5A). The first admin comes from datapipelines.auth.bootstrap-admin-email either way (Auth §4.4). A provider entry whose client-id is empty is ignored with a WARN, not counted. The client-id/secret env var names are chosen by the deployment (GOOGLE_CLIENT_ID, OKTA_CLIENT_ID, …) — they are the one deliberate exception to the derivation rule above.

   The operator's first-admin story with local accounts (no IdP): set datapipelines.auth.bootstrap-admin-email to the admin address, enable datapipelines.auth.local.enabled, and seed the one-time credential — preferably as a pre-computed hash from ./gradlew :modules:auth:hashPassword into datapipelines.auth.local.bootstrap-password-hash (the plaintext form exists for demos). Hand the password to the admin out-of-band; the app forces a change at first login (Auth §5A.2), and a deployment still running the seeded credential announces itself with a startup WARN. Every account after the first is created by an admin on the user-administration screen — there is no self-registration and no self-service email reset: a forgotten password means an admin resets it there, which issues a new one-time credential under the same forced-change rule. With outbound mail configured (DATAPIPELINES_MAIL_HOST + DATAPIPELINES_MAIL_FROM — any SMTP server, Postmark's included; the nine variables are Configuration §3.27 and deploy/secrets.env.example names them) that credential is emailed to the user and sys-ops (DATAPIPELINES_MAIL_OPS_TO) hears about every new user, local or first social login (Auth §5A.8); the admin screen then shows where the password went, not what it was. Mail is on exactly when those two are set — there is no flag — and it then also requires DATAPIPELINES_AUTH_BASE_URL (the login URL in the mail).

Everything else has a default and is optional.

5.2 Everything else

Optional keys — executor concurrency, staging memory, result TTLs and caps, SSE heartbeat and disconnect grace, rate limits, idempotency TTL, template cache, UI theme, retention windows, observability — are cataloged with their defaults in Configuration §3. Resolution precedence (env > profile YAML > base YAML, plus the two per-entity runtime overrides) is Configuration §4.

The keys an operator most often changes at deploy time are datapipelines.result.max-size-bytes (Redis sizing, §4.2.2), datapipelines.executor.max-concurrent-executions-per-instance and datapipelines.staging.h2.max-memory-mb (heap sizing, §6.6), and datapipelines.executor.execution-timeout-seconds (the wall clock that bounds any single execution).

5.3 Full config file

Every setting is an environment variable, and deploy/env/defaults.env lists all the non-secret ones with their shipped values (deploy/secrets.env.example names the rest) — Environments is the operator's page for what to set and where. Operators who prefer a YAML file can mount application.yml at /etc/datapipelines/application.yml (configurable via SPRING_CONFIG_ADDITIONAL_LOCATION); this is the supported route for more than one OIDC provider, whose nested list is awkward as variables. Every key is expressible either as YAML or as its derived env var; the OIDC provider list is a nested structure and is normally supplied as YAML with ${...} placeholders for the secrets. A complete annotated template is in Configuration §5.

6. Deployment Patterns

6.1 Single-instance (dev / small team)

One instance handles everything — UI, API, MCP, pipeline execution. Docker Compose or bare JVM.

Suitable for: dev, evaluation, small teams (< 20 users).

6.2 Multi-instance horizontal scaling (production)

The application is stateless for all CRUD operations, UI, MCP, and auth. Multiple instances run behind a load balancer and serve requests interchangeably.

                    ┌─────────────┐
                    │ Load Balancer│
                    └──┬───┬───┬──┘
                       │   │   │
              ┌────────┘   │   └────────┐
              ↓            ↓            ↓
        ┌──────────┐ ┌──────────┐ ┌──────────┐
        │Instance A│ │Instance B│ │Instance C│   ← stateless web/API/MCP/UI
        └────┬─────┘ └────┬─────┘ └────┬─────┘
             │            │            │
             └────────────┴────────────┘
                      │     │
              ┌───────┘     └────────┐
              ↓                      ↓
        ┌──────────┐           ┌──────────┐
        │ Postgres │           │  Redis   │
        │ (shared) │           │ (shared) │
        └──────────┘           └──────────┘

No sticky sessions required. Each request is independent.

Client addresses behind the load balancer (datapipelines.auth.trusted-proxies). Every instance sees the LB's address as remoteAddr, so anything keyed or recorded on it — the per-IP login rate limiter, every audit source_ip — collapses to one deployment-wide value: without configuration, the login budget (default 10/min) is a single shared bucket any client can exhaust for everyone. Set the key to the CIDRs of proxies you control (a bare IP is a host CIDR):

datapipelines:
  auth:
    trusted-proxies: 10.0.0.0/8   # the LB's network

The resolution is spoof-safe: when the direct peer is NOT in the list, the X-Forwarded-For header is ignored entirely — an untrusted peer cannot forge its way past the limiter by setting it. When the peer IS trusted, the client is the rightmost header entry that is not itself trusted. Each entry must parse as a CIDR or startup is refused (Configuration §3.4/§7) — a typo'd range must not silently widen proxy trust. The shipped default is EMPTY: a deployment with no proxy in front behaves exactly as before, and the header stays ignored.

What's stateless (any instance serves any request)

• REST API (all CRUD, list, detail, execute endpoints)
• MCP server
• UI (all screens — Thymeleaf server-rendered)
• Auth (JWT sessions, API key validation — both backed by Postgres)
• Template rendering (reads from Postgres, per-instance LRU cache)
• Pipeline validation (reads from Postgres)

What's instance-local (tied to the instance that started it)

• In-flight pipeline executions. When POST /pipelines/{id}/execute hits Instance A, the execution runs on Instance A's JVM (coroutines + per-execution H2 staging). The SSE stream is a direct connection from Instance A to the client.
• Implication: if the SSE connection drops mid-execution, there is no reconnection or resumption path. Instance A starts a grace timer (datapipelines.sse.disconnect-grace-seconds, default 30) and, if the execution has not reached a terminal event by the time it elapses, cancels it — the execution ends ABORTED (REST API §6.8). A client that loses its stream should assume the abort and re-execute. This is deliberate: an execution nobody is waiting for must not keep holding source-database connections and staging memory.
• Cross-instance cancel works anyway. DELETE /api/v1/executions/{id} may land on Instance B, which has never heard of the execution. Instance B writes a Redis cancellation flag; Instance A honors it on its next heartbeat tick or node boundary — worst-case latency ≈ one heartbeat interval (REST API §10.4, DAG Executor §8.3.1). No sticky sessions are needed for cancellation to be reliable.
• Completed executions are not instance-local at all. Results live in Redis for their TTL and are readable from any instance via the cursor (REST API §7); execution metadata is in Postgres; the 1-hour event log is in Redis. Only the running execution is pinned to one JVM.
• Instance crash: loses only that instance's in-flight executions (H2 staging is in-memory and non-recoverable). Their rows are swept to ABORTED by the stale-execution sweep (marked pipeline.execution.instance_lost once they are older than datapipelines.executions.stale-timeout-minutes; every replica runs the idempotent sweep, so no surviving-instance coordination is needed). Completed results and history are unaffected; the lost work must be re-executed.

Multi-instance checklist

6.3 Docker Compose (dev / evaluation)

Reference compose file provided in deploy/compose.yml. Single instance + Postgres + Redis.

Compose is one loader of the environment-variable contract, not the contract itself (Environments §3): the same settings drive a bare JAR, systemd, Kubernetes, ECS and Nomad. It reads TWO env files, in this order: deploy/env/defaults.env, which is tracked and carries every non-secret default, then deploy/secrets.env, which is git-ignored and holds every credential plus this deployment's own overrides (DATAPIPELINES_POSTURE=hardened, DATAPIPELINES_ENV=prod, the real database URL, the OIDC client).

docker compose -f deploy/compose.yml \
  --env-file deploy/env/defaults.env \
  --env-file deploy/secrets.env up -d

The app service passes every DATAPIPELINES_* variable the app binds, each with the same default application.yml ships — so a key that works against a host-run app (by exporting the variable) reaches the container too, and leaving it unset yields the shipped default. scripts/compose-env-audit.sh diffs the compose block against application.yml's placeholders — and the two tracked env files — and fails on a missing pass-through, a diverged default, a variable declared in both tracked files or in neither, or a posture-dependent key passed with a default instead of in the valueless form. It runs on every ./gradlew build, not when someone remembers to run it.

6.3A Promotion: a receiver deployment (055)

A second deployment that receives released content from the first (Versioning §10). Both sides are the same image in different postures — there is no separate build.

On the RECEIVER (e.g. uat):

DATAPIPELINES_ENV=uat
DATAPIPELINES_POSTURE=hardened                            # authoring defaults OFF here
DATAPIPELINES_DEPLOYMENT_PROMOTION_SERVER_KEY=<the shared secret>

On the SENDER (e.g. dev):

DATAPIPELINES_ENV=dev
DATAPIPELINES_POSTURE=development                         # this is where content is built
DATAPIPELINES_DEPLOYMENT_PROMOTION_TARGET_URL=https://uat.example.com
DATAPIPELINES_DEPLOYMENT_PROMOTION_TARGET_KEY=<the same shared secret>

DATAPIPELINES_ENV is your name for each deployment and nothing branches on it; the receiver records the sender's label as the source_env of what it received. DATAPIPELINES_POSTURE is what actually changes behaviour — hardened defaults authoring-enabled to false, which is the receiver's whole configuration. Set DATAPIPELINES_DEPLOYMENT_AUTHORING_ENABLED explicitly if this one box must do both (Environments §1).

Generate the secret the way every other one here is generated — openssl rand -base64 32 — and set the identical value on both sides. It is a bearer credential: it belongs in deploy/secrets.env with the rest of the secrets, never inline in a compose file or a chart's values.

What each setting buys, and what goes wrong without it:

Minting the receiver's key (091): on the RECEIVER, sign in as an admin → API → New key → kind Server → name it after the sender (dev → uat) → copy the plaintext, shown once. Paste it into the SENDER's DATAPIPELINES_DEPLOYMENT_PROMOTION_TARGET_KEY. Rotation is then receiver-side and needs no restart anywhere: mint a second server key, set it on the sender, revoke the first. (With the deprecated config value instead: set the new value on both sides and restart both.) No user account is involved either way, so offboarding a human can never break production promotion, and revoking the key revokes nothing else.

Direction is one-way by construction. The receiver validates; it never calls the sender. A compromised receiver cannot reach back into the authoring environment.

Running two stacks on one machine (a rehearsal, or CI): APP_COMPOSE_PROJECT gives ./app.sh a second isolated compose project with its own image tag, volumes and containers, and APP_HOST_PORT moves its published port off the first one's 8080 (the container port never moves). Both are lane knobs, not app config — that is why neither carries the DATAPIPELINES_ prefix.

./app.sh --start                                             # dev, the sender, on :8080
APP_COMPOSE_PROJECT=uat APP_HOST_PORT=8081 ./app.sh --start  # uat, the receiver, on :8081

The sender reaches the receiver from INSIDE its container, so target.base-url is not the localhost URL your browser uses: on Docker Desktop it is http://host.docker.internal:8081, and on Linux it is the host's address on the docker bridge. Two compose projects are two networks; a service name from one does not resolve in the other.

6.4 Kubernetes (recommended for production)

Reference Helm chart in deploy/helm/. Includes:

• Deployment (N+ replicas, behind a Service).
• Externalized Postgres (managed recommended).
• Externalized Redis (managed recommended).
• HorizontalPodAutoscaler (scales on CPU + memory).
• PodDisruptionBudget (availability during node drains).

The pod keeps its root filesystem read-only and mounts a disk-backed emptyDir at /tmp. This supports JVM native libraries and the default LAKE spill directories. Set temporaryStorage.sizeLimit to bound the volume (empty by default), and set ephemeral-storage requests/limits through resources for the workload. Do not use a memory-backed volume for large LAKE spills: that would consume the memory budget spilling is intended to relieve. Each pod has its own volume; Kubernetes removes it with the pod. Size it for concurrent queries and see Configuration §3.24 for explicit paths and cleanup after a process crash. Docker Compose uses the image's writable /tmp; custom read-only deployments must mount a writable temporary directory too.

No sticky session affinity needed. Standard ClusterIP service with round-robin or random load balancing.

6.5 Bare JVM

For development:

java -Duser.timezone=UTC -jar datapipelines-app.jar \
  --spring.datasource.url=jdbc:postgresql://localhost:5434/datapipelines \
  --datapipelines.redis.host=localhost \
  --datapipelines.redis.port=6381

-Duser.timezone=UTC is required here too (§3.1) — the image sets it for you, a bare JVM does not.

Not recommended for production.

6.6 Resource sizing

Two numbers matter: JVM heap, and the container memory limit that must contain it.

Heap. The dominant consumer is H2 staging — each executing pipeline holds up to datapipelines.staging.h2.max-memory-mb (default 1024 MB), or its own settings.tempdb.config.max_memory_mb override:

heap ≥ (staging max-memory-mb × concurrent executions on THIS instance) + ~512 MB baseline

The multiplier is the per-instance execution ceiling max-concurrent-executions-per-instance (050/R2): every execution runs on exactly one instance, and this instance can legitimately hold the full setting's worth at an unbalanced moment. The default 100 × 1024 MB is the worst case ONE box must absorb — size the container for it (with N replicas behind a round-robin LB the typical share is setting / N, but size for the ceiling, not the share; cap the exposure by lowering the setting — or max-memory-mb — per box, or scale on smaller per-instance limits). The ~512 MB baseline covers Spring context, Hikari pools, the template cache, and SSE buffers.

Worked example — 4 concurrent executions per instance at the 1024 MB default: 4 × 1024 + 512 ≈ 4.6 GB heap.

Container memory limit.

container limit ≈ heap × 1.5

The 0.5 covers what the heap number does not: metaspace, code cache, thread stacks, and — significant here — JDBC direct/native buffers, which scale with result-set width and fetch size. A limit set equal to -Xmx gets the container OOM-killed by the kernel rather than getting a clean OutOfMemoryError, which is strictly worse to diagnose. For the example above: ~7 GB limit.

JVM flags. Either -XX:MaxRAMPercentage=65 (heap tracks the container limit — preferred for k8s, where limits change without an image rebuild) or an explicit -Xmx. Do not set both. Always -Duser.timezone=UTC (§3.1).

Servlet threads (MCP blocking calls). An MCP pipelines_execute call (MCP Server §6.2.3) is a single blocking HTTP request that holds one servlet thread until the execution reaches a terminal state or datapipelines.executor.execution-timeout-seconds (default 600) elapses. Per-user concurrency is bounded (max-concurrent-executions-per-user, default 10) but the default Tomcat pool is 200 threads, so on the order of ~20 concurrent long-running MCP callers can saturate it and starve REST/UI traffic on the same instance for minutes. Size server.tomcat.threads.max at or above the expected count of simultaneously-blocking MCP executions plus normal REST concurrency, or isolate /mcp on its own connector/instance. This is in addition to raising proxy/LB idle timeouts above execution-timeout-seconds (MCP Server §6.2.3).

CPU. Execution is coroutine-based and largely I/O-bound on source databases; 2 vCPU per instance is a reasonable floor. Scale out on max-concurrent-executions-per-instance pressure, not CPU.

6.7 Marketing site & in-product docs

Since v1.4 the app serves the marketing site and the documentation itself — the site and the product are ONE deployment (owner decision 2026-08-31). There is no separate static deploy to keep available, and the docs shipped in the jar always match the version running them.

• GET / — the marketing site (public). Template templates/site/index.html, assets under static/site/**, referencing the app's vendored design system at /vendor/design-system/** (the retired website/ directory carried a second vendored copy — the app copy is now the single sync target of scripts/sync-design-system.sh). The only dynamic fact (the MCP tool count) is a compile-time constant baked at render time; public routes touch no database.
• GET /dashboard — the signed-in dashboard, moved off /. There is no auto-redirect: signed-in users hitting / get the marketing page.
• GET /docs, GET /docs/{slug} — the packaged spec set (docs/*.md minus the contributor/research exclusions, packaged by processResources in modules/web/build.gradle.kts). Public since 073: the viewer renders Markdown out of the jar and reaches no principal, no workspace and no datastore, and the identical content is already public in the AGPL repository — so exposing it moves ~25 pages of documentation into this domain's search index instead of GitHub's. Anonymous readers get the public site chrome; signed-in readers keep the application chrome.
• GET /mcp-server-for-sql-databases, /mcp-server/{engine}, /add-mcp-server-to-claude-code, /ai-data-pipeline, /text-to-sql-agent, /compare/{airflow,dbt}, /federated-query — the intent-cluster pages (073), same shape as /: GET-only, anonymous, constant content, no database. See UI Screens §4.15.
• GET /robots.txt, GET /sitemap.xml — crawler surfaces (073). robots.txt is a static file; sitemap.xml is generated from the page registry and the packaged doc slugs, with lastmod taken from build-info.properties when it is on the classpath.
• Public-surface defence is cache headers, not a rate limiter. / is Cache-Control: public, max-age=300; /site/** is public with a 1-hour TTL plus Last-Modified revalidation. The login rate limiter is deliberately NOT applied here (033/D1): the content is constant between deploys, so a shared-cache TTL costs nothing per request — and the T46 remoteAddr-keying concern is closed regardless: the limiter now resolves the CLIENT address through datapipelines.auth.trusted-proxies (§6.2), so pointing it at / would no longer create an LB-address-wide bucket.
• Allowlist. / and /site/** join the permitAll list in SecurityConfig with their reasons inline; 073 added the seven cluster-page matchers (enumerated, not globbed), /docs + /docs/*, /robots.txt and /sitemap.xml, each with its own comment. Nothing else was widened, and anyRequest().authenticated() still closes the list.

Search Console and the sitemap (operator note, 073)

Indexing is not automatic and is not a deploy step — it is a one-time verification plus an occasional check.

1. Verify the domain in Google Search Console (and Bing Webmaster Tools, if you care about it). Use the DNS TXT method: it verifies the whole domain including subdomains, and it survives redeploys — an HTML-file method would need the file added to static/, which is a code change for a DNS problem. Do this before any launch announcement, so the first crawl after the first inbound links is already attributed.
2. Submit https://<your-host>/sitemap.xml once. It regenerates on every request from the page registry and the packaged docs, so a page added in a later release is in it the moment that release is deployed — there is nothing to resubmit.
3. robots.txt allows everything, and deliberately lists no private paths: it is a public file, so naming internal routes in it advertises them. What keeps the private surface private is SecurityConfig's anyRequest().authenticated(), not this file.
4. A self-hosted deployment emits datapipelines.co canonicals. That is deliberate — rel="canonical" names the one address that should be indexed, and a private instance asking Google to index it is the failure mode the tag exists to prevent. If you WANT your own deployment indexed under your own domain, change SITE_ORIGIN in modules/web/.../ui/site/SitePages.kt and rebuild.
5. What to look at afterwards. Coverage (are the doc pages indexed?), Performance filtered to /mcp-server/ (are the engine pages taking their measured queries?), and any Core Web Vitals note. Nothing here needs a schedule; a look a month after launch and a look a quarter later is the whole practice.

S3/CloudFront cold fallback (kept, drop if it rots unused). If the app is down but marketing must stay up, the same template renders to static files with facts baked:

./gradlew :modules:web:websiteExport          # → modules/web/build/website-export/
aws s3 sync modules/web/build/website-export/ s3://YOUR-BUCKET/ --delete
aws cloudfront create-invalidation --distribution-id YOUR_DIST_ID --paths "/*"

The export renders /-rooted links (/site/..., /vendor/design-system/...), which the bucket layout mirrors exactly — upload the CONTENTS of website-export/, not the folder. S3 static website hosting (or CloudFront with an origin access control) serves index.html as the index document. Nothing fingerprinted: keep TTLs short for index.html and longer for vendor/** (those files change only when the design system is re-vendored). This is an emergency procedure, not a second primary deploy — the app is the primary.

7. Persistence & Backup

7.1 Metadata DB

• Backup: operator's standard Postgres backup practice (pg_dump, managed snapshots, etc.).
• Restore tested regularly by operator (runbook provided).
• Critical data: pipelines, templates, datasources (with credentials), audit log.
• Disposable data: executions table (can be truncated; only used for history).

7.2 Redis

• Volatile by design. Every key is TTL-bounded: results, idempotency records, cancellation flags, the 1-hour event log (§4.2).
• No backup needed. A Redis restart loses unexpired results (clients re-execute), unexpired idempotency records (a retry may execute a second time), and pending cancellation flags. All are recoverable by re-running; none is a durable record. The durable records — pipelines, templates, datasources, execution history and execution_events — are all in Postgres.
• Persistence (RDB/AOF) is not required and not recommended; it buys nothing that a re-run does not.
• maxmemory-policy noeviction is required regardless (§4.2.1). "Volatile" means TTL-expiring, not evictable — eviction breaks correctness in a way that restart does not, because it happens silently during normal operation.

7.3 H2 staging

• In-memory only. Per-request. No persistence. No backup. No recovery.

7.4 Restore drill

Operators should run a quarterly restore drill: restore metadata DB from backup to a sandbox instance, verify pipelines still execute. Documented in the runbook (future).

8. Upgrades

8.1 Versioning

• Semantic versioning: MAJOR.MINOR.PATCH.
• PATCH: bug fixes only. Safe to upgrade in place.
• MINOR: new features, backward-compatible. Schema migrations applied automatically on app startup.
• MAJOR: breaking changes. Migration guide published.

8.2 Database migrations

• Flyway for metadata DB migrations.
• Migrations are versioned SQL files in app/src/main/resources/db/migration/.
• Applied automatically on app startup.
• Forward-only; no rollback (restore from backup if needed).

8.3 Upgrade procedure (recommended)

1. Review release notes for breaking changes.
2. Backup metadata DB.
3. Signal shutdown and let the instance drain (§8.3.1 — this is automatic, not a manual step).
4. Start the new version (migrations apply on startup).
5. Verify /health returns UP.
6. Restore traffic.

For k8s: rolling update via kubectl rollout. Each terminated pod flips its readiness and cancels its in-flight executions on the way out (§8.3.1); the preStop and terminationGracePeriodSeconds settings in §8.3.2 keep that orderly.

8.3.1 Graceful shutdown mechanism

Shutdown is a defined sequence, not "stop the process and hope". On SIGTERM the instance, in this order:

1. Fails readiness first. ReadinessState.REFUSING_TRAFFIC is published before anything is cancelled, so /ready starts returning 503 while the process is still fully up and the load balancer / k8s Service bleeds traffic off. The order is the contract: flipping after the drain would keep fresh work flowing into an instance that is already cancelling it. /health keeps reporting UP, so nothing kills the pod for being unhealthy mid-drain.
2. Cancels every in-flight execution through the ordinary cancellation path (DAG Executor §8.3): Statement.cancel() on every registered statement first — which is what actually stops the query on the source database, so the drain is more than a status update — then the root Job cancellation, execution_aborted (reason: "shutdown") emitted to connected streams, tempdb dropped and connections released in finally, and the row written ABORTED. The drain cancels; it does not wait for executions to finish — an execution in flight at SIGTERM ends ABORTED, not SUCCESS.
3. Waits for the flush, bounded (20 seconds). An execution leaves the live count only after its ABORTED status and events are written, so the wait means the bookkeeping reached Postgres and Redis before exit; the bound means a wedged execution meets the kubelet's deadline rather than hanging shutdown forever.
4. Exits. The web server's own graceful shutdown (server.shutdown: graceful) runs after the drain, so SSE clients are still connected while their terminal events arrive, and in-flight ordinary requests complete rather than being cut off.

The accepted loss, stated plainly: an execution running when an instance stops is cancelled, not preserved. It ends ABORTED with reason: "shutdown", is visible as such in execution history, and its client must re-execute. There is no execution hand-off to another instance and no resumption — in-memory H2 staging makes migration impossible, and pretending otherwise would be worse than the honest abort. Bounded, visible loss beats a silent hang.

One residual race, honestly: a request that reaches the instance between the readiness flip and the web server stopping can still launch an execution. The drain re-cancels on every flush tick, so such an execution is cancelled within ~100ms of starting — but the right answer is that traffic should have stopped at the readiness flip, which is what the preStop in §8.3.2 exists to give the endpoints controller time to do.

8.3.2 Kubernetes pod lifecycle

terminationGracePeriodSeconds: 30      # covers the bounded drain flush (§8.3.1 step 3)
lifecycle:
  preStop:
    exec:
      command: ["sleep", "5"]

• terminationGracePeriodSeconds: 30. The drain cancels rather than waits (§8.3.1), so the only clock that matters is the bounded flush — 30 seconds covers it with margin. Anything shorter risks the kubelet SIGKILLing mid-flush: executions die without their finally blocks, so no execution_aborted event and no status update — rows left RUNNING until the stale-execution sweep marks them ABORTED (§6.2).
• preStop: sleep 5 closes the standard k8s race: endpoint removal and SIGTERM are concurrent, so without it the pod can flip readiness microseconds before the Service stops routing to it. Five seconds of overlap is enough for Endpoints propagation — and it is what makes step 1's readiness flip actually take traffic off the pod before step 2 cancels.
• A PodDisruptionBudget (§6.4) keeps node drains from taking every replica's drain at once.

8.4 Rollback

• App rollback: redeploy previous image. Schema migrations are forward-only — if a migration was applied, you can't run the old app version against the new schema.
• For MINOR upgrades: schema changes are additive; rollback usually works.
• For MAJOR upgrades: take a DB backup before upgrade; restore it to roll back.

9. Security Hardening Checklist (Deployment)

• [ ] TLS termination at load balancer / proxy (let it handle cert renewal).
• [ ] Metadata DB password rotated and not in source control.
• [ ] DATAPIPELINES_JWT_SECRET is high-entropy (≥ 32 bytes random).
• [ ] DATAPIPELINES_DB_ENCRYPTION_KEY is high-entropy (32 bytes random) and stored in a secret manager, not a plaintext env file.
• [ ] Redis password set if Redis is networked (requirepass on the server, datapipelines.redis.password on every app instance — they must match).
• [ ] Redis maxmemory-policy noeviction (§4.2.1) — correctness, not tuning.
• [ ] OIDC client secrets from a secret manager, not a plaintext env file; at least one provider configured (§5.1).
• [ ] NetworkPolicy restricts app's egress.
• [ ] Service account / IAM role has least privilege.
• [ ] Container runs as non-root user (enforced in Dockerfile).
• [ ] Container filesystem read-only except for configured volume mounts.
• [ ] Resource limits set (CPU + memory) per deployment.
• [ ] Audit log retained per compliance policy.
• [ ] No /actuator/* path reachable on the application port; /actuator/prometheus on the management port (management.server.port), cluster-internal only (Observability §4.2). The management port is never published to a host or load balancer; MANAGEMENT_SERVER_ADDRESS defaults to loopback — setting it to 0.0.0.0 (required for k8s scraping) demands an accompanying NetworkPolicy confining the port to the monitoring namespace.
• [ ] Internet-exposed deployments may prefer to omit -Pdatapipelines.commit at build time — a public /info commit hash maps the instance to exact source revisions.
• [ ] The lib/ driver drop-in mount is read-only in the container, populated at image build or by a trusted init container, never writable by the app user; LOADER_PATH, if set, comes from the image — never inherited from the deployment environment (a writable lib/ is code-execution-by-file-drop).
• [ ] Production Redis requires requirepass (or ACLs) and TLS, or is confined to a private network segment with a NetworkPolicy — it holds fully materialized caller results for up to an hour (D9). The app logs a structured WARN at startup when the Redis password is empty and the host is not loopback (Configuration §7).

10. Distribution License

• Code: AGPL-3.0 (see LICENSE; contributions under the CLA).
• Dependencies: only those with compatible licenses bundled by default. Oracle and MySQL drivers optional via Gradle profile (operator's responsibility to accept their licenses).

LICENSE and NOTICE files at repo root.

11. What's Out of Scope for v1

• Managed / SaaS deployment: future commercial offering.
• Multi-tenant isolation: single-tenant v1.
• High-availability Postgres in our chart: operator provides managed PG.
• Backup automation: operator responsibility; we provide runbook.
• Air-gapped deployment: should work (no phoning-home telemetry) but not explicitly tested in v1.
• Federated deployments (independent installations sharing state or federating queries across each other): not supported. Multi-instance horizontal scaling (§6.2) is a different thing and is supported.
• Execution hand-off / resumption: an execution is pinned to the instance that started it and is aborted rather than migrated (§6.2, §8.3.1).

Appendix A: Reference compose.yml Sketch

This sketch boots — it satisfies every §5.1 startup requirement. Removing any of the marked items produces a container that exits during context startup, not one that runs degraded. deploy/compose.yml is the real file; every secret below comes from deploy/secrets.env under the same name the app binds, so the same file is sourceable by a bare java -jar, a systemd EnvironmentFile or a Kubernetes Secret (Environments §4).

services:
  datapipelines:
    image: ghcr.io/datapipelines/datapipelines:1.0.0
    ports:
      - "8080:8080"
    environment:
      SPRING_DATASOURCE_URL: jdbc:postgresql://postgres:5432/datapipelines
      SPRING_DATASOURCE_USERNAME: datapipelines
      SPRING_DATASOURCE_PASSWORD: ${SPRING_DATASOURCE_PASSWORD}

      DATAPIPELINES_REDIS_HOST: redis
      # REQUIRED whenever redis runs with --requirepass. Must be the SAME value
      # the redis service below is started with, or every Redis call fails at runtime.
      DATAPIPELINES_REDIS_PASSWORD: ${DATAPIPELINES_REDIS_PASSWORD}

      # REQUIRED, no fallback. Generate once: openssl rand -base64 32
      DATAPIPELINES_JWT_SECRET: ${DATAPIPELINES_JWT_SECRET}
      # REQUIRED, no fallback. Exactly 32 bytes base64: openssl rand -base64 32
      # Losing this makes every stored datasource credential unrecoverable.
      DATAPIPELINES_DB_ENCRYPTION_KEY: ${DATAPIPELINES_DB_ENCRYPTION_KEY}

      # REQUIRED: at least ONE authentication method (ConfigValidator §7) — an OIDC
      # provider configured below, or local accounts (auth.md §5A: set
      # DATAPIPELINES_AUTH_LOCAL_ENABLED=true plus a one-time seed for
      # DATAPIPELINES_AUTH_BOOTSTRAP_ADMIN_EMAIL, which names the first admin either way).
      # These names are referenced by the providers list in application.yml below;
      # Google is only an example — Microsoft/Okta/Keycloak/any OIDC IdP works.
      GOOGLE_CLIENT_ID: ${GOOGLE_CLIENT_ID}
      GOOGLE_CLIENT_SECRET: ${GOOGLE_CLIENT_SECRET}
    volumes:
      # The OIDC provider list is a nested structure — supply it as YAML.
      - ./application.yml:/etc/datapipelines/application.yml:ro
    depends_on:
      - postgres
      - redis
    restart: unless-stopped

  postgres:
    image: postgres:16-alpine
    environment:
      POSTGRES_DB: datapipelines
      POSTGRES_USER: datapipelines
      POSTGRES_PASSWORD: ${SPRING_DATASOURCE_PASSWORD}
    volumes:
      - postgres-data:/var/lib/postgresql/data
    restart: unless-stopped

  redis:
    image: redis:7-alpine
    # noeviction is REQUIRED (§4.2.1): eviction silently destroys results,
    # idempotency keys, and cancellation flags. Same password as the app above.
    command: >
      redis-server
      --requirepass ${DATAPIPELINES_REDIS_PASSWORD}
      --maxmemory 512mb
      --maxmemory-policy noeviction
    restart: unless-stopped

volumes:
  postgres-data:

The mounted application.yml needs only the provider list (everything else has a default or is set above):

datapipelines:
  auth:
    oidc:
      providers:
        - name: google
          client-id: ${GOOGLE_CLIENT_ID}
          client-secret: ${GOOGLE_CLIENT_SECRET}
          issuer-uri: https://accounts.google.com
          display-name: "Sign in with Google"

Format and further provider examples: Auth §5.1 and §11.1. Full key template: Configuration §5.

Appendix B: Demo Quickstart — the Published Sample Data

The sample data is published, versioned artifact sets (one per family), not something the app downloads: loading it is a deployment step (sample-data design D5). Any deployment that pulls the same version gets the same databases, byte for byte in content. The build scripts that produce them live at scripts/sample-data/ (nyc family), scripts/sample-data-trade/ (trade family) and scripts/sample-data-lake/ (dp-lake); everything below is the consuming side.

Three independent families, each behind its own compose profile and app.sh flag — an engineer spins up exactly the data they need:

Any combination is fine — the app's bootstrap keys are comma-separated lists built from the active families, and the MySQL service is shared.

The public marketing site's /demo-data page renders these published manifests as documentation — every table, row count, source and licence per family, generated from the same manifest files the loader verifies (vendored at modules/web/src/main/resources/site/demo/, pinned to the versions above). It is the page to hand anyone who asks what the demo contains.

The lake family is different in kind from the other two and the difference is worth stating before the commands: it has a compose profile and a --demo flag like its siblings, but no loader service, because nothing is loaded. What --profile demo-lake turns on is the datasource registration and the manifest import; the data itself stays in the bucket and is read at query time — see "dp-lake — the family with no loader" below.

One command

The base URLs and the pinned versions live in deploy/env/defaults.env, which is tracked — a data change is a new version directory and a commit, so what a deployment loads is visible in the repository. From a checkout that builds its own image, the whole thing is:

./app.sh --start --demo nyc,trade,lake

For the raw compose path, set the family ON markers your invocation wants and pass the same env files app.sh does — secrets last:

SAMPLE_NYC_ON=1 SAMPLE_TRADE_ON=1 SAMPLE_LAKE_ON=1 docker compose \
  -f deploy/compose.yml -f deploy/compose.local-build.yml \
  --env-file deploy/env/defaults.env \
  --env-file deploy/secrets.env \
  --profile demo-nyc --profile demo-trade --profile demo-lake up -d --wait

--wait returns when the app is healthy, and by then the artifacts have been downloaded, every checksum verified, the databases restored, the SELECT-only demo login created, and the families' datasources registered.

The nyc and trade families also build the jar with -Pmysql (see the driver note below); lake alone does not need it. ./app.sh --stop and --status take the same --demo list, so the demo services are not left running invisibly. ./app.sh --clean [--yes] is the clean slate: it stops and removes the project's containers, then deletes the METADATA volume (<project>-postgres-data — pipelines, templates, executions, users, API keys, workspaces, datasource rows) by explicit name, and keeps <project>-mysql-data (demo source data) and <project>-sample-data (the downloaded artifacts). It asks for the project name back on a terminal, or needs --yes without one; the next --start migrates a fresh database and re-seeds the demo. Sign-in users are re-created on their next login; API keys must be minted again. Demo is a flag, not an environment (Environments §5): DATAPIPELINES_DEMO is the source of truth, --demo merely sets it, and the hardened posture refuses it at boot. The per-family --demo-nyc / --demo-trade switches are gone; the families are nyc, trade and lake, and an unknown name is refused by name.

MySQL driver. MySQL Connector/J is GPL with a FOSS exception and is not in the default build (§3.5, Datasources §10.2). The sample-weather and sample-trade-world datasources are MYSQL, and bootstrap registration fail-fasts startup with datasource.driver_not_loaded without it. Build with the driver:

./gradlew -Pmysql :modules:app:bootJar && docker build -t datapipelines:local .

or drop the jar into lib/. ./app.sh --start --demo nyc[,trade[,lake]] does the -Pmysql build for you.

Point an agent at it — three steps

1. Log in with the account ./app.sh --start prints. The demo needs no OIDC client at all (Auth §5A): local password accounts are enabled and the first admin gets a GENERATED one-time password, written to deploy/secrets.env at first scaffold and forced to change at first sign-in. app.sh reads the seeded account back out of the database after the stack is healthy and prints the login that actually exists — the seed fires once, at row creation, so a DATAPIPELINES_AUTH_BOOTSTRAP_ADMIN_EMAIL changed later names an account that was never created (Environments §8). The first login provisions your personal workspace and seeds the example pipelines into it.
2. Mint an API key from the UI (or POST /api/v1/auth/api-keys). The secret is shown exactly once.
3. Give the agent the MCP endpoint http://localhost:8080/mcp and that key. It can list the seeded pipelines, read the three sample datasources' schemas, and execute nyc/mobility/revenue_by_borough or nyc/mobility/rainy_vs_dry_ridership immediately — see MCP Server.

What the demo profile turns on

The profiles add a mysql service and the families' one-shot loaders, and point the app at the files they place on a read-only volume. The scaffolded deploy/secrets.env enables local password accounts so the demo needs no OIDC client, and names the account the credential lands on; the password itself is a secret and is GENERATED into that same file — never a constant in a tracked file, on an app that binds every interface (Auth §5A.2). There is now exactly ONE place the credential is written and ONE place it is read back from, which is why app.sh can print a login that works. It also sets the §7 demo posture: auto-per-user provisioning (every visitor gets their own workspace) and member-datasources-enabled=false (an open datasource form on a public server is an SSRF and port-scan primitive — demo users get the seeded datasources only). Without --profile demo none of it exists: both datapipelines.bootstrap.* keys are paths and empty means off, so the non-demo stack is configured exactly as it was.

The demo datasources are protected in three independent layers: workspace-scoped access, the is_readonly flag on the datasource row (Datasources §5.7), and a database login granted SELECT and nothing else — created by the loader, never assumed. The SQLite entry additionally opens its file with the driver's read-only mode (Datasources §8A.4). On a public demo, add the §9 hardening item that belongs with this posture: egress-restrict the app container's network, so a datasource anyone can reach cannot become a path to anywhere else.

The loader

deploy/sample-data/load.sh <base-url> <version> is what the one-shot services run. Its contract, in order: download every artifact the manifest names, verify every checksum, and only then touch an engine — a corrupted download can never leave a half-loaded database behind. Each engine then gets a _sample_meta(version) marker, written last, so a re-run of a loaded deployment skips it and a failed engine is retried cleanly on the next start.

It runs as two services because no pinned image carries both a Postgres and a MySQL client, and installing one at container start would put an unpinned package fetch in the one place that has to be reproducible.

Demo Postgres sizing (T200, 109 §E)

The postgres service that the nyc family restores into carries the demo's biggest table — trips, ~804 MB, ~4.9M sampled rows plus rollups — against the Postgres image's compiled defaults (shared_buffers=128MB, random_page_cost=4.0 tuned for spinning disks). Under load, a simple year-2024 aggregate therefore ran in ~21 s: nearly every page came from the OS cache, but the planner kept choosing plans as if each random page cost a seek. The compose file sets:

command: postgres -c shared_buffers=512MB -c effective_cache_size=1536MB
                -c work_mem=32MB -c random_page_cost=1.1

with the container's memory reservation raised to 1 GB to match. The numbers are shaped by the data: shared_buffers ≈ half the restored set, effective_cache_size ≈ shared_buffers plus the OS page cache the box actually has, work_mem enough for the demo's hash joins and sorts, and random_page_cost=1.1 because the volume sits on SSD. This is demo sizing, not production advice — the operator's own datasource is their own DBA's, and the app never tunes a database it did not bring.

Measured on a quiet box (109 §E), year-2024 airport-count query (pu_location_id IN (1, 132, 138), three runs cold→warm): compiled defaults 42.7 s → 23.9 s → 8.6 s (buffers read=43,871 — nearly every page from disk); sized flags 20.5 s → 9.6 s → 8.1 s (buffers hit=46,795, read=0 — the working set fits the 512 MB). The win is the cold/cache-pressured run, ~2.1×; warm steady-state is unchanged, as expected when the OS cache already holds the table. A composite index (pu_location_id, pickup_date) on the artifact itself (the datasource is readonly to agents, so the artifact is the only place an index can come from) takes the same query to ~0.95 s cold / 0.09 s warm (index-only scan, 196 buffer hits) — see the 109 handback for the artifact decision.

dp-lake — the family with no loader

sample-data/lake/<version>/ is different in kind from the two above and the difference is the point: it is read in place. Nothing is downloaded at demo start, nothing is restored into an engine, and there is no one-shot loader service. The query engine reads the Parquet objects over HTTPS and fetches only the ones a query's predicates name.

What is published (build runbook: scripts/sample-data-lake/README.md):

Egress is the operating cost to watch, and it is bounded by the query, not by the artifact. A demo session running the shipped pipeline at its default one-month window reads a few MB. The exposure is a visitor who writes SELECT * FROM hvfhv_trips with no date predicate — a ~7 GB full scan at S3 egress prices, per run. Set a bucket request-rate or budget alarm before the lake demo is announced.

Because there is no loader, the consuming side of this family is a datasource registration — shipped in round 089 as the lake demo family: the sample-lake LAKE datasource is registered create-if-absent from deploy/sample-data/bootstrap-datasources-lake.yml, which also imports the published manifest.json's tables[] into the dp-catalog registry (Datasources §8A.1, §8C). The showcase pipeline ships in its own examples file (scripts/sample-data/content/examples-lake.json) and the seeder loads it exactly when sample-lake is registered (the requires_datasources gate — environments.md §5). It needs egress to S3 (or a configured mirror) whenever a lake query runs.

Resetting an engine volume desyncs the demo login

The loader creates the dp_demo_ro login with the passwords from the env files, and bootstrap registration is create-if-absent — it never updates an existing datasource row (§8A). So if you delete an engine's data volume (docker volume rm dp-mysql-data) or rotate a SAMPLE_* password, the freshly created login no longer matches the credential the app stored at the original bootstrap, and that datasource fails validation in the UI while the engine itself is fine. The repair is to re-run registration for the stale entry only:

docker exec dp-postgres-1 sh -c \
  'psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -c "delete from datasources where name='"'"'sample-weather'"'"';"'
docker restart dp-datapipelines-1   # bootstrap re-registers with the current env password

(2026-08-30: hit live — a recreated MySQL volume left sample-weather failing UI validation until its row was re-registered this way.)

Re-publishing is a new version

Version directories are immutable (design §4). Any change to the data — a wider window, a different sample, a corrected lookup, even a typo — is published as v2 under a new prefix; v1 is never edited. Consumers pin the version (SAMPLE_VERSION), and nothing references a latest alias. This is what makes "the same version means the same data" a fact rather than a hope, and it is why the loader refuses a manifest whose version does not match the directory it came from.

Publish confirmation & release rehearsal — the drift guards

Two guards bracket every publish and every release rehearsal. Both exist because of T70 (2026-09-02): the published v1 examples.json still carried the ${} interpolations 042 had already migrated out of the repo copy, the demo 500ed on every fresh first login for two days, and nothing compared the published bytes with the repo's.

1. Before confirming an upload (and in every release rehearsal), run the published-drift guard for the version the demo pins:

   ./scripts/sample-data/check-published.sh v2                  # nyc (mobility), the default family
   ./scripts/sample-data/check-published.sh --family trade v2   # the trade family
   ./scripts/sample-data-lake/check-published.sh --family lake v1   # dp-lake (a different contract, below)
   

   It fetches the published manifest and examples.json, and fails unless the published copy, the published manifest's declared checksum, and the family's repo content/examples.json all agree. --family selects all three of the repo copy, the base-URL variable (SAMPLE_BASE_URL for nyc, SAMPLE_TRADE_BASE_URL for trade) and the default published prefix. Against an unpublished version it fails on the manifest fetch — which is the upload gate itself. Set the family's base-URL variable to check a mirror or a locally staged build (file://…/scripts/sample-data/work, or the local-serve recipe in app.sh). Network by nature, so it is a rehearsal step, never part of ./gradlew build. The lake family's guard is a separate script because it checks a different thing. That family has no examples.json — its content lives with the nyc family — so what can drift is not content but the objects themselves. It fetches the published manifest, refuses a manifest whose version disagrees with its directory, fails if any provenance row still carries license_verified: null, samples one object per table and compares SHA-256, and fetches the Iceberg metadata file to confirm every location it records sits inside the published prefix — an Iceberg table that still names a build machine's directory is unreadable, and nothing about an object listing shows it. Folding two unrelated contracts behind one --family flag would make "check-published passed" mean two different things.

2. The repo copy is validated in build — the templates module's SampleDataExamplesContentTest runs every shipped template and pipeline through the app's own save-time validators (049 C1), so content the seeder would refuse cannot merge. verify.sh step 5 remains structural only; the two checks are different jobs and each says which it does.

Licence gate — before serving this data publicly

Every provenance row in manifest.json ships license_verified: null. The build verifies no licence and claims none; the licence strings are research claims carried with their evidence links in each family's sources.lock (scripts/sample-data/, scripts/sample-data-trade/, scripts/sample-data-lake/).

Publishing with any license_verified still null blocks go-live (design §8). Verify each source's current terms, record the date, and swap — do not ship — any dataset that fails. This applies to datapipelines.co; a self-hosted evaluation loading the artifacts for its own use is a different question, and one for the operator.

Appendix C: Change Log