# REST API + SSE Specification

**Status:** v2.107 (frozen contract — additive-only changes after this point; see the 2026-09-20, 2026-09-24 and 2026-09-26 (v2.36, v2.37) rows for the deliberate breaks)
**Owner:** datapipelines.co core
**Depends on:** [Type System spec](/docs/type-system.md), [Pipeline Contract spec](/docs/pipeline-contract.md), [Auth spec](/docs/auth.md)
**Last updated:** 2026-11-02

---

## 1. Purpose

This spec defines the **HTTP surface** of datapipelines.co: every REST endpoint, request/response shape, error format, the SSE event stream for pipeline execution, and the uniform Redis-backed result-delivery cursor.

It is the contract for:
- Browser-based UI (the pipeline editor, dashboard, execution views)
- Direct API clients (.NET services, Python scripts, etc.)
- The MCP server (which is a thin adapter over these endpoints — see [MCP spec](/docs/mcp-server.md))

---

## 2. Design Principles

1. **JSON-first.** Every request and response is JSON unless explicitly otherwise (binary upload, SSE stream, Arrow/CSV result pages).
2. **Envelope consistency.** Every success response uses the same envelope shape. Every error response uses the same error envelope.
3. **SSE for execution, REST for everything else.** Pipeline execution is the only long-running, event-emitting operation. It uses Server-Sent Events. All other endpoints are synchronous request-response.
4. **One result path.** Every completed execution's caller result is materialized in Redis and read through one cursor endpoint (§7). `data_ready` carries the first page inline, so small results still cost a single round-trip. There is no inline-vs-claim-check split.
5. **Idempotency where it matters.** Pipeline execution supports idempotency keys (retries don't re-execute) — see §3.5. Write operations on pipelines and templates do not (each write creates a new version).
6. **Pagination everywhere.** List endpoints paginate. Result data pages through the result cursor (§7).
7. **HTTP status codes used correctly.** 2xx success, 4xx client error, 5xx server error. No overloading.

---

## 3. Common Conventions

### 3.1 Base URL

```
https://{host}/api/v1
```

`{host}` is deployment-specific. Self-hosted deployments expose whatever they configure.

`/api/v1` is the API root. Version is in the path (not header) for simplicity and cacheability.

### 3.2 Authentication

Every `/api/v1/**` endpoint requires authentication via one of:

- **Session cookie** (`dp_session`) — for browser-based UI flows. Set by the OIDC login flow (`GET /oauth2/authorization/{provider}` → callback), not by any REST endpoint. There are no `/auth/login` or `/auth/refresh` endpoints — see [Auth §5](/docs/auth.md#5-oidc-login-flow).
- **API key** — for programmatic clients, in the header `DP-API-Key: dpk_...`: an `endpoint` key or a `server` key, each confined to its own routes (below). The MCP key is refused here.

The permission every operation declares is defined once in the [Auth §7.6 permission catalog](/docs/auth.md#76-operation-matrix--the-permission-catalog-authoritative); what a principal holds is a ROLE (scopes were removed, #215). **This REST API is a SESSION's surface, plus two confined key kinds:** an `endpoint` key on the published tree and its own executions, a `server` key on the promotion routes (§18). The MCP key connects an MCP client to `/mcp` and nothing else — on every route here it is refused with `403 endpoint.key_kind_refused`, `details.reason = "user_key_off_surface"` (B2, [Auth §7.7](/docs/auth.md#77-key-kinds-and-published-endpoint-bindings)). Key management is §16.

### 3.3 Content negotiation

- Default: `application/json`.
- SSE endpoints: `text/event-stream`.
- Binary upload (templates): `multipart/form-data` or `application/octet-stream`.
- Claim-check download: `application/json` (default) or `application/vnd.apache.arrow.ipc` (via `Accept`).

### 3.4 Correlation

Every request may include the `DP-Correlation-Id` header. The server echoes it in the response and includes it in logs. If absent, the server generates one and returns it in the response header. Adoption is conditional on shape: an inbound value that is not a well-formed UUID is replaced with a generated id, because the header is attacker-controlled text that is persisted (UUID column) and echoed on every response (added v1.4).

### 3.5 Idempotency

**Only `POST /pipelines/{id}/execute` supports idempotency** via the `Idempotency-Key` header (a de-facto standard header — deliberately not `DP-`-prefixed). The server caches the response reference for that key + request-hash for `datapipelines.idempotency.ttl-seconds` (default 24h); a retried request with the same key returns the original execution instead of re-executing — including a retry that lands while the original is still inside its start window (its first event not yet written, up to the 10 s lifecycle write bound), which WAITS for it (the attach follows the log for ~15 s) and then streams the original's events; a key whose original was refused before its first event (the per-user slot or any pre-start failure) is released immediately, so the same key re-executes. A follower that attached before that release, or an original that died without releasing, still answers `410 result.expired` with `reason: original_not_started` and NO `execution_id` (an id that does not resolve is never disclosed), while an original whose event log has expired keeps `reason: event_log_expired` with the original's id; a log read Redis does not answer is neither expiry nor absence — the attach answers `503 result.storage_unavailable` with the id, before any stream (§6.1.3, #505). Same key + different body → `409 idempotency.key_reused_for_different_request`.

CRUD writes do NOT accept idempotency keys in v1 — each write deliberately creates a new version, and version history makes accidental duplicates visible and removable.

### 3.6 Custom header registry

All datapipelines custom headers use the `DP-` prefix:

| Header | Direction | Purpose |
|---|---|---|
| 2026-09-25 | v2.34 | 233 landing (#233) the limiter's principal | §12.1: limits are per PRINCIPAL — since keys v2 (A13) an mcp key is its own robot identity, so its request budget is its own, not its creator's (the pre-v2 sentence "an API key draws from its owner's budget, so minting more keys does not raise any limit" stopped being true with V37); `PerUserRateLimitE2eTest` (232) now proves the separation on the wire. No field or code changed. |
| 2026-09-25 | v2.33 | keys v2 (#233) | **§16.1 rewritten**: the ONE creation path for every kind — `kind` REQUIRED (`400 auth.key_kind_not_mintable` with no default kind, A15), `mcp` (renamed from `user`, A19) offered with a CHOSEN role under the subset rule (A14; `mcp_key.create` + `mcp_key.revoke_own`), a live name conflict answered `409 auth.key_name_taken` (A18), `GET /auth/api-keys/mine` gone with the login mint. **§19.3**: the result paging documented under the business path (`GET /api/<cat>/v<n>/<path>/executions/{id}[/result]`, keys-v2 A16) — a session refused there as on the serve route. |
| `DP-API-Key` | request | API-key authentication (§3.2) |
| `DP-Correlation-Id` | both | Log/trace correlation (§3.4) |
| `DP-CSRF-Token` | request | CSRF token for cookie-authenticated state-changing requests ([Auth §8.4](/docs/auth.md#84-api-endpoints-auth-via-api-key-or-jwt)) |
| `DP-Result-TTL-Seconds` | request | Client-requested result TTL, clamped by the server (§7.4) |
| `DP-Result-Page-Rows` | request | Client-requested size of the INLINE first page, clamped to `datapipelines.result.page-max-rows`. **One contract on two surfaces** (ruling R-EP4): `POST /pipelines/{id}/execute`'s `data_ready` and a published endpoint's `200` body (§19) |
| `DP-Execution-Id` | response | The execution a published endpoint's answer belongs to (§19.4) — present on both the `200` and the `202` |
| `DP-Promotion-Key` | request | The promotion peer's pre-shared server key, on `/api/v1/promotion/**` and nowhere else (§18, [Versioning §10.6](/docs/versioning.md#106-the-promotion-peer-credential--a-shared-server-key-ratified-2026-09-01)) |

Standard headers used as-is: `Idempotency-Key`, `Retry-After`, `RateLimit-*` (§12), `Authorization` (Bearer on `/mcp` only — [Auth §8.5](/docs/auth.md#85-mcp-endpoint-mcp)).

---

## 4. Response Envelopes

### 4.1 Success envelope

Every success response (except SSE and raw binary) uses this shape:

```json
{
  "schema_version": 1,
  "correlation_id": "uuid",
  "data": { ... }
}
```

- `schema_version` — response envelope version. Currently `1`.
- `correlation_id` — for log/trace correlation.
- `data` — the operation-specific payload.

### 4.2 Error envelope

Every 4xx and 5xx response uses this shape:

```json
{
  "schema_version": 1,
  "correlation_id": "uuid",
  "error": {
    "code": "pipeline.validation.cycle_detected",
    "message": "Pipeline dependency graph contains a cycle: fetch_orders → revenue → fetch_orders.",
    "user_message": "Your pipeline has a circular dependency. Remove one of the arrows.",
    "details": {
      "cycle_path": ["fetch_orders", "revenue", "fetch_orders"]
    },
    "doc_url": "https://datapipelines.co/docs/pipeline-contract#131-pipeline-validation-write-time"
  }
}
```

- `code` — error code from the [Pipeline Contract §13 catalog](/docs/pipeline-contract.md#13-error-code-catalog). Always lowercase, dot-separated.
- `message` — technical message for developers. English. Includes specifics.
- `user_message` — non-technical message safe to display to end users. May be localized in future.
- `details` — structured, code-specific. Each error code documents its `details` shape.
- `doc_url` — link to the public docs: the [Pipeline Contract §13 catalog](/docs/pipeline-contract.md#13-error-code-catalog) at the anchor of the section listing the code's family (there is no per-code page).

One envelope answers **before any handler**: a request body over the platform cap ([Configuration §3.31](/docs/configuration.md#331-web-request-limits)) is refused at a servlet filter with `413 request.body_too_large` (`details.limit_bytes`) on `/api/v1` and `/mcp` alike — the pre-handler 413 (the parameter engine's evaluate response budget answers 413 too — pipeline-contract §13.20), carried before authentication and before any parser reads the body (pipeline-contract §13.21).

A body **within** the cap that cannot be read — not JSON, or nested past §13.21's stated depth of 100 — is the route family's malformed-body `400` with `details.reason: malformed_json` on EVERY route, including the ones that take their body as text and parse it themselves (#291): the templates routes (create, update, the lifecycle verbs, render, evaluate, import), pipelines create, update, import and the release plan body (#517), execute, and the parameter-set routes (create, update, evaluate, import — #323; the family's code is `parameter.validation.body_invalid`). Until #291 a malformed body on templates create/update/import and pipelines create/update/import reached the `500` backstop, and those routes parsed under Jackson's inherited depth of 1000; the parameter-set routes did the same until #323.

**Scalar shapes (#333).** A body that reads but carries a JSON value of the wrong SHAPE — a number or boolean where the contract declares a string, a string or float where it declares an integer — is refused with the path and the expected shape, never bound as text and never echoing the value (`"name": 12` is no longer the key named "12"). The answer is each family's own: `pipeline.validation.schema_version_unsupported` with `details.reason: "wrong_type"` on a pipeline body, `template.contract_invalid` with `details.rule: "wrong_type"` on a template body or a script block (including the `input` of `…/evaluate`), and `pipeline.execution.invalid_parameter_type` with `details.reason` `wrong_type` / `unknown_key` / `missing` and `details.path` on the five routes that bind a typed request body (`POST /auth/api-keys` §16.1, `POST /endpoints` and `POST /endpoints/bindings` §19.5, `POST /promotion/push` §18.2, `POST /dashboards/bindings` §23.2). Every echoed path or key is clipped to 160 characters. The status stays `400`; see [Pipeline Contract §13.21](/docs/pipeline-contract.md#1321-request-limits).

### 4.3 Pagination envelope

List endpoints return:

```json
{
  "schema_version": 1,
  "correlation_id": "uuid",
  "data": {
    "items": [...],
    "pagination": {
      "offset": 0,
      "limit": 50,
      "total": 237,
      "has_more": true
    }
  }
}
```

Query parameters: `?offset=0&limit=50`. Max `limit` is 200 (configurable).

---

## 5. Pipeline Endpoints

**The promoter lens (178).** Every read in this section — §5.2, §5.3, §5.4, §5.7 (both shapes),
§5.9 and the checks read — answers a **promoter** (session or key) through the lens ([Auth §11A.1](/docs/auth.md#11a1-the-404-rule)):
only pipelines that are RELEASED and newer than the promotion target's inventory entry
([Versioning §10.2](/docs/versioning.md#102-the-listing-rule-what-the-ui-shows)) exist for it. A hidden
pipeline is `404 pipeline.execution.not_found` on every by-id read, exactly as an id from another
workspace is — and so is a DRAFT version of a visible one, on §5.3's version read and on the
version's checks read (178b: a status is never probeable by number); the listings, the folder
level and its `total` and `pipeline_count` are the lensed set. While the promotion target cannot be read the lens fails closed — empty listings, 404s —
never a `502` on a read. Every other role reads as before.

### 5.1 Create pipeline

```
POST /pipelines
Content-Type: application/json

{
  "schema_version": 1,
  "name": "acme/finance/monthly_revenue",
  "display_name": "Monthly Revenue",
  "description": "...",
  "parameters": {...},
  "settings": {
    "tempdb": {"engine": "H2"}
  },
  "nodes": [
    {
      "id": "fetch_orders",
      "type": "DQL",
      "source": "pg-prod",
      "template": {"id": "acme/finance/fetch_orders.sql", "version": 2},
      "output": {"target": "tempdb", "table": "stg_orders"},
      "depends_on": []
    },
    ...
    {
      "id": "final_report",
      "type": "DQL",
      "source": "tempdb",
      "template": {"id": "acme/finance/final_report.sql", "version": 1},
      "output": {"target": "caller"},
      "depends_on": ["revenue_by_customer"]
    }
  ]
}
```

Note: no `terminal_node_id` field. The result node is the (at most one) node resolving to `output.target: "caller"` — explicitly or by omitting `output`. See [Pipeline Contract §9](/docs/pipeline-contract.md#9-the-caller-node-result-node).

**`name` is a folder path** (since 2026-09-05, 067; a folder is mandatory since 077): **2–10** `/`-separated segments, each `[a-z0-9][a-z0-9_.-]{0,63}`, ≤ 200 chars total — the same grammar template ids take ([Pipeline Contract §3.2](/docs/pipeline-contract.md#32-field-reference), [Template Hierarchy §14](https://github.com/msabiransari/datapipelines/blob/main/docs/template-hierarchy-design.md)). `finance/payments/daily_settlement` is valid; a bare `monthly_revenue` is not — the tree root holds folders only, and an experiment goes under `test/`. Folders are virtual — there is no folder resource, no folder id and nothing to create — and a name is immutable, so the folder is chosen once, at creation. A malformed name is `400 pipeline.validation.name_invalid`, whose `details.reason` is `folder_required` when a folder is the only thing missing and `grammar` otherwise.

**This changes no route.** A pipeline is addressed by UUID everywhere (`/pipelines/{id}`), so the encoded-slash problem that forced templates' §8 dual addressing ([Template Hierarchy §9.6](https://github.com/msabiransari/datapipelines/blob/main/docs/template-hierarchy-design.md)) cannot arise here: no pipeline route carries a name in a path segment.

Response: `201 Created`

```json
{
  "schema_version": 1,
  "correlation_id": "uuid",
  "data": {
    "id": "pipeline-uuid",
    "version": 1,
    "name": "acme/finance/monthly_revenue",
    ...
  }
}
```

Server assigns: `id`, `version` (starts at `1`), `owner` (from auth), `created_at`, `updated_at`.

**Version 1 lands as a DRAFT** (versioning §3.2, ruling D55, 2026-09-08). The response carries
`status: "DRAFT"`, `version: 1`, that version's `body_hash` (the precondition token for the next
`PUT`), `current_version: null` — nothing is released yet — and the `draft` pointer, exactly as a
`PUT` response does. Releasing it is `POST /pipelines/{id}/release` (§5.10) like any other draft,
and it is a human action (D4). The pipeline is executable immediately regardless: `execute` with no
`version` runs the working version (§6.1).

The two create paths that are NOT authoring keep landing RELEASED: `POST /pipelines/import` (§5.8,
promotion) and the seeders that ride it. `POST /templates` (§8.1) mirrors this section exactly.

### 5.2 Get pipeline (working version)

```
GET /pipelines/{id}
```

Response: `200 OK` with full pipeline JSON (including server-assigned fields). The default
body is the **working version** (versioning §7, since 039): the DRAFT when one exists, else
the current released version — authoring reads must show the draft, or an editor rebases on
released content and quietly discards it. The response states which `version` and `status`
it returned, carries that version's `body_hash` (the precondition token for the next
write), `current_version` (the latest RELEASED version — the execute-default pointer,
unmoved), and a `draft` pointer — `{version, body_hash, updated_by, updated_at}` — when a
draft exists. Since 078 the body's `parameters` also lists the pipeline's **derived execute
inputs** — one entry per key a CALCULATOR node writes (121: every mapped `context_keys`
value of a multi-output kind too),
`{"type": <the key's output wire type, or "ANY">, "required": false, "derived": true}` — because a
calculator key is an implicit optional input of the execute endpoint (pipeline-contract
§4.10: supply it and the node is skipped). Declared parameters carry no `derived` flag;
derived on read, never stored.

### 5.3 Get pipeline (specific version)

```
GET /pipelines/{id}/versions/{version}
```

### 5.4 List pipeline versions

```
GET /pipelines/{id}/versions
```

Returns metadata only (no body JSON) for each version: `version`, `status`
(`DRAFT`/`RELEASED`/`DISCARDED`), `body_hash`, `created_at`, `created_by`, `released_at`.

### 5.5 Update pipeline (writes the draft)

```
PUT /pipelines/{id}
If-Match: <body_hash of the version this edit is based on>
Content-Type: application/json

{full pipeline body, excluding server-assigned fields}
```

Since the version lifecycle (versioning §3.2/§5) a PUT **always writes the DRAFT branch**:
the first write after a release copies the released version to a draft (`version: N+1`,
`status: "DRAFT"`); later writes overwrite that same draft in place. A PUT never appends a
released version. The `If-Match` header carries the hash precondition (§4.2 of versioning):
the DRAFT's hash for an in-place write, the current RELEASED row's hash for a first write —
absent/blank is `400 pipeline.execution.invalid_parameter_type` with
`details.reason = "precondition_missing"`; stale is `409 pipeline.version.conflict` with the
current hash/author in `details`. A draft renaming onto a taken name fails HERE with
`pipeline.validation.duplicate_name` (versioning §3.5), not at release.

Response: `200 OK` with the draft version (`version`, `status: "DRAFT"`, `body_hash`,
`current_version` — the released pointer, unmoved). A PUT whose body is identical to the
released one is a **no-op** (versioning §5.1): no draft is created, no version number is
consumed, and the response reports the current state — `status: "RELEASED"`, the released
version's `body_hash`, no `draft` pointer. Not a 4xx: the write was well-formed and the
outcome is "already in that state".

### 5.10 Release pipeline

```
POST /pipelines/{id}/release
If-Match: <the draft's body_hash>
[?override_checks_reason=<text — only when a check is failing>]
[?release_pinned_templates=true — the legacy consent: release the DRAFT template versions the body pins directly]
[Content-Type: application/json — optional release plan body (#517):]
[{"tree_hash": "<preview digest>", "include_dependencies": ["n1", "n2"], "check_overrides": [{"node_id": "n1", "reason": "…"}]}]
```

Locks the draft: the version flips to RELEASED, `pipelines.current_version` moves to it,
and the index row's name/display_name/description adopt the released body's values
(metadata rides the release — versioning §3.5). Preconditions evaluated server-side:
§12 re-validation on the draft body; every pinned template version RELEASED
(`409 pipeline.release.dependency_not_released` naming it); the hash guard
(`409 pipeline.version.conflict`); no draft at all (`409 pipeline.version.not_draft`);
and — when the body carries [`checks[]`](/docs/pipeline-contract.md#33-release-checks-checks) —
**the release-check gate** (versioning §5.3 precondition 4): a fresh check run whose every
verdict must be `pass`, else `409 pipeline.check.failed` with `details.checks` (expected,
observed and message per failing check). Releasing past a failing check needs
`override_checks_reason`, a non-blank reason of ≥ 10 characters, and is recorded:
`pipeline.version.released` gains `checks_overridden: [ids]` and `override_reason`. A
version with no checks releases exactly as before.
**Dependencies (142, #517, versioning §5.3):** the release takes the draft's whole release closure into
account — every template version it pins and every child pipeline version a PIPELINE node pins, transitively,
within the configured bounds ([configuration §3.35](/docs/configuration.md#335-release-dependencies)). A release flips
only RELEASED-ready work: every dependency must already be RELEASED or be flipped by the same request.
Anything else refuses `409 pipeline.release.dependency_not_released` before anything is written, in the
**shared blockers shape** every family's `<family>.release.dependency_not_released` uses (pipeline-contract
§13, release-dependencies record §14):

```json
{"root": {"node_id": "n0", "family": "pipeline", "name": "acme/daily", "version": 3},
 "blockers": [{"node_id": "n2", "family": "template", "name": "acme/one.sql", "version": 2,
               "status": "DRAFT", "reason": "not_included", "paths": ["nodes[1].template"]}],
 "blockers_truncated": false}
```

`node_id` is an opaque locator inside this one closure snapshot, never an entity id; `status` is `DRAFT`,
`DISCARDED` or `MISSING`; `reason` is `not_included` (a DRAFT the request did not consent to), `discarded` or
`missing`, and on a restore's refusal (§5.13) `not_released` (a DRAFT a restore cannot include); `paths` names the pin fields that reach the node (`nodes[1].pipeline/nodes[0].template`). A pin that is
absent and a pin the caller may not see are the same `MISSING` blocker, byte for byte.
At most `max-blockers` are listed in closure discovery order; the rest set `blockers_truncated`.

Three request shapes consent to flipping dependencies:

- **No plan, no flag** — only the pipeline flips; every dependency must already be RELEASED.
- **`release_pinned_templates=true`, no plan (the legacy alias)** — the root's DIRECT DRAFT template pins are
  released at their pinned versions in the SAME transaction as the pipeline flip, templates first. A deeper
  DRAFT (a child pipeline, a child's template) is a `not_included` blocker. The alias is admitted without a
  plan until the release dialog sends `tree_hash` (#522); a successful response carries release 142's bytes.
- **A plan body** — `tree_hash` is the preview digest of the closure the caller reviewed, and
  `include_dependencies` lists the DRAFT nodes it consents to flip (each a node of that snapshot, none twice,
  never the root, only DRAFTs). Every reachable DRAFT must be selected. `check_overrides` gives a node its own
  release-check override reason (the root's may instead ride `override_checks_reason`, never both). The digest
  is compared BEFORE any validation or check work and again inside the transaction under the workspace
  writer guard; any drift between preview and submit — a new pin, a status change, a body changed in place —
  refuses `409 pipeline.release.plan_conflict` and nothing flips. The caller must hold each flipped family's
  release permission (auth §7.6). The response appends `dependencies_released: [{family, name, version}]` in
  release order; a request without a plan never carries the field.

A body whose bytes cannot be read — not JSON, or nested past the request bound — is the pipeline family's
malformed-body 400 (`details.reason: malformed_json`, [§13.21](/docs/pipeline-contract.md#1321-request-limits)), refused before any lookup like
every string body. Plan refusals: `428 pipeline.release.plan_required` — a plan field without `tree_hash`; `400
pipeline.release.plan_invalid` — a body that is not a JSON object, a field the plan does not have, a node id that
is not a DRAFT dependency of the snapshot, a duplicate, an override for a node the plan does not flip, or the
alias together with `include_dependencies` (`details: {field, reason}`, the field a locator such as
`include_dependencies[2]`, never the submitted value); `422 pipeline.release.dependency_limit` — the closure or
the body exceeds a configured bound (`details: {limit, maximum}`, `limit` one of `depth`, `versions`, `edges`,
`wire_bytes`); `409 pipeline.release.dependency_cycle` — the pins form a cycle (`details.path`, the visible
nodes' identities); `409 pipeline.release.busy` — another write in the workspace held the writer guard longer
than `lock-wait-seconds` (`details.lock_wait_seconds`; nothing was written, retry). Every metadata write that
changes what a release resolves — a save, a discard, a restore, a purge, a manual switch, an import, a
visualization test session's evidence — takes the same workspace writer guard first, so each can answer its own
family's `<family>.release.busy` the same way ([versioning §5.3](/docs/versioning.md#53-release-lock)).

Every release — with or without a plan — is ONE metadata transaction: the dependencies flip first (each at the
hash the closure read), the pipeline last, and one `*.version.released` audit row per actual flip is written
in that same transaction, strictly (an audit store that refuses the row refuses the release). Any refusal —
a dependency's own preconditions, a stale `If-Match`, a concurrent write, the audit row — rolls the whole
release back. The audit rows of one release share `release_id`, carry `scope: "version"`, the cascaded rows
`cascade_root: {family, id, version}`, and the pipeline's row `dependencies_released`; the 142 keys stay —
each cascaded template row keeps `cascade_from_pipeline_id` / `cascade_from_version`, and the pipeline's row
`templates_released: [{template_id, version}]` (empty when nothing cascaded). The same role holds both release
verbs (auth §7.6), so neither the flag nor a plan grants a new capability.
UI-driven in practice — agents never release (versioning D4); no MCP tool exists. Over REST
the verb needs `pipeline.release` — an author's or workspace admin's session (release is the
author's since 2026-09-20, D8); no key reaches it (the MCP key is confined to `/mcp` since #215
B2). **Audited** as `pipeline.version.released` with the version and `via` (`session` /
`api_key`) — the record of who made the D4 decision and through what (T187).

Response: `200 OK` with the released version's full shape (`status: "RELEASED"`) **plus
`warnings`** (7e, [transform-nodes design](https://github.com/msabiransari/datapipelines/blob/main/docs/superpowers/specs/2026-09-09-transform-nodes-design.md) §8.2) —
ALWAYS present, `[]` on a clean release, one `{code, message, template, version}` per pinned
template version that reads `needs_review` (it cites a retired learned fact, [Templates §3.4](/docs/templates.md#34-implements-and-drift)):

```json
"warnings": [
  { "code": "pipeline.release.template_needs_review",
    "message": "Template 'nyc/weather/rainy_days.jsonata' v2 cites a retired fact: 1b2e… — superseded by 7c4a…. Released anyway — re-cite the successor (or drop the citation) with templates_update.",
    "template": "nyc/weather/rainy_days.jsonata", "version": 2 }
]
```

A warning never refuses and never changes the status: the release happened, the fact edit is
the releaser's to know about ([Pipeline Contract §13.13](/docs/pipeline-contract.md#1313-versioning--draft-release-lifecycle--promotion)
catalogues the code with HTTP `—`). The release dialog shows the same facts on its pin rows.

**`caller_output` (#328, additive):** the body also carries the one word
`recorded` | `declared` | `none` | `not_observed` — what the release did about its caller
node's result columns ([pipeline-contract §3.3.1](/docs/pipeline-contract.md)): `recorded` — the
flip copied the version's latest run's schema; `declared` — a script caller, the pinned
contract is the answer; `none` — no caller node; `not_observed` — a SQL caller with nothing
observed, the release succeeded anyway and a dashboard save skips the input-contract check
for it (dashboards §4.3). The audit row carries the same word, never the column list.

### 5.11 Purge pipeline draft

```
POST /pipelines/{id}/draft/discard
If-Match: <the draft's body_hash>
```

**Purges** the draft (101: the route keeps its historical spelling for the editor's
button): the row is hard-deleted **together with its executions** — development runs of a
thing that never shipped are not history — and when it was the sole version the entity row
goes too. The number returns to the pool (nothing surviving outranks it). **Session-only**
(an API key is refused `403 auth.session.required`); audited as `pipeline.version.purged`.

Response: `204 No Content`. Errors as §5.10 (`not_draft` / `version.conflict`; a stale hash
refuses before the pin check), plus `409 pipeline.version.pinned`: live parent pipeline versions
pinning the exact draft, and live dashboard versions pinning it. A sole-draft purge takes the
entity and checks stored dashboard pins to any version (DISCARDED included for a dashboard with
a live version); parent scope remains live/exact. `details.pinned_by` names `{pipeline, version, node}`;
`details.referencing_dashboards` names `{dashboard, version, status}` when non-empty. Refusal
precedes all deletion and preserves executions, hash and pointer.

### 5.12 Discard pipeline version

```
POST /pipelines/{id}/versions/{v}/discard
```

Discards RELEASED version v (101): the row flips to `DISCARDED` — reversible via §5.13,
its executions and promotion history stay — and the sticky pointer recomputes only when v
WAS the pointer (D60: highest eligible live version, else NULL; a DRAFT is eligible only
under development posture). Refusals: `409 pipeline.version.not_released` (a draft is
purged, never discarded; an already-discarded version needs restore), `409
pipeline.version.pinned` (a live parent version — or, since #320, a live dashboard version whose source — exact-pins v; `details.pinned_by` names the pipelines, `details.referencing_dashboards` the dashboards), `404
pipeline.execution.not_found` (unknown version), `403 pipeline.authoring.disabled`
(hardened). **Session-only**; audited as `pipeline.version.discarded` (pointer
before/after in `details`).

Response: `200 OK` with the pipeline's full shape at the new pointer.

### 5.13 Restore pipeline version

```
POST /pipelines/{id}/versions/{v}/restore
```

Returns DISCARDED version v to RELEASED (101): the original `released_at`/`released_by`
return untouched, the discard stamps clear, and the pointer moves only when v > current or
current is NULL. `409 pipeline.version.not_discarded` otherwise. Under the workspace writer guard the
version's outbound pins are re-judged first (#517, versioning §3.5.3): a missing or DISCARDED template or
child pin is `400` with §12's validation code (`template_not_found`, `template_version_not_found` with
`reason: discarded`, `pipeline_not_found`, `pipeline_version_not_found`, `pipeline_reference_not_released`),
a present-but-DRAFT one `409 pipeline.release.dependency_not_released` in §5.10's blockers shape
(`reason: not_released`); a refusal writes nothing. **Session-only**; audited as `pipeline.version.restored`.

Response: `200 OK` with the pipeline's full shape.

### 5.14 Purge pipeline version

```
DELETE /pipelines/{id}/versions/{v}
```

**Purge, drafts only** (101): the version row and its executions are deleted,
irreversibly; the sole-draft case takes the entity with it. A RELEASED target is `409
pipeline.version.last_release` — a release is never purged; discard is per version and
reversible. **Session-only**; audited as `pipeline.version.purged`.

Response: `204 No Content`. Also refuses `409 pipeline.version.pinned` with the named parent
pipelines and dashboards and the same scope as §5.11: live/exact for a nonsole draft; for a
sole draft live/exact parents plus stored any-version dashboard pins. All rows and executions
remain untouched on refusal. An unknown target is `404 pipeline.execution.not_found`; a
DISCARDED target also refuses `pipeline.version.last_release`; authoring-disabled and
session/key/role restrictions are unchanged.

### 5.15 Switch pipeline current version

```
POST /pipelines/{id}/current
{"version": 3}
```

The **manual switch** (101, D60): `current = version`, which must be a live,
posture-eligible version (`409 pipeline.version.not_eligible` for a DISCARDED target or a
DRAFT under a hardened posture). NOT authoring-gated — this is the promotion receiver's
rollout/rollback lever — but still **session-only**; audited as
`pipeline.current_switched` (from/to in `details`).

Response: `200 OK` with the pipeline's full shape at the new pointer.

### 5.16 Release checks (140)

```
POST /pipelines/{id}/versions/{version}/checks/run
GET  /pipelines/{id}/versions/{version}/checks
```

A pipeline body may carry [`checks[]`](/docs/pipeline-contract.md#33-release-checks-checks) —
server-run cross-checks that gate §5.10's release. The agent writes the query and the
expectation; **only the server's own run produces `observed`** — there is no endpoint,
and no tool, that records an observed value from a caller.

**Run** (`pipeline.run_checks` — it writes `pipeline_check_runs` rows, like execute writes
executions): runs the version's checks NOW against their datasources, bound with the
pipeline's declared parameters (defaults filled the way §6's execute fills them — the
calculator context is NOT available to a check), through the same bounded probe path as
`sql_probe` (`via = rest`). An optional `parameters` JSON body overrides the defaults for
this run. Response `200 OK`:

```json
{ "version": 3,
  "runs": [ { "check_id": "manhattan_share_reconciles", "name": "Manhattan rideshare share, Q4 2024",
              "expected": { "kind": "value", "value": 74.62, "tolerance": 0.01 },
              "observed": "74.62", "verdict": "pass", "message": null, "ran_at": "…" } ] }
```

`verdict` is `pass` | `fail` | `error` (enums.md §20): `error` means no verdict could be
formed — the datasource was unreachable, the statement was refused, or it returned a shape
the expectation cannot compare (a `value`/`range` check requires exactly one row and one
column) — with the reason in `message`. A version with no checks answers `200` with an
empty `runs` list.

**Latest** (`pipeline.read`): the version's check definitions each with their LATEST run
(latest per `(version, check_id)` of the append-only `pipeline_check_runs`), or `null`
runs when none was ever recorded. This is what the UI's release dialog and version page
read.

### 5.6 Delete pipeline (the entity purge)

```
DELETE /pipelines/{id}?include_exclusive_draft_templates=false
```

**The entity purge** (101 — replaces the V1 soft delete, retired in V19): allowed only
when the pipeline's ONLY version is a DRAFT (`409 pipeline.version.last_release`
otherwise); the entity row, the draft and its executions go, irreversibly.
`include_exclusive_draft_templates=true` also purges the draft-only templates this
pipeline exclusively pins; the response carries the offered set either way, so the UI can
show the cleanup. **Session-only**; audited as `pipeline.purged`.

Response: `200 OK` with `{"purged": true, "exclusive_draft_templates": [...],
"exclusive_draft_templates_purged": bool, "kept_draft_templates": [...]}`. Since #320
"exclusive" spans every aggregate that pins templates: a draft-only template that a parameter
set or a visualization ALSO pins is never offered and never purged — `kept_draft_templates` lists
each one as `{"id", "referencing_parameter_sets"?, "referencing_visualizations"?}` (the keys
`template.in_use` uses), so the caller sees what the purge left and why
([Versioning §3.5.3](/docs/versioning.md#353-the-reverse-arrows-into-other-families-320)).

### 5.7 List pipelines

```
GET /pipelines?owner={user-id}&datasource={name}&q={search}&offset=0&limit=50
```

Filters:
- `owner` — limit to pipelines owned by user.
- `datasource` — limit to pipelines using this datasource name.
- `q` — full-text search on name, display_name, description. Since names are paths, `q` matches across the **full path**: `q=finance/pay` finds `finance/payments/daily_settlement`. `q` is trimmed; blank (or whitespace-only) means no search. A term over 200 characters after the trim is refused `400 request.query_too_long` (#504b, [pipeline-contract §13.21](/docs/pipeline-contract.md)) with `details.limit` and `details.length`, before any read; the term is never echoed.
Each row carries `version` — the **working** version, the draft's number when the pipeline has a
draft and the latest released otherwise — and `status` (`DRAFT` or `RELEASED`), because since D55 a
freshly authored pipeline has no released version to name (both are `null` for a pipeline whose sole
draft was discarded).

- `prefix` — browse ONE level of the folder tree instead of listing flat (067; same contract as `pipelines_list {prefix}`, [MCP §6.2.1](/docs/mcp-server.md)). Present-but-empty (`?prefix=`) is the ROOT. The `data` payload becomes `{prefix, folders, pipelines, total, has_more}`: `folders` lists the prefix's direct sub-folders as `{path, segment, pipeline_count}` (subtree counts), `pipelines` its direct leaves as the same metadata rows as the flat list, `total`/`has_more` page the leaves via `offset`/`limit`. An unknown or illegal prefix answers an EMPTY level with `200` — never a `400`, never a query error. `owner`/`datasource`/`q` are ignored while `prefix` is present: browse and search are different presentations.

### 5.8 Import pipeline

```
POST /pipelines/import
Content-Type: application/json

{full pipeline JSON, possibly including id}
```

Response: `201 Created` or `200 OK` (if updating existing pipeline).

Fails with `pipeline.import.missing_datasource` or `pipeline.import.missing_template` if dependencies unmet.

**Preserved-version import** (versioning §9.2, D5 — version numbers are global identities
and imports never renumber): a payload carrying `version` (export and promotion send it)
is honored at that EXACT version. `body_hash` must ride along and is recomputed from the
payload body — a mismatch (or a missing declaration) is
`409 pipeline.import.hash_mismatch`. Target-side rules: absent ⇒ insert as RELEASED at that
version (`released_at` from the payload's, if present); present+RELEASED+same hash ⇒
idempotent `200`; present+RELEASED+different hash / present as DRAFT / present as DISCARDED
⇒ `409 pipeline.import.version_conflict` with both hashes and the target status in
`details`. A version-less payload keeps the allocate-next behavior above.

### 5.9 Export pipeline

```
GET /pipelines/{id}/export?include_templates=true
```

Returns a bundle:

```json
{
  "schema_version": 1,
  "correlation_id": "uuid",
  "data": {
    "pipeline": {full pipeline JSON},
    "templates": [
      {full template JSON for each referenced template version}
    ],
    "manifest": {
      "pipeline_id": "...",
      "pipeline_version": 3,
      "template_count": 4,
      "exported_at": "..."
    }
  }
}
```

`include_templates=true` is the default. Set `false` to export the pipeline only.

**Released only.** An export is what a promotion import consumes, and §9.2's import lands RELEASED
content, so a pipeline with no released version (D55: every freshly created one) is refused with
`409 pipeline.promotion.not_released` — "release it from the UI first" — rather than exporting a
draft or 404-ing on an absent body.

The exported pipeline object carries its lifecycle fields (`version`, `status`,
`body_hash`, `released_at`) so a preserved-version import on the target can verify and
re-stamp them; bundled template versions carry their `version`, `status` and `body_hash`.

**The bundle is the pipeline's pins, not lensed; the root is** (the owner's ruling of 2026-10-02, #344,
[Auth §11A.1](/docs/auth.md#11a1-the-404-rule)'s lens clause). The promoter lens decides whether the pipeline exports at all — a
pipeline it hides is the absent `404` — but never what the bundle carries: a pinned template it hides still rides.

---

## 6. Pipeline Execution — SSE

### 6.1 Endpoint

```
POST /pipelines/{id}/execute
Authorization: ...
Content-Type: application/json
Accept: text/event-stream
Idempotency-Key: ...   (strongly recommended)

{
  "version": 3,                    // optional; defaults to the WORKING version (§6.1.1)
  "parameters": {                  // required values per pipeline.parameters schema
    "start_date": "2026-01-01",
    "end_date": "2026-03-31",
    "include_cancelled": false
  }
}
```

#### 6.1.1 The default version

With no `version` in the body, the execution runs the **working version** (versioning §7.2, ruling
D56): the DRAFT when one exists, else the latest RELEASED version — "always run the last version".
On a development server that may be a draft (drafts have been executable since 039); where
`authoring-enabled=false` no draft can exist at all, so it is always a release
([environments.md](/docs/environments.md)). An explicit `version` is exact and is never clamped: an
unknown number is `404 pipeline.execution.not_found` (`details.pipeline_version` names it), not a
silent run of the latest.

#### 6.1.2 The body is read before the pipeline is looked up

Since #291 the whole body is read and checked before anything is looked up, the pipeline included: a
body that is not JSON, or nests past pipeline-contract §13.21's depth, is `400
pipeline.execution.invalid_parameter_type` with `details.reason: malformed_json` whatever the id —
so it reveals nothing about which pipelines exist, and costs no query. A JSON **number** whose
magnitude no plain decimal rendering can carry (`1e10000` — the execution echoes its parameters as
plain decimals, and Jackson refuses a scale past ±9999) is the same `400` with `details.parameter`
naming it and `details.reason: too_many_digits`; send a large value as a string within
pipeline-contract §6.3's digit cap. (Until #291 that number answered `500`.)

The execution record pins the version that actually ran, and every executions surface marks a draft
run — REST `draft_run` (§10.2), and a `DRAFT` label on the executions list and detail screens
([ui-screens §4.8](/docs/ui-screens.md)). The one pipeline that cannot be run at all is one with no
version, reachable only by discarding the sole draft of a never-released pipeline (versioning §3.4);
it answers that same `404 pipeline.execution.not_found`.

Response: `200 OK` with `Content-Type: text/event-stream`.

The response is a **stream of SSE events**, one per execution milestone. The stream closes when execution completes (success or failure).

#### 6.1.3 Refusals before the first event

A refusal decided before the first event is never a stream: it answers the §4.2 JSON envelope
(`Content-Type: application/json`) with its own status, also to a client whose `Accept` is
`text/event-stream` only. Most are decided by the handler itself (the `400`s of §6.1.2, the `404`s
of §6.1.1). Two are decided AFTER the handler has opened the stream,
off the request thread, still before any byte is sent — and answer exactly the same way:

| Status | Code | When |
|---|---|---|
| `429` | `pipeline.execution.concurrency_limit` | The executor has no execution slot for the caller (`details.scope`: `per_user` or `global`, `details.limit`; §12.1). The reservation of an `Idempotency-Key`, if one was sent, is released before this response (§3.5, #403), so the same key re-executes once a slot is free; only a retry that attached before the release meets the next row. |
| `410` | `result.expired` | An idempotent retry (§3.5) attached to an original that never started: no event appeared within the follow's patience (~15 s). `details.reason: original_not_started` and NO `execution_id` — that id never resolves. Re-execute with a fresh `Idempotency-Key`. |
| `503` | `result.storage_unavailable` | An idempotent retry (§3.5) whose follow met eight consecutive event-log reads Redis did not answer (~18 s at the 2 s command timeout) before serving an event (#487). `details.reason: event_log_unavailable`, NO `execution_id` (a fault cannot say whether it resolves). Retry with the same key. A follow that already served events ends instead with a final `event_log_unavailable` SSE comment — the status is spent once a frame went out. Faults never count toward the row above's patience. Since v2.93 the attach's own first read answers this row too (#505): a fault there throws the same `503` WITH the `execution_id` (the idempotency reservation already handed the retry caller the id), before any stream is committed — a terminal and a running original alike; only an ANSWERED empty read still consults the execution row, which keeps the two `410`s (the original's `event_log_expired`, §3.5, and the never-started row above). |

Until v2.80 both answered `401 auth.api_key.missing` ("No credentials provided") to a fully
authenticated caller: the stream's completion re-enters the server as an async dispatch, and the
authorization interceptor judged that dispatch against its empty security context (#404,
[auth §8.2](/docs/auth.md#82-filter-order-per-request)). A refusal after the first event is not an
HTTP status at all — it is the stream's own terminal event (§6.5).

### 6.2 Why SSE, not WebSocket

- SSE is unidirectional server → client, which is exactly what we need (the client sends one request, the server streams progress).
- SSE works over standard HTTP — proxies, load balancers, auth headers all work natively.
- WebSocket would require a custom protocol, custom proxy config, custom auth. Overkill.

Note: stream *resumption* (`Last-Event-Id`) is NOT supported — a dropped stream means the execution will be cancelled after the disconnect grace period (§6.8).

### 6.3 SSE event format

Each event follows the SSE wire format:

```
event: {event_type}
id: {event_id}
data: {json_payload}

```

(Terminated by blank line.)

`event_id` is monotonic per execution. Used for gap detection (§6.7).

### 6.4 Event types

Every event payload additionally carries `correlation_id` (normative — [Observability §3.3](/docs/observability.md#33-correlation-id-propagation)); the examples below omit it for brevity except where shown.

#### 6.4.1 `execution_started`

Emitted once at the start.

```json
{
  "execution_id": "exec-uuid",
  "pipeline_id": "pipeline-uuid",
  "pipeline_version": 3,
  "started_at": "2026-08-05T14:30:00.123Z",
  "correlation_id": "uuid",
  "parameters": {"start_date": "2026-01-01", ...}
}
```

#### 6.4.2 `node_started`

Emitted when a node begins executing (after its dependencies completed).

```json
{
  "execution_id": "exec-uuid",
  "node_id": "fetch_orders",
  "started_at": "2026-08-05T14:30:00.234Z",
  "attempt": 1
}
```

#### 6.4.3 `node_completed`

Emitted when a node finishes successfully.

```json
{
  "execution_id": "exec-uuid",
  "node_id": "fetch_orders",
  "started_at": "2026-08-05T14:30:00.234Z",
  "completed_at": "2026-08-05T14:30:01.500Z",
  "duration_ms": 1266,
  "rows_out": 12453,
  "bytes_out": 4567890
}
```

A `PIPELINE` node's `node_completed` additionally carries `"child_execution_id": "exec-uuid"` — the execution its child ran as — so a stream consumer can follow the link to the child's own stream and record (composition, [DAG Executor §6.6](/docs/dag-executor.md#66-pipeline-composition-direct-delivery-slots-and-cancellation)). The field is absent for every other node type. The same value appears in the node's `node_stats` entry on the terminal events.

#### 6.4.4 `node_failed`

Emitted when a node fails. Execution then halts (fail-fast); a `pipeline_failed` event follows.

The `error` object is the **failure record** (057/T85): one object, completed once at the failure site, carried unchanged into this event, the terminal `pipeline_failed` event, and `pipeline_executions.error_json` — so the live stream, `GET /executions/{id}` and the execution detail page all show the same thing. `correlation_id` matches the top-level stamp. `node`, `sql` and `exception` are present under `datapipelines.executions.error-detail=full` (the default) and omitted under `structured`; the SQL is always the rendered statement in `:name` form — bound parameter **values never travel in the error object** (a driver message may echo a value; that is the driver's text and ships as-is, exactly as the log prints it). The `caused_by` chain is outermost-first, **root cause LAST**; frames are capped at 40 per level and the chain at 16 levels.

```json
{
  "execution_id": "exec-uuid",
  "node_id": "fetch_orders",
  "correlation_id": "corr-uuid",
  "started_at": "2026-08-05T14:30:00.234Z",
  "failed_at": "2026-08-05T14:30:00.500Z",
  "duration_ms": 266,
  "error": {
    "code": "pipeline.node.datasource_connection_failed",
    "message": "Could not acquire connection to 'pg-prod': Connection refused.",
    "user_message": "We couldn't reach the 'pg-prod' database. Check that the database is online and reachable from this server.",
    "details": {
      "datasource_name": "pg-prod",
      "underlying_error": "java.net.ConnectException: Connection refused"
    },
    "doc_url": "https://datapipelines.co/docs/pipeline-contract#134-node-execution",
    "correlation_id": "corr-uuid",
    "node": {
      "id": "fetch_orders",
      "type": "DQL",
      "datasource": "pg-prod",
      "dialect": "POSTGRES",
      "template": "acme/finance/fetch_orders.sql",
      "template_version": 4
    },
    "sql": "SELECT * FROM orders WHERE placed_on >= :start_date",
    "exception": {
      "class": "java.sql.SQLException",
      "message": "Connection refused",
      "frames": ["org.postgresql.Driver.connect(Driver.java:262)", "..."],
      "caused_by": [
        {"class": "java.net.ConnectException", "message": "Connection refused", "frames": ["..."]}
      ]
    }
  }
}
```

#### 6.4.5 `pipeline_completed`

Emitted when execution finishes successfully, immediately before `data_ready`.

```json
{
  "execution_id": "exec-uuid",
  "pipeline_id": "pipeline-uuid",
  "pipeline_version": 3,
  "started_at": "2026-08-05T14:30:00.123Z",
  "completed_at": "2026-08-05T14:30:02.500Z",
  "duration_ms": 2377,
  "status": "SUCCESS",
  "node_stats": [
    {"node_id": "fetch_orders", "duration_ms": 1266, "rows_out": 12453, "bytes_out": 4567890, "status": "SUCCESS"},
    {"node_id": "fetch_customers", "duration_ms": 850, "rows_out": 5400, "bytes_out": 1200000, "status": "SUCCESS"},
    {"node_id": "revenue_by_customer", "duration_ms": 200, "rows_out": 4500, "bytes_out": 800000, "status": "SUCCESS"},
    {"node_id": "final_report", "duration_ms": 60, "rows_out": 4480, "bytes_out": 780000, "status": "SUCCESS"}
  ]
}
```

#### 6.4.6 `pipeline_failed`

Emitted when execution halts due to any node failure. Its `error` object is the SAME failure record `node_failed` carried for the failed node (§6.4.4), unchanged — and is also what `pipeline_executions.error_json` stores, so `GET /executions/{id}` and the execution detail page show it after the fact. A timeout or setup failure carries the same shape minus `node`/`sql` (neither exists for it).

```json
{
  "execution_id": "exec-uuid",
  "pipeline_id": "pipeline-uuid",
  "pipeline_version": 3,
  "started_at": "2026-08-05T14:30:00.123Z",
  "failed_at": "2026-08-05T14:30:00.500Z",
  "duration_ms": 377,
  "status": "FAILED",
  "failed_node_id": "fetch_orders",
  "error": {
    "code": "pipeline.node.datasource_connection_failed",
    "message": "...",
    ...
  },
  "node_stats": [
    {"node_id": "fetch_orders", "duration_ms": 266, "rows_out": 0, "status": "FAILED", "error_code": "..."},
    {"node_id": "fetch_customers", "duration_ms": 0, "status": "ABORTED"},
    {"node_id": "revenue_by_customer", "duration_ms": 0, "status": "ABORTED"},
    {"node_id": "final_report", "duration_ms": 0, "status": "ABORTED"}
  ]
}
```

#### 6.4.7 `data_ready`

Emitted after `pipeline_completed`, only when the pipeline has a caller node ([Pipeline Contract §9](/docs/pipeline-contract.md#9-the-caller-node-result-node)). By the time this event is emitted the full result is materialized in Redis; the event carries the schema, the **inline first page** (up to `datapipelines.result.page-size-rows`), and the cursor for the rest.

```
event: data_ready
id: 7
data: {
  "execution_id": "exec-uuid",
  "pipeline_id": "pipeline-uuid",
  "schema": [
    {"name": "customer_id", "type": "INTEGER"},
    {"name": "total_amount", "type": "BIGDECIMAL", "precision": 18, "scale": 2},
    {"name": "first_order_at", "type": "TIMESTAMP"}
  ],
  "rows": [
    [1, "12345.67", "2024-01-15T00:00:00Z"],
    [2, "67890.12", "2024-02-03T00:00:00Z"]
  ],
  "row_count": 2,
  "total_rows": 2,
  "has_more": false,
  "result_url": "https://{host}/api/v1/executions/exec-uuid/result",
  "expires_at": "2026-08-05T14:35:02Z",
  "ttl_seconds": 300,
  "warnings": []
}
```

- `rows` — the first page. For results ≤ one page, `rows` IS the whole result (`has_more: false`) and no cursor read is needed.
- `result_url` + `expires_at` — cursor for paging the full result within TTL (§7). Always present, small results included — a client may re-fetch or download in another format within the TTL.
- There is no `delivery_mode` field — the delivery model is uniform (v1.3; the former inline/claim-check split is gone).

#### 6.4.8 `execution_aborted`

Terminal event when an execution is cancelled: explicit `DELETE /executions/{id}` (§10.4), client disconnect beyond the grace period (§6.8), or server shutdown. Emitted to any still-connected stream (e.g., a UI watching an execution another session cancelled).

```json
{
  "execution_id": "exec-uuid",
  "pipeline_id": "pipeline-uuid",
  "aborted_at": "2026-08-05T14:30:01.000Z",
  "reason": "client_disconnect" | "cancelled" | "shutdown",
  "status": "ABORTED",
  "node_stats": [...]
}
```

#### 6.4.9 `node_progress`

A **measured sample of a node's operation** (149): what the node is doing, where its output goes, the state it is in at the observation instant, its cumulative counts and the wall time spent in each state so far. Zero or more per node, strictly between that node's `node_started` and its `node_completed`/`node_failed` (or the execution's terminal event when the node ends by cancellation). Never terminal on the wire; `event_id` continues to count it like any other event. Additive: a client on the eight-event vocabulary sees exactly the events it always did, in the same order, with these interleaved.

```json
{
  "execution_id": "exec-uuid",
  "node_id": "fetch_orders",
  "attempt": 1,
  "sequence": 3,
  "operation": "stage",
  "destination": {"kind": "tempdb", "table": "orders"},
  "state": "writing",
  "started_at": "2026-08-05T14:30:00.234Z",
  "observed_at": "2026-08-05T14:30:05.535Z",
  "elapsed_ms": 5301,
  "timings_ms": {"connecting": 12, "executing": 410, "fetching": 3400, "waiting_output": 22, "writing": 1457},
  "rows_fetched": 12000,
  "rows_written": 11000,
  "batches_written": 11,
  "correlation_id": "corr-uuid"
}
```

| Field | Meaning |
|---|---|
| `sequence` | Ordinal of this sample within the operation — strictly increasing per (`execution_id`, `node_id`, `attempt`). Repeated batches and statements are distinguished by `sequence` and `batches_written`, never by a second destination. |
| `operation` | `stage` (source cursor → tempdb table), `ctas` (tempdb `CREATE TABLE … AS` — query and materialisation are ONE indivisible statement), `materialize` (caller result → the result store, or `direct` to an invoking PIPELINE node), `writeback` (cursor → external datasource table, one transaction), `statement` (DML/DDL), `child` (a PIPELINE node: the child execution, then the parent's own output write). CALCULATOR nodes have no operation and emit no samples. |
| `destination` | The REAL output destination: `{"kind":"tempdb","table":…}`, `{"kind":"datasource","datasource":…,"table":…}` (`table` absent on a DML/DDL statement — no SQL lineage is inferred), `{"kind":"caller"}`, `{"kind":"parent"}` (a child's caller rows streamed to the invoking node), `{"kind":"none"}` (DDL, or a node that writes nothing). Identifiers only — never a connection string ([Observability §9.3](/docs/observability.md)). |
| `state` | `connecting` (a SOURCE connection is being obtained), `executing` (the statement is submitted and has not returned — a returned cursor is not a fetched row), `fetching` (advancing the source cursor / decoding rows), `waiting_output` (an OUTPUT connection was requested and is not yet held), `writing` (inside a destination write, holding its connection), `finalizing` (commit, row count, budget check, result meta), then exactly one terminal `completed` \| `failed` \| `aborted`. |
| `started_at`, `observed_at`, `elapsed_ms` | Operation start; this sample's ORIGINAL observation instant (replay preserves it); the honest wall time between them. |
| `timings_ms` | Cumulative measured wall time per state actually observed, the open state counted up to `observed_at`. A state never entered is absent — not zero. Wall time at the executor's boundary, never engine CPU or exclusive network time. |
| `rows_fetched` | Rows read off the source cursor (or received from the child) so far. ABSENT when not observed (`ctas`, `statement`). |
| `rows_written` | Rows ACCEPTED by the destination so far — a tempdb batch inserted, a write-back batch executed, a result-store page pushed, a DML update count. Absent when not observed (DDL; a CTAS until its count). Accepted is not committed: see `committed`. |
| `batches_written` | Destination batches executed so far. |
| `committed` | **Terminal samples only**, and only for a destination that has something to commit. It is COMMIT EVIDENCE, independent of the node's outcome: `true` when the writer confirmed the write durable (write-back after `commit()` returned, tempdb after the last batch and the budget check, the result store after its meta key, DML after autocommit) — **also on a `failed`/`aborted` sample**, because a node can fail AFTER its write became durable (a connection close that throws once `commit()` returned; a cancellation landing between the commit and the node's completion) and a failed node is not evidence that its side effects were undone; `false` when the write was confirmed undone (`rolled_back` is then `true`: a write-back transaction rolled back after a failure that came BEFORE the commit attempt, a failed stage's partial table dropped). **Absent on a terminal sample when neither was observed** — a driver that never returned from `commit()` before the node's deadline, a cancellation before the commit whose implicit rollback nobody witnessed, and **any failure thrown by `commit()` itself**: the transaction may be durable with only its acknowledgement lost, a `rollback()` that succeeds afterwards undoes nothing, so the writer reports neither — the outcome is unknown, and clients must show it as such ("commit not observed"), never as "not committed". Absent while running — there is no "committed" before the commit — and absent for DDL or a child with no output. **Limit:** an abandoned driver body (§6.4.6 timeouts) may still commit AFTER the terminal sample; the sealed operation publishes nothing about it, so an unknown outcome on a timed-out write means exactly that — check the target. |
| `rolled_back` | Present (`true`) only when a failed write-back transaction was rolled back or a failed stage's partial table was dropped. |
| `child_execution_id` | `child` operations, once the child's id is minted. |

There is no percentage: no operation knows its total. A slow source query is `executing` (or `fetching`) time, a full tempdb pool is `waiting_output` time, a slow destination is `writing` time — the split is measured, not inferred. The measurement limits are stated in [DAG Executor §10](/docs/dag-executor.md#10-sse-event-integration): the source's server-side time is inside `executing` until the driver returns the first result, a driver that pre-buffers (`source-fetch-size: 0`, MySQL without streaming) moves source time into `executing`, and `fetching` for `materialize` includes JSON encoding.

**Cadence.** A sample is emitted at each FIRST entry into a state (taken at the entry instant, so a `writing` interval shorter than the executor's 250 ms pump tick is still seen), then at most once per `datapipelines.executor.progress-sample-interval-seconds` (default 1) while something changed, then exactly one terminal sample. Per operation that is at most `states + duration / interval + 1` events — never per row, never per batch. Every sample is persisted like every other event (§10.3).

### 6.5 Event ordering guarantee

Within a single execution stream, events are ordered:
1. Exactly one `execution_started` (first).
2. For each node: zero or one `node_started` → zero or more `node_progress` (§6.4.9) → zero or one of (`node_completed` | `node_failed`). A node's terminal `node_progress` sample (`completed`/`failed`/`aborted`) precedes its `node_completed`/`node_failed`; no `node_progress` for a node follows the node's terminal event, and none follows the execution's.
3. Exactly one terminal sequence: (`pipeline_completed` [→ `data_ready` if a caller node exists]) | `pipeline_failed` | `execution_aborted`.
4. Stream closes after the terminal event.

For parallel nodes, events are emitted in real-time as they occur (interleaved). Order between parallel nodes is non-deterministic — on the live stream and, independently, in the §10.3 replay, whose order between parallel nodes is the persistence order; a node's own events keep their order in both.

### 6.6 Heartbeat (keepalive)

To prevent load balancer / proxy idle-timeout kills (AWS ALB default 60s, nginx default 65s), the server sends SSE comment lines every 15 seconds when no events have been emitted:

```
: heartbeat
```

These are SSE comments — ignored by the `EventSource` parser and by our `fetch`-based consumer. They exist solely to keep the TCP connection alive during periods of no event flow (e.g., a slow source query taking 30+ seconds).

The heartbeat interval is configurable via `datapipelines.sse.heartbeat-interval-seconds` (default: 15).

### 6.7 Event idempotency

`event_id` is monotonic per execution. Clients can use it to detect dropped events (gap in sequence). Stream resumption is not supported (§6.8).

### 6.8 Client disconnect

**A disconnected client cancels its execution.** If the SSE connection drops mid-stream, the executing instance starts a grace timer (`datapipelines.sse.disconnect-grace-seconds`, default 30). If no terminal event has been reached when the grace period elapses, the execution is cancelled — in-flight statements are interrupted via `Statement.cancel()`, held datasource connections are released, and the execution finishes as `ABORTED` ([DAG Executor §8.3](/docs/dag-executor.md#83-cancellation)). Rationale: an execution nobody is waiting for must not keep occupying source-database connections and staging memory.

**A revoked subscriber's stream is cut; the execution is not** (#230, security-assurance P4). Before every write — event or heartbeat — an open stream re-judges its subscriber's authority to read the execution, with the same predicate a new request would run ([Auth §11.4](/docs/auth.md#114-api-key-validation-cache)); when the answer turns no, the stream ends at that write, carrying a final `revoked` SSE comment and nothing after, while the execution itself keeps running to its terminal state. The re-judged workspace is the one the stream OPENED in — the `DP-Workspace` header's resolution, not the token's stamped claim (#263) — and a write at or past the token's own `exp` ends the stream the same way: the same final comment, the same kept-running execution, with the duration timer recording `expired` beside `revoked` ([Observability §4.1](/docs/observability.md#41-metric-naming)).

Consequences clients must design for:

- There is no reconnection or resumption path — `Last-Event-Id` is ignored. A client that loses its stream should assume the execution will be aborted and re-execute (with an `Idempotency-Key`, a retry within the idempotency TTL that arrives before the abort completes attaches to nothing — the original is gone; the retry starts a fresh execution).
- A disconnect **after** the terminal event costs nothing: the execution is complete and its result lives out its TTL in Redis — fetch it via §7.
- Detached (fire-and-forget) execution is intentionally not offered in v1; async execution with webhooks is a ROADMAP item.

Completed executions remain visible: metadata via `GET /executions/{execution_id}` (durable, [Metadata DB](/docs/metadata-db.md)), events replayable for 1 hour via §10.3 (Redis event log), results within their TTL via §7.

---

## 7. Result Delivery

### 7.1 Model

Every completed execution with a caller node has its full result **materialized in Redis before `data_ready` is emitted**. One storage model, one retrieval path:

- `data_ready` carries the schema + inline first page + `result_url` (§6.4.7). Small results need no further call.
- The cursor endpoint below pages the stored result — for ANY execution, any size, within the TTL, in JSON / Arrow / CSV.
- Because the result is fully materialized before the cursor exists, row order is stable across pages.

**Hard cap:** a caller result larger than `datapipelines.result.max-size-bytes` (default 100 MB) fails the execution with `result.too_large`. **Result delivery is not the bulk-data path** — pipelines producing large datasets should write them back with `output.target: "datasource"` and return a summary (or nothing) to the caller. Explicit NOT-goals: durable result storage beyond the TTL, and result delivery as an ETL mechanism.

**Redis unavailable at result-write time** fails the execution with `result.storage_unavailable` — there is no fallback to inline-only delivery (that would reintroduce a second path).

### 7.2 Cursor endpoint

```
GET /executions/{execution_id}/result?offset=0&limit=10000&format=json
```

Auth: the `execution.result.read` row (D11, 177): the caller's OWN execution — `executed_by = self` (an `endpoint` key's identity owns the runs it started, #215 A.5; a person does not own a run a key they created started) — or any execution with `execution.read_all` (a workspace admin's); a promoter is refused by role, and another member's execution is `404 result.execution_not_found`, never 403. The URL is not a capability — an unauthenticated request 401s ([Auth §7.6](/docs/auth.md#76-operation-matrix--the-permission-catalog-authoritative)).

### 7.3 Response (JSON format, default)

```json
{
  "schema_version": 1,
  "correlation_id": "uuid",
  "data": {
    "execution_id": "exec-uuid",
    "schema": [...],
    "rows": [
      [1, "12345.67", "2024-01-15T00:00:00Z"],
      ...
    ],
    "row_count": 10000,
    "offset": 0,
    "limit": 10000,
    "total_rows": 12450000,
    "has_more": true,
    "expires_at": "2026-08-05T14:35:02Z"
  }
}
```

Pagination: `offset` + `limit`. Default `limit` is `datapipelines.result.page-size-rows`; maximum is `datapipelines.result.page-max-rows`.

### 7.4 TTL — fixed, client-influenced, clamped

The client may request a TTL on the execute call:

```
POST /pipelines/{id}/execute
DP-Result-TTL-Seconds: 900
```

Effective TTL = `clamp(requested, datapipelines.result.ttl-min-seconds, datapipelines.result.ttl-max-seconds)`; if the header is absent, `datapipelines.result.ttl-default-seconds` (300). The clamp is non-negotiable — an unbounded client-controlled TTL would let one caller pin gigabytes in Redis.

The expiry is **fixed at result-write time** — page reads do NOT extend it (predictable memory; a result can never be kept alive indefinitely by polling). The effective `expires_at` is reported in `data_ready` and every cursor response. After expiry: `410 result.expired` — re-run the pipeline.

### 7.5 Other formats

```
GET /executions/{execution_id}/result?format=arrow    # application/vnd.apache.arrow.ipc
GET /executions/{execution_id}/result?format=csv      # text/csv, header row
```

Arrow: binary IPC stream with embedded schema, full result in one response (no pagination). CSV: header row; big integers/decimals as their wire-string form (Type System rules); full result, no pagination. Both respect the same TTL and auth.

**v1 delivery note (added v1.4):** `format=arrow` is recognized but not served in v1 — no Arrow IPC encoder ships in the v1 dependency set, and a hand-rolled one is unverifiable. The request is answered `400 result.format_unsupported` with `details.supported = ["json", "csv"]`. Arrow delivery remains tracked in §14.

### 7.6 Endpoint errors

Registry of record: [Pipeline Contract §13.10](/docs/pipeline-contract.md#1310-result-retrieval).

| Code | HTTP | Description |
|---|---|---|
| `result.execution_not_found` | 404 | Execution ID unknown |
| `result.execution_incomplete` | 409 | Execution has not completed yet |
| `result.execution_failed` | 410 | Execution ended in failure — no result to retrieve |
| `result.expired` | 410 | TTL elapsed; result no longer available |
| `result.format_unsupported` | 400 | Unknown `format` parameter |
| `result.too_large` | 500 | (During execution) result exceeded the size cap; execution failed |
| `result.storage_unavailable` | 503 | Redis unavailable: during execution the execution fails; on a read the request is refused (§10.3's replay, `details.reason: event_log_unavailable`) and may be retried. 500 until v2.91 (#487) |

---

## 8. Template Endpoints

Templates are first-class entities. See [Templates spec](/docs/templates.md) for the entity model.

**The promoter lens (178)** applies to every read here as it does in §5: a **promoter** sees only
templates with a current RELEASED version newer than the promotion target's ([Auth §11A.1](/docs/auth.md#11a1-the-404-rule),
[Versioning §10.2](/docs/versioning.md#102-the-listing-rule-what-the-ui-shows) — the template arm); a hidden
template is `404 template.not_found` by name (§8.2, §8.3), and absent from §8.5's page, level,
`total` and `template_count`. Fail closed while the target is unreadable; every other role unchanged.

**Addressing (v2.0 — template-hierarchy-design §9.6): the template name NEVER travels in a URL
path segment.** Since v2.0 a name may contain `/` (`acme/finance/monthly_revenue`), and an
encoded `%2F` in the path is refused `400` by the container below routing and below the
security chain (measured on the pinned Tomcat — no handler can reach past it). Every route in
this section therefore carries the name as a **query parameter or a body field**; the pre-v2.0
`/{id}` path forms are removed, not kept alongside. `GET /templates` answers **three shapes on
one route**: the single-resource envelope with `404 template.not_found` when `name` is present
(§8.2), one tree level (folders with counts + leaves) when `prefix` is present (§8.5), and the
paged list envelope when neither is (§8.5) — an exact-match filter returning an empty list
would make "no such template" indistinguishable from "empty result", so the miss keeps its code.

### 8.1 Create template

```
POST /templates
Content-Type: application/json

{
  "id": "nyc/mobility/fetch_orders.sql",   // optional; auto-generated under test/ if omitted
  "type": "sql",                    // optional (default); "html" takes NO dialect (046)
  "dialect": "POSTGRES",
  "display_name": "Fetch Orders in Date Range",   // required (templates.md §3.2)
  "description": "Fetch orders in date range. Expects start_date and end_date in the render context.",
  "imports": [],
  "body": "SELECT order_id, customer_id, total_amount, order_date\nFROM orders\nWHERE order_date BETWEEN '${start_date}' AND '${end_date}'"
}
```

Templates declare no parameter schema — variables are declared by the pipelines that reference the template, and validated there by dry-render ([Pipeline Contract §7.4](/docs/pipeline-contract.md#74-template-variable-resolution)).

`type` is chosen here and never changes afterwards: `sql` (the default) requires `dialect` and is what pipeline nodes reference; `html` declares no `dialect`, renders through an auto-escaping engine configuration, and cannot be referenced by a pipeline node (templates.md §3.2, 046); `jsonata` and `javascript` are the script types (7b) — `engine: "none"`, no `dialect`, no `imports`, `is_library: false`, and the three blocks `contract` / `invariants` / `tests` are required on create and update (templates.md §3.3; JavaScript requires the installed isolate, otherwise `script.js.unavailable`).

A script body may carry **`implements`** (7e, [Templates §3.4](/docs/templates.md#34-implements-and-drift)): an array of learned-fact ids — WORKSPACE `definition`/`exclusion`/`preference` facts of this workspace, at most 50 — the rules this script implements. Not content: outside `body_hash`. An id the workspace cannot cite is `400 template.implements_unresolved` (`details.fact_id`, `details.reason`); on `sql`/`html` the field is `400 template.blocks_not_allowed`.

Response: `201 Created` with full template (including `type`, version `1`, `created_at`, and on a script `implements` / `needs_review`).

### 8.2 Get template (working version)

```
GET /templates?name={path}
```

Response carries the **working version's** projection (versioning §7, since 039 — the
template mirror of §5.2): the DRAFT when one exists, else the latest released version. The
projection states its `version`, `status` and `body_hash`, and a `draft` pointer is present
when a draft exists. Every template projection here and in §8.3/§8.5 carries **`needs_review`**
(7e — `true` when a cited learned fact has been retired, computed on read, never stored) and,
on a script, **`implements`** (its cited fact ids, sorted) and — when marked —
**`retired_facts`** (`[{fact_id, retired_reason, superseded_by}]`) ([Templates §3.4](/docs/templates.md#34-implements-and-drift)).
The pipeline editor's SCRIPT card reads `needs_review` off §8.3's read.

### 8.3 Get template (specific version)

```
GET /templates/versions?name={path}&version={N}
```

### 8.4 Update template (writes the draft)

```
PUT /templates
If-Match: <body_hash of the version this edit is based on>

{
  "id": "acme/finance/fetch_orders.sql",   // REQUIRED here — the §9.6 addressing form (the name never travels in the path)
  "dialect": "POSTGRES",            // OPTIONAL since 136 §D (T289a): omitted, the working version's is inherited; a DIFFERENT one is refused 400 template.validation.dialect_invalid, details naming dialect / established_dialect / template_id — a template's dialect is fixed by the version you edit (135 §C's MCP rule, on REST)
  "type": "sql",                    // OPTIONAL too: omitted, the working version's is inherited; a different one is template.validation.type_immutable
  "display_name": "Fetch Orders in Date Range",   // required (templates.md §3.2)
  "description": "...",
  "imports": [{"id": "acme/lib/date_filters.sql", "version": 2, "alias": "dates"}],
  "body": "..."
}
```

The template mirror of §5.5 (versioning §3.2/§6): the first write after a release copies
to a draft, later writes overwrite it in place; `If-Match` carries the hash precondition;
stale is `409 template.version.conflict`. `dialect` and `type` are inherited from the
working version when the body omits them (136 §D, T289a) — before 136 this endpoint
REQUIRED `dialect` and silently wrote a changed one; now a changed dialect is
`400 template.validation.dialect_invalid` naming both, before validation or any write. The draft versions the CONTENT fields —
`display_name`/`description` move on the index row at save time (versioning §6's
documented asymmetry: they are not part of the versioned artifact).

Response: `200 OK` with the draft version's projection (`version`, `status: "DRAFT"`,
`body_hash`). Content identical to the released version is a no-op (versioning §5.1): the
response carries `status: "RELEASED"` and no draft was created — though
`display_name`/`description` still moved at save time (the §6 asymmetry: they are not part
of the hashed artifact).

**`implements` on update (7e, [Templates §3.4](/docs/templates.md#34-implements-and-drift)).** Not content, so it never decides whether a
draft opens. Stated, it replaces the written version's citations (`[]` clears); **absent, it is
inherited** from the version the edit is based on (owner ruling 2026-09-25). A body equal to the
released content that states `implements` is the no-op above for content, and the citations land
on the RELEASED version — the one post-release write templates.md §5.1 allows; that is how a
`needs_review` version is cleared (cite the successor). Refusals as on create.

### 8.9 Release template

```
POST /templates/release
If-Match: <the draft's body_hash>

{
  "name": "acme/finance/fetch_orders.sql"
}
```

Locks the template draft (§5.10's mirror; templates lock BEFORE pipelines — versioning
§6's pin rule is enforced at pipeline release). Errors: `template.version.not_draft`,
`template.version.conflict`, the template validator's §13.9 codes re-run on the draft
content, or `template.release.busy` — the flip runs in one metadata transaction under the
workspace writer guard (#517, §5.10). **Audited** as `template.version.released` (`template_id`, `version`, `via`). Response: `200 OK` with the released version.

### 8.10 Purge template draft

```
POST /templates/draft/discard
If-Match: <the draft's body_hash>

{
  "name": "acme/finance/fetch_orders.sql"
}
```

**Purges** the draft (101 — historical route spelling): always a hard delete (nothing
references a template version by FK — versioning §6), and the sole-draft case takes the
template entity with it. **Session-only**; audited as `template.version.purged`.
Response: `204 No Content`. **Refused `409 template.in_use`** (#320) while a pipeline, a parameter
set or a visualization pins the draft — the exact version when the template keeps other versions,
any version when the draft is the only one and the purge takes the template with it; `details`
carry `pinned_by`, `referencing_parameter_sets` and `referencing_visualizations`
([Versioning §3.5.3](/docs/versioning.md#353-the-reverse-arrows-into-other-families-320)). Before #320
this route ran no pin check.

### 8.11 Discard / restore / purge a template version, and the manual switch (101)

```
POST /templates/version/discard          {"name": "...", "version": 2}
POST /templates/version/restore          {"name": "...", "version": 2}
DELETE /templates/version?name=...&version=2
POST /templates/current                  {"name": "...", "version": 2}
```

The pipeline twins by name (§5.12–§5.15; §9.6: the name never travels in a path segment):
discard flips a RELEASED version to DISCARDED (`409 template.version.not_released`;
`409 template.in_use` while any stored pipeline version — discarded included — pins it), restore brings it back
(`template.version.not_discarded`), purge is drafts-only
(`template.version.last_release`), and `current` is the manual switch
(`template.version.not_eligible`). The template's sticky pointer follows the same D60
rules. All **session-only**, audited as the `template.version.*` / `template.current_switched`
events. Responses mirror §5.12–§5.15 at template shape.

### 8.5 List templates

```
GET /templates?dialect={dialect}&type={sql|html|jsonata|javascript}&q={search}&offset=0&limit=50
```

The second shape on this route: answers only when `name` AND `prefix` are both ABSENT (§8's addressing note). The `type` filter (046) is optional; an unknown value is refused `400 pipeline.execution.invalid_parameter_type` naming the supported values. **`&implements={fact_id}`** (7e, [Templates §3.4](/docs/templates.md#34-implements-and-drift)) keeps the templates whose listed version cites that learned fact — a parameterised join through the template's own workspace, so another workspace's fact id matches nothing; a malformed id is `400 pipeline.execution.invalid_parameter_type`. It applies to this flat shape only (like `q`). `q` is trimmed; blank (or whitespace-only) means no search. A term over 200 characters after the trim is refused `400 request.query_too_long` (#504b, [pipeline-contract §13.21](/docs/pipeline-contract.md)) with `details.limit` and `details.length`, before any read; the term is never echoed.

```
GET /templates?prefix={folder}&dialect={dialect}&type={sql|html|jsonata|javascript}&offset=0&limit=50
```

The third shape: answers when `name` is absent and `prefix` is PRESENT — browse ONE level of the template tree (067; same contract as `templates_list {prefix}`, [MCP §6.2.6](/docs/mcp-server.md)). Present-but-empty (`?prefix=`) is the ROOT. The `data` payload becomes `{prefix, folders, templates, total, has_more}`: `folders` lists the prefix's direct sub-folders as `{path, segment, template_count}` (subtree counts), `templates` its direct leaves as the same rows as the flat list, `total`/`has_more` page the leaves via `offset`/`limit`. `dialect`/`type` narrow both halves, so a folder whose whole subtree is filtered out is absent rather than empty; `q` is ignored while `prefix` is present (browse and search are different presentations). An unknown or illegal prefix answers an EMPTY level with `200` — never a `400`, never a query error.

### 8.6 Delete template (the entity purge, 101)

```
DELETE /templates?name={path}
```

**The entity purge** (replaces the V1 soft delete, retired in V19): allowed only when the
template's ONLY version is a DRAFT (`409 template.version.last_release` otherwise) and no
stored pipeline version — draft, released or discarded — pins any version of it (`409 template.in_use`, naming the pinners).
Existing pipelines referencing any version continue to work (we never hard-delete template
versions); **session-only**, audited as `template.purged`.

### 8.7 Validate template (render against sample context)

```
POST /templates/render

{
  "name": "acme/finance/fetch_orders.sql",
  "version": 1,
  "context": {
    "start_date": "2026-01-01",
    "end_date": "2026-01-31"
  }
}
```

Response: rendered SQL string. Useful for UI editor preview and for LLM-assisted authoring. A **script type** (`jsonata`/`javascript`) has nothing to render — refused `400 template.render_not_applicable` with `details.use: "templates_evaluate"` (7b).

### 8.7A Evaluate a script template (7b)

```
POST /templates/evaluate

{
  "name": "acme/finance/apply_rules.jsonata",
  "version": 1,
  "input": {
    "rows": [ { "order_id": 1, "amount_cents": 1250, "customer_id": null } ],
    "inputs": { "tz": "UTC", "min_total": 0.00 }
  }
}
```

The evaluate route — the MCP `templates_evaluate` tool's REST twin, the same service, the same bounded evaluation pool, the same timeout (`datapipelines.script.evaluate-timeout-seconds`). `version` omitted = the working version (the draft when one exists, else the latest released); `input` is the template contract's input object (in row mode `rows` is the batch and `inputs` holds the value inputs only; in table/value mode `inputs` holds every input, tables as arrays); an optional `now` (ISO-8601) pins the clock for `$now()`/`$millis()` (absent, those builtins refuse). Response:

```json
{
  "output": { "rows": [], "rejects": [ { "row": { "order_id": 1, "amount_cents": 1250, "customer_id": null }, "reason": "customer_id missing" } ] },
  "rejects": [ { "row": { "order_id": 1, "amount_cents": 1250, "customer_id": null }, "reason": "customer_id missing" } ],
  "invariants": [ { "name": "one_to_one", "passed": true, "message": "every input row is accepted or rejected, never lost" } ],
  "console": []
}
```

`console` is always present: a list such as `[{"level":"log","text":"mapped 2 rows"}]`, empty when unused. A failed evaluation retains lines emitted before failure under `details.console`. A refusal is the code with its detail — the contract's input check, the type gate, or the engine's own (the record's §7 mapping; §13.18 lands with 7c). Permission: `template.evaluate` (the render row's twin — auth.md §7.6).

### 8.8 Import template library

```
POST /templates/import
Content-Type: application/json

{
  "templates": [
    {"id": "acme/lib/aggregate.sql", "dialect": "POSTGRES", "body": "<#macro aggregate ...>...</#macro>"},
    ...
  ]
}
```

Library templates (Freemarker macros usable via `#import`) are stored like regular templates; they're just referenced by other templates rather than by pipelines directly. See [Templates spec](/docs/templates.md).

A script entry may carry `implements` (7e). The import keeps the ids that resolve to citable facts in THIS workspace and drops the rest without refusing (owner ruling 2026-09-25 — learned facts are environment-local and never promoted, so a promotion from another deployment lands with none); the kept ids land on the version the import wrote, and an idempotent re-import that wrote nothing writes no citations ([Templates §3.4](/docs/templates.md#34-implements-and-drift)).

---

## 9. Datasource Endpoints

Datasources are environment-specific connections. See [Datasources spec](/docs/datasources.md) for the entity model.

### 9.1 Register datasource

```
POST /datasources
Content-Type: application/json

{
  "name": "pg-prod",
  "display_name": "Production Postgres",
  "dialect": "POSTGRES",
  "jdbc_url": "jdbc:postgresql://host:5432/db",
  "credential": {                   // §3.4 — WHAT the credential is
    "kind": "password",             // password | token | private_key | service_account_json | none
    "username": "readonly_user",
    "secret": "..."                 // write-only; never returned in GET
  },
  "introspection_include_schemas": ["apex_reporting"],  // OPTIONAL — §9.7 escape hatch for the
                                                        // system-schema exclusion (exact names,
                                                        // no patterns; lowercased at bind)
  "properties": {
    "hikari": {
      "maximumPoolSize": 10,
      "connectionTimeout": 30000
    },
    "jdbc": {
      "sslmode": "verify-full"
    }
  }
}
```

**The credential ([Datasources §3.4](/docs/datasources.md#34-credential-kinds)).** Send EITHER the `credential` object or the legacy top-level `username`/`password` pair, which means `kind: "password"` and stays accepted under the frozen-shape rule. A body carrying BOTH is `400 datasource.validation.properties_invalid` — the two can disagree and there is no defensible winner. `kind: "none"` carries neither field and is the shape for a file database or an IAM role; `kind: "token"` carries a secret and may name a username. A kind the dialect's pinned driver cannot authenticate with is `400 datasource.validation.properties_invalid` naming what the dialect accepts. A missing secret on create is `400 datasource.validation.password_missing` for every kind but `none`.

`properties` has exactly two reserved namespaces — `hikari` (HikariCP's own camelCase property names, durations in milliseconds) and `jdbc` (driver connection properties) — validated by a test pool build at save time. See [Datasources §5](/docs/datasources.md#5-connection-pool-configuration).

`introspection_include_schemas` (optional, [Datasources §3.3](/docs/datasources.md#33-field-reference)) exempts exact schema names from the §9.7 system-schema exclusion; entries are legal schema identifiers — letters, digits, `_`, `$`, `#`, lowercase (entries outside the alphabet — wildcards, quoted identifiers, qualified `db.schema` names — are rejected with `400 datasource.validation.properties_invalid`), and the stored list is normalized on save — trim, lowercase, drop blank-after-trim entries, deduplicate (first-seen order) — so what a GET projects always survives an unmodified PUT round-trip. Omitted from the response when empty.

**Workspace binding (workspaces D8):** the optional `global` (boolean, admin-only — creating shared infrastructure is an admin act) and `workspace` (string, a workspace the caller can access) fields set the binding; with neither, the datasource binds to the caller's ACTIVE workspace. `readonly` (boolean, default `false`) forbids the three write-shaped pipeline uses ([Datasources §5.7](/docs/datasources.md#57-readonly-datasources-flag-semantics-and-enforcement-layers)). A non-admin sending `global: true`, binding to a workspace they are not in, or any member write while `member-datasources-enabled` is off, is `400 datasource.validation.workspace_forbidden`. Datasource NAMES stay a flat global namespace: a collision with another workspace's datasource is still `409 datasource.validation.duplicate_name` (by design — `name` is the PK and the pool-registry key).

Response: `201 Created` with the datasource entity (excluding password).

### 9.2 List datasources

```
GET /datasources?dialect={dialect}&offset=0&limit=50
```

Returns the §4.3 pagination envelope (§2 principle 6 — list endpoints paginate). **Workspace-scoped** (workspaces §5.3): the listing shows the active workspace's bound datasources plus all global ones; the predicate runs in the repository's SQL, so `total` counts exactly what the caller can see. A workspace-bound datasource of another workspace is absent — not filtered client-side, not paged-past.

**`facts` and `definitions` on every row (138 §B, additive).** Each item carries the two learned-fact blocks §9.7A defines — `facts`, the datasource-wide kinds (window, sampling), and `definitions`, the active workspace's rules (`definition`, `exclusion`, `preference`) that name this datasource — as the MCP `datasources_list` has since 136: a rule is read BEFORE the tables are chosen, so it rides the call every client makes first. Both are `[]` when nothing is recorded, never absent.

### 9.3 Get datasource (sensitive fields redacted)

```
GET /datasources/{name}
```

Returns everything except the credential SECRET — neither `password` nor `credential.secret` — plus the additive `credential` (`{kind, username?}`, §3.4), `workspace` (the bound workspace's name, `null` = global), `readonly` (workspaces design §9) and `last_test` fields. `password_set` is derived from `credential.kind`: `false` exactly for `kind: "none"`, which genuinely has no stored credential. Top-level `username` is still returned and is `null` for the kinds that have none. `last_test` is the outcome of the last connection test — `{"tested_at": "...", "ok": true|false, "message": "..."}`, or `null` when the datasource has never been probed ([Datasources §8.1B](/docs/datasources.md#81b-the-last-tests-outcome-is-stored-and-listed)); it is also on every `GET /datasources` row, which is what lets a reader see a credential has stopped working without running an execution. A name bound to another workspace behaves as not-found (`404 datasource.not_found`).

**`pool` (094, additive).** Beside `properties`, the response carries the connection pool's EFFECTIVE settings — one entry per tunable HikariCP key ([Datasources §5](/docs/datasources.md#5-connection-pool-configuration)) as `{"value", "unit", "source"}`, where `source` is `configured` (this row set it), `dialect_default`, `application_default` or `hikari_default`. The keys keep HikariCP's camelCase spelling, deliberately: they are the exact identifiers a caller writes back under `properties.hikari.*`. This is NOT a duplicate of `properties.hikari` — that map is what the row STORES and is empty for almost every datasource, while `pool` is what the pool RUNS with, which is what a pool-acquisition timeout is explained by.

```json
"pool": {
  "maximumPoolSize": {"value": 10, "unit": "count", "source": "hikari_default"},
  "minimumIdle": {"value": 2, "unit": "count", "source": "application_default"},
  "connectionTimeout": {"value": 5000, "unit": "ms", "source": "configured"}
}
```

### 9.4 Update datasource

```
PUT /datasources/{name}
```

Updates connection details. The credential secret is optional — omit `credential.secret` (or the legacy `password`) to keep the stored one. `credential.kind` is part of the body like every other field: changing it to `none` clears the stored credential, and changing it to ANY other kind requires a secret in the same request — `400 datasource.validation.password_missing` otherwise, because keeping the stored secret would relabel it as something it is not ([Datasources §3.4](/docs/datasources.md#34-credential-kinds)). The body is the §9.1 shape (name immutable); `introspection_include_schemas` is replaced wholesale when present and dropped to empty when absent. The `global`/`readonly` flags are optional — absent keeps the stored value, present attempts a gated write: `global` (either direction) and mutating a global datasource are admin-only, and `readonly` on a GLOBAL datasource is admin-only (workspaces design §6 last paragraph); a member may flip `readonly` on their workspace-bound datasource when the D8 gate is on. An accepted flag write crosses the same registry save path as every update — the connection pool is RETIRED and rebuilds under the new settings at the next lease. Retired, not closed (094, [Datasources §5.2](/docs/datasources.md#52-pool-lifecycle)): the old pool stops taking new leases immediately and closes once the statements already running on it finish, so an update never cuts a query that is mid-flight. Errors: `400 datasource.validation.workspace_forbidden` for the refusals.

### 9.5 Delete datasource

```
DELETE /datasources/{name}
```

Fails with `datasource.in_use` if any non-deleted pipeline references it **in any version it has ever stored** — DRAFT, RELEASED and DISCARDED alike ([Datasources §6.2](/docs/datasources.md#62-in-use-check-on-delete)). Pipeline versions are immutable and executable by explicit version, so a reference from a released v1 that a later v2 dropped is a real reference; the guard that read each pipeline's current version only let that delete through and failed v1's next execution at connect.

`details` carries `datasource_name`, `referencing_pipelines` (the distinct pipeline names) and `references` — one entry per referencing node, `{"pipeline", "node_id", "pipeline_version", "version_status"}` — the same shape `template.in_use` uses (§8.6), because the version is what tells the operator which body to go and edit.

D8-gated like update: deleting a global datasource requires admin; a member needs the `member-datasources-enabled` gate for a bound one.

A successful delete **retires** the pool on the deleting instance and, over the §5.7 invalidation channel, on every other one (094, [Datasources §5.2](/docs/datasources.md#52-pool-lifecycle)): new leases miss it at once, and each instance closes its pool when the queries already running on it finish — or at `datapipelines.datasources.retire-ceiling-seconds` if one hangs. A delete therefore never cuts a running execution.

The UI's Delete action asks this same in-use question BEFORE it offers anything, and renders the `references` rows ([UI Screens §4.5](/docs/ui-screens.md#45-datasource-list)).

### 9.6 Test connection

```
POST /datasources/{name}/test
```

Returns `200 OK` with the full wire form of `TestResult` ([Datasources §8.1](/docs/datasources.md#81-post-apiv1datasourcesnametest)) — `connected`, `tested_at`, `latency_ms`, `server_version`, `error`, `error_class` — on success and on failure alike (note: not an HTTP error — connection test failure is a normal outcome, not a server error).

The probe also **records its outcome on the datasource row**, so the result shows up as `last_test` on the list and get responses and on the datasources screen ([Datasources §8.1B](/docs/datasources.md#81b-the-last-tests-outcome-is-stored-and-listed)). The write touches the three outcome columns only: it does not move `updated_at` and is not a datasource edit.

### 9.7 Schema introspection

```
GET /datasources/{name}/schemas
GET /datasources/{name}/tables?schema={schema}
GET /datasources/{name}/tables/{table}/columns?schema={schema}
```

Read-only live schema metadata ([Datasources §7A](/docs/datasources.md#7a-schema-introspection)) over JDBC `DatabaseMetaData`, with column types mapped to the canonical Type System types. Scope: `author` ([Auth §7.6](/docs/auth.md#76-operation-matrix--the-permission-catalog-authoritative)) — same precedent as the connection test, since each call opens a live connection.

Responses (the §4.1 envelope around `data`):

```json
// GET /datasources/{name}/schemas
{ "data": { "schemas": ["public", "sales"], "truncated": false } }

// GET /datasources/{name}/tables
{ "data": { "tables": [ {"schema": "public", "name": "orders", "type": "TABLE", "remarks": "customer orders"} ], "truncated": false } }

// GET /datasources/{name}/tables/{table}/columns
{ "data": [
  {"name": "id", "type": "INTEGER", "nullable": false, "source_type": "int4", "warnings": [], "remarks": "surrogate primary key"},
  {"name": "amount", "type": "DECIMAL", "precision": 10, "scale": 2, "source_type": "numeric", "warnings": []}
] }
```

Notes:

- `GET /schemas` returns the driver-reported schema names with the engine's system schemas excluded, as a page (`{"schemas": [...], "truncated": bool}`); on MySQL the databases arrive as JDBC catalogs, so the listing reads them from `getCatalogs()`. An empty list is a valid result on schemaless datasources (SQLite, single-db DuckDB). The listing is capped at 2000 schemas; `truncated: true` means the cap dropped some (on MySQL catalog routing the walk would otherwise span every database the server grants).
- `type` in a column descriptor is the canonical wire type; `source_type` is the driver's own type name. `precision`/`scale`/`nullable` are omitted when the metadata does not report them (the envelope convention — omitted is not null). `warnings` carries the ingress type mapper's warning messages, empty when the mapping was clean. `remarks` in a table or column descriptor is the engine-stored comment (JDBC REMARKS), omitted when the driver/database has none; the schemas listing carries no remarks (`getSchemas()` has none).
- `type` in a table descriptor is the driver's raw JDBC table type (`TABLE`, `VIEW`, `BASE TABLE`, ...).
- The tables listing is capped at 2000 tables; `truncated: true` means tables were dropped. Without a `schema` parameter the tables listing spans schemas — pass each table's reported `schema` to `/columns`.
- Pass the table name exactly as `/tables` returned it — JDBC metadata name matching is case-sensitive. `table` and `schema` filters are exact-match identifiers, not LIKE patterns (`_`/`%` are escaped); a present-but-empty `?schema=` binds to `""` and is treated as absent, so the default applies rather than a match-nothing empty filter. System schemas are excluded everywhere; `/columns` without a `schema` parameter defaults to the connection's current schema (routed per dialect, [Datasources §7A](/docs/datasources.md#7a-schema-introspection)) so same-named tables in different schemas cannot merge their columns — and when the datasource reports **no current schema** (e.g. a database-less MySQL URL), that default is impossible, so `/columns` fails with `400 pipeline.execution.parameter_required` instead of returning a merged answer; list `/schemas` and pass one explicitly. An unfiltered `/tables` carries each row's own schema and cannot merge — it keeps working on such a datasource, with no guard.
- An unknown `schema`/namespace filter on a listing matches nothing and returns an empty list. The columns read is table-ADDRESSED (123 §A): a table absent from the namespace's listing is `404 datasource.table_not_found`, with a "Did you mean …?" line naming the nearest listed table when one is close (complete-catalog dialects assert "does not exist"; privilege-filtered ones say the table does not exist OR the datasource's credentials cannot see it — [Datasources §7A](/docs/datasources.md#7a-schema-introspection)). An existing table with zero readable columns is still an empty list. An unknown datasource name is `404 datasource.not_found`. A connection failure against the datasource is `502 pipeline.execution.datasource_unreachable` (the customer's database being down is not a server error).
- No pagination: the tables and schemas listings are bounded by their 2000-row cap (`truncated` flags the drop), and per-table listings are naturally bounded.

### 9.7A Learned facts on the introspection endpoints (118)

`GET /datasources/{name}`, `GET /datasources/{name}/tables` and `GET /datasources/{name}/tables/{table}/columns` carry the **learned facts** agents recorded about the objects they return ([learned-semantic-layer design](https://github.com/msabiransari/datapipelines/blob/main/docs/superpowers/specs/2026-09-11-learned-semantic-layer-design.md) D-S7, §7.2) — the same block, from the same code, as the MCP twins ([MCP §6.2.18a](/docs/mcp-server.md#6218a-the-learned-fact-block-on-introspection-responses-118)). Additive: a client that never reads `facts` sees exactly what it saw before.

```json
// GET /datasources                   → data.items[i].facts and data.items[i].definitions (138 §B): the same two blocks as the detail; [] when none
// GET /datasources/{name}            → data.facts: the datasource-wide kinds (window, sampling); data.definitions: the workspace's rules naming this datasource (138 §B); [] when none — each rule carries implemented_by (7e)
// GET /datasources/{name}/tables     → data.tables[i].facts: TABLE-grain facts (a ref with no column); omitted when none
// GET /datasources/{name}/tables/{table}/columns → data[i].facts: the facts with a ref on that column; omitted when none
{
  "id": "…", "scope": "DATASOURCE", "kind": "unit",
  "fact": "value is already in the unit named by unit_col — never tenths",
  "trust": "observed", "drift": "column X no longer exists",
  "evidence_summary": "unit=°C, value=21.4 | unit=mm, value=0.8",
  "recorded_via": "mcp", "recorded_at": "2026-09-11T10:15:00Z",
  "from_this_workspace": true,
  "source_pipeline": {"id": "…", "name": "finance/revenue"},
  "conflict": true
}
```

Notes:

- `drift`, `evidence_summary`, `source_pipeline` and `conflict` are omitted-when-absent (the §3.2 envelope convention). `trust` is `asserted` / `observed` / `verified` / `needs_review` / `stale` (a `definition`, `exclusion` or `preference` lands `asserted` even with evidence — a choice is confirmed by a person, never observed; datasources.md §7E); retired facts are never served here.
- **The drift check runs at read** (design §6): `/columns` recomputes each fact's per-table fingerprint from the columns it just read, and a COMPLETE `/tables` listing (no `namespace`/`schema` filter, not truncated) checks each fact's tables; a demotion (`needs_review`, `stale`) is written back to the row before it is served — idempotent, one-way; nothing re-maps. `GET /datasources/{name}` opens no connection and serves facts as stored.
- **`implemented_by` (7e, [Templates §3.4](/docs/templates.md#34-implements-and-drift)):** every rule under `definitions` carries `[{template_id, version}]` — the live template versions citing it as the script implementing it, under the caller's `template.read` lens (a promoter's: RELEASED versions of the templates it may read); `[]` when none. DATASOURCE facts never carry it — nothing can cite them.
- **Visibility is the store's one predicate:** every DATASOURCE fact on a datasource the workspace can see, plus the workspace's own WORKSPACE facts. **Provenance stops at the reader's visibility (D-S9):** `from_this_workspace` says whether the active workspace recorded it, and `source_pipeline` is present only when the caller's workspace can read that pipeline. **Conflicts coexist (D-S5):** two live facts of one kind on the same refs both carry `conflict: true`.
- Recording, listing and retiring facts are MCP verbs in round 1 (`semantics_record` / `semantics_list` / `semantics_retire`, [MCP §6.2.37–39](/docs/mcp-server.md#6237-semantics_record)); there is no REST write for facts yet. The refusal codes are [Pipeline Contract §13.15](/docs/pipeline-contract.md#1315-learned-semantics).

### 9.8 Lake tables (the dp-lake catalog)

```
POST   /datasources/{name}/tables
DELETE /datasources/{name}/tables/{namespace}/{table}
POST   /datasources/{name}/tables/import
GET    /datasources/{name}/lake-tables
```

The registry of tables a **LAKE**-dialect datasource serves ([metadata-db §4.15](/docs/metadata-db.md#415-lake_tables), the 2026-09-07 lake-datasource design record §2). A LAKE datasource reads object storage in place and the engine cannot LIST a bucket — these rows are its catalog. Scope: `author` for the three writes ([Auth §7.6](/docs/auth.md#76-operation-matrix--the-permission-catalog-authoritative)); mutating a GLOBAL datasource's registry additionally requires admin (workspaces D8, enforced in-handler). The listing is `read`. Every operation refuses a non-LAKE datasource with `400 datasource.validation.lake_dialect_required`.

**Register one table** — `POST /datasources/{name}/tables`, 201:

```json
{
  "namespace": ["nyc", "mobility"],
  "name": "hvfhv_zone_day",
  "format": "parquet",
  "location": "s3://datapipelines-co/sample-data/lake/v1/hvfhv_zone_day/part-0.parquet",
  "partition_column": null
}
```

- `namespace` — required, 1–9 segments; an array of segments or the dotted shorthand `"nyc.mobility"`. Each segment follows the pipeline/template segment grammar (`[a-z0-9][a-z0-9_.-]{0,63}`) **without `.`** (the dotted form must round-trip). Violations are `400 datasource.validation.lake_namespace_invalid`.
- `name` — required, one segment of the same grammar; `400 datasource.validation.lake_name_invalid`.
- `format` — `parquet` | `iceberg`; `400 datasource.validation.lake_format_invalid`.
- `location` — `s3://bucket/prefix[/glob]` or a `file://` path. For `format: iceberg` the location is the table's **current metadata FILE** (`…/metadata/00042-<uuid>.metadata.json`), not the table root — the measured rule of [Datasources §8C.7](/docs/datasources.md#8c7-the-iceberg-location-rule--register-the-metadata-file). **No other schemes**, and no quotes, backslashes, whitespace or control characters anywhere — the value is later interpolated into the engine's `CREATE VIEW`, so the refusal is total (`400 datasource.validation.lake_location_invalid`).
- `partition_column` — optional; a plain column identifier.
- Re-registering a taken (namespace, name) triple is `409 datasource.lake_table_duplicate`.

**Unregister** — `DELETE /datasources/{name}/tables/{namespace}/{table}`, 204; `{namespace}` is the dot-joined form (`nyc.mobility`). An absent triple is `404 datasource.lake_table_not_found`, never a silent no-op. The bucket's objects are untouched.

**Bulk import** — `POST /datasources/{name}/tables/import`. The body is EITHER a 088-style manifest block inline:

```json
{
  "namespace": ["nyc", "mobility"],
  "publish_prefix": "s3://datapipelines-co/sample-data/lake/v1",
  "tables": [
    {"name": "hvfhv_zone_day", "format": "parquet", "path": "hvfhv_zone_day/part-0.parquet"},
    {"name": "hvfhv_trips", "format": "parquet", "location": "s3://datapipelines-co/sample-data/lake/v1/hvfhv_trips/pickup_date=*/part-*.parquet", "partition_column": "pickup_date"}
  ]
}
```

OR `{"manifest_url": "https://…", "namespace": "nyc.mobility"}` — a URL fetched **server-side, and only from the datasource's own bucket/endpoint** (derived from its declared `dialect.endpoint` / `catalog.ref`; an `s3://` URL is translated to the matching https form; with neither declared, only AWS S3 hosts over https). Anything else is `400 datasource.validation.lake_manifest_url_forbidden` — there is no arbitrary URL fetch (SSRF). A failed fetch is `502 pipeline.execution.datasource_unreachable`. Entry fields follow register's rules (`path` is resolved against `publish_prefix` / `access.https_base`; an entry's own `namespace` beats the shared one; an entry with neither is a 400). Import is **idempotent**: already-registered triples come back in `already_registered`, not as errors, so bootstrap can re-run it. The response is `{registered: [...], registered_count, already_registered: [...], already_registered_count}`.

**List the registry** — `GET /datasources/{name}/lake-tables` (`read`): the catalog's own rows, `{datasource, tables: [...], count}`. This is deliberately a separate route from §9.7's `GET /{name}/tables`, the introspection listing — which for a LAKE datasource is itself served from this registry (tables reported as `VIEW` with the format in remarks; [Datasources §8C.3](/docs/datasources.md#8c3-introspection-reads-the-registry-not-jdbc-metadata)), so the two routes agree by construction.

Every successful write evicts the datasource's connection pool and publishes the §5.7 invalidation, so per-connection state is rebuilt on the next lease ([Datasources §5.7](/docs/datasources.md)).

---

## 10. Execution History

### 10.1 List executions

```
GET /executions?pipeline_id={id}&status={status}&offset=0&limit=50
```

Filters:
- `pipeline_id` — limit to one pipeline.
- `status` — `RUNNING | SUCCESS | FAILED | ABORTED`.
- `started_after` / `started_before` — ISO 8601 timestamp range.

Each page item carries the §10.2 metadata projection minus `result_url` /
`result_expires_at` (present only on the single-execution read, and only while the
result is unexpired):

| Field | Type | Description |
|---|---|---|
| `execution_id` | uuid | The execution's id |
| `pipeline_id` | uuid | The pipeline that ran |
| `pipeline_version` | int | The version of the pipeline that ran |
| `status` | string | `RUNNING` \| `SUCCESS` \| `FAILED` \| `ABORTED` |
| `draft_run` | bool | The version that ran was a draft at the time (or still is / was discarded) — an informational history label, never execution behaviour or promotion eligibility ([Versioning §8](/docs/versioning.md#8-executing-drafts)) |
| `parameters` | object | The execution-time parameters the caller sent |
| `started_at` | timestamp | Start of execution |
| `completed_at` | timestamp \| null | Terminal timestamp; null while `RUNNING` |
| `duration_ms` | int \| null | Wall-clock duration; null while `RUNNING` |
| `node_stats` | array \| null | Per-node stats (rows read/written, timings) |
| `error` | object \| null | The mapped failure (`code`, `message`) on `FAILED` |
| `failed_node_id` | string \| null | The node the failure is attributed to |
| `correlation_id` | uuid \| null | The caller-supplied correlation id |
| `executed_by` | uuid | The user the run belongs to (D11, 177): the session's user, or a key's OWNER. Was `triggered_by` until 2026-09-20 — renamed, not re-derived; every row kept its actor |
| `executed_by_key_kind` | string \| null | Which KIND of credential started it when a key did — `user` \| `endpoint` \| `server` ([enums §18A](/docs/enums.md#18a-executedbykeykind--which-kind-of-credential-started-an-execution)); `null` for a signed-in session. An `endpoint` run is nobody's OWN: it lists for workspace admins only, and for the endpoint key itself (§19) |
| `triggered_via` | string | `UI` \| `REST` \| `MCP` \| `PIPELINE` \| `ENDPOINT` \| `SCHEDULE` (#9 — a schedule fired it; `executed_by` is then the system identity, [Auth §4.5](/docs/auth.md#45-the-system-service-account-r7)) \| `DASHBOARD` (#10 — a dashboard refresh started it; `executed_by` is the refreshing person, [Dashboards §5](/docs/dashboards.md#5-the-runtime); it writes no stored result and is cancelled only through its refresh's abort) |
| `result_row_count` | int \| null | Rows in the caller result; null for a zero-caller pipeline and for a `direct`-delivered child |
| `result_size_bytes` | int \| null | Size of the materialized caller result |
| `parent_execution_id` | uuid \| null | The execution whose PIPELINE node spawned this one; null for a root ([§10.2](#102-get-execution-metadata)) |
| `parent_node_id` | string \| null | That node's id; null for a root |
| `root_execution_id` | uuid | The family's top ancestor; equals `execution_id` for a root |

Ownership (roles design D11, ratified 2026-09-20 — [Auth §7.6](/docs/auth.md#76-operation-matrix--the-permission-catalog-authoritative) `execution.read`): a **workspace admin** (or super admin — `execution.read_all`) reads every execution in the workspace, optionally pipeline-narrowed; a **viewer** or **author** reads only their OWN runs — `executed_by = self` and not started through an endpoint key; a **promoter** is refused the list, the read, the result and the replay by role (`403 auth.role_required`) before any row is consulted. The filter is SQL, so the page is cut after it and `has_more` is honest. Another member's execution is `404 result.execution_not_found` on the single reads, never 403 ([Auth §11A.1](/docs/auth.md#11a1-the-404-rule)).

**Scheduled runs (#9, ruling R3).** An execution a schedule fired (`triggered_via = SCHEDULE`) is nobody's own — it ran as the system identity — so it is visible to **every member whose role reaches `execution.read`**, on every surface: this list and §10.2, §10.3, §10.3A and the result read, the UI's execution lists (the executions screen and the dashboard, #250; the search palette's executions group and the pipeline and template explorers' Runs tabs, #275), and MCP's `executions_list`, `executions_get` and `executions_get_result` and its executions resource listing (#250, #275 — the key's own role decides, keys v2). A role WITHOUT `execution.read` that still reaches a runs pane through `pipeline.read` / `template.read` — the promoter, on the explorers' Runs tabs and the search palette — sees its own runs only: the scheduled arm is `execution.read`'s, never the pane's. The read is one SQL predicate (`findVisible` — own runs `OR triggered_via = 'SCHEDULE'`, workspace-scoped), so every surface cuts its page after visibility and the surfaces cannot disagree. It lifts visibility, never ownership: cancelling one (§10.4) still needs `execution.cancel_all`.

### 10.2 Get execution metadata

```
GET /executions/{execution_id}
```

Returns the execution record (without rows — use §7 for result data):

```json
{
  "schema_version": 1,
  "correlation_id": "uuid",
  "data": {
    "execution_id": "exec-uuid",
    "pipeline_id": "pipeline-uuid",
    "pipeline_version": 3,
    "status": "SUCCESS",
    "parameters": {...},
    "started_at": "...",
    "completed_at": "...",
    "duration_ms": 2377,
    "node_stats": [...],
    "result_url": "...",            // present while the result is unexpired (absent for zero-caller pipelines)
    "result_expires_at": "...",
    "result_row_count": 1204,       // null for a zero-caller pipeline, and for a `direct`-delivered child
    "result_size_bytes": 48213,
    "executed_by": "user-uuid",           // D11: the run's user — the session's, or the key's OWNER
    "executed_by_key_kind": null,         // "user" | "endpoint" | "server" when a key started it; null for a session
    "triggered_via": "UI" | "REST" | "MCP" | "PIPELINE" | "ENDPOINT" | "SCHEDULE" | "DASHBOARD",

    "parent_execution_id": "exec-uuid",   // the execution whose PIPELINE node spawned this one; null for a root
    "parent_node_id": "run_leaf",         // that node's id; null for a root
    "root_execution_id": "exec-uuid"      // the family's top ancestor; equals execution_id for a root
  }
}
```

**Composition lineage.** The three lineage fields are the composition family's links ([Pipeline Contract §8.5](/docs/pipeline-contract.md#85-pipeline-nodes), [Metadata DB §4.6](/docs/metadata-db.md#46-pipeline_executions)) and are always present — `null` on a root rather than omitted, so "this is a root" and "this server does not report lineage" stay distinguishable. They answer the question a client is left with when a `node_completed` names a `child_execution_id` (§6.4.3): fetch that id here, and `parent_execution_id` / `parent_node_id` say where it came from. `root_execution_id` is never null (it is the execution's own id for a root), so grouping a family needs no special case. §10.1's listing carries the same fields — it is the same projection, minus `result_url` / `result_expires_at`.

### 10.3 Replay SSE stream

```
GET /executions/{execution_id}/events
Accept: text/event-stream
```

Re-emits the SSE event stream from the Redis event log, in original order with original timestamps — `node_progress` samples included, each with the `observed_at` it was taken at (§6.4.9). Useful for debugging pipelines after the fact. The same revoked-subscriber cut as the live stream applies per chunk (§6.8, #230): a replay re-checks the caller's authority before each chunk it serves.

Availability: the Redis event log lives **1 hour** past completion (not configurable); afterwards this endpoint returns `410 result.expired`. A log read Redis does not answer — Redis down, or the read slower than the client's command timeout (`datapipelines.redis.command-timeout`, 2 s) — is NOT an expiry: it answers `503 result.storage_unavailable` with `details.reason: event_log_unavailable` and the `execution_id`, before any byte of the stream, and the request may be retried (#487). The log is read once per request: the answer and the stream come from the same read. Measured on the shipped client (#487): the whole-log read of 100,000 `node_progress` entries (~600 bytes each) takes ~0.1–0.15 s, and the largest log the shipped defaults produce is a few thousand entries. The durable per-event record survives 7 days in the `execution_events` table (`datapipelines.executions.event-retention-days`) and is queryable via ordinary execution metadata — only the *replayable stream* expires at 1 hour. The replay is also the answer to "I was not attached (or left early) while it ran": the live stream's delivery guarantee runs only to a connected consumer (§10.4), and everything else is read back from here.

### 10.3A Durable event record (JSON)

```
GET /executions/{execution_id}/events?format=json&after=0&limit=200
```

The execution's **durable** event record — the `execution_events` rows ([Metadata DB §4.7](/docs/metadata-db.md#47-execution_events)) — as JSON, for as long as they are retained (`datapipelines.executions.event-retention-days`, 7 days past completion by default), within §10.3's hour or after it (#9; the scheduler design revision's §5.4). It is how a scheduled run's messages are read: nobody was attached to its stream. `format=json` selects it; without it the route is §10.3's replay, unchanged.

Paged by `event_id`, oldest first: `after` (default `0`) is the last `event_id` you hold, `limit` 1–500 (default 200). Response `data`:

| Field | Type | Description |
|---|---|---|
| `execution_id` | uuid | The execution |
| `events` | array | `{event_id, event, timestamp, data}` — `event` is the §6.4 event type, `data` its payload as it was persisted |
| `next_after` | int \| null | The `event_id` to pass as `after` for the next page; null on an empty page |
| `has_more` | bool | Another page exists |

Visibility and refusals are the metadata read's (§10.2, §10.1's ownership and R3 paragraph; session-only, like every execution read since keys v2). A completed execution whose rows the retention job has removed answers `410 result.expired` with `details.reason = event_record_expired`; its metadata stays readable at §10.2. A running execution with no rows yet answers an empty page.

### 10.4 Cancel execution

```
DELETE /executions/{execution_id}
```

Cancels a RUNNING execution: in-flight statements are interrupted (`Statement.cancel()`), connections released, status set to `ABORTED`, and `execution_aborted` (§6.4.8) emitted to any connected stream. Scope: `execute` + ownership (`admin` may cancel any) — [Auth §7.6](/docs/auth.md#76-operation-matrix--the-permission-catalog-authoritative).

Works from **any** instance: the request writes a Redis cancellation flag that the executing instance honors within ~one heartbeat interval ([DAG Executor §8.3.1](/docs/dag-executor.md#831-the-registry)). The `204` acknowledges the cancellation *request*; the `execution_aborted` event marks its completion.

**Live delivery guarantee.** A consumer that stays connected receives every event on its live stream, terminal event included, and the stream is closed by the server after the terminal sequence (§6.5). The terminal frame may legitimately lag the `204`: up to the executor's cancel re-issue horizon (~2 s) on the executing instance — the courtesy the abort unwind allows a driver that is slow to raise — and about one heartbeat interval cross-instance (§8.3.1). A consumer that abandons the stream early (its own fallback timer, a proxy idle timeout, a reload) has no live-delivery claim for anything written after it left; that gap is exactly what §10.3's replay — and, past the replay's hour, §10.2's durable record — exist to close.

Response: `204 No Content`. Cancelling an already-terminal execution returns `409 Conflict` with `pipeline.execution.not_running`.

---

## 11. Health & Diagnostics

### 11.1 Health check

```
GET /health
```

Returns `200 OK` with service status. No auth required.

```json
{
  "status": "UP",
  "version": "1.2.3",
  "components": {
    "database": "UP",
    "redis": "UP",
    "h2_factory": "UP"
  }
}
```

### 11.2 Readiness check

```
GET /ready
```

Returns `200 OK` when the service is ready to accept traffic, `503` otherwise. Used by orchestrators (k8s).

### 11.3 The agent manual (public routes)

```
GET /skill.md
GET /skill/{name}.md
```

The manual an agent reads for THIS deployment, unauthenticated and `text/markdown` — it holds
no secret, and requiring a key would mean an agent cannot learn to use its key correctly until
after it has one (mcp-server.md §15). `/skill.md` is the operating core
(`core`); `/skill/{name}.md` serves any document of the rendered set by its flat name
(`pipelines`, `pipelines-dag`, `core-error-codes`, … — `docs_list` over MCP is the catalog).
The one-release aliases of the previous deliveries' names (`skill`, `authoring-playbook`,
`connecting`, …) answer with their successors' bytes; an unknown name is a `404` in the §4.2
envelope with `details.reason: "skill_reference_not_found"`. The bytes are the same rendered
`DocSet` the MCP tools and resources serve ([mcp-server.md §15](/docs/mcp-server.md#15-the-manual-how-an-agent-learns-this-server)).

---

## 12. Rate Limiting

### 12.1 Limits

All limits are **per principal** — a signed-in person, or a key acting as its own robot identity ([Auth §7.4](/docs/auth.md#74-issuance), keys v2 A13): a key's budget is the key's, never its creator's, so a person's session and their mcp key are two budgets, and every key a role may create is one more. (Before keys v2 a key drew from its owner's budget; since 2026-09-25 the creator's ROLE gates how many principals a workspace holds.) Key names and defaults in [Configuration §3.7](/docs/configuration.md#37-rate-limiting) and §3.2 (executor concurrency):

- Requests: `rate-limit.requests-per-second` (100), `rate-limit.requests-per-minute` (1000).
- Pipeline execution: `executor.max-concurrent-executions-per-user` (10) → `pipeline.execution.concurrency_limit`.
- SSE connections: `sse.max-streams-per-user` (50) concurrent streams per user — execution (§6), refresh (§23.3) and observed-evaluation (§21.5) streams count together. Each of the three registries counts the other two families' open streams, so the one cap holds whichever family a user's streams belong to.

The SSE counters are **per instance**: each of the three registries keeps its open streams in memory, so in a multi-instance deployment `sse.max-streams-per-user` is a per-instance cap, not a cluster-wide one. The check and the later stream registration are not atomic, and neither are the cross-family reads, so under simultaneous admissions a user can briefly exceed the cap (a bounded overshoot, each extra stream ending with its run's timeout or the disconnect grace).

The request-rate counters (`rate-limit.requests-per-second`, `rate-limit.requests-per-minute`) are tracked in Redis, so those limits hold across instances in a multi-instance deployment. The executor concurrency limit is the executor's own (`executor.max-concurrent-executions-per-user`).

### 12.2 Headers

Every response includes the IETF draft rate-limit headers:

```
RateLimit-Limit: 100
RateLimit-Remaining: 87
RateLimit-Reset: 1691234567
```

On limit exceeded: `429 Too Many Requests` with `Retry-After` header and the single system-wide code `rate_limit.exceeded` ([Pipeline Contract §13.11](/docs/pipeline-contract.md#1311-rate-limiting--idempotency)) — the same code at every layer (REST, MCP, login).

### 12.3 When the limiter itself is unavailable

The limiter **fails closed**. If its Redis cannot be reached, the request is refused — `429 Too Many Requests`, `Retry-After: 1`, and the **distinct** code `rate_limit.unavailable` (§13.11), in the standard §4.2 envelope. `/mcp` answers with the same code, because one filter meters both surfaces.

The two codes exist so a client can tell the cases apart: `rate_limit.exceeded` is the caller's own budget and is fixed by slowing down, `rate_limit.unavailable` is the service's dependency and is fixed by the operator. The response body for the unavailable case carries no `limit` — the caller spent nothing — and never names the failing dependency.

Failing open was rejected: it would make "make the limiter's Redis fail" the cheapest way past every limit in the system. There is **no configuration switch** to restore it. The outage is visible to operators on `/health` (the `redis` component the limiter shares — [§11.1](#111-health-check)) and as a single WARN per outage per instance ([Observability §3.2](/docs/observability.md#32-levels)).

---

## 13. CORS

### 13.1 Default policy

- `Access-Control-Allow-Origin`: configured per deployment (default: same-origin).
- `Access-Control-Allow-Methods`: `GET, POST, PUT, DELETE, OPTIONS`.
- `Access-Control-Allow-Headers`: `Authorization, DP-API-Key, DP-Correlation-Id, DP-CSRF-Token, DP-Result-TTL-Seconds, Content-Type, Idempotency-Key`.
- `Access-Control-Allow-Credentials`: `true` (for cookie-based UI auth).

### 13.2 SSE-specific

SSE endpoints must include CORS headers on the stream response. Browsers won't consume SSE without them.

---

## 14. Open Questions / Future Additions

Out of scope for v1:

- **Streaming result delivery via SSE**: stream rows through the SSE channel itself in `data_chunk` events, in addition to the stored-result cursor. Useful for very large results the client wants to process incrementally.
- **Async / detached execution with webhooks**: execute-and-return-immediately with delivery via webhook; would relax the §6.8 cancel-on-disconnect rule for explicitly detached runs.
- **GraphQL**: a GraphQL endpoint mirroring the REST surface, for clients that prefer it. Not in v1.
- **Webhook callbacks**: register a webhook URL and have us POST execution events there instead of (or in addition to) SSE. Useful for non-interactive integrations.
- **Result caching**: optional TTL-based caching of pipeline results keyed by `pipeline_id + version + parameters hash`. Useful for expensive pipelines that are queried with the same inputs repeatedly.
- **Arrow as default**: if Arrow IPC adoption grows, default result format could become Arrow IPC with JSON as fallback.
- **Multi-tenant deployment**: separate rate limits, datasources, and pipelines per tenant.

---

## 15. OpenAPI Specification

A complete OpenAPI 3.1 spec lives in `docs/api/openapi.yaml` and is published at `/openapi.json` on every running instance. This document is the normative reference; the OpenAPI is generated from it.

(The OpenAPI yaml is generated as a build artifact; not hand-edited.)

---

## 16. Auth & User Admin Endpoints

Flows and rules are specified in [Auth](/docs/auth.md) (§4.7 key identities, §7.4 issuance, §7.5 key roles, §7.6 the permission catalog); this section defines the HTTP surface. All endpoints below live under `/api/v1`.

### 16.1 API keys (creation is on the Keys page — keys v2)

```
GET /auth/api-keys
```
Lists the keys the caller CREATED — their own MCP key and any key they minted (id, name, `kind`, `role`, `identity`, `created_by`, created_at, expires_at, last_used_at, is_revoked). Never returns secrets. (`mcp_key.own` — every role.)

```
POST /auth/api-keys
Content-Type: application/json

{"name": "nightly-sync", "kind": "endpoint", "role": "api_caller", "bindings": ["/nyc"], "expires_at": "2027-08-07T00:00:00Z"}
```
The ONE creation path for every kind (keys v2 A15 — the login mint is retired; the same route serves the Keys page's dialog). **`kind` is REQUIRED** — naming no kind is `400 auth.key_kind_not_mintable` (there is no default kind to fall back to). The create PERMISSION follows the kind: `mcp_key.create` for `mcp` (author, promoter, workspace admin — the LOWEST of the three create permissions, so the route's floor), `api_key.create` for `endpoint` (workspace admin), `server_key.create` for `server` (super admin). On an `mcp` key the requested `role` must also pass **the subset rule** (A14): a creator may give a key only a role whose permission set is a subset of their own in that workspace — a role outside it is `403 auth.role_required` (`details.required`, `details.held`). The key is created with its own identity in one transaction ([Auth §4.7](/docs/auth.md#47-key-identities)). Response `201`:

```json
{
  "schema_version": 1,
  "correlation_id": "uuid",
  "data": {
    "id": "dpk_ab12cd34ef56",
    "name": "nightly-sync",
    "kind": "endpoint",
    "role": "api_caller",
    "identity": {"id": "uuid", "display_name": "nightly-sync"},
    "created_by": "uuid",
    "bindings": ["/nyc"],
    "key": "dpk_ab12cd34ef56.9f8e7d6c...",
    "created_at": "2026-09-08T09:00:00Z",
    "expires_at": "2027-08-07T00:00:00Z"
  }
}
```

`key` is the full plaintext, returned **exactly once** — it is never retrievable again.

**`kind`** ([Auth §7.7](/docs/auth.md#77-key-kinds-and-published-endpoint-bindings), [Enums §8A](/docs/enums.md#8a-apikeykind--what-an-api-key-is)) is `mcp`, `endpoint`, `server`, or `dashboard` (L5). It changes what the rest of the body means:

| `kind` | `role` | `bindings` | Who may create |
|---|---|---|---|
| `mcp` (V37, renamed from `user` — A19) | one of `author` \| `promoter` \| `workspace_admin`, chosen under the subset rule (A14) — never viewer (A15), never `super_admin` (B1) | Refused | Author (author only), promoter (promoter only), workspace admin (any of the three), super admin (any) — `mcp_key.create` + the subset rule |
| `endpoint` | `api_caller` (the only one offered, record A1); another role is `403 endpoint.key_kind_refused` | The endpoint-tree nodes it authorises, validated BEFORE the key is minted — and since 2026-09-21 (#191) each must be the root or lie at or above a path the CALLER'S workspace publishes (`400 endpoint.path_invalid`, naming only the caller's own tree) | Workspace admins and super admins (`api_key.create`) |
| `server` | `promotion_receiver`; another role is `403 endpoint.key_kind_refused` | Refused | Super admin only (`server_key.create`) |
| `dashboard` (L5, #367) | `dashboard_viewer`; another role is `403 endpoint.key_kind_refused` | Refused — the dashboard FOLDERS are a separate verb, `dashboard.key.bind` on the key's card after mint (§23.2's binding routes) | Workspace admins and super admins (`api_key.create` — the lane's decision, the endpoint key's floor) |

**`scopes` is refused by name** (`400 pipeline.execution.invalid_parameter_type`, `details.field = "scopes"`): scopes were removed (#215, PK8), and a caller who still sends them believes the key will carry them. An unknown `role` token is the same 400 with `details.supported`. **A name already taken by a live key in the workspace is `409 auth.key_name_taken`** (keys v2 A18; revoking the old key frees the name). **An unbound published path is unservable until a key is bound to it** (#215 B3) — a session is refused on the published tree, and no other key kind reaches it.

**Body shape (#333).** The body is read through the strict request mapper: a number or boolean for `name`, `role`, `kind` or a `bindings` entry is `400 pipeline.execution.invalid_parameter_type` (`details.reason: "wrong_type"`, `details.path`, `details.expected`), an unknown key is the same code with `reason: "unknown_key"` (the key, clipped, in `path`), and an absent or null `name` is `reason: "missing"` — never the key named "12" and never a silently dropped field. Before #333 an unreadable body here answered the pipeline family's `schema_version_unsupported` stand-in.

An unknown `kind` is `403 endpoint.key_kind_refused`, naming the supported values. A `server` key is the credential a SENDING deployment presents as `DP-Promotion-Key` (§18); it authenticates nothing on this API — presented as `DP-API-Key` it is refused on every route with `403 endpoint.key_kind_refused` and `details.reason = "server_key_off_surface"`.

```
DELETE /auth/api-keys/{key_id}
```
Revokes a key the caller CREATED (creator-scoped; `mcp_key.revoke_own`, author and above — the service checks `created_by`). Revoking any key of the workspace is the workspace admin's `api_key.revoke` (a `server` key's is `server_key.revoke`). Revoking an `endpoint` or `server` key deactivates its identity in the same transaction; revoking a member's key ends with their removal (A17/B6). Effective ≤ cache TTL, ~60s. `204 No Content`.

### 16.2 Current principal

```
GET /auth/me
```
Returns the authenticated principal: `user_id`, `email`, `display_name`, `role` (the role it is judged as — the active membership's, `super_admin`, or a key role; informative), `auth_method`, `key_id` (when key-authenticated). A session's surface — the MCP key is refused here (B2).

### 16.3 User administration (super admin — `user.manage`)

```
GET  /auth/users?q={search}&offset=0&limit=50     — list users
GET  /auth/users/{user_id}                         — user detail
POST /auth/users/{user_id}/deactivate              — is_active = false (effective ≤ ~60s: every session and key of that user then answers 401 auth.principal_deactivated, Auth §11A.3)
POST /auth/users/{user_id}/activate                — is_active = true
POST /auth/users/{user_id}/grant-admin             — is_admin = true
POST /auth/users/{user_id}/revoke-admin            — is_admin = false
```

All return the standard envelopes; mutations return the updated user record and write the corresponding `auth.user.*` audit events ([Auth §10.1](/docs/auth.md#101-events)). Every route answers people only: a key's identity or the System row is the unknown-user `404` (`details.reason = "user_not_found"`), looked up BEFORE any mutation ([Auth §4.7](/docs/auth.md#47-key-identities)), and the list returns `human` rows only. There is no user-create endpoint — users are provisioned by OIDC first login only ([Auth §4.2](/docs/auth.md#42-user-provisioning)). The `q` search term is trimmed and blank means no search; its own `%` and `_` are matched literally. A term over 200 characters after the trim is refused `400 request.query_too_long` (#504b, [pipeline-contract §13.21](/docs/pipeline-contract.md)) with `details.limit` and `details.length`, before any read; the term is never echoed.

### 16.4 Logout (browser session)

```
POST /logout
```
Clears the `dp_session` cookie ([Auth §6.5](/docs/auth.md#65-logout)). Root-level (not under `/api/v1`), CSRF-protected, listed here for completeness.

---

## 17. Workspace Endpoints

Workspaces are the unit of team isolation ([workspaces design](https://github.com/msabiransari/datapipelines/blob/main/docs/superpowers/specs/2026-08-16-workspaces-design.md) §9; [Auth §5.6](/docs/auth.md#56-workspace-resolution--the-dp-workspace-header) resolves the ACTIVE workspace per request). Every endpoint below lives under `/api/v1`. Scope minimums are in [Auth §7.6](/docs/auth.md#76-operation-matrix--the-permission-catalog-authoritative); the role/mode gates (owner-or-admin, provisioning mode, `open-join`) are enforced in the service layer, default-deny.

**The no-oracle rule** ([pipeline-contract §13.12](/docs/pipeline-contract.md#1312-workspace-resolution)): for anyone but a global admin, an unknown workspace name and a workspace the caller is not a member of are the SAME `403 workspace.membership_required` — a name cannot be probed. A global admin (who could otherwise see any workspace) gets a real `404 workspace.not_found`. A member who is not the owner of a workspace they ARE in gets the same 403 for management operations — role probing is an oracle too.

### 17.1 List own workspaces

```
GET /workspaces
```
The caller's memberships (design §9 "list-own"): `{name, role, active, joined_at}` rows, `role` one of `viewer` \| `author` \| `promoter` \| `workspace_admin` ([enums §8C](/docs/enums.md#8c-workspacerole--the-one-role-a-membership-holds); D1, 2026-09-20 — the three booleans V23's flags had put on the wire are gone). **Every member's read** — it is the list the chrome's switcher draws from, so it sits on the switcher's row (`workspace.switch`), not on the page's (`workspace.read`, which D13 narrowed to admins: §17.2 and §17.6). A super admin gets every workspace on the instance, deactivated ones marked `active: false`.

### 17.2 Get workspace

```
GET /workspaces/{name}
```
`{name, display_name, is_personal, created_at, active, deactivated_at}`. A workspace admin of that workspace, or a super admin (D13) — everyone else gets the 403/404 split above.

### 17.3 Create workspace

```
POST /workspaces
{"name": "team-etl", "display_name": "Team ETL"}
```
`display_name` optional (defaults to `name`). **Super admins only** (D-R11): provisioning modes were retired with round 1, and with them `workspace.creation_forbidden` — a non-super-admin is refused by the matrix's own `auth.role_required`. The creator takes NO membership row: a super admin is an implicit member of every workspace (D-R8), and an explicit one would be indistinguishable from a granted membership in the audit. Errors: `400 workspace.validation.name_invalid` (`[a-z0-9_-]+`, 1–63), `409 workspace.validation.duplicate_name` (global namespace, soft-deleted included).

### 17.4 Update workspace

```
PUT /workspaces/{name}
{"display_name": "Team ETL (renamed)"}
```
Renames the display name; `name` is immutable v1. An absent `display_name` keeps the current one. Owner or global admin.

### 17.5 Delete workspace

```
DELETE /workspaces/{name}
```
Soft delete. `409 workspace.in_use` while the workspace still owns non-deleted pipelines, templates or (workspace-bound) datasources — `details.counts` names what blocks, by kind. Owner or global admin.

### 17.5a Deactivate / reactivate a workspace

```
POST /workspaces/{name}/deactivate
POST /workspaces/{name}/reactivate
```
Super admin (`workspace.lifecycle`). Deactivate, never delete (D-R10): nothing is purged; the workspace stops being selectable, its published endpoints become unknown paths to every caller, keys pinned to it answer `404 auth.key_workspace_inactive` and a server key pinned to it is refused by the promotion peer — all within the ~60 s liveness window ([Auth §11A.3](/docs/auth.md#11a3-deactivation)). Returns the §17.2 record (`active: false`, `deactivated_at` set). Reactivate restores every credential; reactivating an already-active workspace is `404 workspace.not_found` (the 404 rule: the surface owes no "it was not deactivated"). Both are audited (`workspace.deactivated` / `workspace.reactivated`).

### 17.6 List members

```
GET /workspaces/{name}/members
```
TWO arrays, never mixed ([Auth §4.6](/docs/auth.md#46-invitations)) — a client that counts members must not count ghosts:

- `members[]`: `{user_id, email, display_name, role, joined_at}` rows, oldest membership first — `role` is the member's ONE role (D1). A workspace admin of that workspace, or a super admin (D13).
- `invitations[]`: `{email, role, invited_by, invited_at}` rows — the pending, email-keyed invitations ([Auth §4.6](/docs/auth.md#46-invitations)) whose `users` row does not exist yet, each carrying the one role it will materialise with (D20). No `user_id`: there is no user yet; the email IS the identity.

### 17.7 Add member

```
POST /workspaces/{name}/members
{"email": "bob@example.com", "role": "author"}
```
Workspace admin or super admin (`workspace.members.manage`; D-R11 retired the `open-join` self-service path). `role` is optional, one of `viewer` \| `author` \| `promoter` \| `workspace_admin`; absent = `viewer` — a body that says nothing asks for the least a membership can be. A value outside the four is the surface's bad-parameter 400 (`pipeline.execution.invalid_parameter_type`, `details.field = "role"`, `details.allowed` the four) — never stored. The email is normalized lowercase before every lookup and store (the [Auth §4.2](/docs/auth.md#42-user-provisioning) rule). Adding an existing member is idempotent: the existing membership is returned unchanged — changing a role is §17.10.

TWO outcomes, distinguishable by status AND body, so a client never mistakes a ghost for a member:

- **`200`** with the membership row — the `users` row existed; the person is now (or already was) a member.
- **`202`** with `{"invited": true, "email", "role"}` — nobody by that email exists yet; an invitation ([Auth §4.6](/docs/auth.md#46-invitations)) was created, upserting over any earlier one (the latest admin decision wins, audited), and their first login materialises it.

A missing or non-textual `email` is the surface's generic bad-parameter 400 (`pipeline.execution.invalid_parameter_type`, `details.field = "email"`). An email the §4.3 domain allowlist would refuse at login is refused HERE with the SAME code the login would use — `auth.login.domain_not_allowed` at a 400 — because an invitation that could never be honoured is a trap. A deactivated workspace is `workspace.inactive` (404).

### 17.8 Remove member

```
DELETE /workspaces/{name}/members/{user_id}
```
Workspace admin or super admin. Removing YOURSELF is refused with `409 workspace.self_membership` (#208 — nobody administers their own membership; another admin does). Removing the LAST workspace admin is refused with `409 workspace.last_admin` — a workspace with no admin is unmanageable, and the caller's next step is "give someone else the workspace admin role first" (§13.12). A `user_id` that names no member of the workspace is the workspace's own 404 (`workspace.not_found`), so the member list cannot be probed one id at a time. The member's login-minted key pinned to this workspace is revoked in the same act (§17.11's effect, reason `member_removed`; [Auth §7.4](/docs/auth.md#74-issuance)) — a key is tied to user + workspace.

### 17.9 Revoke an invitation

```
DELETE /workspaces/{name}/invitations/{email}
```
Workspace admin or super admin (`workspace.members.manage`). Removes a pending invitation ([Auth §4.6](/docs/auth.md#46-invitations)) — `204` on success. An email with no invitation in this workspace is `404 workspace.invitation.not_found`: the workspace itself resolved, so the not-found thing is the invitation, and one answer for "revoked already", "never created" and "another workspace's" keeps an admin from probing which emails hold pending invitations. The email is normalized lowercase, so revoking `Bob@Company.com` finds the row the invite of `bob@company.com` created. A super admin may revoke inside a DEACTIVATED workspace (its pending invitations wait; cleanup before reactivation is the point).

### 17.10 Set a member's role

```
PUT /workspaces/{name}/members/{user_id}
{"role": "promoter"}
```
Workspace admin or super admin. REPLACES the membership's ONE role (D1, 2026-09-20; the three booleans of the flags era are gone from this body — see the change log). `role` is one of the four; absent = `viewer`, unknown = the §17.7 400. Changing your OWN role is `409 workspace.self_membership` (#208). Demoting the LAST workspace admin is `409 workspace.last_admin`. Returns the §17.6 member row. The audit event is `workspace.member_role_changed` (`from`, `to`).

### 17.11 Revoke a member's key

```
DELETE /workspaces/{name}/members/{user_id}/key
```
Workspace admin or super admin (`workspace.members.manage`, the §17.8 row). Revokes the member's login-minted `user` key pinned to this workspace WITHOUT removing them (#200; roles record §3.7, ruling 3; [Auth §7.4](/docs/auth.md#74-issuance)). `204` — also when the member holds no live key: the verb is IDEMPOTENT, "already revoked" is the success it reports, so an admin retrying after a timeout cannot turn cleanup into a failure. A `user_id` that names no member is the workspace's own 404 (`workspace.not_found`, the §17.8 rule).

The member's SESSION is untouched — revoking a key is not deactivation — and their next login or workspace entry mints a fresh key. There is no automatic rotation on password or identity events (§3.7 ruling 2: the product cannot tell a forgotten password from a compromise, so recovery is an admin act — this verb, or §17.8's removal). The audit event is `auth.api_key.revoked_by_admin` with `reason: admin_revoked`; the removal path's revoke carries `reason: member_removed`. No key id, prefix or plaintext appears in any response.

---

## 18. Promotion Endpoints (receiver)

The RECEIVER half of promotion ([Versioning §10](/docs/versioning.md#10-promotion-ui-driven-separate-use-case)). Two endpoints, and they are the only routes under `/api/v1/promotion/`.

**Authentication is different here, and deliberately narrower.** These routes are gated by the server key of §10.6, presented as `DP-Promotion-Key`. The header is read on this prefix and no other, so the credential authenticates nothing anywhere else and grants no read access outside this pair. The converse also holds: an ordinary API key or a session cookie does **not** open these routes — promotion is a deployment-to-deployment channel, not a privileged human one.

**How the receiver mints one (091; the page moved in 179).** On the receiver, a super admin opens the API keys page (`/api-keys`), creates a key of kind **`server`** (the `promotion_receiver` role, no bindings — the form hides bindings for this kind) and copies the plaintext, which is shown once. That value goes into the SENDER's `datapipelines.deployment.promotion.target.server-key`. Equivalently over REST, on the receiver:

```
POST /auth/api-keys   {"name": "uat receiver", "kind": "server"}
→ 201 {"data": {"id": "dpk_…", "kind": "server", "role": "promotion_receiver", "key": "dpk_….<secret>"}}
```

Rotation is then a receiver-side act with no restart on either side: mint a second `server` key, set it on the sender, revoke the first. The older pre-shared config value (`datapipelines.deployment.promotion.server-key`) is still accepted for one release and WARNs at boot ([Configuration §3.19](/docs/configuration.md#319-deployment)).

**Fail closed.** A receiver with no configured value AND no live `server` key refuses every request here. Every refusal on this prefix — missing header, malformed header, wrong key, a key of the wrong KIND, a revoked or expired key, and "nothing configured on this deployment" — is the same `401 auth.promotion.key_invalid` with the same body, so a caller can tell none of them apart.

There is **no MCP twin** for either endpoint and no schedule that calls them ([Versioning §10.1](/docs/versioning.md#101-policy) D8: promotion is a human action).

### 18.1 Promotion inventory

```
GET /promotion/inventory?workspace={name}
```

What this deployment already holds in `{name}` — the sender's whole delta input (§10.2) and the datasource set §10.5 pre-validates against, in one round trip.

```json
{
  "schema_version": 1,
  "correlation_id": "...",
  "data": {
    "deployment": "uat",
    "authoring_enabled": false,
    "workspace": "acme",
    "pipelines": [
      { "name": "acme/finance/daily_revenue", "current_version": 3, "body_hash": "sha256-..." }
    ],
    "templates": [
      { "name": "finance/revenue.sql", "current_version": 2, "body_hash": "sha256-..." }
    ],
    "datasources": ["sales_db", "warehouse"],
    "parameter_sets": [ { "name": "acme/sales/region_filters", "current_version": 4, "body_hash": "sha256-..." } ],
    "visualizations": [ { "name": "acme/charts/monthly_revenue", "current_version": 1, "body_hash": "sha256-..." } ],
    "dashboards": [ { "name": "acme/dash/revenue_overview", "current_version": 2, "body_hash": "sha256-..." } ]
  }
}
```

`parameter_sets` (#194 lane D, the record's §8.3), `visualizations` and `dashboards` (#10 L1c) are the same `(name, current_version, body_hash)` triple for the workspace's sets and released artifacts; a sender that predates an engine reads an inventory whose extra lists it ignores.

**The sender caches this answer (178).** The promoter lens evaluates §10.2 on every read a promoter makes, so the sender reads this endpoint through a per-workspace cache (`datapipelines.deployment.promotion.inventory-cache-ttl-seconds`, default 60 s — [Configuration §3.19](/docs/configuration.md#319-deployment)); an unreachable answer is remembered for the same window, a successful push invalidates the entry, and the push path itself always reads fresh (§10.3).

All five inventory arms carry only objects whose current pointer names a RELEASED version. The pointer may name a draft after a development-posture discard fallback or manual switch (versioning §3.4, §10.2); such an object is not promotable and its version and hash are omitted, as are never-released objects. `authoring_enabled` lets a sender refuse a misconfigured target before building a batch instead of after pushing one.

Workspaces are addressed by **name**, because names are a global namespace and ids are not — a name is the one identifier that means the same thing on both deployments. An unknown name is `404 workspace.not_found`, not the no-oracle `403` an ordinary caller gets (§17): the peer is a trusted deployment, and "you do not have that workspace" is the answer its operator needs.

| Error | HTTP | When |
|---|---|---|
| `auth.promotion.key_invalid` | 401 | Any credential failure on this prefix, including no key configured here |
| `workspace.not_found` | 404 | This deployment has no workspace with that name |

### 18.2 Promotion push

```
POST /promotion/push
```

Apply one batch. **All of it, or none of it** ([§10.4](/docs/versioning.md#104-push-order-dependency-closure)): the receiver applies the batch in a single transaction, so a mid-batch failure cannot leave pipelines whose template pins or child pipelines do not resolve. The batch is read through the strict request mapper (#333): a number or boolean where it declares a string (`source_env`, `key_fingerprint`, `workspace`, an endpoint entry's `path`, `pipeline`, `description` or `bindings` entry) is `400 pipeline.execution.invalid_parameter_type` (`details.reason: "wrong_type"`, `details.path`). Unknown keys stay tolerated — the batch is a cross-version wire, so a newer sender's additive field is ignored, not refused.

```json
{
  "source_env": "dev",
  "key_fingerprint": "sha256:1a2b3c4d5e6f",
  "workspace": "acme",
  "templates": [ { "id": "finance/revenue.sql", "version": 2, "body_hash": "...", "body": "...", "...": "..." } ],
  "pipelines": [ { "id": "…uuid…", "version": 4, "body_hash": "...", "name": "acme/finance/daily_revenue", "nodes": [] } ],
  "parameter_sets": [ { "id": "3f2a…", "name": "acme/sales/region_filters", "version": 4, "body_hash": "…", "released_at": "…", "parameters": [] } ],
  "visualizations": [ { "id": "9c1b…", "name": "acme/charts/monthly_revenue", "version": 1, "body_hash": "…", "released_at": "…", "renderer": { "…": "…" } } ],
  "dashboards": [ { "id": "51e0…", "name": "acme/dash/revenue_overview", "version": 2, "body_hash": "…", "released_at": "…", "sources": [] } ]
}
```

`parameter_sets` (#194 lane D) are the §21.4 envelopes' `parameter_set` NODES (a whole envelope as an entry is refused by the strict bind), applied in the same one transaction AFTER the templates (a set's pins must resolve) and BEFORE the pipelines — the record's §8.3 order. A pin the batch does not bring is `400 parameter.import.missing_template`; the id is KEPT (P24) and an id held by another workspace's set refuses `409 parameter.version.conflict` / `id_taken` (C29).

`visualizations` and `dashboards` (#10 L1c, [Dashboards §3.3](/docs/dashboards.md#33-export-and-import)) are the versions' payloads — the body with its lifecycle fields, NEVER the §22/§23 export envelopes (the transform templates ride the batch's template closure) — applied LAST in the receiver's order (§10.4: templates → sets → pipelines → endpoints → visualizations → dashboards, D61), bound through the families' strict readers and landed inside the same transaction, where the just-landed rows are what the import lens resolves against (neither validator opens a customer datasource, so there is no probe wall — unlike the sets, no out-of-transaction validation phase). **The arms are per family** (the L1c-b round, O1): a dashboard's pinned visualizations travel as entries of the `visualizations` arm — deduplicated by name + version + body hash against the explicit visualization roots and against each other — and the `dashboards` arm carries dashboards ALONE, because every entry an arm carries is bound by that family's reader (a visualization payload in the dashboard arm refuses `dashboard.validation.body_invalid` and rolls the batch back). Each arm is COUNT-bounded before any member binds (O7): at most `datapipelines.visualization.max-visualizations-per-dashboard` entries, refused with the family's `body_invalid` and `reason: too_many` before the transaction opens. **The ceiling is AGGREGATE** (the L1c-c round): the value also caps each WHOLE arm — and, on the import surfaces, each whole envelope array — as an ADDITIONAL ceiling on top of the per-document bound the key's name describes, not a consequence of it. Two dashboards that are each individually valid — 30 distinct pins apiece under a 50 ceiling — can together present 60 entries, and such a batch refuses WHOLE (one refusal naming the configured key and the count, nothing landed, no automatic split); a genuine batch at exactly the configured ceiling lands whole. A landing is judged by the RELEASE rules (O2): a batch entry lands RELEASED, so a pin it carries that exists here but is not RELEASED — a draft, or a discarded version — refuses with the family's release code (`visualization.release.dependency_not_released` / `dashboard.release.dependency_not_released`); a pin this deployment does not hold at all is `400 dashboard.import.missing_dependency`; and a refusal at ANY entry after the first rolls the whole batch back — templates included (witnessed end to end: earlier batch members land, the dashboard refuses on landing, every preceding write is absent — the L1c-c round's receive case). The sender re-raises a receiver refusal verbatim — the code, the message and the details — so the promote verb's answer names the bound, the count and the key; the promotion page renders a `body_invalid` flash as a toast naming the same.

A **template-backed set is received** (#302): each entry's full parameter-set validation — the selector probe among it, which opens a CUSTOMER-datasource connection — runs BEFORE the one transaction opens, on this deployment's own datasources, with the batch's template payloads overlaying the receiver's registry so a pin the same batch brings resolves. A refusal at that phase is the entry's own catalogued code and NOTHING has landed; the transaction then lands the pre-validated entries, and a refusal there (a brought template's hash mismatch, C29's `id_taken`) rolls the batch back whole, templates included.

`templates` and `pipelines` arrive **in push order** — template versions in the transitive `imports_json` closure first, then child pipelines, then their parents (children before parents) — and are applied in the order given. The receiver does not re-derive the closure: the sender owns that rule (§10.4), and a second implementation of it here would be a second thing to keep correct. Entries already present at the same version and hash are omitted by the sender and are an idempotent no-op if sent anyway.

Each entry is the ordinary portable body plus the [§9.2](/docs/versioning.md#92-import-with-preserved-versions) lifecycle fields (`version`, `body_hash`, `released_at`), so the receiver's preserved-version import path applies unchanged: version numbers are honored, never renumbered; `body_hash` is recomputed and must match; a local DRAFT is never clobbered; a same-hash re-import is a no-op.

`source_env` is the sender's `deployment.name` and `key_fingerprint` a truncated SHA-256 of the key it presented — **never the key**. Both are recorded by the receiver against the import (`auth.promotion.accepted`), so a promoted row's provenance survives in the audit trail. Every promoted row is stamped with the receiver's own system service account ([Auth §4.5](/docs/auth.md#45-the-system-service-account-r7)) — the payload carries no user id that would mean anything locally.

```json
{
  "schema_version": 1,
  "correlation_id": "...",
  "data": { "workspace": "acme", "source_env": "dev", "templates": 1, "pipelines": 2, "parameter_sets": 1, "visualizations": 2, "dashboards": 1 }
}
```

| Error | HTTP | When |
|---|---|---|
| `auth.promotion.key_invalid` | 401 | Any credential failure on this prefix |
| `workspace.not_found` | 404 | This deployment has no workspace with that name |
| `pipeline.promotion.target_is_authoring` | 409 | This deployment has `authoring-enabled: true` — dev is where drafts live (§10.1 D7) |
| `pipeline.import.missing_datasource` | 400 | A pushed pipeline references a datasource not registered here. The sender pre-validates (§10.5) and refuses with `pipeline.promotion.missing_datasources` before pushing; this is the receiver's own end of the same guard |
| `pipeline.import.missing_template` | 400 | A pushed pipeline pins a template version not present here and not in the batch |
| `pipeline.import.hash_mismatch` | 400 | A declared `body_hash` does not match its body — transfer corruption or canonicalization drift |
| `pipeline.import.version_conflict` | 409 | That version exists here with different content, as a local DRAFT, or was DISCARDED (§9.2's table) |
| `parameter.import.missing_template` | 400 | A pushed set pins a template version neither in the batch nor stored here — refused before anything lands |
| `parameter.validation.*` | 400 | A pushed set failed its validation on this deployment (the selector probe included — it runs BEFORE the batch's transaction, on this deployment's own datasources); `details.failures[]` lists every one |
| `parameter.version.conflict` | 409 | A pushed set's landing refused: the id held by another workspace (`details.reason = id_taken`), a hash mismatch, or a taken version number — the batch rolls back whole, templates included |

The whole batch rolls back on any of the above. The sender's own refusals — `pipeline.promotion.not_released`, `.not_newer`, `.missing_datasources`, `.target_is_authoring` — happen before the request is made and never reach this endpoint.

---

## 19. Published Endpoints

A released pipeline served as a **`GET` API** at `/api/<category>/<version>/<path…>`, whose
response is the `data_ready` payload — the first page plus the cursor — with no event handling on
the client (this section is the contract). Ruling R-EP5: the **category** is the engineer's
namespace — a business domain, a team, a product line — and the product's own API is simply its
reserved namespace: a category matching `v[0-9]+` (`/api/v1/…` today, any `v<n>` tomorrow) or the
literal `api` can never be published, so a published path cannot shadow a product route *by
construction*. There is **one** handler for the whole subtree — its mapping constrains the
category segment itself — and no runtime route registration: a published endpoint is a row, not
a mapping.

### 19.1 The path grammar

`path_pattern` is 3–10 segments — **category, version, path** — each a literal
`[a-z0-9][a-z0-9_.-]{0,63}` or a variable `{name}` matching the parameter grammar
`[a-z_][a-z0-9_]*`; at most 200 characters; a leading `/`, no trailing slash, no wildcards. The
category and the version are always **literal**: variables are allowed only after the version.
The version is one free-form segment — no pattern is enforced; `v1` is the convention. The stored
form is always the part after `/api`: a pattern handed in *with* one `/api` prefix is normalised,
not refused — which is why `/api/api/…` is refused (its category is `api`, a reserved segment).
It is the same segment grammar hierarchical template names
use ([Template hierarchy §4](https://github.com/msabiransari/datapipelines/blob/main/docs/template-hierarchy-design.md)) — one grammar to learn.

**Reserved categories.** A category matching `v[0-9]+` or equal to `api` is refused at publish
with `400 endpoint.path_reserved` naming the segment, and re-checked when the serve registry is
built — a row written around the publish path is never served. The reservation is the routing
contract: the catch-all handler's own pattern excludes those categories, so an unknown
`/api/v1/…` path answers the product's 404, not the endpoint one.

**Ambiguity is refused, not resolved.** `/nyc/v1/{x}` and `/nyc/v1/b` both match
`GET /api/nyc/v1/b`, so
publishing the second is `409 endpoint.path_conflict` naming the first. Literal-beats-variable
precedence is deliberately NOT offered in v1: because ambiguity cannot be published, at most one
pattern can match a request, and a reader of the registry can tell what a URL does by finding the
one row it matches. `UNIQUE (path_pattern)` is deployment-wide — a URL is global.

### 19.2 What may be published

The pipeline must exist in the endpoint's workspace and have a **released** version, and it must
be **side-effect-free**: every node of the current released version is `DQL` with `output.target`
in {`tempdb`, `caller`}, a `DML`/`DDL` node whose `source` is `tempdb`, or a `PIPELINE` node whose
pinned child satisfies the same rule transitively. `tempdb` is the execution's own in-memory H2,
created for the run and discarded with it — a statement against it cannot outlive the request, so
it has no effect a `GET` needs to be safe from (#171). A `DML`/`DDL` node whose `source` is a
registered datasource — or a DQL node writing back to a datasource, which is a write wearing a
read's type — refuses publication with `409 endpoint.pipeline_not_readonly` naming the node and
the datasource.

That rule is what makes `GET` safe here, and it is not a formality: `GET` is retried on timeout,
preloaded by browsers and followed by crawlers. It is **re-checked on every serve**, because an
endpoint pins a pipeline and serves its latest released version — a later release can put a write
under a live URL.

Every path variable must name a declared parameter of that version
(`400 endpoint.path_variable_unknown`); the pipeline may declare more, and they come from the
query string. `timeout_seconds` is clamped to `datapipelines.endpoints.timeout-min-seconds` /
`datapipelines.endpoints.timeout-max-seconds` ([Configuration §3.22](/docs/configuration.md#322-published-endpoints)).

### 19.3 The request

Authentication is an API key (`DP-API-Key`), never a browser session — this is a machine surface,
and a session is `401`. Authorisation is the key's hierarchical bindings
([Auth §7.7](/docs/auth.md#77-key-kinds-and-published-endpoint-bindings)): an `endpoint` key (role
`api_caller`) serves what is bound to it, the MCP key is refused here by kind, and so **a published
path with no binding on any ancestor serves no one until a key is bound to it** (#215 B3).

The accepted parameters are the released version's declared ones: **path variables** bind by
name, and the **query string** supplies the rest. Validation is strict and **reports every defect
at once** — one `400 endpoint.request.invalid` whose `details.errors[]` carries
`{parameter, code, message}` per problem, because a client fixes a request once:

- an unknown query parameter is `endpoint.request.parameter_unknown` — deliberately not ignored,
  since a typo that silently ran the default would return plausible, wrong rows;
- a repeated key is `endpoint.request.parameter_repeated`, as is a query key that also came from
  the path (the URL decides, not the query string);
- a required parameter with no value is `pipeline.execution.parameter_required`, and a value that
  is not its declared type is `pipeline.execution.invalid_parameter_type` — the existing codes.
  Nothing is trimmed: `?amount=%2012.50%20` is refused exactly as the execute body refuses
  `" 12.50 "` (the v2.36 break, #194). A value that breaks a declared rule of its parameter —
  a `constraints` bound, length or pattern, or the declared precision/scale — is
  `pipeline.execution.parameter_constraint_violation` (v2.37, pipeline-contract §6.2); a value
  judged against a STORED declaration today's save refuses (a body saved before #194 with a
  `constraints` block or `cardinality` then ignored) is `409 pipeline.execution.parameter_declaration_invalid`
  (v2.45, #268) — the pipeline must be re-saved;
- a value over 4 KB is `endpoint.request.value_too_large`.

Headers: `DP-Result-TTL-Seconds` (§7.4's clamp) and `DP-Result-Page-Rows` (R-EP4, clamped to
`page-max-rows`) are honoured; an unparseable value reads as absent, because the server clamps
anyway. `Accept` must admit `application/json` (`*/*` and an absent header do), else `406`.

**The result paging rides the business path** (keys v2 A16 — the framework reads are session-only
now): a caller that started a run pages its result under the SAME path it called —
`GET /api/<category>/v<n>/<path>/executions/{execution_id}` and
`GET /api/<category>/v<n>/<path>/executions/{execution_id}/result` — with the same handlers'
semantics as the framework's `GET /executions/{id}`[`/result`] ([REST §10](#10-execution-history)), served
to the `api` key that STARTED the run and bound to this path. A run another key (or another
workspace's key) started is the 404 rule ([Auth §11A.1](/docs/auth.md#11a1-the-404-rule)); a BROWSER
SESSION is refused here exactly as it is on the serve route — this is a machine surface, and the
session gets the same `401 auth.session.required` shape. The routes are delegated from the serve
catch-all (the business path is variable-length, so no separate mapping can exist), and
`EndpointAuthorizer`'s bound-walk runs before any read.

### 19.4 The response

`200` — the body is the `data_ready` payload of [§6.4.7](#647-data_ready), verbatim:
`execution_id`, `schema`, `rows`, `row_count`, `total_rows`, `has_more`, `result_url`,
`expires_at`, `ttl_seconds`. Headers: `DP-Correlation-Id`, `DP-Execution-Id`, and
`Cache-Control: no-store` — a result is per-execution, so a shared cache holding one would hand a
second caller another caller's rows.

The execution runs **in-process** through the same recording path an MCP execution uses — never
HTTP-to-self, never SSE parsing — as the endpoint's workspace, with `triggered_via = ENDPOINT`,
`executed_by` = the key's owner and `executed_by_key_kind = endpoint` (D11: the run is the
ENDPOINT's, not the owner's — it lists for workspace admins only, and for this key through the
serve audit). Every serve is audited with the key id, the endpoint, the execution id and the outcome.

**`202` on timeout (ruling R-EP3).** When the endpoint's `timeout_seconds` elapses the execution
is **not** cancelled — it keeps running, and the response is `202` with
`{execution_id, result_url, status_url, expires_at}`. The cursor `404`s until the result exists
and `GET /executions/{id}` reports status. A `504` would be a lie in both directions: the upstream
is fine, and the answer is coming.

### 19.5 Managing endpoints

`POST` / `GET` / `DELETE /api/v1/endpoints` (`author`), addressed by `?path=` — never by a path
segment, because a `path_pattern` contains `/` and an encoded `%2F` is refused below routing on
the pinned Tomcat (the measured reason §8 moved templates to query addressing). The `GET` listing
answers a **promoter** through the lens (178, [Auth §11A.1](/docs/auth.md#11a1-the-404-rule)): an endpoint is
listed iff its pipeline is visible to that principal, so a hidden pipeline never leaks through
the endpoint that publishes it.

Both bodies (publish, and the binding in `POST /endpoints/bindings`) are read through the strict request mapper (#333): a number or boolean where a string is declared (`path`, `pipeline`, `description`, `path_prefix`, `api_key_name`, `api_key_id`) or a string or float for `timeout_seconds` is `400 pipeline.execution.invalid_parameter_type` with `details.reason: "wrong_type"` and `details.path`; an unknown key is `reason: "unknown_key"`; a missing required key is `reason: "missing"`. Before #333 a missing `path` answered `endpoint.path_invalid` with `reason: malformed_json`.

**Legacy rows (#274).** A stored row whose `path_pattern` predates the current grammar — fewer
than three segments, saved before R-EP5 — is **retired, never fatal**: it cannot boot-refuse the
deployment ([metadata-db §4.13](/docs/metadata-db.md#413-published_endpoints)) and it never serves. The
listing answers it after the valid rows with the keys `path`, `pipeline`, `enabled`,
**`legacy: true`** and **`reason`** — the path grammar's own refusal, bounded — plus `url`; it
carries none of the valid shape's parse-derived fields (`path_variables`, `bindings`,
timestamps), because there is no parse. Once V41 has run its `enabled` is `false`; a single read
(`?path=`) resolves valid rows only and answers the legacy path `404`, like the serve path. The
one verb the legacy row supports is **`DELETE`** — the same unpublish route, unchanged — which
removes it; republishing at a current-grammar path is the replacement. The API console page lists
the row flagged with the same reason and offers only that verb. The flag is the parse verdict taken on every read, not the `retired_reason` column (an audit marker); a legacy `path` (and its `url`) is echoed as stored up to the grammar's 200 characters and cut there (#286 — `path_pattern` is `TEXT` with no CHECK, so only a database write can store a longer one; every API write has been held to 200 since the registry was born, so every row the product wrote is echoed exactly and stays addressable by `DELETE ?path=`, and a longer one is unpublished by its full stored path; the cut falls on a code-point boundary, and such a row's `reason` is the length sentence — it never quotes more of the stored path than the echo shows (#298) — and the console offers no Unpublish on it (#293)), and a row malformed in a way V41's two-segment predicate does not name is listed `legacy: true` with `enabled: true` and is never served either. A **promotion** batch (§10.5) never carries a legacy row — the target's grammar would refuse it — and the sender logs the omission (`event=endpoint.promotion_legacy_omitted`, the count over the promoted pipelines, never a path); the promotion result does not list it.

Bindings are `POST` / `DELETE /api/v1/endpoints/bindings`, and since 179 (D17) they are a
**workspace admin's verb** (`api_key.bind` — associating a credential with an endpoint tree is
no longer the publisher's). A prefix is refused with `endpoint.path_invalid` unless it is the
root or lies at or above a path the CALLER'S workspace publishes (2026-09-21, #191 — the refusal
names only the caller's own tree), and at serve time a binding decides only for an endpoint of
its own workspace ([Auth §7.7](/docs/auth.md#77-key-kinds-and-published-endpoint-bindings)). The key is
named by **`api_key_id`** (the `/api-keys` page's shape —
unambiguous, workspace-scoped, `endpoint` kind required) or, kept for REST compatibility, by
**`api_key_name`** — which resolves within the caller's OWN keys only, exactly as before:
`api_keys.name` carries no uniqueness constraint, so "the key named `ci`" is ambiguous
deployment-wide and a surface that resolved it across owners would pick one person's credential
to widen. Promotion carries endpoint rows and their bindings **of the source workspace, by key
name** — the bindings are read with the workspace predicate in the query (2026-09-21, #199), so a
neighbouring workspace's key bound at an equal node (legal: only the exact `path_pattern` is
unique deployment-wide, path trees may overlap) never rides into the batch; a target missing
that key name refuses the batch with `endpoint.promotion.key_missing` before anything is pushed.
The same operations exist as MCP tools ([MCP Server §6.2](/docs/mcp-server.md#62-tool-definitions)).

### 19.6 Status codes, complete

| Code | When |
|---|---|
| 200 | Result inline — the first page plus the cursor |
| 202 | The endpoint's timeout elapsed; the execution continues (R-EP3) |
| 400 | Request validation — every defect at once in `details.errors[]` |
| 401 | No or invalid API key; a browser session |
| 403 | Key not bound / wrong workspace / wrong key kind |
| 404 | No endpoint matches. **The same body whether the path is unknown or the endpoint is disabled** — otherwise the registry is enumerable one URL at a time |
| 405 | Any method but `GET`, with `Allow: GET` |
| 406 | An `Accept` this surface cannot satisfy |
| 429 | Rate limit or concurrency limit, with `Retry-After` |
| 500 / 502 | Pipeline failure, mapped by [Pipeline Contract §13](/docs/pipeline-contract.md#13-error-code-catalog) (datasource connection failures are `502`) |
| 503 | The pipeline has no released version, or is no longer side-effect-free |

### 19.7 Not in v1

`POST` endpoints (parameters in a body, side-effecting pipelines — the read-only rule is what
makes `GET` safe, and a write surface needs its own ruling); anonymous or embed tokens for public
dashboards; response caching across callers (results are per execution by design); CSV/Arrow by
`Accept` (the cursor's `format` already serves them); custom domains.

### 19.8 The demo API — the worked example (#224)

The demo workspace the product ships doubles as this section's worked example. When a deployment
seeds a demo family (`--demo nyc,trade,lake`), the bootstrap seeder **publishes every seeded demo
pipeline** as an endpoint under the `demo` category, deriving the path from the pipeline name by
one rule: the name's first segment is the version segment, and underscores fold to hyphens —

| Seeded pipeline | Published endpoint |
|---|---|
| `nyc/mobility/revenue_by_borough` | `GET /api/demo/nyc/mobility/revenue-by-borough` |
| `trade/balance_by_partner` | `GET /api/demo/trade/balance-by-partner` |
| `nyc/mobility/taxi_vs_rideshare` (the lake showcase) | `GET /api/demo/nyc/mobility/taxi-vs-rideshare` |

Re-seeding is idempotent: an endpoint already published at a derived path over the same pipeline
is left alone, a changed pipeline set adds and retires paths, and a family without seeded
pipelines publishes nothing.

**One key bound to all of them.** The seeder also mints exactly ONE `endpoint` key named
`demo-public-key` — an ordinary `api_caller` ([Auth §7.7](/docs/auth.md#77-key-kinds-and-published-endpoint-bindings)),
acting as its own identity, bound to every published demo path. Its plaintext comes from
configuration, `datapipelines.bootstrap.demo-api-key`
([Configuration §3.18](/docs/configuration.md#318-bootstrap)), and the site's
[demo-data page](https://datapipelines.co/demo-data) renders the same value, so an outsider can
copy the key and call any demo endpoint with `DP-API-Key` — no account:

```bash
curl -H "DP-API-Key: dpk_…" https://datapipelines.co/api/demo/nyc/mobility/revenue-by-borough?year=2024
```

```json
{"execution_id": "…", "schema": ["column", "…"], "rows": [["value", "…"]], "row_count": 1,
 "total_rows": 1, "has_more": false, "result_url": "…", "expires_at": "…"}
```

The key is **public by design** and its reach is exactly this section's surface: the bound demo
paths, plus the executions it started. On anything else it is refused like any other `api_caller`
— an unbound published path is `403 endpoint.key_not_bound`, `/mcp` and the product's own routes
are `403 endpoint.key_kind_refused`, and it can create nothing. Blank the setting and the public
API is retracted on the next boot (no key minted, the demo endpoints unpublished, the page's
section hidden); change the value and the next boot is the rotation — the old key revoked, the
new one minted and bound.

**The per-key request budget.** The serve path applies a budget to EVERY `api_caller` key —
`datapipelines.endpoints.key-request-budget` ([Configuration §3.22](/docs/configuration.md#322-published-endpoints)),
default 60 requests per 60-second window per key, per instance. The (N+1)th request inside the
window answers `429 rate_limit.exceeded` with `Retry-After` — the same code §12 meters with —
before the pipeline runs, and the key serves again when the window rolls over. A budget is a
product property of a public credential first; the demo key merely needs it first. It is also the
lake family's gate: the lake showcase's endpoint pays S3 egress on every call, so it is published
and bound only while a budget stands behind it (`max-requests` > 0).

---

## 20. Schedules

A **schedule** runs a registered executor's job — in v1 the `pipeline` executor, which runs a pipeline — at the occurrences of a five-field cron in an IANA timezone (#9; behaviour in [Scheduler](/docs/scheduler.md), the one owner of these semantics). Schedules are not versioned artifacts: no draft, no release, never promoted. Every route is under `/api/v1/schedules`, works in the caller's active workspace, and is **session-only in slice 1** — every key kind is refused here by §3.2's confinement (an application credential is a later slice). Permissions are [Auth §7.6](/docs/auth.md#76-operation-matrix--the-permission-catalog-authoritative)'s `schedule.*` rows: every member reads; authors, workspace admins and super admins create, edit, pause/resume/unblock, delete and Run now. A schedule fires as the **system identity**, never as its creator — a creator later demoted to viewer, or removed, does not stop it. Errors are [Pipeline Contract §13.19](/docs/pipeline-contract.md#1319-schedules)'s `schedule.*` codes; an id from another workspace (or one the promoter lens hides) is `404 schedule.not_found`.

**The schedule object** (every response that returns a schedule; the `ETag` header carries `"<revision>"`):

| Field | Type | Description |
|---|---|---|
| `id` | uuid | The schedule |
| `name` | string | A folder path, `finance/daily/revenue` — the pipeline/template name grammar, unique among the workspace's live schedules; rename/move keeps the id and the history |
| `revision` | int | Starts at 1; every edit, pause, resume, unblock and delete a person makes bumps it (a block the scheduler sets does not) |
| `executor` | string | The registered executor (`pipeline`) |
| `payload_schema_version` | int | The executor's payload schema version (1) |
| `payload` | object | The executor's payload. For `pipeline`: `{"pipeline": "<name>", "version": "current"}` — `current` follows the pipeline's current-version pointer at each run, drafts included; nothing else is accepted in v1. Since slice 3 the payload MAY also carry the optional, additive `parameter_bindings` object (§20.2) |
| `parameters` | object | Literal values for the pipeline's declared parameters, validated by the binder an interactive run uses |
| `target_ref` | string | The executor's reference to what it runs (`pipeline:<name>`) |
| `cron` | string | Five fields, Unix style (`minute hour day-of-month month day-of-week`) |
| `timezone` | string | IANA region id (`America/New_York`) |
| `missed_run_policy` | string | `skip` (default) \| `latest` |
| `notifications` | object | `{recipient_count, events, delivery: {state: "off", reason: "disabled"}}`; `recipients` is present ONLY when the workspace context permits `schedule.update`. Every other reader gets count/events/delivery and no addresses. Events serialize in order `start`, `success`, `failure`, `unknown`, `blocked` (#442a) |
| `enabled` | bool | `false` while paused |
| `condition` | string | `enabled` \| `paused` \| `blocked` — blocked wins |
| `blocked` | object \| null | `{reason, at, run_id}` while blocked (`run_unknown`, or an executor refusal such as `pointer_null`) |
| `next_due_at` | timestamp \| null | The next occurrence the dispatcher will record; ignored while paused or blocked (resume and unblock recompute it from now) |
| `created_by`, `updated_by` | uuid | People |
| `created_by_name`, `updated_by_name` | string | The display names of the people the ids name (#261) — one batched read per response, the same resolution the pipelines explorer uses; a stamp whose user row no longer exists falls back to the id's first eight characters |
| `created_at`, `updated_at` | timestamp | |

**The run object** (§20.10–§20.12):

| Field | Type | Description |
|---|---|---|
| `id` | uuid | The run |
| `schedule_id` | uuid | Its schedule |
| `origin` | string | `cron` \| `catch_up` \| `manual` |
| `scheduled_at` | timestamp \| null | The occurrence (UTC); null for a manual run |
| `reference_at`, `reference_timezone` | timestamp, string | The run's frozen reference time — the occurrence for cron and catch-up runs, the accepted request time for a manual one |
| `admit_by` | timestamp | The end of its admission window (`datapipelines.scheduler.lateness-seconds`) |
| `schedule_revision` | int | The revision it was recorded under |
| `state`, `reason` | string, string \| null | [Scheduler §5](/docs/scheduler.md#5-runs-states-and-reasons) |
| `execution_id` | uuid \| null | The execution it launched (read it at §10.2, its messages at §10.3A); null when none was |
| `prepared` | object \| null | The executor's frozen snapshot — for `pipeline`, `{pipeline_id, version, body_sha256}` plus `resolved_parameters` (slice 3): the literal map the run actually executed with, bindings included |
| `requested_by` | uuid \| null | The person who pressed Run now |
| `requested_by_name` | string \| null | That person's display name (#261); the same resolution as the schedule's people |
| `attempts` | int | Capacity retries |
| `created_at`, `claimed_at`, `started_at`, `finished_at` | timestamp | Lifecycle; the last three null until they happen. `claimed_at`/`started_at` are claim and launch stamps, and `finished_at` is when the RECONCILER recorded the terminal state — up to one tick after the execution ended — so none of them answers "how long did it take" |
| `execution_started_at`, `execution_completed_at` | timestamp \| null | The EXECUTION's own start and end, copied by the reconciler from the execution row it already reads (#258); null until a terminal carrying them is recorded (runs that never launched, and runs finished before the field existed) |
| `execution_duration_ms` | int \| null | `execution_completed_at − execution_started_at`: how long the execution took, answered in this one read for every member who may read the schedule — no `execution.read` needed, and no per-run execution read on a runs list |

### 20.1 List schedules

`GET /schedules?prefix=finance&offset=0&limit=50` — `schedule.read`. The workspace's live schedules by name, `prefix` narrowing to a folder's subtree — a folder path of 1–9 segments with no trailing `/` (`finance`, `finance/daily`; anything else is `400 schedule.validation.request_invalid`, `details.field = prefix`); the §4.3 page envelope. A promoter sees only schedules whose target the promoter lens admits.

`condition` (#641, additive) narrows the list to one of the three words a schedule's `condition` reads: `enabled` (not blocked, enabled), `paused` (not blocked, paused) or `blocked` (blocked, paused or not — blocked wins, as the field does). The predicate is the database read's, before the promoter lens, so `has_more` and the offset count filtered rows; absent or empty is no filter, and any other value — including another case or surrounding space — is `400 schedule.validation.request_invalid` with `details.field = condition` and `details.expected = "one of enabled, paused, blocked"`. `GET /schedules?condition=blocked&limit=50` lists what the Schedules home's Blocked tile counts.

`outcome` (#725, additive) narrows the list to the schedules whose NEWEST run — by `created_at`, ties broken by run id, the same run the Schedules home reads — falls in one of the five Last run buckets: `succeeded` (`succeeded`), `failed` (`failed`, `aborted`, `unknown`), `in-progress` (`queued`, `starting`, `running`), `did-not-run` (`not_started`, `skipped`, `cancelled`) or `never-ran` (no run at all). Like `condition`, the predicate is the database read's, before the promoter lens, so `has_more` and the offset count filtered rows; the two compose (both must hold), and each composes with `prefix`. Absent or empty is no filter, and any other value — including another case, surrounding space or a run state that is not a bucket's word (`aborted`, `not_started`) — is `400 schedule.validation.request_invalid` with `details.field = outcome` and `details.expected = "one of succeeded, failed, in-progress, did-not-run, never-ran"`. `GET /api/v1/schedules?outcome=failed&limit=50` lists what the Schedules home's Failed bar counts. A schedule item carries no run state, so a run that starts or ends after the read moves the schedule between buckets only on the next read.

### 20.2 Create a schedule

`POST /schedules` — `schedule.create`. Body: `name`, `payload`, `cron`, `timezone` (required); `executor` (default `pipeline`), `parameters` (default `{}`), `missed_run_policy` (default `skip`). Optional `Idempotency-Key`, held durably: a replay of the same request answers `200` with the original schedule; a replay with a different body is `409 idempotency.key_reused_for_different_request`. `201` with the schedule and its `ETag`. Refusals: the `schedule.validation.*` family, `409 schedule.name_taken`, `409 schedule.limit.per_workspace`, and a parameter refusal exactly as an interactive run's (§13.4).

**`notifications`** (#442a) is optional: absent or JSON null creates no recipients and events `failure`, `unknown`, `blocked` (start/success off). A present object accepts optional `recipients` (list of strings) and `events` (list of the five event names); an omitted member uses its create default. Example: `{"recipients":["a@example.com"],"events":["failure","unknown","blocked"]}`. Addresses are trimmed, domains lower-cased and duplicates removed case-insensitively, preserving first spelling and order; events deduplicate. One WHATWG valid e-mail address grammar, at most 254 characters, refuses header injection. The configured recipient limit defaults to 20. The idempotency hash includes normalized effective settings; case-only duplicates and reordered event choices replay the same meaning. Pre-V49 create keys still replay when the effective notification settings are the old defaults (no recipients and the three default events); different settings cannot match that legacy hash. A schedule's `notifications` in a response is `{recipient_count, events, delivery}`, plus `recipients` for a `schedule.update` holder only; `delivery` is the DEPLOYMENT's computed state, never stored (#442b, [Scheduler §2.3](/docs/scheduler.md#23-notification-settings-442)): `{"state":"on"}`, or `{"state":"off","reason":…}` with `disabled` (the enable flag is false — the default), `mail_unconfigured` or `no_allowlist` (the `development` posture with an empty recipient-domain allowlist). Settings are saved in every state; while off nothing is queued or sent.

A refused `notifications` object is `400 schedule.validation.notifications_invalid` (#442a; the catalogue row is pipeline-contract §13.19). `details` contains only `{field, reason}`: `notifications`, `notifications.recipients`, an indexed recipient or event field; `not_an_object`, `not_a_list`, `not_a_string`, `syntax`, `too_long`, `too_many`, `unknown_event`, and `domain_not_allowed` (#442b: a recipient outside a non-empty `recipient-domains` allowlist, [Configuration §3.29](/docs/configuration.md#329-scheduler-9)). No address is echoed

**`payload.parameter_bindings`** (#9 slice 3) — an optional, ADDITIVE key of payload schema 1. One binding per declared `DATE` parameter:

```json
{"pipeline": "<name>", "version": "current",
 "parameter_bindings": {
   "as_of_date":    {"source": "keyword", "name": "TODAY"},
   "previous_date": {"source": "keyword", "name": "YESTERDAY"},
   "label":         {"source": "literal", "value": "TODAY"}
 }}
```

Keywords are an exact allowlist — `TODAY` and `YESTERDAY`, uppercase — and are matched ONLY inside an explicit `{"source": "keyword"}` binding, never by scanning strings: a literal STRING `"TODAY"` in `parameters` reaches the pipeline as the string. Each run resolves its bindings on the run's frozen reference time (§20.11's `reference_at`/`reference_timezone`) — the SCHEDULE'S timezone and its logical occurrence time, not the run's actual start: a run due 23:55 that starts 00:05 the next day still resolves `TODAY` to the day it was due for (`YESTERDAY` is a calendar day before, not 24 hours). A keyword binding is validated structurally at save; a keyword with no reference at run time is `binding_invalid`/`no_reference`. Refusals: a bound name the version does not declare, or that is not `DATE` (`400 schedule.validation.binding_invalid`, `details.reason` `unknown_parameter` / `type_mismatch` / `unknown_keyword` / `unknown_source` / `literal_invalid`); the same name in `parameters` and `parameter_bindings` (`400 schedule.validation.binding_conflict` — give it one source; there is no precedence rule). A `reference`, `reference_at` or `reference_timezone` key inside the payload or a binding is `400 schedule.validation.payload_invalid` (`details.reason` `unknown_field`): the run's time is the scheduler's frozen context, never a client field.

**The save-time placeholder (#269).** To bind a required keyword-bound `DATE` parameter at save, the save binds a PLACEHOLDER DATE — today, UTC — for each keyword-bound name, never persisted; each run resolves its own real value on its frozen reference. Consequence for constrained DATE parameters: a `min`/`max` constraint that excludes today makes the SAVE pass or refuse on a date no run will ever bind. The save's parameter checks that mean anything for a bound name are the structural ones (declared, DATE, the keyword allowlist, the conflict rule); the prepare-time bind is the real gate for values. A schedule whose bindings stop fitting the current version refuses its run `not_started` / `parameters_invalid` and blocks, exactly as a literal that no longer binds; a run whose frozen snapshot somehow carries no resolved map for a bindings-carrying schedule is refused `not_started` / `snapshot_unresolved` and does NOT block (a rolling deploy between prepare and start; the next occurrence re-prepares).

### 20.3 Preview a pattern

`GET /schedules/preview?cron=30%202%20*%20*%20*&timezone=America/New_York&count=5` — `schedule.read`. The next `count` (1–20, default 5) occurrences of a pattern before any save, from the same function the dispatcher uses: `{cron, timezone, occurrences: [{at, local, offset}]}` — `at` the UTC instant, `local` the wall-clock time, `offset` the zone offset then in force. The DST rule shows here: a time inside a spring-forward gap runs at the transition, a time inside a fall-back overlap runs once, at its first pass.

### 20.4 Get a schedule

`GET /schedules/{id}` — `schedule.read`. The schedule, `ETag: "<revision>"`.

### 20.5 Edit a schedule

`PUT /schedules/{id}` with `If-Match: "<revision>"` — `schedule.update`. Body as §20.2 (the whole schedule), `parameter_bindings` included — replacing the saved bindings wholesale, like every other body field. A stale revision is `409 schedule.revision_conflict` (`details.current_revision`); a missing `If-Match` is refused as on §5.5. Changing the cron or timezone recomputes `next_due_at` from now — occurrences of the old pattern are not "missed". Runs already recorded keep the revision they were recorded under.

**Additive frozen-contract rule (#442a):** absent or JSON-null `notifications` KEEPS the stored settings, so clients written before this field cannot wipe them. A present object replaces each supplied member; an omitted member keeps its stored value. `recipients: []` turns mail off; `events: []` selects no events. Neither omission resets a member to its create default.

### 20.6 Delete a schedule

`DELETE /schedules/{id}` with `If-Match` — `schedule.delete`. `204`. A soft delete: the schedule disappears from every read and records no more occurrences, a run still queued ends `skipped` / `schedule_deleted` when its turn comes, its history and any running execution stay, and its name is free again.

### 20.7 Pause and resume

`POST /schedules/{id}/pause`, `POST /schedules/{id}/resume` — `schedule.pause`; both idempotent, both return the schedule. Occurrences due while paused are not missed and are not caught up: resume recomputes from now. Resume never clears a block.

### 20.8 Unblock

`POST /schedules/{id}/unblock` — `schedule.pause`. A schedule blocks when a run's outcome is `unknown` (nobody can say whether its work happened) or an executor refusal asks for it (the pipeline has no current version, the parameters no longer bind, …). Unblock re-validates the saved payload and parameters first — a refusal there is the same `400` a save would get — then clears the block, recomputes `next_due_at` from now and writes an `unblocked` row on the blocking run's trail. `409 schedule.not_blocked` when it is not blocked.

### 20.9 Upcoming occurrences

`GET /schedules/{id}/upcoming?count=5` — `schedule.read`. As §20.3, for the saved schedule.

### 20.10 List runs

`GET /schedules/{id}/runs?offset=0&limit=50` — `schedule.read`. Its runs, newest first, the §4.3 page envelope. A deleted schedule's runs are not reachable here (its id is `404`); their executions stay readable at §10.

### 20.11 Get a run

`GET /schedules/{id}/runs/{run_id}` — `schedule.read`. The run plus its frozen `payload` and `parameters` and its `trail`: `[{seq, kind, reason, at, worker, details}]`, append-only, `seq` 1, 2, 3, … The execution's own events are not duplicated here — read them at §10.3A with the run's `execution_id`. When the run was prepared, `prepared.resolved_parameters` (slice 3) is the literal map — bindings already resolved to dates — that the execution was launched with; the frozen `parameters` stay exactly what the schedule holds.

**`deliveries`** (#442b) — the run's scheduler mail, one row per notice and recipient, in transition then recipient order: `[{id, notice, state, attempts, resend_count, next_attempt_at, sent_at, last_error_code, recipient}]`. `notice` is the policy outcome the message announces (`started`, `completed`, `failed`, `could_not_start`, `cancelled`, `aborted`, `unknown`, `unknown_update`, `blocked`); `state` is `pending` \| `accepted` \| `failed` \| `uncertain` ([Enums §44](/docs/enums.md#44-deliverystate--one-recipients-copy-of-a-notice-442b)); `last_error_code` is the code of the last failed attempt or refusal (`connect_failed`, `smtp_refused`, `domain_not_allowed`, `interrupted`, … — the same section). `recipient` is present ONLY when the workspace context permits `schedule.update`, exactly as the schedule's `recipients` (§20 schedule object); every other reader gets the row with no part of the address — not its domain. `accepted` means the relay took the message, not that it reached an inbox. The sends also appear in `trail` as `notification_sent` / `notification_failed` / `notification_uncertain` rows, whose `details` carry ids, the attempt count and the code, never an address.

### 20.12 Run now

`POST /schedules/{id}/run` — `schedule.run`. Records a manual run (`origin = manual`, `requested_by` = you) and answers `202` with it; it then runs like any other — as the system identity, through the same capacity and admission. Allowed on a paused schedule; refused on a blocked one (`409 schedule.blocked`) and while one of its runs is queued, starting or running (`409 schedule.run.overlap`). Optional `Idempotency-Key`, durable: the same key answers `200` with the original run.

### 20.13 Resend a notification

`POST /schedules/{id}/runs/{run_id}/deliveries/{delivery_id}/resend` — `schedule.update` (the permission that edits the recipients sends to them; session only, with the session's CSRF header). Queues a `failed` or `uncertain` delivery again (#442b): it becomes `pending` and due now, its `attempts` KEPT and `resend_count` + 1, under its own Message-ID; the sender re-checks the deployment's mail state and the recipient-domain allowlist when it sends, so a resend while mail is off ends `failed` with the reason. `200` with the delivery object (§20.11's row, `recipient` included). When the delivery was `uncertain` the body also carries `"warning": "may_duplicate"` — the relay may already hold the first copy, and the recipient can receive the message twice (one Message-ID, so a mail client may thread or collapse them). A `pending` or `accepted` delivery is `409 schedule.notification.not_resendable` (`details.state`); a delivery id not on that run is `404 schedule.notification.not_found`; an unknown schedule or run is the usual `404`. A resend at the deployment pending cap is `409 schedule.notification.backlog_full` (`details.pending`, `details.backlog_max`): serialized admission leaves the row unchanged and writes no audit row. Every successful resend writes one `schedule.notification_resent` audit row ([Enums §15](/docs/enums.md#15-authauditevent--auth-audit-log-events)): ids and the prior state, never the address.

---

## 21. Parameter Sets

The parameter engine's REST surface (the [parameter-engine design record](https://github.com/msabiransari/datapipelines/blob/main/docs/superpowers/specs/2026-09-21-parameter-engine-design.md) §9.2 — the **pipelines** routes are the mould, addressed by **id** per P24; a multi-segment NAME never travels in a path segment, it appears only in bodies and `?prefix=`). Authorization is the nine `parameter_set.*` rows of [Auth §7.6](/docs/auth.md#76-operation-matrix--the-permission-catalog-authoritative); every route is session-or-key like §5 (the MCP key refused as everywhere off `/mcp`). Envelopes per §4; every refusal is a `parameter.*` code of [Pipeline Contract §13.20](/docs/pipeline-contract.md#1320-parameter-sets). The first-party Parameter Sets workspace ([ui-screens §4.23](/docs/ui-screens.md)) reads and evaluates through this section; the observed evaluation stream is §21.5. Its pages are UI routes, not part of this contract — `GET /parameter-sets`, `GET /parameter-sets/{id}?version=&tab=` and the tree fragment `GET /partials/parameter-sets/tree` are session-authenticated HTML on `parameter_set.read` (the promoter through the lens), and the page's live form streams §21.5 (the observed evaluation, #383) with the VIEWED version always explicit — §21.3 stays the plain route every other client calls. **Audit:** no new event kinds — the routes ride the request-interceptor logging and, for MCP, the dispatcher's `mcp.tool.called`; one evaluate writes exactly that call row (the datasource reads it causes are the selector statements themselves).

### 21.1 The entity and its document

A **parameter set** is a versioned definition: the §3 document (`name`, `display_name`, `description`, `parameters[]`) stored with the folder-path grammar and the **templates' version lifecycle** (draft / release / discard / restore / purge / switch / import — [Versioning §3.5](/docs/versioning.md#35-the-lifecycle-table)). The name is immutable; the id is a UUID the server assigns at creation and an import KEEPS (P24), so it is stable across environments.

Reads follow the working-version rule ([Versioning §7.1](/docs/versioning.md#71-authoring-reads-return-the-working-version-039)): a GET answers the DRAFT when one exists, else the current RELEASED version, and says which — `version` and `status` name the returned row, `current_version` the latest release, and a `draft` pointer carries the draft's `body_hash` whenever one exists. Every mutation takes `If-Match: <body_hash>` ([Versioning §4.2](/docs/versioning.md#42-the-precondition-protocol)); a stale hash is `409 parameter.version.conflict`.

### 21.2 Routes

| Route | Permission | What |
|---|---|---|
| `POST /api/v1/parameter-sets` | `parameter_set.create` | Create: validate the §3 document in full (the record's §4, including the metadata execution of every source template — a set whose datasource is down is not saved), land version 1 **DRAFT** with a server-assigned id. `201`. |
| `GET /api/v1/parameter-sets` | `parameter_set.read` | The flat listing (offset/limit) — EVERY set of the workspace the caller's lens admits, at its listed version, paged against a lens-true `total` (#312: a flat read, never the tree's ROOT level, which the folder grammar keeps empty) — or, with `?prefix=`, ONE level of the set tree: `folders` with their subtree counts and the level's sets (the pipelines §5.7 shape; `parameter_set_count` per folder). `total` counts the parameter **sets** the caller's lens admits — on the flat listing the workspace's admitted sets, on the browse the level's — never the folders (folders are navigation; the sets are what `has_more` pages) and never sets outside the lens. `has_more` is `offset + page size < total` — truthful on both listings. `?prefix=` (empty) is the ROOT and is a different request from an absent `prefix`. An optional `q` limits the flat listing to the sets whose name, display name or description contains it, case-insensitively (names are paths, so `q` matches across the **full path**); blank/absent `q` is the plain listing, and `q` is ignored while `?prefix=` is present (browse and search are different presentations). `q` is bounded at **200 characters after the trim** (#490, the shared rule since #504b, [pipeline-contract §13.21](/docs/pipeline-contract.md) — the same bound every surface's search applies and the web UI's boxes truncate to): a longer one is refused `400 request.query_too_long` with `details.limit` and `details.length`, before any read, and the term is never echoed. |
| `GET /api/v1/parameter-sets/{id}` | `parameter_set.read` | The working version's full JSON (the document + `id`, `version`, `status`, `body_hash`, `current_version`, `draft`). |
| `GET /api/v1/parameter-sets/{id}/versions` | `parameter_set.read` | Version metadata, newest first; no bodies. |
| `GET /api/v1/parameter-sets/{id}/versions/{version}` | `parameter_set.read` | One version. A version other than RELEASED is 404 under a narrowing lens (the promoter's). |
| `PUT /api/v1/parameter-sets/{id}` | `parameter_set.update` | The draft write (copy-on-write, then in-place; `If-Match` required). The hash precondition is checked BEFORE the body is parsed. An identical body is a no-op and reports the current state. |
| `POST /api/v1/parameter-sets/{id}/release?release_pinned_templates=` | `parameter_set.release` | Release (lock) the draft. Every pinned template version must be RELEASED (`409 parameter.release.dependency_not_released`, `details.pins_not_released`) — or, with `release_pinned_templates=true`, is released WITH the set in one transaction, templates first (142). Release re-runs the §4 source validation against the pins as they are now. |
| `POST /api/v1/parameter-sets/{id}/draft/discard` | `parameter_set.version.manage` | Purge the DRAFT (versioning §5.4); the sole draft takes the set with it. Session-only. **`409 parameter.in_use`** while a dashboard pins it (#320) — see the note below the table. |
| `POST /api/v1/parameter-sets/{id}/versions/{version}/discard` | `parameter_set.version.manage` | Discard a RELEASED version (reversible); the pointer falls back per D60. `409 parameter.in_use` while a live dashboard version pins it (#320). |
| `POST /api/v1/parameter-sets/{id}/versions/{version}/restore` | `parameter_set.version.manage` | Restore a DISCARDED version; the pointer moves only above-current-or-NULL. Its selector-template pins are re-judged first under the writer guard (#517): missing or DISCARDED → `400 parameter.validation.template_not_found` / `template_version_not_found`; DRAFT → `409 parameter.release.dependency_not_released` (§5.10's shape, `reason: not_released`). |
| `DELETE /api/v1/parameter-sets/{id}/versions/{version}` | `parameter_set.version.manage` | Purge a DRAFT version. A release is discarded, never purged (`409 parameter.version.last_release`); `409 parameter.in_use` while a dashboard pins the draft (#320). |
| `POST /api/v1/parameter-sets/{id}/current` | `parameter_set.switch_version` | The manual switch (the promotion receiver's rollout/rollback lever): body `{"version": n}`. A missing `version` — or one that is not an integer — refuses `400 parameter.validation.body_invalid` (`details.path = "version"`, `details.reason = "missing"` / `"wrong_type"`); the set lookup happens only after the body's shape is judged. Session-only. |
| `DELETE /api/v1/parameter-sets/{id}` | `parameter_set.delete` | The entity purge — only when the set's only version is a DRAFT; `409 parameter.in_use` while ANY stored dashboard version of a live dashboard pins any version of the set (#320). |
| `GET /api/v1/parameter-sets/{id}/export` | `parameter_set.read` | The export bundle (§21.4). Released-only: a never-released set is `404 parameter.not_found` naming the set. Lensed (auth §7.6's `lens` cell, §11A.1): a set the promoter lens hides is the same `404` — never a body, never an existence oracle. The bundled closure is the set's pins, not lensed; the root is (#344). |
| `POST /api/v1/parameter-sets/import` | `parameter_set.import` | Import the bundle (§21.4): templates first, then the set; the exported id is KEPT. |
| `POST /api/v1/parameter-sets/{id}/evaluate` | `parameter_set.evaluate` | **Evaluate** (§21.3) — the whole set re-rendered against the submitted selections. |

Every read and write is workspace-scoped: a set of another workspace — or one the promoter lens hides — answers the same `404 parameter.not_found` an absent id gets; nothing confirms existence across workspaces.

**`parameter.in_use` (#320).** A dashboard pins a set by name and exact version (`parameter_set`), and a set a dashboard pins is not purged or discarded out from under it: `409 parameter.in_use`, `details` `{id, version?, pinned_by: [{dashboard, version, status}]}` — the exact version for a discard and for a draft purge that leaves other versions, every stored dashboard version for an entity purge and for a draft purge that takes the entity ([Versioning §3.5.3](/docs/versioning.md#353-the-reverse-arrows-into-other-families-320)). Every verb that can refuse it is an author verb (auth §7.6), so the refusal names dashboards to a caller who may read them all.

### 21.3 Evaluate

```
POST /api/v1/parameter-sets/{id}/evaluate
{ "version": 4,
  "selections": { "country": "USA", "state": "NY", "city": null, "min_order_amount": 250.00 } }
```

- `version` optional: the SERVED version (`current_version`) when absent; an explicit version — a DRAFT by its number, the working-version read rule — is read exactly, never rounded or fallen back. A PRESENT `version` that is not an integer refuses `400 parameter.validation.body_invalid` (`details.path = "version"`, `details.reason = "wrong_type"`) — `{"version":"2"}` never evaluates the served version. An explicit JSON `null` is ABSENT here (the served version) — the switch's `{"version": null}` is `missing`, since a switch without a target is meaningless. `selections` carries EVERY parameter's current value, wire-encoded for its type, a `MULTI` as an array; the first render sends `{}`. A PRESENT `selections` that is not an object refuses the same `body_invalid` (`details.path = "selections"`) — it is never silently read as "nothing chosen". Absent, `null` and `[]` are one signal — nothing chosen — and walk the selection priority (P26); there is no cleared state. An unknown key refuses the whole request (`400 parameter.evaluate.unknown_parameter`). The body's shape is judged before the set is resolved.
- **Request bound (stated):** the body is refused over **1,048,576 UTF-8 bytes** before its JSON is parsed — `413 request.body_too_large` (`details.limit_bytes = 1048576`), the platform's envelope (§4.2, §13.21). The 2 MiB `RequestLimits` filter on the `/api/v1` routes (Configuration §3.31) remains the PRE-READ bound; this cap is the route's own pre-PARSE bound. The largest legal selections document for any set is bounded by the engine's own caps (`max-parameters-per-set`=64 × `max-input-length`=4,096 ≈ 256 KiB), so the cap refuses nothing legal. Per-value validation is the shared validator (P28): `parameter.evaluate.invalid_value_type` / `constraint_violation` (a `DECIMAL(12,2)` submitted `12.345` is refused, `details.reason = "scale"` — never rounded).
- The response is the runtime's JSON **verbatim** inside the §4 envelope — `id`, `name`, `version`, `valid`, `org` (the deployment's currency for `currency` formats), `values` (the consumer payload: one canonical value per parameter, hidden and disabled included) and `parameters[]` (the full definition, `dependents`, and `state`: `value`, `origin` (`client`/`default`/`first`/`source`/`none`), `computed_default`, `reset`, `hidden`, `disabled`, `options`, `errors`). Per-parameter errors ride `state.errors[]` in a **200** (`valid: false`); the whole-request refusals are `parameter.evaluate.timeout` (**504**, the executor-deadline mapping), `parameter.evaluate.response_too_large` (**413**) and `parameter.evaluate.unknown_parameter` (**400**). A selector's datasource failure marks ITS parameter and the form stays whole.
- Cost: one selector statement per template-backed parameter plus one datasource metadata read per evaluate (measured, the record's C25). There is no options cache in round one.

### 21.4 Export and import

```json
{ "parameter_set": { "id": "3f2a…", "name": "acme/sales/region_filters", "version": 4,
                     "body_hash": "sha256-…", "released_at": "…", "display_name": "…", "parameters": [] },
  "templates": [ { "id": "acme/sales/states_of_country.sql", "version": 3, "body": "…", "…": "…" } ],
  "manifest": { "parameter_set_id": "3f2a…", "parameter_set_version": 4, "parameter_set_body_hash": "sha256-…",
                "template_pins": [ { "id": "acme/sales/states_of_country.sql", "version": 3 } ], "exported_at": "…" } }
```

Export bundles the CURRENT release — the set body with its lifecycle fields, the pinned template versions with their transitive `imports` closure, and the manifest. Import reads the same envelope: the `templates` array, when present, is imported first (**templates before sets** — the promotion order), then the set; a pin this deployment still lacks is `400 parameter.import.missing_template`. The id is kept (P24), so an import is environment-stable; an id already held by ANOTHER workspace's set refuses `409 parameter.version.conflict` with `details.reason = "id_taken"` (C29 — the id is refused, never re-issued; re-issuing would break the identity P24 grants). A same-version, same-hash re-import is an idempotent no-op.

| Error | HTTP | When |
|---|---|---|
| `parameter.not_found` | 404 | No such set (or version) in the workspace, hidden by the lens, or another workspace's |
| `parameter.validation.*` | 400 | The document failed the record's §4 (the full failure list rides `details.failures`); `duplicate_name` is **409**. A request body whose shape is wrong — the switch's missing `version`, or a present `version`/`selections` of the wrong JSON type — is `parameter.validation.body_invalid` with `details.path` naming the key and `details.reason` `missing` / `wrong_type`; a body that cannot be read at all — not JSON, or nested past §13.21's depth of 100 — is the same code with `details.reason: malformed_json` on every route (§4.2, #323) |
| `parameter.authoring.disabled` | 403 | A promotion receiver refuses every authoring write (reads, evaluate and import unaffected) |
| `parameter.version.conflict` | 409 | A stale `If-Match`, a taken version number, or a kept id held by another workspace (`id_taken`, C29) |
| `parameter.version.not_draft` / `.not_released` / `.not_discarded` / `.last_release` / `.not_eligible` | 409 | The lifecycle precondition of the verb |
| `parameter.release.dependency_not_released` | 409 | Release with a pinned template version still DRAFT (`details.pins_not_released`) |
| `parameter.import.missing_template` | 400 | An import whose pinned template is absent here (templates are promoted first) |
| `parameter.evaluate.unknown_parameter` | 400 | A `selections` key names no parameter of the set |
| `parameter.evaluate.timeout` | 504 | The evaluate's deadline (`evaluate-timeout-seconds`) passed |
| `parameter.evaluate.response_too_large` | 413 | The response would exceed `max-evaluate-response-bytes` — options are never truncated |
| `request.body_too_large` | 413 | The evaluate body over the route's stated 1,048,576-byte cap (`details.limit_bytes`), refused before its JSON is parsed; the same code the platform filter answers over `datapipelines.web.max-request-bytes` (§4.2) |

### 21.5 The observed evaluation

The Parameter Sets workspace's stream (the [parameter-set workspace spec](https://github.com/msabiransari/datapipelines/blob/main/docs/superpowers/specs/2026-10-02-parameter-set-workspace-spec.md) §4, R1; its page landed with S1, #374, and the page's client streams it — ui-screens §4.23's live form is this route's consumer, #383). It is the §21.3 act — the same engine, the same `SelectorPool` bulkhead, the same caps and the same row, `parameter_set.evaluate` — with the cascade's progress streamed as Server-Sent Events. It is a separate route (the owner's §11.4 ruling) so that §21.3 stays byte-for-byte what it was, pinned by a golden of its response bytes. No ordinary caller attaches an observer: `POST …/evaluate`, the dashboard runtime and the `parameter_sets_evaluate` tool get the completed response and produce no trace events. Unlike the rest of §21 the route is **session-only**: no key kind reaches it, and it has no MCP placement (spec §7).

| Route | Request | Answer |
|---|---|---|
| `POST /api/v1/parameter-sets/{id}/evaluations` | `{version, selections, evaluation_id, instance_id}` | `200` `text/event-stream` (below) |

- `version` is **REQUIRED** — the page is version-explicit. Absent or `null` is `missing`; a value that is not an integer is `wrong_type`. It is read exactly through the caller's lens, never clamped and never fallen back to the served version. `selections` follows §21.3 exactly: absent or `null` is `{}`, a non-object is `wrong_type`, and an unknown key refuses the whole request.
- `evaluation_id` is a fresh **v4 UUID the client mints** per attempt, in its canonical lower-case spelling (the owner's §11.3 ruling). Every frame carries it, and the page drops any frame whose id is not its current attempt's. Missing is `missing`, a non-string is `wrong_type`, and a non-UUID, another version or a non-canonical spelling is `malformed`. An id already open on this instance, or already recorded in the caller's workspace history (it is the history record's key, #376), is `reused`. Reuse is judged against this instance's open streams AND the workspace's history; the history read is the caller's workspace's own, so another workspace's id answers as unused (no cross-workspace signal; its insert conflict is the backstop). A concurrent first use of one id on two instances is that START insert's conflict — logged at ERROR, the loser's record lost, its evaluation still run. `instance_id` (the page instance, for diagnostics only — the server keeps no per-instance state) follows the same rule.
- **Refusals, in order** — a refused observation opens nothing and starts nothing:
  1. The route's row (the scope interceptor, before the handler, as for every route).
  2. The body's shape: the §21.3 pre-parse bound of 1,048,576 UTF-8 bytes (`413 request.body_too_large`), then the fields above (`400 parameter.validation.body_invalid` with `details.path` and `details.reason`, never the value).
  3. The set and the explicit version through the lens. A hidden set or a missing version answers the IDENTICAL `404 parameter.not_found` §21.3 answers.
  4. Every selections key against the set (`400 parameter.evaluate.unknown_parameter`, the evaluator's own judge, the same `details`).
  5. A reused `evaluation_id` — open here or already in the workspace's history (`400 parameter.validation.body_invalid`, `details.path = evaluation_id`, `details.reason = reused`).
  6. The per-user SSE cap (§12.1): execution, refresh and observed-evaluation streams count together (`429 rate_limit.exceeded`).

**The stream.** The execution stream's framing (§6): `event:` name, `id:` monotonic per stream from 1, `data:` JSON, and a `: heartbeat` comment while quiet. Before EVERY write, the heartbeat included, the stream re-judges its subscriber in order:

1. The session's expiry.
2. The user's liveness.
3. The workspace the stream OPENED in, strictly re-resolved and matched by immutable id (another membership preserves nothing).
4. `parameter_set.evaluate`.

A refusal ends the stream at that write with a final `: revoked` comment, and the evaluation runs on to its end (#343's rule). Frames carry identity and progress only — **never a value, an option, SQL, a bind or a driver message**. The events (spec §4.2, FROZEN):

| Event | `data` |
|---|---|
| `evaluation_started` | `{evaluation_id, parameter_set_id, version, deadline_at}` — always first |
| `parameter_waiting` | `{evaluation_id, parameter, waiting_on: [parents]}` — its coroutine awaits its parents (a parameter without parents never waits) |
| `parameter_admitted` | `{evaluation_id, parameter}` — template-backed; the `SelectorPool` running slot is held |
| `parameter_running` | `{evaluation_id, parameter, datasource, template: {id, version}}` — render + run began on the selector's worker (the statement reaches the driver inside it) |
| `parameter_resolved` | `{evaluation_id, parameter, origin, reset, rows?}` — the parameter resolved with no error; `rows` = the option count when a query produced them (a template `SELECT`) |
| `parameter_failed` | `{evaluation_id, parameter, code, detail?}` — the parameter carries an error: the first error's catalogued code; `detail` is that code's own `details.reason` word (≤ 200 characters), never its message. A saturated selector is THIS event with `parameter.evaluate.selectors_saturated` |
| `evaluation_completed` | `{evaluation_id, response}` — `response` is the §21.3 response, byte-identical to what `POST …/evaluate` returns for the same evaluation; always the last frame on success |
| `evaluation_failed` | `{evaluation_id, code}` — the whole-request refusal (`parameter.evaluate.timeout`, `parameter.evaluate.response_too_large`; a server fault carries the §4 backstop's `pipeline.execution.aborted`, as §21.3's 500 does). The spec's optional `message` is never sent. Always the last frame on failure |

**Terminal and coverage rules.** `evaluation_completed` or `evaluation_failed` is always the last frame; nothing follows it. On `evaluation_completed`, every parameter of the set has had exactly one `parameter_resolved` or `parameter_failed`: hidden and disabled parameters included, and constants-resolved and input-typed parameters with no query frames. On `evaluation_failed`, the parameters still unfinished when the request was refused get no per-parameter frame, because their coroutines were cancelled. The client marks every node without a terminal state `failed` from the terminal frame (#375 D3, a clarification of the spec's coverage rule). A stream that ends without a terminal frame is a transport failure ([Dashboards §6.6](/docs/dashboards.md#66-the-states-notifications-and-the-csp)'s rule).

**A disconnected client aborts its evaluation** (the owner's §11.7 ruling, superseding the spec's "a disconnect does NOT cancel"). A client gone past `datapipelines.sse.disconnect-grace-seconds` (§6.8, the same key) has its evaluation's coroutines cancelled: the evaluate deadline's own path runs, and a running statement is abandoned (statement cancel, connection discard). The evaluation ends `ABORTED`, and no frame says so — the table has no aborted event and the catalogue no aborted code; it is a log event. With the defaults the grace (30 s) and `datapipelines.parameters.evaluate-timeout-seconds` (30 s) coincide, so the deadline usually ends an abandoned evaluation first. The abort fires only where the deadline exceeds the grace, and it is built for that case. No abort route exists: the page supersedes an attempt by closing its stream, and the grace does the rest. A stream cut by the authority check is not a disconnect.

| Error | HTTP | When |
|---|---|---|
| `parameter.validation.body_invalid` | 400 | The body is unreadable, or a field is missing (`missing`), mistyped (`wrong_type`), not a canonical v4 UUID (`malformed`), or an `evaluation_id` already open on this instance or already recorded in the caller's workspace history (`reused`) |
| `request.body_too_large` | 413 | The body exceeds 1,048,576 UTF-8 bytes — refused before its JSON is parsed |
| `auth.role_required` | 403 | The caller's role lacks `parameter_set.evaluate` (the promoter) |
| `parameter.not_found` | 404 | No such set, a set the lens hides, or no such version — the identical body §21.3 answers |
| `parameter.evaluate.unknown_parameter` | 400 | A selections key names no parameter of the set — judged before the stream opens |
| `rate_limit.exceeded` | 429 | The per-user concurrent-stream cap (§12.1) — execution, refresh and observed-evaluation streams count together; no stream opened |
| `parameter.evaluate.timeout` / `.response_too_large` | — | Inside the stream only: `evaluation_failed` |

## 22. Visualizations

The visualization artifact's REST surface (#10; the [dashboard implementation spec](https://github.com/msabiransari/datapipelines/blob/main/docs/superpowers/specs/2026-09-28-dashboard-implementation-spec.md) §6.1 — §21's parameter-set routes are the mould, addressed by **id** per P24; a multi-segment NAME never travels in a path segment). The document is [Dashboards §2.1](/docs/dashboards.md#21-visualization); the lifecycle is [Dashboards §3](/docs/dashboards.md#3-the-lifecycle). Authorization is the seven `visualization.*` rows of [Auth §7.6](/docs/auth.md#76-operation-matrix--the-permission-catalog-authoritative); every route is session-or-key like §5 (the MCP key refused as everywhere off `/mcp`), and the version verbs are session-only like §21's. Envelopes per §4; every refusal is a `visualization.*` code of [Pipeline Contract §13.22](/docs/pipeline-contract.md#1322-visualizations). **Audit:** the lifecycle verbs ride the request-interceptor logging (and a refusal's `auth.scope.denied`); the MCP tools ride the dispatcher's `mcp.tool.called`; the transfer routes (below) write `visualization.exported` / `visualization.imported` (enums §15).

### 22.1 Reading and writing

Reads follow the working-version rule ([Versioning §7.1](/docs/versioning.md#71-authoring-reads-return-the-working-version-039)) exactly as §21.1 states it: a GET answers the DRAFT when one exists, else the current RELEASED version; `version` and `status` name the returned row, `current_version` the latest release, and a `draft` pointer (its `version`, `body_hash`, `updated_by`, `updated_at`) rides the whole-view read — never a promoter's. Every read is the caller's LENS: a promoter sees the visualizations a dashboard her lens admits pins (§23.1), RELEASED only, and anything else answers the same `404 visualization.not_found` an absent id gets. Every mutation takes `If-Match: <body_hash>` ([Versioning §4.2](/docs/versioning.md#42-the-precondition-protocol)); a stale hash is `409 visualization.version.conflict`.

**Bodies.** A `POST`/`PUT` body is read through the request copy of the module's strict mapper (§13.21's nesting, string and number bounds) and bound by the visualization READER, which refuses an unknown key, a wrong JSON type or a missing key naming its path, and applies the seven `datapipelines.visualization.*` bounds ([Configuration §3.33](/docs/configuration.md)) before it walks a member; the platform's 2 MiB request cap (§4.2) stands ahead of both. A body that is not JSON — or no body where one is required — is `400 visualization.validation.body_invalid` with `details.reason = "malformed_json"`, never another family's code and never a 500.

### 22.2 Routes

| Route | Permission | What |
|---|---|---|
| `POST /api/v1/visualizations` | `visualization.create` | Create: read and validate the document in full (the renderer's configuration schema, the bindings, the transform pin — a DRAFT pin is accepted at save — and the test cases), land version 1 **DRAFT** with a server-assigned id. `201`. |
| `GET /api/v1/visualizations` | `visualization.read` | The flat listing (offset/limit) — every visualization the caller's lens admits at its listed version, paged against a lens-true `total` (#312's shape) — or, with `?prefix=`, ONE level of the tree: `folders` (with `visualization_count`) and the level's `visualizations`. `total` counts visualizations, never folders; `has_more` is `offset + page size < total` on both. `?prefix=` (empty) is the ROOT. |
| `GET /api/v1/visualizations/{id}` | `visualization.read` | The working version's full JSON: the document + `id`, `name`, `version`, `status`, `body_hash`, `current_version`, `created_at`, `updated_at`, `draft`. |
| `GET /api/v1/visualizations/{id}/versions` | `visualization.read` | Version metadata, newest first; no bodies. RELEASED ones only under a narrowing lens. |
| `GET /api/v1/visualizations/{id}/versions/{version}` | `visualization.read` | One version. A version other than RELEASED is `404` under a narrowing lens. |
| `PUT /api/v1/visualizations/{id}` | `visualization.update` | The draft write (copy-on-write after a release, then in place; `If-Match` required and checked BEFORE the body is parsed). A document naming another visualization is `400 visualization.validation.name_invalid` (`reason: immutable`) — nothing is renamed. |
| `POST /api/v1/visualizations/{id}/release?release_pinned_templates=` | `visualization.release` | Release the draft at `If-Match` ([Dashboards §3.1](/docs/dashboards.md#31-releasing-a-visualization)): at least one test case, the rules re-run against the pin as it is NOW, the transform pin RELEASED — or released WITH the visualization under `release_pinned_templates=true` (D61, the 142 cascade) — then, in ONE transaction, the evidence gate (#352): the LATEST test run of the exact draft must be GREEN and the mechanical test must pass again now — the refusal ladder is below the table. The answer names `templates_released`. |
| `POST /api/v1/visualizations/{id}/draft/discard` | `visualization.version.manage` | Purge the DRAFT at `If-Match` (versioning §5.4); the sole draft takes the visualization. `204`. Session-only. |
| `POST /api/v1/visualizations/{id}/versions/{version}/discard` | `visualization.version.manage` | Discard a RELEASED version (reversible); the pointer falls back per D60. Session-only. |
| `POST /api/v1/visualizations/{id}/versions/{version}/restore` | `visualization.version.manage` | Restore a DISCARDED version; the pointer moves only above-current-or-NULL. Session-only. |
| `DELETE /api/v1/visualizations/{id}/versions/{version}` | `visualization.version.manage` | Purge a DRAFT version (a release is discarded, never purged — `409 visualization.version.last_release`). `204`. Session-only. |
| `POST /api/v1/visualizations/{id}/current` | `visualization.switch_version` | The manual switch: body `{"version": n}`, its shape judged BEFORE the lookup (`400 visualization.validation.body_invalid`, `details.path = "version"`, `details.reason` `missing` / `wrong_type`). Session-only. |
| `DELETE /api/v1/visualizations/{id}` | `visualization.delete` | The entity purge — only when the only version is a DRAFT. `204`. Session-only. |
| `GET /api/v1/visualizations/{id}/export` | `visualization.read` | The export envelope (§12, [Dashboards §3.3](/docs/dashboards.md#33-export-and-import)): the current release's payload — body + lifecycle fields — the pinned transform template's `imports` closure, and the manifest (ids, version, hash, `exported_at`, `evidence: null` until the test sessions land). Lensed (auth §7.6's `lens` cell): a visualization the promoter lens hides is the same `404` an absent id gets — never a body, never an existence oracle (§21.2's mould). The bundled closure is the visualization's pins, not lensed; the root is (#344). Audited `visualization.exported`. |
| `POST /api/v1/visualizations/import` | `visualization.import` | Import the envelope ([Dashboards §3.3](/docs/dashboards.md#33-export-and-import)): the bundled templates FIRST (the promotion order), then the visualization; the exported id is KEPT (P24). The body is read through the §13.21 request mapper, then bound through the visualization READER — the seven `datapipelines.visualization.*` bounds hold on import as on save, and the `templates` array is count-bounded (O7: at most `max-visualizations-per-dashboard` entries, `body_invalid` `reason: too_many` before any member binds) — so a present-but-non-integer `version` and a present-but-non-array `templates` refuse `400 visualization.validation.body_invalid` naming the path, never a silent fallback. Lands RELEASED, judged by the RELEASE rules (O2): a transform pin that exists here but is not RELEASED refuses `409 visualization.release.dependency_not_released`; a pin this deployment lacks is `400 visualization.import.missing_template` (the bundle's own templates land first, in their own transaction — C35's shape, the transfer E2E's case 4 pins it). The workspace-admin verb by the owner's ruling (2026-09-29, the `api_key.bind` cells). Audited `visualization.imported` with `imported_with_evidence` recorded verbatim from the manifest. |
| `POST /api/v1/visualizations/{id}/tests/sessions` | `visualization.update` | Open a test session on the WORKING version (#353; spec §11.2). `201` `{session_id, run_id, visualization_id, version, body_hash, preview_url, expires_at, cases}` — `preview_url` carries the preview capability (`?session=`), shown ONCE and nowhere else; absolute when `datapipelines.auth.base-url` is set, else root-relative. TTL `datapipelines.visualization.session-ttl-minutes` (60). A version without a case: `400 visualization.validation.test_case_invalid` (`reason: no_cases`). |
| `POST /api/v1/visualizations/{id}/tests/sessions/{sessionId}/results` | `visualization.update` | Submit the verdicts: `{cases: [{name, verdict: "green" \| "red", notes?}], environment?: {theme, viewport, browser, locale, renderer_version}}` — notes ≤ 2,000 characters, each environment field ≤ 120, the field set CLOSED. The SAME principal that started submits (another one is `session_not_found`); the role is judged NOW. The server runs the mechanical test and answers `{session_id, run_id, status: GREEN \| RED \| INCOMPLETE, completed_at, mechanical, upload}`; the preview capability is revoked. `upload` is `{url, header: "DP-Upload-Token", token, expires_at}` on a GREEN run — the single-use capability, shown once — else `null`. |
| `POST /api/v1/visualizations/{id}/tests/sessions/{sessionId}/screenshot` | — (the upload capability) | The run's ONE screenshot: a raw `image/png` or `image/webp` body, ≤ 4 MiB (its own cap — not the platform's 2 MiB), the capability in `DP-Upload-Token`, the depicted case optional as `?case=`. No session, no key, no CSRF pair: the capability is the only credential ([Auth §8.3](/docs/auth.md#83-public-endpoints-no-auth-required), §8.4), re-checked against the starter's CURRENT authority — and, since #373, the capability's own deadline is judged BEFORE a byte of the body is read (a past-deadline capability is the `410` below, never a buffered body). Consumed atomically with the stored image: `201` `{run_id, media_type, sha256, width, height, depicted_case, uploaded_at}`; a second upload is refused. |
| `GET /api/v1/visualizations/{id}/tests/runs` | `visualization.read` | The visualization's runs, newest first, capped at the newest **100** (`limit: 100` in the answer; no paging): each the redacted run view — `status`, `cases` (verdicts and notes), `environment` (the agent's claims, never measurements), `mechanical`, `preview_revoked`, `upload_capability: {expires_at, consumed_at?}` — never a hash, never a capability. Lensed: only runs of versions the caller's lens admits. |
| `GET /api/v1/visualizations/{id}/tests/runs/{runId}` | `visualization.read` | One run, the same redacted view. A run of a version the lens hides answers exactly as an unknown run. |
| `GET /api/v1/visualizations/{id}/tests/runs/{runId}/screenshot` | `visualization.read` | The run's stored image, streamed with the media type detected at upload. A run that stores none — never uploaded, or superseded: a DRAFT version keeps only its newest run's screenshot, so an author who re-tests many times keeps the LAST one; a RELEASED version keeps its qualifying run's — is `400 visualization.test.screenshot_invalid` (`reason: no_screenshot`). |
| `POST /api/v1/visualizations/{id}/versions/{version}/check` | `visualization.read` | The §11.3 mechanical test on demand over version v as stored — no session, no write: the report `{ran_at, ok, failures (the first 100), failures_dropped, cases}`. Lensed like `GET …/versions/{version}`. |

**The pin guard.** A visualization version a LIVE (DRAFT or RELEASED) dashboard version pins is never discarded or purged, and the entity is never purged while any of its versions is pinned: `409 visualization.version.pinned`, `details.pinned_by` naming each dashboard as `name@version` (the design record §4.2).

**The test workflow (#352 backend, #353 surfaces; [Dashboards §3.4](/docs/dashboards.md)).** Start a session → open `preview_url` in a browser (the fixtures through the real runtime, no live data) → submit the verdicts → on GREEN, upload one screenshot with the single-use token → a person releases. The two capabilities are 32 random bytes, base64url, stored as purpose-prefixed SHA-256 hashes only; each authorises ONE route for ONE run: the preview its page, until submit or expiry; the upload its screenshot route, once, until the session's original deadline. A capability-authenticated request carries no principal, so the run's starter is re-judged at the moment of use — a member who lost `visualization.update`, a revoked or expired key, a deactivated identity or workspace all void the capability. Every capability failure — absent, malformed, wrong, consumed, another visualization's, a starter who lost the right — is the SAME `404 visualization.test.session_not_found` (`details.reason: session_unknown`): the routes are an oracle for nothing.

**The release's refusal ladder** (the gate inside the release transaction, in this order; each a `409` naming `details.reason` and, past the first two, `details.session_id`):

| Code | `details.reason` | When |
|---|---|---|
| `visualization.release.tests_stale` | `draft_missing` | The candidate's DRAFT moved on after the release began (another release flipped it, or it was purged) |
| `visualization.release.tests_stale` | `draft_changed` | The draft's content changed since the candidate was formed |
| `visualization.release.tests_missing` | `no_runs` | The version has no test run |
| `visualization.release.tests_stale` | `run_expired` | The LATEST run expired — swept, or RUNNING past its deadline though nothing swept it |
| `visualization.release.tests_stale` | `run_hash_mismatch` | The latest run tested other content (an edit since) |
| `visualization.release.tests_missing` | `run_open` | The latest run is still RUNNING: a session without a verdict is not evidence — submit it or let it expire. A session opened after a GREEN run is the latest run and refuses too |
| `visualization.release.tests_red` | `red` / `incomplete` | The latest run is RED (a red verdict or a mechanical failure) or INCOMPLETE (a case without a verdict) |
| `visualization.release.mechanical_failed` | `mechanical` | The mechanical test fails NOW against the current pins (`details.failures`: the first five) |

A body without a test case is refused before the gate (`tests_missing`, `reason: no_cases`).

| Error | HTTP | When |
|---|---|---|
| `visualization.not_found` | 404 | No such visualization (or version) in the workspace, hidden by the lens, or another workspace's |
| `visualization.validation.*` | 400 | The document failed a rule ([Pipeline Contract §13.22](/docs/pipeline-contract.md#1322-visualizations); every failure rides `details.failures`); `name_taken` is **409**. A malformed or wrong-shaped request body is `body_invalid` |
| `visualization.authoring.disabled` | 403 | A promotion receiver refuses every authoring write (reads and the switch unaffected) |
| `visualization.version.conflict` | 409 | A stale `If-Match` |
| `visualization.version.not_draft` / `.not_released` / `.not_discarded` / `.last_release` / `.not_eligible` / `.pinned` | 409 | The lifecycle precondition of the verb, or a live dashboard's pin |
| `visualization.release.tests_missing` / `.tests_stale` / `.tests_red` / `.mechanical_failed` / `.dependency_not_released` | 409 | The evidence gate's ladder above (`details.reason`), or a transform pin not RELEASED without consent (`details.pins_not_released`) |
| `visualization.test.session_not_found` | 404 | No such session or run for this visualization, another principal's session, or a capability that verifies nothing — one answer for all |
| `visualization.test.session_expired` | 410 | The session was submitted, swept or voided (`details.reason`: `not_running`, `content_moved_on`, `version_gone`, `run_expired`, `capability_expired`). On the upload route a `capability_expired` arrives BEFORE any byte of the body is read — and only for a presented token whose hash MATCHES an unconsumed capability; a wrong or consumed token is the `404 session_not_found` above (#373) |
| `visualization.test.screenshot_too_large` | 413 | A screenshot over 4 MiB — by `Content-Length` before any byte reaches the route, or by the counting stream (`details.cap_bytes`). The route's own cap; every other route keeps `request.body_too_large`. After a `Content-Length` refusal the server reads and discards a declared body of up to 6 MiB (the cap + 2 MiB), so a client that writes its whole body before reading still receives the 413 (#467); a larger declaration is closed after Tomcat's bounded swallow, and a client still writing may then see the connection reset instead |
| `visualization.test.screenshot_invalid` | 400 | Not a readable PNG/WebP, or not this run's to store (`details.reason`: `empty`, `unrecognized`, `truncated`, `media_type_mismatch`, `case_unknown`, `already_stored`, `no_screenshot`, …) |
| `visualization.validation.body_invalid` | 400 | A results body of the wrong shape, or an unknown / repeated case, notes over 2,000, an environment field outside the closed set or over 120 (`details.reason`, `details.path`) |

## 23. Dashboards

The dashboard artifact's REST surface (#10; the implementation spec's §6.2) — §22's twin, plus `validate`. The document is [Dashboards §2.2](/docs/dashboards.md#22-dashboard). Authorization is the seven `dashboard.*` rows of [Auth §7.6](/docs/auth.md#76-operation-matrix--the-permission-catalog-authoritative); every refusal is a `dashboard.*` code of [Pipeline Contract §13.23](/docs/pipeline-contract.md#1323-dashboards). The reading, writing and body rules are §22.1's word for word, with the dashboard's own codes (`dashboard.not_found`, `dashboard.version.conflict`, `dashboard.validation.body_invalid`). **Audit:** as §22, with `dashboard.exported` / `dashboard.imported` on the transfer routes.

### 23.1 The promoter's lens

A promoter reads a dashboard only when it is RELEASED, newer than the promotion target's, and her pipeline lens admits EVERY pipeline it sources — so her dashboards can never outrun her pipelines; the visualizations she reads (§22.1) are the ones such a dashboard pins. Since the transfer landed (L1c) the wire carries a dashboard inventory, so the "newer than the target" arm is real: a dashboard the target already holds at the same version and hash is HIDDEN. A dashboard's pins stay readable even when the target holds the same visualization — the dashboard can be promotable on a NEW version while pinning the SAME visualization the target holds, and hiding that pin would blind the read the promotion is judged with. An unreadable target admits nothing (fail closed).

### 23.2 Routes

The seven lifecycle permissions govern the same thirteen routes as §22.2 under `/api/v1/dashboards` — `POST` (create, `201`), `GET` (the flat listing, or `?prefix=` with `folders` carrying `dashboard_count` and the level's `dashboards`), `GET /{id}`, `GET /{id}/versions`, `GET /{id}/versions/{version}`, `PUT /{id}`, `POST /{id}/release`, `POST /{id}/draft/discard`, `POST /{id}/versions/{version}/discard`, `POST /{id}/versions/{version}/restore`, `DELETE /{id}/versions/{version}`, `POST /{id}/current`, `DELETE /{id}` — with these differences:

| Route | Permission | What |
|---|---|---|
| `POST /api/v1/dashboards/{id}/release?release_pinned_visualizations=` | `dashboard.release` | Release the draft at `If-Match` ([Dashboards §3.2](/docs/dashboards.md#32-releasing-a-dashboard)): the rules re-run against the dependencies as they are NOW; the pinned set release RELEASED; every pinned visualization RELEASED — or, under `release_pinned_visualizations=true`, released through ITS OWN release (its gate, and its template drafts NOT consented by this flag — they need that visualization's own `release_pinned_templates`); every source release RELEASED and read-only. Otherwise `409 dashboard.release.dependency_not_released` naming every dependency. One transaction; the answer names `visualizations_released`. Until L4 a DRAFT visualization cannot be released, so the consented cascade answers the visualization's `409 visualization.release.tests_missing`. |
| `POST /api/v1/dashboards/{id}/validate` | `dashboard.update` | Validate the WORKING version against the dependencies' CURRENT state — the §3.2 rules, no write (the agent's loop before a human releases). The verdict is the answer: `200` with `id`, `name`, `version`, `status`, `body_hash`, `valid` and `failures` (each `{code, path, message, details}` — the 400's failure shape, exhaustive). An AUTHOR verb — the implementation spec's §6.2 names `dashboard.read`; the owner ruled it `dashboard.update` (2026-09-29) because the validator reads the pinned pipelines', set's and visualizations' statuses unlensed, so a viewer and a promoter are refused `403 auth.role_required` before the handler. No body. |
| `GET /api/v1/dashboards/{id}/export` | `dashboard.read` | The export envelope (§12, [Dashboards §3.3](/docs/dashboards.md#33-export-and-import)): the dashboard's payload, each pinned visualization's OWN envelope inside it, and the manifest naming the pinned pipelines and set BY REFERENCE — a dashboard export assumes they were promoted first. Lensed before the export (§22.2's mould). The bundled closure — each pinned visualization and its templates — is the dashboard's pins, not lensed; the root is (#344). Audited `dashboard.exported`. |
| `POST /api/v1/dashboards/import` | `dashboard.import` | Import the envelope ([Dashboards §3.3](/docs/dashboards.md#33-export-and-import)): every bundled visualization (D61), then the dashboard; the exported ids KEPT. The same §13.21 read + READER bind as §22.2's import — the two dashboard bounds hold, a non-array `visualizations` refuses, and the bundle is count-bounded (O7: at most `max-visualizations-per-dashboard` entries, `body_invalid` `reason: too_many` before any member binds). ONE transaction around templates → visualizations → dashboard (F1 of the L1c pass): a refused dashboard leaves NOTHING landed. Lands RELEASED, judged by the RELEASE rules (O2): a pinned set or visualization that exists here but is not RELEASED refuses `409 dashboard.release.dependency_not_released`; a pin the deployment lacks is `400 dashboard.import.missing_dependency`; an id another workspace's artifact holds is `409 dashboard.import.id_taken` (C29). The workspace-admin verb (the owner's ruling, exactly `visualization.import`'s row). Audited `dashboard.imported` — and one `visualization.imported` row per LANDED bundled visualization, each naming its id, its version and its own envelope manifest's evidence flag (F1's record honesty: nothing landed is unaudited). |

A source whose pinned release does not DECLARE its caller columns — a SQL caller node; only a script caller node's contract names them — is not judged against its visualization's input contract at save (`input_contract_mismatch` needs declared columns); the runtime judges the real columns ([Dashboards §2.2](/docs/dashboards.md#22-dashboard)). The export/import routes (L1c) are the `/export` and `/import` rows above; the runtime and refresh routes are §23.3, and L5's key-binding routes are here:

| Route | Permission | What |
|---|---|---|
| `POST /api/v1/dashboards/bindings` | `dashboard.key.bind` | Bind a `dashboard` key to a folder of the dashboard name space (L5): `{api_key_id, name_prefix}` — or `{api_key_name, name_prefix}`, the caller's OWN key (kind-filtered: a dashboard binding names no other kind). `name_prefix` is a folder of the name grammar (1–9 segments) or the root `/` (everything); `400 dashboard.binding.path_invalid` off the grammar or when no dashboard of the CALLER's workspace lies at or under it (#191 — the refusal names only the caller's own tree). The key must be a `dashboard` key of the ACTIVE workspace; anything else is `404 dashboard.not_found` — a key id must not be probeable. The body is read through the strict request mapper (#382, §4.2): a string stays a string — a JSON number or boolean for `name_prefix`, `api_key_name` or `api_key_id` is `400 pipeline.execution.invalid_parameter_type` with `details.reason: "wrong_type"`, an unknown key (`unknown_key`) or a missing `name_prefix` (`missing`) is the same refusal, and no value is echoed. Idempotent; `201` with `{name_prefix, api_key_name, api_key_id}`. Audited `dashboard.key_bound`. |
| `DELETE /api/v1/dashboards/bindings?name_prefix=&api_key_id=` | `dashboard.key.bind` | Unbind — the same addressing forms and refusals. `404 dashboard.not_found` when the binding is not there. Audited `dashboard.key_unbound`. |

The DEEPER-BINDING-REPLACES rule (R-EP2, [Auth §7.7](/docs/auth.md#77-key-kinds-and-published-endpoint-bindings)) applies verbatim: a binding at `finance/dashboards/private` hides the one at `finance/dashboards` for that subtree, and an unbound key serves nothing.

| Error | HTTP | When |
|---|---|---|
| `dashboard.not_found` | 404 | No such dashboard (or version) in the workspace, hidden by the lens, or another workspace's |
| `dashboard.validation.*` | 400 | The document failed a rule ([Pipeline Contract §13.23](/docs/pipeline-contract.md#1323-dashboards)); `name_taken` is **409**. A malformed or wrong-shaped request body is `body_invalid` |
| `dashboard.authoring.disabled` | 403 | A promotion receiver refuses every authoring write |
| `dashboard.version.conflict` | 409 | A stale `If-Match` |
| `dashboard.version.not_draft` / `.not_released` / `.not_discarded` / `.last_release` / `.not_eligible` | 409 | The lifecycle precondition of the verb |
| `dashboard.release.dependency_not_released` | 409 | A pinned visualization, set or source not RELEASED (and not cascaded) — `details` names each |

### 23.3 The runtime and the refresh routes

An explicitly selected draft dashboard admits live DRAFT or RELEASED dependencies throughout its graph (#459).
Default and explicitly released runtime views, dashboard release and import remain RELEASED-only. The existing
`dashboard.execute` permission, promoter lens and session-only version selection continue to govern these routes.
Draft configuration identity includes nested pipeline and template/import content; same-version edits invalidate it.

Six routes under `/api/v1/dashboards/{id}`, all `dashboard.execute` ([Auth §7.6](/docs/auth.md#76-operation-matrix--the-permission-catalog-authoritative);
the promoter reads through the lens of §23.1 — a dashboard it hides is `404 dashboard.not_found`). The semantics are
[Dashboards §5](/docs/dashboards.md#5-the-runtime); this section is the wire. Since L5 (#367) a `dashboard` key reaches them too — the runtime and refreshes prefixes are its whole surface, its bindings the lens, one key one budget ([Auth §7.7](/docs/auth.md#77-key-kinds-and-published-endpoint-bindings), [Dashboards §6.5](/docs/dashboards.md#65-credentials-session-and-proxy)) — and no MCP tool executes a dashboard. Since #369 the FOUR runtime routes take an optional `version` query parameter naming a DRAFT or RELEASED version of the dashboard (the draft preview's, `GET /dashboards/{id}/preview?version=`): absent = the current RELEASED version, exactly the behaviour below; a value is a bounded positive integer (`400 pipeline.execution.invalid_parameter_type` otherwise), must resolve for the caller — absent, DISCARDED, or hidden under a narrowing lens is `404 dashboard.not_found` naming `details.version` — and a `dashboard` key never names one (`403 dashboard.key.kind_refused`, `details.reason: version_is_session_only`, before anything is looked up). A request body is read whole before anything is looked up: not JSON, or a
field missing / of the wrong type, is `400 dashboard.validation.body_invalid` with `details.path` and `details.reason`
(`missing`, `wrong_type`, `malformed`, `too_large`, `unknown_value`, `unexpected`, `reused`) — never the value.

| Route | Request | Answer |
|---|---|---|
| `GET /{id}/runtime/config` | — | `200` `{configuration_id, renderer: {bundle}, dashboard: {id, name, version, status}, layout, parameter_set, visualizations: [{name, display_name, description, artifact: {id, name, version}, renderer: {kind, version}, config, bindings, presentation, timeout_seconds}], groups, actions, action_controls, parameter_scopes, parameter_state, timeouts: {refresh_seconds, parameter_lock_seconds, render_seconds}, budgets: {max_bytes_per_source, max_bytes_per_refresh}}` — `renderer.bundle` is `"2d"` or `"3d"`, derived from the pinned visualizations' trace types ([Dashboards §6.4](/docs/dashboards.md#64-the-two-bundles)): the page loads exactly one Plotly bundle, the one this field names. `display_name`/`description` (#473) are the pinned VERSION's own presentation words — the card headings read them, version-scoped, never the artifact record's name; `description` is JSON null when the body omits it |
| `POST /{id}/runtime/parameters` | `{configuration_id, instance_id, selections, intent}` (`intent`: `bootstrap` \| `parent_change` \| `retry`) | `200` the parameter engine's evaluate response ([§21.3](#213-evaluate)) unchanged, plus `overrides_applied`, `parents`, `parameter_revision` |
| `POST /{id}/runtime/visualizations` | `{configuration_id, instance_id, refresh_id, parameter_revision, selections, scope, targets}` — `refresh_id` a fresh v4 UUID the client mints; `scope` `all` \| `targets`; `selections` at most 64 KiB | `200` `text/event-stream` (below) |
| `POST /{id}/runtime/refreshes/{refresh_id}/abort` | `{instance_id}` | `202` `{refresh_id, status: "abort_requested"}` — no wait. An abort arriving while the refresh is still STARTING (no row yet) is the same `202` when the in-flight start is the caller's own — same workspace and dashboard, their principal AND client instance, or `execution.cancel_all` — and the refresh then ends `ABORTED` before any source runs |
| `GET /{id}/refreshes?offset=&limit=` | — | `200` a page of the caller's refreshes, newest first (every refresh with `execution.read_all`) |
| `GET /{id}/refreshes/{refresh_id}` | — | `200` `{refresh_id, dashboard_id, dashboard_version, instance_id, scope, targets, parameter_revision, selections, status, started_at, finished_at, summary, executions?}` — `executions` (`[{source, execution_id, shared}]`) only for a caller holding `execution.read` |

**The stream.** The execution stream's framing (§6): `event:` name, `id:` monotonic per refresh from 1, `data:` JSON, and a
`: heartbeat` comment while quiet; a client that stays away past the disconnect grace (§6.8) aborts its refresh. Events, in the
order they can occur: `refresh_started` `{refresh_id, targets, sources: [{name, shared}], deadline_at}`; per source
`source_started` `{refresh_id, source, execution_id}` (the source is already linked to the refresh), `source_completed`
`{…, rows, bytes}` or `source_failed` `{…, execution_id?, error: {code, message}}`; per target `visualization_status`
`{refresh_id, name, type, state, stage?, reason?}` (`state`: `in-progress`, `error`, `abort`, `no-data`; `stage`: `source`,
`transform`, `budget`, `timeout`, `abort`) and `visualization_data` `{refresh_id, name, type, bindings: {path: [values]}, rows, bytes}`;
and `refresh_completed` `{refresh_id, status, targets: {name: {outcome, stage?, reason?}}}` — always last (`status`: `COMPLETED`,
`PARTIAL`, `FAILED`, `ABORTED`, `TIMED_OUT`; `outcome`: `ok` — the data frame delivered — `no-data`, `error` (with `stage`/`reason`),
or `abort`). A `source_failed` names a code and never a driver or datasource message.

For `POST /runtime/parameters`, `dashboard.execute` authorizes the pinned set evaluation under D50 without separately
requiring `parameter_set.evaluate`. The response remains the evaluator's unchanged response, including its existing
bounded selector diagnostics and safe echo for failed or unreachable selectors. The stream's code-only error rule
applies only to stream frames.

The `execution_id` member on each `source_started`, `source_completed` and `source_failed` frame is present only when
the subscriber's current principal holds `execution.read`; `source_failed` omits the member even when its internal id
is null. Before every event or heartbeat, stream admission and this projection use one current-authority decision. It
strictly resolves the workspace captured when the stream opened and requires the same immutable workspace id; it never
uses session-navigation fallback, and another membership does not preserve access to this stream. The internal event
and durable refresh-execution link still retain the id. A role change is reflected under the auth cache's membership
window. A dashboard execution has no stored result: `GET /api/v1/executions/{id}/result` on it is `404 execution.not_found`, and
`DELETE /api/v1/executions/{id}` on it is `404` for everyone — it is cancelled only through its refresh's abort.

| Error | HTTP | When |
|---|---|---|
| `dashboard.validation.body_invalid` | 400 | The request body is unreadable or a field is missing, mistyped or a reused `refresh_id`; the selections are invalid (`details.reason: selections_invalid`, `details.parameters`) |
| `pipeline.execution.invalid_parameter_type` | 400 | A `version` query parameter that is not a positive integer (#369) |
| `dashboard.not_found` | 404 | A `version` query parameter that names no version the caller may see — absent, DISCARDED, or hidden under a narrowing lens (`details.id`, `.version`) (#369) |
| `dashboard.key.kind_refused` | 403 | A `dashboard` key named a `version` on a runtime route — the version routes are the session page's (`details.reason: version_is_session_only`) (#369) |
| `dashboard.validation.empty_targets` / `.target_not_visualization` | 400 | `scope: targets` naming nothing, or a name that is no visualization occurrence |
| `dashboard.runtime.configuration_stale` | 409 | `configuration_id` is not the current one (`details.configuration_id`); the client reloads |
| `dashboard.runtime.dependency_missing` | 409 | A pinned visualization, set, transform template or source no longer holds: gone, DISCARDED, not released (a released dashboard; an explicitly selected draft admits DRAFT or RELEASED, #459), or no longer read-only (`details.dependency`, `.name`, `.reason`: `not_found`, `not_released`, `not_read_only`). For a draft the walk also covers nested pipelines, node, selector and transform templates and their imports. A transform that changes status after resolution fails its target before evaluation. Inside the stream, the admission run on each exact loaded body (root source and every nested child) immediately before execution fails that source with this code, or with `dashboard.validation.source_not_read_only` when the body is no longer read-only |
| `dashboard.refresh.saturated` | 429 | No room for the refresh within `max-wait-seconds`; carries `Retry-After`; **no row is written** |
| `dashboard.refresh.result_too_large` | — | Inside the stream only: `source_failed` / a target's `reason` at stage `budget` |
| `dashboard.refresh.not_found` | 404 | No such refresh for the caller on this dashboard, or it already finished (abort), or it is still starting for someone else — the four answer the identical body |
| `rate_limit.exceeded` | 429 | The per-user concurrent-stream cap (§12.1) — execution, refresh and observed-evaluation (§21.5) streams count together |

## 24. First-party navigation trees

`GET /api/v1/<family>/tree` and `GET /api/v1/<family>/tree/search` are literal session-only
reads for `pipelines`, `templates`, `dashboards`, `visualizations` and `parameter-sets`.
Each declares its existing read permission and derives the workspace from the principal.
Keys gain no surface. Every response uses the §4.1 envelope and `Cache-Control: no-store`.
An unavailable promotion view returns a retryable `503 navigation.view.unavailable` with
`Retry-After: 5` and no `details`, rather than a completed empty level; the server logs it once
at WARN without a stack. Every refusal is catalogued in [pipeline-contract §13.24](/docs/pipeline-contract.md#1324-navigation-trees).

Browse takes optional `root`, `parent` and `cursor`; omitted parent means root. Empty root is
the family root. A nonempty root names a folder boundary; parent cannot escape it. A page
contains at most 200 immediate folders AND artifacts, folders first, in binary path order.
The client automatically exhausts continuation for this one level; no grandchildren are read.
Search takes `root`, required `q` and `cursor`, and pages all case-insensitive literal substring
matches of canonical or display NAME, never descriptions. The shared search-needle rule (#504b)
gates `q` before any read: it is trimmed, a blank (whitespace-only) `q` answers the browse level
instead of a refusal, and a term over 200 characters after the trim is refused `400
request.query_too_long` with `details.limit` and `details.length` ([pipeline-contract §13.21](/docs/pipeline-contract.md)).
The tree then strips leading/trailing runs and collapses interior runs to one U+0020 using
one fixed whitespace class on both client and server: U+0009–U+000D, U+0020, U+00A0, U+1680,
U+2000–U+200A, U+2028, U+2029, U+202F, U+205F, U+3000 and U+FEFF. This is applied **after**
the shared 200-after-trim rule (whose Kotlin trim includes U+00A0 and U+3000 but excludes
U+FEFF); the first-party client sends `q` already collapsed. An empty collapsed term answers
the browse level. Any control characters remaining after normalization, including U+0085, are refused.
Each page includes all ancestors
needed to connect its matches to root. Search does not crawl browse endpoints.

A 1 MiB serialized response budget may shorten a page before 200 logical results; continuation
then resumes at the last delivered result. It never drops matches or required ancestors. A single
node whose metadata alone exceeds the budget is `503 navigation.metadata.too_large`.

The typed payload has `family`, `mode` (`browse` or `search`), `root`, `parent`, normalized
`query`, server-derived `workspace_id`, `view_token`, `nodes` and nullable `next_cursor`.
Nodes have `key`, `parent_key`, `kind` (`folder` or `artifact`), `name`, canonical `path`,
`has_children` and `match`. Artifacts also have `resource_id`, canonical relative `href`,
listed `version` and nullable `draft_version`. A template artifact also has `needs_review` — true
when its listed version cites a retired or superseded learned fact ([Templates §3.4](/docs/templates.md#34-implements-and-drift)),
false otherwise — read once per page with the release warning's citation read; it is null on
folders and on every other family's nodes. A narrowing view emits released metadata only.
Folder and artifact keys are distinct even at the same path. Search ancestors may repeat
across pages; deduplicate by key. No stored body, SQL, description, fixture or secret travels.

Continuation uses signed keyset ordering, bound to family/mode/root/parent/query, workspace,
caller and actual admitted-name membership. A changed view, foreign or malformed cursor is a
safe `400 navigation.request.invalid` (so is a parent outside its root, or a malformed path) with
a static message that echoes nothing, and must be retried from the beginning. Process restart expires outstanding cursors.
Completeness applies to an unchanged dataset; Refresh revalidates external changes. There is
no immutable snapshot promise. Personalized rows and expansions stay in memory, never in
browser persistence. Clients must reject foreign/stale responses even after aborting fetch.

---

## Appendix A: Change Log

| Date | Version | Author | Change |
|---|---|---|---|
| 2026-11-01 | v2.107 | 758 (#758) | **§24:** fixes the tree search whitespace alphabet on both client and server, after the shared raw-term bound; names the code points and empty-term browse behavior. |
| 2026-10-27 | v2.106 | 464 (#464) | §24: the tree routes' refusals carry catalogued codes ([pipeline-contract §13.24](/docs/pipeline-contract.md#1324-navigation-trees)) instead of the stand-in `pipeline.execution.aborted`: `400 navigation.request.invalid`, `503 navigation.view.unavailable` (now with `Retry-After: 5`, no `details`) and `503 navigation.metadata.too_large`. Statuses are unchanged. |
| 2026-10-27 | v2.105 | 575 (#575) | §18.1: all five wire inventory arms require a RELEASED current pointer; pipeline and template draft pointers and never-released templates are omitted, matching the promotability rule. |
| 2026-10-27 | v2.104 | 725 (#725) | Additive. §20.1: optional `outcome` (`succeeded`, `failed`, `in-progress`, `did-not-run`, `never-ran`) filters the schedule list by each schedule's newest run, in the database read before the promoter lens, composing with `condition` and `prefix`; any other value is `400 schedule.validation.request_invalid`, `details.field = outcome`. |
| 2026-10-25 | v2.103 | 641 (#641) | Additive. §20.1: optional `condition` (`enabled`, `paused`, `blocked`) filters the schedule list in the database read, before the promoter lens; any other value is `400 schedule.validation.request_invalid`, `details.field = condition`. |
| 2026-10-24 | v2.102 | #517 release dependencies, core (S1) | **§5.10:** the release takes the draft's whole release closure (templates and child pipelines, bounded); an optional JSON plan body (`tree_hash`, `include_dependencies`, `check_overrides`) flips the selected dependency drafts in the same transaction and appends `dependencies_released`; the plan refusals `428 plan_required`, `400 plan_invalid`, `422 dependency_limit`, `409 plan_conflict` / `dependency_cycle` / `busy`. **Deliberate break (record §14, ruled 2026-10-18, no deprecation window):** the template-only refusal code (record §14) is `pipeline.release.dependency_not_released`, and its details are the shared blockers shape `{root, blockers, blockers_truncated}` (the old `template_id` / `template_version` / `template_status` / `pins_not_released` are gone). A request without a plan answers a success with release 142's bytes. Every release is one transaction with strict in-transaction audit rows. **§8.9:** `template.release.busy`; the template flip runs in a guarded transaction. **§21:** the parameter set's template-only refusal code is `parameter.release.dependency_not_released` (details unchanged until #519). Round three (#719): **§5.13 / §21** a restore re-judges its outbound pins (400 validation code; 409 `dependency_not_released`), **§5.10** the blockers' `reason` gains `not_released`. |
| 2026-10-25 | v2.101 | 442c (#442) | §20.13: resend at the deployment pending cap is 409 schedule.notification.backlog_full; no delivery change or resend audit row on refusal. |
| 2026-10-24 | v2.100 | 442b (#442) | Additive. §20's schedule object: `notifications.delivery` is the deployment's computed state, `{"state":"on"}` or `{"state":"off","reason":"disabled"\|"mail_unconfigured"\|"no_allowlist"}`. §20.11: a run carries `deliveries` (notice, state, attempts, resend count, times, sanitized `last_error_code`; `recipient` only for `schedule.update` holders). New **§20.13** `POST /schedules/{id}/runs/{run_id}/deliveries/{delivery_id}/resend` (`schedule.update`; `may_duplicate` warning for an `uncertain` delivery; `409 schedule.notification.not_resendable`, `404 schedule.notification.not_found`). |
| 2026-10-22 | v2.99 | 8b (#8) | JavaScript save/evaluate is enabled with the isolate; §8.7A adds the always-present console key and failed-evaluation console detail. |
| 2026-10-21 | v2.98 | #640 the script-node rename (script-node-rename lane, V50, owner ruling 2026-10-20) | **The template-type and node prose say script**: a script caller / type / body, §8.7A "Evaluate a script template", the pipeline editor's SCRIPT card, the key `datapipelines.script.evaluate-timeout-seconds`; no route or field changes. A rename only: no behaviour, route, permission or key change beyond the names, and no alias (the ruling). The visualization block's `transform` key, `transform.template` / `transform.inputs` and the CSS/SVG sense keep their word. |
| 2026-10-16 | v2.97 | 467 (#467) the over-cap screenshot refusal reaches the client | **§22's `visualization.test.screenshot_too_large` row**: the `Content-Length` refusal no longer closes mid-body — the route drains a declaration of up to 6 MiB after the 413 so a write-then-read client receives it; beyond that bound a reset may still replace it. No route, code, status or cap change |
| 2026-10-16 | v2.96 | 533 (#533) the templates tree carries `needs_review` | **§24** (additive): a template artifact node carries `needs_review` — true when its LISTED version cites a retired or superseded learned fact ([Templates §3.4](/docs/templates.md#34-implements-and-drift)), false otherwise; null on folders and on the pipelines, dashboards, visualizations and parameter-sets nodes. The flag comes from ONE citation read per page over the page's own rows (name and listed version) — the read the pipeline release's `template_needs_review` warning uses — so a narrowing view is answered about the released versions it lists. The 1 MiB budget counts the flag on every template artifact. The sidebar's Templates tree renders it as the catalog row's **needs review** marker ([UI Screens §4.6a](/docs/ui-screens.md)). No route, permission or other field changed. |
| 2026-10-11 | v2.95 | 504b (#504b) the shared search-needle refusal on every REST search | **§5.7, §8.5, §16.3, §21.2 and §24**: a `q` over 200 characters after the trim is refused `400 request.query_too_long` with `details.limit` and `details.length`, before any read and never echoed — one rule ([pipeline-contract §13.21](/docs/pipeline-contract.md) v1.58) applied by the service that owns each search, replacing §21.2's parameter-only `parameter.validation.query_too_long` (#490) and giving the pipelines, templates, user and five `/tree/search` listings the same refusal. Blank `q` still means no search everywhere; the UI boxes still truncate and never meet the refusal; the MCP list tools answer the same term as `-32602` ([MCP §6.2.44](/docs/mcp-server.md) v1.74). A narrowing of accepted input only for terms no box could send; no route, field, permission or other code changed. |
| 2026-10-10 | v2.94 | 504a (#504) the users search escapes its LIKE term; the listings trim `q` — renumbered at lane after #505's v2.93 | **§16.3** (`GET /auth/users?q=`): the search term's own `%` and `_` are matched literally — the trimmed, lowercased term is escaped and both LIKEs carry an explicit `ESCAPE '\'` (the house form; the term stays a bound parameter, so nothing about the parameterisation changed). Unescaped, a `q` of `%` matched every user and a `q` of `_` matched every user with at least one character in email or display name. **§5.7 and §8.5** (the pipelines and templates flat listings): `q` is trimmed and a blank (whitespace-only) term means NO search — the whitespace-only behaviour change: a `q` of `" "` used to keep only the pipelines whose name, display name or description contained that space, and to match only the templates with a space in a searched column (`q=""` bound `%%`). No route, field, status code or permission changed. |
| 2026-10-10 | v2.93 | 505 (#505) the idempotent attach reads the replay log's three answers itself | **§6.1.3** (the `503` row's last sentence becomes the attach's rule): the attach's first read of the log now answers the fault row itself — `503 result.storage_unavailable`, `details.reason: event_log_unavailable`, WITH the `execution_id` (the reservation already handed the retry caller the id), before any stream is committed, for a terminal and a running original alike; it had taken a fault for "no log", so a TERMINAL original answered `410 event_log_expired` (v2.92 left that tracked as #505). Only an ANSWERED empty read (`Absent`) still consults the execution row, which keeps the two `410`s. **§3.5** names the 503 beside its two `410`s. No route, field or permission is added. |
| 2026-10-09 | v2.92 | 487 (#487) a replay-log read fault is not an expiry — renumbered at merge after #490's v2.91 | **A deliberate status change on the frozen contract (owner's ruling, 2026-10-09):** `result.storage_unavailable` is **503**, was 500 ([Pipeline Contract §13.10](/docs/pipeline-contract.md#1310-result-retrieval) v1.56; §7.6 mirrors it). **§10.3:** a log read Redis did not answer — down, or slower than the 2 s command timeout (#482) — answered `410 result.expired` as if the log had expired; it now answers `503 result.storage_unavailable` (`details.reason: event_log_unavailable`), before the stream; the log is read once per request (it was read twice — a fault on the second read served an empty stream). **§6.1.3** gains the idempotent attach's row: a follow that meets 8 consecutive unanswered reads before its first event ends with the same 503, id-free; after an event, with a final `event_log_unavailable` comment. Faults no longer count toward the attach's ~15 s never-started patience. Unchanged and tracked as #505: the attach's own first read still takes a fault for no log, so a terminal original answers `410 event_log_expired`. No route, field or permission is added. |
| 2026-10-09 | v2.91 | 490 (#490) the search needle's bound | **§21.2's `GET /api/v1/parameter-sets` row**: the flat listing's `q` is bounded at 200 characters after the trim; a longer one is refused `400 parameter.validation.query_too_long` (`details.limit`, `details.length`; [pipeline-contract §13.20](/docs/pipeline-contract.md) v1.56), before any read and with no echo of the term — it had passed to the search unbounded. The rule lives once in `ParameterSetService`; the web UI's boxes keep truncating to the same bound, and `parameter_sets_list` refuses it as `-32602` ([MCP §6.2.44](/docs/mcp-server.md) v1.73). A narrowing of the accepted input only for terms no box could send; no route, field, permission or other code changed. |
| 2026-10-09 | v2.90 | #473 runtime config card metadata — rebased in recovery after #419's v2.89 | **§23.3** (additive): each `visualizations[]` entry of `GET /{id}/runtime/config` carries the PINNED version's `display_name` and `description` (`display_name` is required on the body; `description` is null when the pinned body has none), read from the version the configuration pins, never today's artifact record. |
| 2026-10-08 | v2.89 | 419 (#419) the parameter-sets flat listing takes `q` — renumbered at recovery after #417 took v2.79 while the lane sat, and at merge after #462's v2.87 and #460's v2.88 | **§21.2's `GET /api/v1/parameter-sets` row**: the existing flat handler gains an optional `q` — additive only, blank/absent `q` is today's listing byte for byte. A non-blank `q` limits the listing to the lensed sets whose name, display name or description contains it, case-insensitively (names are paths, so `q` matches the full path) through the same `ParameterSetService.search` the pages use, and the `total` stays lens-true. A `q` under `?prefix=` is ignored — the §5.7 pipelines rule; no new route, handler, field, status code or permission ([MCP §6.2.44](/docs/mcp-server.md) v1.72). |
| 2026-10-06 | v2.88 | 460 merge follow-up — the section #460 added without a row — renumbered at merge after #462's v2.87 | **§24 added (additive)**: `GET /api/v1/{pipelines,templates,dashboards,visualizations,parameter-sets}/tree` and `/tree/search`, session-only first-party navigation reads under each family's existing read permission — keyset pages of at most 200 immediate folders and artifacts (folders first, binary path order) under a 1 MiB response budget, literal case-insensitive name search with every ancestor a match needs, signed continuations bound to the request, the actor, the workspace and the lens, `Cache-Control: no-store`, and a 503 rather than an empty level when the promotion view is unavailable. No existing route, shape or code changed. |
| 2026-10-06 | v2.87 | #462 draft purge pins | §5.11 and §5.14: 409 `pipeline.version.pinned` on both draft purges, named blockers, sole/nonsole scope, hash precedence and unchanged stored state on refusal. |
| 2026-10-06 | v2.86 | 459 merge follow-up | §23's runtime error table, the `dashboard.runtime.dependency_missing` row (additive, no code or status change): it names the lifecycle per mode (released dashboard RELEASED-only; an explicitly selected draft admits DRAFT or RELEASED), the `.reason` values, the draft walk's reach (nested pipelines, templates, imports), and the stream-side admission before execution, which fails a source with this code or `dashboard.validation.source_not_read_only`. |
| 2026-10-05 | v2.85 | #459 draft dashboard dependencies | §23.3: explicitly selected draft runtime admits live draft dependencies transitively, with nested content in configuration identity. Released views, release/import, permissions and key restrictions remain strict. No new route or wire field. |
| 2026-10-04 | v2.84 | 442a (#442) — renumbered at merge after 403's v2.83 (the lane's row sat atop the older second table) | Saved schedule notification settings: recipients and five event choices, validated and shown by role; omitted PUT settings preserved; mail remains off pending part b. Merge follow-up (review F2): §20.2's refusal is a sentence, not a lone table row. |
| 2026-10-04 | v2.83 | #403 release never-started reservations — renumbered at merge after 382's v2.82 (the lane's row sat atop the older second table) | **§3.5**: pre-start refusals release the caller’s key before completing the error response; same-key retries re-execute. A follower already attached or an original that died without cleanup retains the id-free 410. Started executions retain their reservations. Merge follow-up (review F1): §6.1.3's 429 row says the reservation is released, no longer claimed. |
| 2026-10-04 | v2.82 | #382 the dashboard binding body is read strictly — renumbered at merge after 384's v2.81 | **§23.2** — the `POST /api/v1/dashboards/bindings` row names the strict body reader (#333's, the fifth DTO-bodied route it had omitted): a number or boolean for a string field, an unknown key and a missing `name_prefix` are the standard `400 pipeline.execution.invalid_parameter_type` (`wrong_type` / `unknown_key` / `missing`), the value never echoed; before this a number bound as text (`name_prefix: 12` was the folder "12") and an unknown key was dropped. No route, scope, status code or success shape changed. **§4.2** (merge follow-up, #382 review F1): the strict-reader sentence names five routes, `POST /dashboards/bindings` §23.2 added. |
| 2026-10-04 | v2.81 | #384 the one SSE cap spans all three families | **§12.1** — the execution registry now counts refresh and observed-evaluation streams beside its own, so `sse.max-streams-per-user` is the one per-user cap the sentence already promised (#375 D7): a user at the cap through refresh or evaluation streams is refused a new execution stream (`rate_limit.exceeded`), and the doc stops claiming every counter lives in Redis — the request-rate counters are Redis-backed, but the three SSE registries count **per instance**, so the SSE cap is per-instance in a multi-instance deployment, and check-then-open (the cross-family reads included) is non-atomic, a bounded overshoot. No route, field, code or permission changed. Wire-proven by `CrossFamilySseCapE2eTest`. |
| 2026-10-03 | v2.80 | #404 the pre-first-event refusals reach the wire | **New §6.1.3** names the execute route's two refusals decided after the stream opened and before its first event — `429 pipeline.execution.concurrency_limit` (no execution slot, §12.1) and the idempotent retry's never-started `410 result.expired` (`reason: original_not_started`, no `execution_id`, §3.5) — as §4.2 JSON envelopes with their own status, also under an SSE-only `Accept`. Both were documented before (§3.5, §12.1) but answered `401 auth.api_key.missing` on the wire: the authorization interceptor judged the stream's async completion dispatch against an empty security context (auth v3.40 §8.2). A bug fix, not a contract change: no route, field or code is added. Wire-proven by `IdempotentAttachRowOrderE2eTest`. |
| 2026-10-03 | v2.79 | #417 the observed evaluation refuses a recorded `evaluation_id` | **§21.5** — a reused `evaluation_id` is now also an id the caller's workspace history already holds (a closed stream's, or one recorded on another instance), refused before the stream opens and before the cap with the same `400 parameter.validation.body_invalid` (`details.path = evaluation_id`, `details.reason = reused`); the sentence "across instances it becomes the history row's key conflict when S3 (#376) lands" is corrected (the insert conflict is logged and the evaluation runs unrecorded, so the id is now refused up front), the refusal order's step 5 and the `body_invalid` row name both cases. The read is workspace-scoped: another workspace's id answers as unused. No route, field, code or permission changed. |
| 2026-10-03 | v2.78 | #399 the Visualizations workspace — renumbered at merge after the #383 follow-up's v2.77 | **§22's intro** — the sentence "No UI page exists yet — this section and the MCP tools … are the whole surface" leaves: the first-party workspace ([ui-screens §4.24](/docs/ui-screens.md)) reads and drives the lifecycle verbs through the same services, its page and partial routes named as UI routes on the `visualization.*` rows (auth.md §7.6), outside this contract. No route, field, code or permission changed. |
| 2026-10-03 | v2.77 | #383 merge follow-up — the page's client streams | **§21's intro** — the sentence "the page's live form calls §21.3 exactly as any client does" became false when #383 moved the live form onto §21.5; it now names §21.5 as the page's route and §21.3 as every other client's. **§21.5's** opening (#383) names the page's client as the route's consumer. No route, field or code changes. |
| 2026-10-02 | v2.76 | S1 (#374) the Parameter Sets pages | **§21's intro** — the sentence "No UI page exists (the record's §13)" is replaced (workspace spec §14, row C1): the first-party workspace ([ui-screens §4.23](/docs/ui-screens.md)) reads and evaluates through §21, and its page routes (`GET /parameter-sets`, `GET /parameter-sets/{id}`, `GET /partials/parameter-sets/tree`) are named as UI routes on `parameter_set.read`, outside this contract. **§21.5** — "its page lands with S1" now reads as landed, with the client half (#383) named. No route, field, code or permission changed: the page's form calls the existing §21.3 evaluate. |
| 2026-10-02 | v2.74 | 328 (#328) the release response carries `caller_output` | Additive only. **§5.10:** the release body gains the one word `caller_output` (`recorded` \| `declared` \| `none` \| `not_observed`) — what the release did about its caller node's result columns (pipeline-contract §3.3.1); the `pipeline.version.released` audit row carries the same word and never the column list. No new route, no status change, no MCP schema change (`dashboards_validate`'s input is unchanged). |
| 2026-10-02 | v2.73 | 324 (#324) the idempotent retry waits for the row | **§3.5**: the retry's behaviour inside the original's START WINDOW is stated — a retry landing before the original's first event (up to the 10 s lifecycle write bound) WAITS for it (the attach follows the log for ~15 s, above the bound) and then streams the original's events, instead of the premature `410 result.expired` with an id whose row did not resolve yet. An original that never started answers `410 result.expired` with `reason: original_not_started` and NO `execution_id` (an unresolvable id is never disclosed); a genuinely expired log keeps `reason: event_log_expired` with the id. No route, scope or status-code change; the attach consults the execution row it already has. |
| 2026-10-02 | v2.72 | 333 (#333) the strict contract readers | **§4.2 gains the scalar-shapes paragraph**: a JSON number or boolean where a body declares a string (a string or float where it declares an integer) is refused with the path and the expected shape, never bound as text, on the pipeline and template bodies, the transform blocks and the four typed-DTO routes (§16.1 `POST /auth/api-keys`, §18.2 `POST /promotion/push`, §19.5 `POST /endpoints` and `/endpoints/bindings`). Those four now answer `pipeline.execution.invalid_parameter_type` with `details.reason` `wrong_type` / `unknown_key` / `missing` and `details.path` — **behaviour change** on the two request DTOs: an unknown key is refused (it was dropped) and a missing required key answers this code (it answered the unreadable-body stand-in). Template-body refusals gain `details.rule: "wrong_type"`. No route, scope or status change. |
| 2026-10-02 | v2.71 | S2 (#375) the observed parameter-set evaluation | Additive. **§21.5 (new): `POST /api/v1/parameter-sets/{id}/evaluations`**, the observed evaluation (the parameter-set workspace spec §4, R1; the owner's rulings §11.3, §11.4 and §11.7) — the §21.3 act on `parameter_set.evaluate`, its cascade streamed as Server-Sent Events: `version` REQUIRED, a client-minted v4 `evaluation_id` (`reused` when already open on this instance), the refusal order, the eight FROZEN events, the terminal and coverage rules (on `evaluation_failed` the unfinished parameters carry no per-parameter frame — #375 D3), the per-write authority re-judgement, and the abort of an evaluation whose client stayed away past `disconnect-grace-seconds` (no frame; with the defaults the 30 s deadline usually ends it first). `evaluation_failed` is code-only. **§21.3 is unchanged byte for byte** (a golden pins it). **§12.1, §23.3:** the per-user stream cap counts observed-evaluation streams beside execution and refresh streams. No new code, no new key. |
| 2026-10-02 | v2.70 | 373 (#373) the upload gate judges expiry before the body | §22.2: the screenshot route names the pre-read expiry judgement (a past-deadline capability is the 410 before any byte of the body); the `session_expired` error row states its order — 410 only for a presented token whose hash matches an unconsumed capability, wrong or consumed keeps the one 404. Additive clarifications; no status or route changes. |
| 2026-10-02 | v2.69 | 344 (#344) the bundle is the artifact's pins | **§5.9 and the three export rows (§21.2's set, §22.2's visualization, §23.2's dashboard)** say what the routes already did: the bundled closure is the artifact's pins, read without the promoter lens; only the root is lensed (the owner's ruling of 2026-10-02, auth §11A.1's lens clause). No route, status code, permission or wire shape changed. |
| 2026-10-01 | v2.68 | L5 (#367, #10) the `dashboard` key kind | **§23.2 gains the two binding routes** — `POST`/`DELETE /api/v1/dashboards/bindings` under `dashboard.key.bind`, the `api_key.bind` twin (`400 dashboard.binding.path_invalid` off the grammar or the caller's own tree; the key by id or own-name, kind-filtered; audited `dashboard.key_bound`/`dashboard.key_unbound`). **§16.1's `kind` table gains `dashboard`** — `dashboard_viewer` fixed, bindings refused at create (they are the binding verb's), `api_key.create` holders. **§23.3:** the runtime routes' caller sentence names the key kind — its bindings are the lens, one key one budget. |
| 2026-10-01 | v2.67 | L4b (#353) the visualization test workflow | Additive. **§22.2: seven routes** — `POST /{id}/tests/sessions` and `POST /{id}/tests/sessions/{sessionId}/results` (`visualization.update`), `POST /{id}/tests/sessions/{sessionId}/screenshot` (no permission: the single-use `DP-Upload-Token` capability alone, the raw PNG/WebP under the route's own 4 MiB cap), `GET /{id}/tests/runs` (the newest 100), `GET /{id}/tests/runs/{runId}`, `GET /{id}/tests/runs/{runId}/screenshot` and `POST /{id}/versions/{version}/check` (`visualization.read`, lensed). The release row names the installed gate; the refusal ladder with every `details.reason` (`draft_missing`, `draft_changed`, `no_runs`, `run_expired`, `run_hash_mismatch`, `run_open`, `red` / `incomplete`, `mechanical`) and the four `visualization.test.*` codes are tabled. The stale "until L4" and "not routes yet" sentences are gone. |
| 2026-10-01 | v2.66 | #356 the abort that arrives before the row | **§23.3:** an abort arriving while the refresh is still starting — before `insertRunning` writes its row — is honoured: the in-flight start registers a transient, TTL'd marker (bounded per principal by the stream cap, removed on the start's every exit), a matching caller (same workspace and dashboard, own principal AND instance, or `execution.cancel_all`) is answered the SAME `202 {refresh_id, status: "abort_requested"}`, the intent is recorded under the id like any abort flag, and the refresh ends `ABORTED` before any source runs (the engine's check at job start). A stranger's, another instance's, another dashboard's, an unknown and a finished id all answer the identical `404 dashboard.refresh.not_found`. No route, body or response shape changed; the frozen §8.4 wire is untouched. |
| 2026-09-30 | v2.65 | #343 stream workspace authority | Recheck dashboard refresh events and heartbeats only in the stream's opening workspace, matched by immutable id; other memberships do not preserve access, and the refresh remains running. |
| 2026-09-30 | v2.64 | #343 dashboard runtime residue — renumbered at merge after L1c-c's v2.63 | Clarify unchanged bounded parameter-evaluation diagnostics and current `execution.read` projection of source-frame execution ids; include transform-template pins in runtime dependency refusals. |
| 2026-09-30 | v2.63 | L1c-c (#10) transfer limits and evidence | Documentation and wiring honesty inside existing verbs (no new route, no §13 code, no key). **§18 (A2):** the count ceiling is stated as AGGREGATE — `max-visualizations-per-dashboard` caps each whole arm (and each whole envelope array on the import surfaces) on top of the per-document bound; two individually valid dashboards can together exceed it, the batch refuses whole (no partial writes, no split), a batch at exactly the ceiling lands whole — E2E-proven both ways. **§18 (A1):** the receive's refused-dashboard rollback is witnessed end to end (earlier batch members land, the dashboard refuses on landing, every preceding write is absent — the transfer E2E's case, the L1c-b round's open follow-up closed). **§18:** the sender re-raises the receiver's refusal verbatim (pinned by test), and the promotion page's `body_invalid` flash renders its toast (unmapped = silent before; the `missing_datasources`/`key_invalid` branches' `&#39;` entities inside fragment literals were a latent render-500, fixed). **C1:** the transfer bean receives the operator's configured ceiling (the constructor default stood in silently — an operator's override was ignored at the default).
| 2026-09-30 | v2.62 | L1c-b (#10) the transfer round's corrections | Behavioural, inside existing verbs (no new route, no §13 code). **§18 (O1):** a dashboard's pinned visualizations travel as entries of the batch's `visualizations` arm — deduplicated by name + version + hash against the roots — and the `dashboards` arm carries dashboards alone (a visualization payload there refused `dashboard.validation.body_invalid` and rolled every such batch back). **§18 (O7):** each family arm is count-bounded (`max-visualizations-per-dashboard`, `reason: too_many`) before the transaction opens. **§18 (O2):** the receive judges a landing's pins by the RELEASE rules — a present-but-not-RELEASED pin refuses the family's `release.dependency_not_released`. **§22.2 (O2, O7):** the import's `templates` array count-bounded; a DRAFT transform pin refuses `visualization.release.dependency_not_released`. **§23 (F1, O2, O7):** the dashboard import is ONE transaction — a refused dashboard leaves nothing landed — the bundle count-bounded, the release rules judged, and the audit carries one `visualization.imported` row per landed bundled visualization. The two-deployment E2E's Orders 60–61 carry `@Test` and run (the L1c handback's 18/18 claim corrected: they never had one). |
| 2026-09-29 | v2.61 | L1c (#10) the transfer — export/import routes and the promotion wire — renumbered at merge after L3a-c's v2.60 | Additive. **§22/§23: four routes** — `GET /{id}/export` on the `*.read` rows (lensed BEFORE the export; the envelope of [Dashboards §3.3](/docs/dashboards.md#33-export-and-import), its manifest carrying `evidence: null` until L4) and `POST …/import` on the NEW `*.import` rows (the owner's ruling, the `api_key.bind` cells: workspace admin + super admin only, never an author — the verb lands RELEASED with no evidence re-run). Import binds through the families' READERS, so the document bounds hold off the save path (a non-integer `version`, a non-array bundle: `body_invalid` naming the path); a malformed body is the family's `body_invalid`. The transfer audits `*.exported` / `*.imported` with `imported_with_evidence` verbatim (enums §15). **§18: the inventory and the batch carry `visualizations` and `dashboards`** — the inventory the same `(name, current_version, body_hash)` triples; the batch the versions' PAYLOADS (never the export envelopes), applied LAST in the receiver's order (§10.4, D61) inside its one transaction, the import lens resolving against the just-landed rows; the applied report counts both. **§23.1:** the dashboard lens's "newer than the target" arm is real — a target-held dashboard at the same version and hash is hidden. No §13 code added. |
| 2026-09-30 | v2.60 | L3a-c (#10) the stream's target outcomes named | **§23.3:** the `refresh_completed` frame's per-target `outcome` values are now enumerated — `ok` (the data frame delivered), `no-data`, `error` (with `stage`/`reason`), `abort` — the server's closed `TargetOutcome` vocabulary, which the client runtime consumes (dashboards §6.6). Documentation of what the writer already sent; no route, payload or behaviour changed. |
| 2026-09-29 | v2.59 | L2 (#10) the dashboard runtime — renumbered at merge after 320's v2.57 and 264's v2.58 | Additive. **New §23.3** — the six runtime and refresh routes (`GET /runtime/config`, `POST /runtime/parameters`, `POST /runtime/visualizations` — the SSE stream, `POST /runtime/refreshes/{refresh_id}/abort`, `GET /refreshes`, `GET /refreshes/{refresh_id}`), all `dashboard.execute`, with the stream's events, the request-body reasons and the runtime's errors (`dashboard.runtime.configuration_stale` / `.dependency_missing`, `dashboard.refresh.saturated` with `Retry-After` and no row, `.not_found`). **§10 / §11's `triggered_via`** gains `DASHBOARD`: the execution's `executed_by` is the refreshing person, it writes no stored result (its `/result` is 404) and only its refresh's abort cancels it (`DELETE /executions/{id}` is 404 for it). |
| 2026-09-29 | v2.58 | 264 (#264, #265) parameter values at every entry — renumbered at merge after 320's v2.57 | **Deliberate tightening at save, release and import (v2.36–v2.37's promise completed): a `PIPELINE` node's parameter literals are now judged by the child parameter's WHOLE declaration**, not its type alone — a literal that coerces but breaks the child's `constraints` bound, length or pattern, or its declared precision/scale, is refused at save (and at release and import, which run the same rules) with the new `400 pipeline.validation.pipeline_parameter_invalid` (`details.reason` names the rule; pipeline-contract §12.9). Until now such a body saved and the value failed only when the composition ran (`child_execution_failed`). **A same-type `${ref}` into a child parameter is accepted only when the parent parameter's descriptor widens losslessly into the child's** (parameter-engine record §6.4): a `DECIMAL(12,4)` parent feeding a `DECIMAL(12,2)` child is now `pipeline_parameter_invalid` with `reason: narrowing` — it used to save and fail at run whenever the parent value had more than two places. **Existing bodies: a stored parent that saves cleanly today may refuse its next save** (never a run of an already-stored version); the shipped demo set and every documented example were replayed — 12 bodies, 3 composition mappings, 0 refused (the lane's replay guard pins the corpus). Calculator `context_key` targets and org/platform tiers carry no descriptor and stay type-only. **`sql_probe` parameters are judged by the shared strict coercion (#265)** — v2.36's no-trim rule reaches the probe, which still trimmed: `" 12 "` for an `INTEGER` and `"TRUE"` for a `BOOLEAN` are now `-32602` argument faults naming the parameter, never the value; the BIG-number digit cap (#278) applies to the probe's values before any parse. |
| 2026-09-30 | v2.57 | 320 (#320) dependency guards | **§5.6:** the pipeline entity purge's response gains `kept_draft_templates` — the draft-only templates the exclusive offer skipped because a parameter set or a visualization also pins them. **§8.10:** the template draft purge is refused `409 template.in_use` while a pipeline, set or visualization pins the draft (it ran no pin check before). **§21.2:** the parameter-set purge and discard routes refuse the new `409 parameter.in_use` while a dashboard pins the set. |
| 2026-09-29 | v2.56 | 321 (#323) the parameter-set routes read their body the #291 way — renumbered at merge: L1b took v2.55 | **§4.2 and §21.4's error table:** a parameter-set request body that cannot be read — not JSON, or nested past §13.21's depth of 100 — is `400 parameter.validation.body_invalid` with `details.reason: malformed_json` on create, update, evaluate and import (the four routes that parse their own text, now through the request mapper) and on the switch (bound by the message converter; the malformed-body handler had no row for the family and answered the pipeline family's `schema_version_unsupported`). The four text routes answered the `500` backstop before (reproduced on the wire: `{` as the body). Well-formed bodies are unchanged; no route, permission or §13 code added. |
| 2026-09-29 | v2.55 | L1b (#10) the visualization and dashboard surfaces | Additive. **New §22 Visualizations and §23 Dashboards** — the parameter-set routes' shape addressed by id (P24): create, the flat listing and the `?prefix=` browse (lens-true totals, #312), the working-version read, versions, the `If-Match` draft write, release (the visualization under the evidence gate, which refuses `visualization.release.tests_missing` / `gate_not_installed` until L4; the dashboard's D61 cascade under `release_pinned_visualizations`), the draft and version purges, discard/restore, the switch and the entity purge — and **`POST /api/v1/dashboards/{id}/validate`**, the §3.2 rules against current state with the verdict as a `200` answer, an author verb on `dashboard.update` (owner ruling 2026-09-29; the spec named `dashboard.read`). The pin guard (`visualization.version.pinned`) and the promoter lens over both families (§23.1) are stated. A malformed body is the family's own `body_invalid` 400. Export/import (L1c), the test sessions (L4), the runtime and refresh routes (L2) and key bindings (L5) are not routes yet. The §13.22/§13.23 codes existed (L1a); none added. |
| 2026-09-29 | v2.54 | 312 (#312) the flat listing lists — renumbered from v2.53 at merge, which 302 takes | **§21.2, one row:** `GET /api/v1/parameter-sets` without a prefix now lists EVERY set the caller's lens admits at its listed version, paged against a lens-true `total` — a flat repository read (the pipelines §5.7 mould). It had reused the ROOT tree level's read, and the name grammar's `folder_required` kept that level empty: every workspace answered `[]` / `total 0`. The tree routes (`?prefix=`) are unchanged; no permission, code or route added. |
| 2026-09-29 | v2.53 | 302 (#302) the template-backed promotion receive | **§18.2: a template-backed parameter set is received whole.** The batch sets' validation (the full §4 — the selector probe included, a customer-datasource connection) moved BEFORE the one transaction, on the receiver's own datasources, with the batch's template payloads overlaying the receiver's registry; the transaction lands pre-validated entries. A refusal at validation is the entry's own catalogued code (`parameter.import.missing_template`, `parameter.validation.*`) and lands nothing; a refusal at landing (C29's `id_taken`, a hash mismatch) rolls the batch back whole, templates included. The §18.2 error table names the set-family codes. No route, permission or §13 code added. |
| 2026-09-28 | v2.52 | the 300 merge's security pass (#300) | **§21.3, one clause:** an explicit JSON `null` `version` on evaluate is absent (the served version), while the switch's `null` is `missing` — the two routes' deliberate divergence stated. Docs only. |
| 2026-09-28 | v2.51 | 300 (#300, the 194d security-pass residue) | **§21.3: the evaluate request's bound answers the platform's `413 request.body_too_large`** (`details.limit_bytes`) — the 400 `parameter.evaluate.unknown_parameter` stand-in with `details.reason = "request_too_large"` is retired; the 2 MiB `RequestLimits` filter stays the pre-read bound. **§21.2/§21.3: request-body shape is judged before resolution** — the switch's missing or non-integer `version` and evaluate's present wrong-typed `version`/`selections` refuse `400 parameter.validation.body_invalid` (`details.path`, `details.reason = missing` / `wrong_type`) instead of a 404 that reads as if the set were absent, or a silent fallback to the served version; the listings' `total` counts the level's lens-admitted sets (never the folders, never sets outside the lens) and `has_more` is truthful on both. Statuses only; no route, permission or code added. |
| 2026-09-28 | v2.50 | 298 (#298, #293) the legacy row's echo | §19.5: a legacy row's echoed `path`/`url` is cut on a code-point boundary (a supplementary character straddling 200 is dropped whole, never half), its `reason` is the length sentence whenever the echo is cut so it quotes no more of the stored path than the echo does (a stored `/api/…` path of 201–204 characters used to quote 1–3 characters past the cut), and the console withholds Unpublish on such a row. No code, route or permission changed. |
| 2026-09-28 | v2.49 | 298 (#291) request constraints on every body | **§4.2**: a within-cap body that cannot be read — not JSON, or nested past §13.21's depth of 100 — is the route family's malformed-body 400 (`details.reason: malformed_json`) on every route, including those that parse their own text body (templates create/update/lifecycle/render/evaluate/import, pipelines create/update/import, execute); a malformed body on six of them answered 500 before. **New §6.1.2**: execute reads its body before the pipeline lookup (a malformed body is the 400 whatever the id), and a parameter number no plain decimal can carry (`1e10000`) is `400 pipeline.execution.invalid_parameter_type` with `details.parameter` and `details.reason: too_many_digits` (was 500). No code, route or permission added. |
| 2026-09-28 | v2.48 | the 194d merge's security pass | **§21: the promoter lens on two reads.** `GET /parameter-sets/{id}/export` resolves through the lens (a hidden set is `404 parameter.not_found`, the pipelines mould — it had read by workspace alone); `GET /parameter-sets/{id}` attaches the `draft` pointer only under the whole view (a promoter never sees a draft's number, hash or editor — 178b, `PipelineService.findDraft`'s rule; it had been re-read unlensed after the lensed working read). |
| 2026-09-28 | v2.47 | 194d (#194) — renumbered at merge after 286's v2.46; parameter engine lane D — the surfaces | Additive. **New §21 Parameter Sets** — the pipelines-shaped routes addressed by id (P24): create, the `?prefix=` browse and flat list, the working-version read, versions, the `If-Match` draft write, release (the 142 cascade consent, §4 steps 4–6 re-run), draft/version purge, discard/restore, the switch, the entity purge, export/import (templates first; the id KEPT, C29 refuses a taken id), and **`POST /parameter-sets/{id}/evaluate`** (§21.3): the whole set re-rendered per submission, the selection priority P26, the request bounded at 1 MiB before parsing, `parameter.evaluate.timeout` 504 / `response_too_large` 413 / `unknown_parameter` 400 as whole-request refusals, per-parameter errors in a 200's `state.errors[]`. **§18 promotion gains parameter sets** (§21.4's bundle is the batch entry): templates → sets → pipelines, the promoter lens over set rows. The `parameter.*` §13.20 family is live from routes (§13's statuses mapped in the catalog: `duplicate_name` 409, `not_found` 404, `version.confirm_mismatch` 400). |
| 2026-09-28 | v2.46 | 286 (#268) a stored declaration save refuses | **Additive code.** Every API surface that binds pipeline parameters (execute, `pipelines_execute`, release checks, schedules) answers `409 pipeline.execution.parameter_declaration_invalid` (pipeline-contract §13.3) when a supplied value or an applied default is judged against a stored declaration today's save refuses — `details.reasons` names the §12.7 codes. A published endpoint keeps its caller-facing envelope: the same refusal is reported inside `400 endpoint.request.invalid`, `details.errors` carrying the binder's re-save sentence (the 286 merge's security pass, observation 1). Before, such a declaration (only a body saved before #194 can hold one) reached the validator's `IllegalArgumentException`: a 500. |
| 2026-09-28 | v2.45 | 286 (#286) legacy endpoint rows — the security pass's follow-ups | No route or field changes. **§19.5**: a legacy row's `path` and `url` are echoed cut at the grammar's 200 characters (they were echoed unbounded — only a database write can store a longer path, and the model keeps the stored one because unpublish is BY PATH); a promotion batch omits a legacy row and the sender now LOGS the omission with its count (before #274 the batch threw; after it the row was left out without a word). The repository's deployment-wide `findLegacy(null)` read, which had no caller, is gone. |
| 2026-09-28 | v2.44 | 286 (#275) — renumbered at merge after 279's v2.43 R3 on the four lists that still read own-only | No route, field or code changes. **§10.1's R3 paragraph stands and names the four surfaces it now covers**: the search palette's executions group, the pipeline and template explorers' Runs tabs, and MCP's executions resource listing read `findVisible` for a member with `execution.read` (they read own-only `findByUser` until #275 — narrower than the sentence, never wider), `findAll` with `execution.read_all`. The paragraph also says what the panes' route permission implies: the explorers' Runs tabs and the palette are `pipeline.read` / `template.read` routes, so a promoter reaches them without `execution.read` and sees her own runs only — R3's scheduled arm is `execution.read`'s (`ScheduledRunSurfacesE2eTest`, whose promoter case is falsified with the arm opened). |
| 2026-09-28 | v2.43 | 279 (#279) the request-body cap | **§4.2 names the envelope's one pre-handler answer.** A request body over `datapipelines.web.max-request-bytes` ([Configuration §3.31](/docs/configuration.md#331-web-request-limits)) is refused at a servlet filter with `413 request.body_too_large` (`details.limit_bytes`) on `/api/v1` and `/mcp`, before authentication and before any parser reads the body — the new §13.21 (pipeline-contract v1.38). Jackson's `StreamReadConstraints` (nesting 100, string 4M chars, number 1000 digits) are stated on the request mappers of both surfaces rather than inherited; a within-cap body the parser refuses still answers the malformed-body 400. No route, permission or existing code changed. |
| 2026-10-02 | v2.75 | #369 the dashboard draft preview — renumbered at merge (main sat at v2.74) | Additive. **§23.3:** the four runtime routes take an optional `version` query parameter naming a DRAFT or RELEASED version (R2, owner-confirmed) — absent is byte-identical with the previous behaviour; three new refusals (`pipeline.execution.invalid_parameter_type` for a non-positive-integer value, `dashboard.not_found` naming `details.version` for a version the caller may not see, `dashboard.key.kind_refused`/`version_is_session_only` for a `dashboard` key naming one). The pin rule is unchanged on a draft (R1): a draft pinning a DRAFT pin is the existing `dashboard.runtime.dependency_missing`/`not_released`, its message now carrying the release hint. The preview PAGE (`GET /dashboards/{id}/preview?version=`) is a first-party page on `dashboard.read` — [auth.md §7.6](/docs/auth.md), not this section. |
| 2026-09-27 | v2.42 | 274 (#274) legacy endpoint rows, retired never fatal | **§19.5's listing carries legacy rows.** A stored `published_endpoints` row whose path predates R-EP5 (fewer than three segments — the shape that made the row mapper throw inside the demo seeder's conflict check and refuse the boot) is now retired: the `GET /api/v1/endpoints` listing answers it flagged (`legacy: true`, `reason`, `enabled`) after the valid rows, in the MCP listing too; a single read (`?path=`) and the serve path answer it `404`; the existing `DELETE /api/v1/endpoints` removes it — the one verb the row supports, offered by the API console with its reason. No route, permission or status code changed; the wire shape of a valid row is byte-identical to v2.40. V40 (metadata-db §4.13) disables such rows in the database and records `retired_reason`; the repository's defensive mapping makes the boot itself immune regardless. |
| 2026-09-27 | v2.41 | scheduler follow-ups (#258, #261) — numbered after origin/main's v2.38 (scheduler-3) at dispatch | Additive. **§20's run object carries the EXECUTION's own timing** — `execution_started_at`, `execution_completed_at`, `execution_duration_ms` — copied by the reconciler from the execution row it already reads (V40's two nullable columns on `schedule_runs`); the run's own `finished_at` stays the reconciler's stamp and is documented as never answering "how long did it take". A runs list answers duration per run in one read, for every member who may read the schedule, including readers without `execution.read`. **§20 names people beside their ids** — `created_by_name` / `updated_by_name` on the schedule, `requested_by_name` on the run (#261): display names resolved in one batched read per response (the pipelines explorer's `ActorNames`), a stamp whose user row is gone falling back to the id's short form. The trail JSON no longer carries `worker` (#253). |
| 2026-09-27 | v2.40 | 250 (#250, #259) R3 on every surface — the split ends | No route, field or code changes. **§10.1's R3 paragraph now states "every surface"**: the UI's execution lists (the executions screen, the dashboard's recent executions and run figures) and MCP's `executions_list`/`executions_get`/`executions_get_result` read the same `findVisible` predicate the REST listing does — own runs plus every `triggered_via = SCHEDULE` run of the workspace — so the temporary split (REST only) is closed. **§4.3b of ui-screens**: the pipelines explorer's Usage tab gains a Schedules heading (#259) — the schedules whose `target_ref` names the pipeline, read through `ScheduleService.listByTarget` under the promoter lens; a schedule is not refusal evidence and the tab badge does not count it. Numbered v2.40 after origin/main's v2.39 (262) at lane time. |
| 2026-09-26 | v2.39 | 262 (#263, with #262) | §6.8's revoked-subscriber paragraph extended — no route, status or wire shape changed. The re-judged workspace is the one the stream OPENED in (a `DP-Workspace` header switch's resolution, not the stamped claim), so a header-switched stream keeps reading the workspace it was opened in while that membership holds; and a write at or past the session token's `exp` ends the stream with the same final `revoked` comment (the duration timer's `close_reason` gains `expired`, [Observability §4.1](/docs/observability.md#41-metric-naming)). |
| 2026-09-26 | v2.38 | scheduler lane 3 (#9) — numbered after origin/main's v2.37 (194a) at merge | Additive. **§20.2: `payload.parameter_bindings`** — an optional, additive key of payload schema 1: one binding per declared `DATE` parameter, `{"source": "keyword", "name": "TODAY"\|"YESTERDAY"}` or `{"source": "literal", "value": …}`; keywords matched only inside an explicit binding (a literal STRING `TODAY` stays the string); each run resolves on its frozen reference — the schedule's timezone, its logical occurrence time, not the actual start. New §13.19 codes `schedule.validation.binding_invalid` (400, `details.reason`/`details.parameter`) and `schedule.validation.binding_conflict` (400); a `reference*` key in the payload or a binding is `payload_invalid`. **§20.11/§20's run object: `prepared.resolved_parameters`** — the literal map the run executed with. §20.5 documents that an edit replaces the bindings wholesale. |
| 2026-09-26 | v2.37 | 194a (#194) parameter engine lane A — the shared validator | **Deliberate break — a pipeline parameter's declared precision and scale now bind the values it accepts.** Every surface that binds pipeline parameters (`POST /pipelines/{id}/execute`, `pipelines_execute`, release checks, schedules, published endpoints §19.3) judges a value with the shared `ParameterValueValidator`: a `DECIMAL`/`BIGDECIMAL` value with more decimal places than the declared `scale` (`12.345` into `DECIMAL(12,2)`), or more integer digits than `precision − scale`, is `400 pipeline.execution.parameter_constraint_violation` with `details.reason` `scale` / `precision` — never rounded. Until now a declared precision/scale was descriptive only and such values bound unchanged; a stored default or schedule input that breaks its declaration now fails when applied. **Additive:** parameters may declare `constraints` (`min`, `max`, `min_length`, `max_length`, `pattern` — pipeline-contract §6.1) and `cardinality` (`SINGLE`; `MULTI` refused until the dashboard round); a value that breaks one is the same new code with its own `details.reason` (`min`, `max`, `min_length`, `max_length`, `pattern`, `pattern_budget`). Save refuses what execute would: `default_invalid`, `constraint_not_applicable`, `constraint_invalid`, `pattern_invalid`, `cardinality_unsupported` (pipeline-contract §12.7). `null` and an absent key are unchanged — the default, then `parameter_required`. |
| 2026-09-26 | v2.36 | 194a (#194) parameter engine lane A | **Deliberate break — parameter values are no longer trimmed.** Until now a `BIGINTEGER`/`BIGDECIMAL` value with surrounding whitespace was trimmed before parsing (`" 12.50 "` bound as `12.50`) on every surface that binds pipeline parameters — `POST /pipelines/{id}/execute` (§6.1), `pipelines_execute`, release checks, schedules — and a published endpoint (§19.3) additionally trimmed its `INTEGER`, `DECIMAL` and `BOOLEAN` query values (`?limit=%2010`). Every one of those is now `400 pipeline.execution.invalid_parameter_type` with the coercion's existing wording (at an endpoint, inside `endpoint.request.invalid`). The parameter engine's rule (P19/P28): the server never trims, rounds or normalises what the caller sent, and every place a parameter value arrives shares one strict coercion, now in `typesystem`. A caller that sent padded values must send them unpadded. Headers (`DP-Result-TTL-Seconds`, `DP-Result-Page-Rows`, `Accept`) are unchanged. |
| 2026-09-26 | v2.35 | scheduler lane 1 (#9) — numbered after origin/main's v2.34 (197/232) | Additive. **New §20 Schedules** — list, create (durable `Idempotency-Key`), preview, get (`ETag` = revision), edit and delete (`If-Match`), pause/resume, unblock, upcoming, runs, one run with its trail, Run now (`202`); session-only in slice 1. **New §10.3A** — `GET /executions/{id}/events?format=json`, the durable event record paged by `event_id`, `410 result.expired` (`event_record_expired`) past retention. §10.1/§10.2: `triggered_via` gains `SCHEDULE`; a scheduled run is visible to every member with `execution.read` (R3), on the REST list and the single reads. |
| 2026-09-25 | v2.32 | 7e (#7) the semantic link | Additive. **§5.10: the release response carries `warnings`** — `[]` when clean, one `pipeline.release.template_needs_review` `{code, message, template, version}` per pinned version citing a retired learned fact; never a refusal. **§8.1/§8.4: a transform accepts `implements`** (outside `body_hash`; inherited when an update omits it; lands on a released version without a draft; `400 template.implements_unresolved` / `template.blocks_not_allowed`). **§8.2/§8.3/§8.5: every projection carries `needs_review`**, a transform's `implements` and, when marked, `retired_facts`; **§8.5 gains `implements={fact_id}`**. §8.8: import keeps only the ids that resolve in the importing workspace (owner ruling 2026-09-25). **§9.7A: rules under `definitions` carry `implemented_by`.** Status caught up (it read v2.30 after v2.31's row). |
| 2026-09-24 | v2.31 | 224 (#224) demo API | New **§19.8**: the demo family seeding publishes every seeded demo pipeline under `/demo/…` (the name-to-path mapping table), mints one configured `api_caller` key (`demo-public-key`) bound to all of them, and the demo-data page renders the same plaintext; the serve path's per-key request budget (`datapipelines.endpoints.key-request-budget`, 60/60, `429 rate_limit.exceeded` + `Retry-After`, per instance, every `api_caller` key; the lake endpoint's gate). §19.7 drops "per-endpoint rate limits" — a per-KEY budget now exists. |

| 2026-09-24 | v2.30 | 215b (#215) key identities, key roles | **Deliberate breaks, owner-ruled (2026-09-24):** (1) **the MCP key is refused on every REST route** (§3.2 — B2: "MCP key should be only MCP"; `403 endpoint.key_kind_refused`, `details.reason = user_key_off_surface`) — REST is a session's surface plus the two confined key kinds; (2) **an unbound published path is unservable** until a key is bound (B3). §16.1: create takes `role` (never `scopes` — refused by name) and returns `role`, `identity {id, display_name}` and `created_by`; the list is the keys the caller created; revoking an endpoint/server key deactivates its identity. §16.2 `/auth/me` returns `role`, not `scopes`. §16.3: every user-admin route answers the unknown-user 404 for a non-person row, before any mutation. §5.16/§7.2: permissions named, not scopes. |
| 2026-09-24 | v2.29 | 215a (#215) the permission catalog | No route, field or status changes. The inline authorization names are the §7.6 catalog's `<functionality>.<permission>` permissions instead of the retired operations (`RELEASE_VERSION` → `pipeline.release`, `READ_EXECUTIONS` → `execution.read`, `MANAGE_API_KEYS` → `api_key.create` / `api_key.bind`, …); §5's release sentence corrected — release is the author's since 2026-09-20 (D8), so a promoter's key cannot release (the text still described the pre-D8 rule). A `role_required` refusal's `details.held` is the role the caller was judged as (auth §9). Status caught up with the change log (it read v2.27). |
| 2026-09-23 | v2.28 | 213 (#213) show-once MCP key | §16.1: `/mine`'s `copyable` is false once the key has been copied — the first read of the copy endpoint destroys the copyable secret in the same statement (auth.md §7.4). No wire shape changed. |
| 2026-09-22 | v2.27 | #212 doc_url | §4.2: `doc_url` is `https://datapipelines.co/docs/pipeline-contract#<section anchor>` — the catalog page at the code's §13 section. The previous shape (`https://docs.datapipelines.co/errors/<code>`) named a host that does not exist. |
| 2026-09-22 | v2.26 | #208 self-membership | Additive. §17.8 / §17.10 / §17.11: a caller addressing their OWN membership — role, login-minted key, removal — is refused with the new `409 workspace.self_membership` (pipeline-contract §13.12); another workspace admin or a super admin does it. |
| 2026-09-21 | v2.25 | 200 (#200) membership-bound keys | Additive. NEW §17.11 `DELETE /workspaces/{name}/members/{user_id}/key` — a workspace admin or super admin revokes a member's login-minted key without removing them (idempotent `204`; the member's session keeps working, the next login mints fresh; [Auth §7.4](/docs/auth.md#74-issuance)). §17.8 states the removal path's new effect: the member's key is revoked in the same act (`auth.api_key.revoked_by_admin`, reason `member_removed` \| `admin_revoked`). |
| 2026-09-21 | v2.24 | 199 (#199) | Security, no wire change. §19.5: a promotion batch carries the bindings **of the source workspace** only — the sender read every workspace's `endpoint_key_bindings` rows and matched by node, so a neighbouring workspace's key bound at a node equal to a promoted pattern rode into the batch by NAME and, on the target, bound the target's key of that name or refused the whole batch for a name the source never chose. The read is now filtered by `workspace_id` in the query (the #191 rule for the serve path, applied to the sender). Numbered on base db12e043; the merger renumbers if another lane took v2.24. |
| 2026-09-21 | v2.23 | 180 (#180) roles R4 | Additive. §16.3: user deactivation names its effect — every session and key of the user answers `401 auth.principal_deactivated` within the liveness window (new §13.7 code; `auth.api_key.invalid` no longer means "owner deactivated"). New §17.5a documents the existing `POST /workspaces/{name}/deactivate` and `/reactivate` routes (super admin) and the 404 rule they keep. |
| 2026-09-21 | v2.22 | 178 (#178) roles R2 | **The promoter lens** ([Auth §11A.1](/docs/auth.md#11a1-the-404-rule)) on every pipeline, template and endpoint read (§5, §8, §19.5): a promoter's session or key sees released objects newer than the promotion target's and nothing else; hidden ones are the same `404` an id from another workspace gets; listings, levels and counts are the lensed set; fail closed (never a `502` on a read) while the target is unreadable. §18.1: the sender caches the inventory per workspace (`inventory-cache-ttl-seconds`). No wire shape changed. |
| 2026-09-20 | v2.21 | 177 (#177) roles R1 | **Two deliberate breaks under the ratified [roles design](https://github.com/msabiransari/datapipelines/blob/main/docs/superpowers/specs/2026-09-20-roles-permissions-design.md)**, both on surfaces whose only callers are the product's own UI and E2E suites: (1) §17 — the membership wire says **`role`** (`viewer` \| `author` \| `promoter` \| `workspace_admin`), replacing the `author`/`promoter`/`admin` booleans on §17.1, §17.6, §17.7 (request and both responses) and §17.10 (renamed *Set a member's role*; body `{"role"}`); an unknown role is a 400. §17.2/§17.6 are a workspace admin's reads (D13); §17.1 (list-own) stays every member's on the switcher's row. (2) §10 — `triggered_by` → **`executed_by`** on the list and read projections, plus **`executed_by_key_kind`** (`user` \| `endpoint` \| `server` \| null); `triggered_via` documents `ENDPOINT`. Ownership rewritten to D11: own unless workspace admin, promoter refused by role, endpoint-key runs admin-only. `/promotion` (UI) is readable by authors too (rule 13). |
| 2026-09-19 | v2.20 | 172 (#172) | **§19 re-rooted (BREAKING for a surface with zero callers)**: published endpoints serve at `/api/<category>/<version>/<path…>` — at least three segments; the category is the engineer's namespace with `v[0-9]+` and `api` reserved (`400 endpoint.path_reserved`, re-checked when the serve registry is built); the version is one free-form literal segment; variables live after it. One `/api` prefix on a submitted pattern is normalised away, never stored. The catch-all mapping constrains the category, so `/api/v1/…` stays the product's — its unknown paths answer the product's 404, not `endpoint.not_found`. Create/read responses gain the full served `url`. Prod held zero published endpoints at the ruling, so nothing migrates. |
| 2026-09-18 | v2.19 | 171 (#171) | §19.2: a published endpoint's `DML`/`DDL` node is allowed when its `source` is `tempdb` (the execution's own in-memory H2, discarded with the run) and refused, naming the datasource, for any registered datasource — previously every `DML`/`DDL` node was refused regardless of source. Re-checked at serve (§5.1/§19.4) through the same rule; no wire or status-code changes. |
| 2026-09-17 | v2.18 | #143/#130 live-stream delivery | §10.4 gains the **live delivery guarantee** paragraph (no new client behaviour): a connected consumer receives the terminal event and the server-closed stream; the frame may lag the `204` by up to the cancel re-issue horizon (~2 s) same-instance, ~one heartbeat interval cross-instance; a consumer that leaves early reads what it missed from §10.3's replay (then §10.2's record). §10.3 names itself that answer. No wire changes. |
| 2026-09-17 | v2.17 | 149 correction / #125 review R149-4 | §6.4.9: a failure thrown by `commit()` itself is ambiguous (the transaction may be durable, the acknowledgement lost) — `rolled_back` is evidence only when the failure came BEFORE the commit attempt; a lost acknowledgement leaves `committed`/`rolled_back` absent. |
| 2026-09-17 | v2.16 | 149 correction / #125 review R149-1 | §6.4.9 `committed` is commit EVIDENCE, independent of the node outcome: `true` survives a `failed`/`aborted` sample when the writer confirmed the commit before the node failed; `false` only with a confirmed rollback (`rolled_back`); **absent on a terminal sample when neither was observed** (unknown — never "not committed"); the abandoned-driver limit stated. |
| 2026-09-16 | v2.15 | 149 / #125 node_progress | New **§6.4.9 `node_progress`** (additive): a measured sample of a node's operation — `operation` (stage/ctas/materialize/writeback/statement/child), the real `destination`, one lifecycle `state`, `started_at`/`observed_at`/`elapsed_ms`, per-state `timings_ms`, `rows_fetched`/`rows_written`/`batches_written` (absent when unobserved), `committed`/`rolled_back` on the terminal sample only, `child_execution_id`; never a percentage. §6.5: zero or more per node between `node_started` and the node's terminal event, the terminal sample first; §10.3 replays them with their original `observed_at`. Cadence: first entry per state, then `datapipelines.executor.progress-sample-interval-seconds`. |
| 2026-09-15 | v2.14 | 142 release cascade | §5.10 gains `?release_pinned_templates=true`: consent to release the DRAFT template versions the body pins in the same transaction as the pipeline flip (templates first; any refusal rolls all back); the refusal's `details` gain `pins_not_released`; audit `template.version.released` per cascaded template (`cascade_from_pipeline_id`, `cascade_from_version`) and `templates_released` on `pipeline.version.released`. Additive; no route removed; no MCP tool releases anything. |
| 2026-09-14 | v2.13 | 140 release checks | New **§5.16** — `POST /pipelines/{id}/versions/{version}/checks/run` (`execute` scope; writes `pipeline_check_runs`; optional `parameters` body; `pass`/`fail`/`error` verdicts) and `GET …/checks` (`read`; definitions with the latest run per check). §5.10 gains the **release-check gate**: a body with `checks[]` releases only past an all-pass fresh run, else `409 pipeline.check.failed` with `details.checks`; `override_checks_reason` (≥ 10 chars, audited as `checks_overridden` + `override_reason` on `pipeline.version.released`) is the only way past. Only the server's run produces `observed` — no endpoint records one from a caller. All additive; no route removed. |
| 2026-09-08 | v2.8 | 099 draft-first (D55/D56) | **§5.1** — `POST /pipelines` lands v1 as a **DRAFT**: `status: "DRAFT"`, `current_version: null`, the `draft` pointer, and release is `POST …/release` like any other draft ([Versioning §3.2](/docs/versioning.md)). **New §6.1.1** — execute with no `version` runs the WORKING version (the draft when one exists, else the latest release); an explicit version stays exact and never clamped. **§5.7** — listing rows carry the working `version` plus a new `status` field. **§5.9** — export is released-only and refuses a never-released pipeline with `409 pipeline.promotion.not_released`. §8.1 (templates) mirrors §5.1. Response VALUES change; no request shape and no route does. |
| 2026-09-02 | v1.18 | 051 auth/config sweep | §10.1 gains its field table (T19): the listing's items were documented only by cross-reference to §10.2. The table is the shared metadata projection minus `result_url`/`result_expires_at`, including the fields §10.2's example omitted (`draft_run`, `error`, `failed_node_id`, `correlation_id`), plus the ownership sentence |
| 2026-08-05 | v1.0 | initial draft | Initial REST API + SSE specification: endpoints, envelopes, SSE event schemas, claim-check pattern, pagination, rate limits, CORS |
| 2026-08-05 | v1.1 | propagation | Updated create-pipeline example to v1.1 Pipeline Contract shape (no `terminal_node_id`, no `datasources_used`, node has `type`/`output`). |
| 2026-08-05 | v1.2 | SSE hardening | Added SSE heartbeat (§6.6) for LB idle-timeout prevention. Updated stream reconnection (§6.8) for multi-instance without sticky sessions: execution continues on originating instance; client polls/fetches result via REST if SSE reconnects to different instance. |
| 2026-08-07 | v1.3 | consistency campaign | **D9:** §7 rewritten — every caller result materialized in Redis, `data_ready` = schema + inline first page + cursor, `DP-Result-TTL-Seconds` clamped TTL (fixed expiry), 100MB cap, `result.too_large`/`result.storage_unavailable`/`result.expired`; inline/claim-check split removed. **D7:** §6.8 inverted — client disconnect cancels the execution after grace; `Last-Event-Id` resumption language removed; new `DELETE /executions/{id}` (§10.4) + `execution_aborted` event (§6.4.8) + `pipeline.execution.not_running`. **D10:** `DP-` header sweep + header registry (§3.6); `RateLimit-*` headers. **D5/D15:** per-user rate limits, scope matrix references. New §16: API-key CRUD, `/auth/me`, user admin; deleted stale `/auth/login`//`/auth/refresh` references. §3.5 idempotency scoped to execute only. `jdbc_url` removed from `node_failed` details (redaction). See [SPEC-REVIEW-2026-08](https://github.com/msabiransari/datapipelines/blob/main/docs/SPEC-REVIEW-2026-08.md) |
| 2026-08-11 | v1.4 | P6a gate C doc-sync | Additive implementation-reality notes from the web module's Gate C: §3.4 correlation-id adoption is shape-conditional (non-UUID inbound values are replaced, not echoed); §7.5 `format=arrow` recognized but not served in v1 (`result.format_unsupported`, supported=[json,csv] — tracked in §14); §9.2 datasources list takes offset/limit and returns the §4.3 envelope (§2 principle 6); §10.3's post-expiry 410 named as `result.expired`. §13 gained `template.not_found` / `datasource.not_found` (404) — see pipeline-contract v1.3. |
| 2026-08-14 | v1.5 | v1.1 introspection build | New **§9.7 schema introspection**: `GET /datasources/{name}/schema`, `/tables?schema=`, `/tables/{table}/columns?schema=` — read-only JDBC metadata with canonical type mapping, `author` scope, 200-table snapshot cap, empty-list-for-unknown-filter. Sourced from datasources §7A; three MCP twins per mcp-server §6.2.16–18. |
| 2026-08-15 | v1.6 | surface restructure (part 1) | §9.7: `GET /datasources/{name}/schema` removed (bundled whole-schema snapshot deleted; table listings stay lightweight so more tables fit in one response). Snapshot example and cap notes dropped. |
| 2026-08-15 | v1.7 | surface restructure (part 2) | §9.7: new `GET /datasources/{name}/schemas` — the flow's entry point; system schemas excluded, `getCatalogs()` on MySQL, empty list valid on schemaless datasources. Tables note now states the unfiltered listing spans schemas and each table's schema belongs in `/columns`. |
| 2026-08-15 | v1.8 | semantics via remarks | §9.7: table and column descriptors gain `remarks` (JDBC REMARKS, omitted when none); schemas listing carries none by construction. |
| 2026-08-15 | v1.9 | hardening round 3 (005 review fix-cycle) | §9.7: a present-but-empty `?schema=` binds to \"\" and is treated as absent (the default applies, not a match-nothing empty filter); a datasource reporting **no current schema** (e.g. database-less MySQL URL) makes `/columns` and an unfiltered `/tables` fail with `400 pipeline.execution.parameter_required` instead of a merged answer (list `/schemas`, pass one); the `/schemas` response becomes a page `{\"schemas\": [...], \"truncated\": bool}` capped at 2000 (was a bare array); blank remarks are omitted, never `\"\"`. §9.1/§9.4: optional `introspection_include_schemas` array (exact lowercase names, no patterns — `400 properties_invalid` otherwise; projected when non-empty; PUT replaces it wholesale). |
| 2026-08-16 | v1.10 | hardening round 4 (007 review fix-cycle) | §9.1/§9.4: `introspection_include_schemas` entries carrying `*` or `%` are rejected as patterns at save (`400 properties_invalid`); `_` stays a legal name character. |
| 2026-08-16 | v1.11 | hardening round 4 (007 review fix-cycle) | §9.7: an unfiltered `/tables` no longer fails on a datasource reporting no current schema — the 400 parameter_required is scoped to `/columns` alone (each tables row carries its own schema; a listing cannot merge). |
| 2026-08-16 | v1.12 | hardening round 5 (008 review fix-cycle) | §9.1/§9.4: the include-schemas list is normalized on save — trim, lowercase, drop blank-after-trim entries, deduplicate (first-seen order) — so a row that landed dirty (restore, manual JSONB edit) reads back clean and an unmodified GET→PUT round-trip succeeds instead of 400ing on blank entries. |
| 2026-08-16 | v1.13 | hardening round 5 (008 review fix-cycle) | §9.1/§9.4: include-schemas entries are validated against the legal-identifier alphabet of the supported dialects (letters, digits, `_`, `$`, `#`, lowercase) instead of a per-character wildcard denylist — `?`, glob ranges, quoted identifiers, and qualified `db.schema` entries are now rejected (`400 properties_invalid`) rather than storing inert. |
| 2026-08-16 | v1.14 | pipeline composition | §10.2: `triggered_via` gains `"PIPELINE"` — a child execution spawned by a parent's PIPELINE node appears in execution history like any other row (enums §18, metadata-db §4.6 V3 lineage columns). |
| 2026-08-17 | v1.15 | pipeline composition | §6.4.3: a PIPELINE node's `node_completed` carries `child_execution_id` (absent for all other node types); the same value appears in the terminal events' `node_stats` entries. §10.1's history surfaces render the lineage: a child row shows its `parent_execution_id` link. |
| 2026-08-28 | v1.16 | workspaces surfaces | New **§17 workspace endpoints** (list-own/read/create per mode/update/delete with `workspace.in_use`/members sub-resource with `open-join`) — §13.12's CRUD codes go live. §9 re-grounded on the workspaces model: §9.1 binding fields (`global` admin-only, `workspace` accessible-to-caller, default = ACTIVE workspace) + `readonly`; §9.2 listing is workspace-scoped with exact totals; §9.3 gains additive `workspace`+`readonly` fields; §9.4/§9.5 D8 gates (member CUD behind `member-datasources-enabled`, global CUD admin-only) with pool-rebuilding flag writes; by-name access to another workspace's datasource is not-found. T23: template duplicate name is `409 template.validation.duplicate_name`. T31: unauthenticated HTML-accepting requests 302 to `/login`; `/api/**`+`/mcp` keep the exact 401 JSON envelope. |
| 2026-09-02 | v2.2 | 055 promotion | New **§18 promotion endpoints** (receiver): `GET /promotion/inventory?workspace=` and `POST /promotion/push`, additive under v2.0. Authenticated ONLY by the §10.6 pre-shared server key on the `DP-Promotion-Key` header (§3.6's registry gains it) — read on that prefix and no other, and an API key or session cookie does not open the routes. The push is one transaction on the receiver; entries carry §9.2's preserved-version fields and arrive in §10.4 push order. No MCP twin and no schedule (§10.1 D8). |
| 2026-09-02 | v2.1 | 046 typed templates | Additive: §8.1's create gains optional `type` (`sql` \| `html`, default `sql`, fixed at creation — an `html` template takes no `dialect`); every template response echoes it; §8.5's list gains the `type` filter. No route changes. Per Templates §11.2's amended clause, the `dialect` conditional-requirement relaxation is claimed here explicitly: no existing payload becomes invalid (every stored template backfills to `sql` with its dialect intact). |
| 2026-09-02 | v2.0 | 043 template addressing | **BREAKING — one addressing form (template-hierarchy-design §9.6).** §8's eight `/{id}` path-addressed template routes are REMOVED and replaced by name-in-query/name-in-body forms (`GET /templates?name=`, `GET /templates/versions?name=&version=`, `PUT /templates` with `id` in the body, `POST /templates/release` + `POST /templates/draft/discard` with `name` in the body, `DELETE /templates?name=`, `POST /templates/render` with `name`+`version` in the body). Measured reason: on the pinned Tomcat an encoded `%2F` in the path is refused `400` below routing and below the security chain, so a hierarchical name (`acme/finance/report`, legal since this round's §4.1 grammar) cannot travel in a path segment at all. `GET /templates` now answers two shapes on one route (single-resource + `404 template.not_found` with `name`, paged list without). Sanctioned break of the v1.4 freeze: the owner confirmed zero callers outside this repo (2026-09-01), so the promise was protecting a population of zero. Datasources and pipelines keep path addressing — their names cannot contain `/`. |
| 2026-09-05 | v2.2 | 067 pipeline folders | Additive and route-free: §5.1 records that a pipeline `name` is now a **folder path** (the template grammar, [Pipeline Contract §3.2](/docs/pipeline-contract.md#32-field-reference)) — every pre-067 name is still valid as a one-segment path, so no request shape changes and no client breaks. §5.7 records that `q` matches across full paths, and that folder BROWSING deliberately stays off REST: it is served by `GET /partials/pipelines?prefix=…` and by MCP `pipelines_list {prefix}`. **No route changes**, because a pipeline is UUID-addressed — the `%2F` problem that forced v2.0 for templates cannot arise here. |
| 2026-09-05 | v2.3 | 074 published endpoints | New **§19**: a released, side-effect-free pipeline served as a `GET` endpoint, answering the `data_ready` payload verbatim. One catch-all handler over a registry — never runtime route registration (R-EP1). Ambiguous paths are refused at publish (`endpoint.path_conflict`) rather than resolved by precedence, so at request time at most one pattern matches. Validation reports every defect at once and is strict about unknown query parameters. `202` on timeout with the execution still running (R-EP3), never `504`. §3.6's registry gains `DP-Result-Page-Rows` (R-EP4 — one contract, honoured by §6's execute too) and the `DP-Execution-Id` response header. Management is `/api/v1/endpoints`, addressed by `?path=` for the same measured reason §8's templates are. |
| 2026-09-05 | v2.4 | 077 mandatory folders | §5.1: a pipeline `name` needs a **folder** — 2–10 segments, not 1–10 ([Pipeline Contract §3.2](/docs/pipeline-contract.md#32-field-reference)). `POST /api/v1/pipelines` and `POST /api/v1/templates` answer `400` with the existing codes (`pipeline.validation.name_invalid`, `template.validation.id_invalid`) for a flat name, and `details.reason` now separates `folder_required` from `grammar`. **A narrowing, not an addition** — the one kind of change §11 forbids after the freeze — taken pre-release on the owner ruling of 2026-09-05: with no rename (Template Hierarchy §4.5), a root-level name created after the tag is permanent, and the root would accrete scratch with no way to tidy it. No route changes. |
| 2026-09-07 | v2.5 | 087 connector seams | §9.1 gains the `credential` object ([Datasources §3.4](/docs/datasources.md#34-credential-kinds)) — `{kind, username?, secret?}`; the legacy top-level `username`/`password` pair still works and means `kind: password`, and a body carrying both is `400 datasource.validation.properties_invalid`. §9.3's response gains `credential: {kind, username?}` and derives `password_set` from the kind (`false` for `none`); top-level `username` is now nullable. §9.4: `credential.kind` is part of the body — moving to `none` clears the stored credential. §9.7's `/tables` and `/tables/{table}/columns` accept `?namespace=` (repeated or dotted) beside `?schema=`, and every table row gains a `namespace` array beside `schema`; `/schemas` gains `entries: [{namespace, label}]` beside the legacy `schemas`. All additive. |
| 2026-09-08 | v2.6 | 089 dp-lake registry (recorded with the §G corrections) | New **§9.8 Lake tables (the dp-lake catalog)** — `POST /datasources/{name}/tables`, `DELETE …/tables/{namespace}/{table}`, `POST …/tables/import` (inline `tables[]`, or a `manifest_url` fetched server-side and restricted to the datasource's own bucket/endpoint — the SSRF boundary), and `GET …/lake-tables` (`read`). The writes are `author`, with a GLOBAL datasource's registry admin-only as a workspaces D8 rule; every write evicts the pool and publishes the §5.7 invalidation. Two corrections landed with this row: the Iceberg `location` is the table's current metadata FILE, not its root (the measured rule, datasources.md §8C.7), and §9.7's introspection listing is registry-backed for LAKE as shipped (datasources.md §8C.3) — §9.8's "a later phase" sentence was written before 089 §C landed. All additive. |
| 2026-09-15 | v2.13 | 138 §B definitions on the REST datasource reads | §9.2 / §9.3 / §9.7A: `GET /datasources` rows and `GET /datasources/{name}` carry `definitions` beside `facts` — the two blocks the MCP `datasources_list` / `datasources_get` have served since 136, from the same one-store-read; the listing used to carry neither and the detail `facts` alone. Additive; `[]` when nothing is recorded. |
| 2026-09-14 | v2.12 | 136 §D PUT /templates inherits dialect and type | §8.4: `dialect` and `type` are optional on update — omitted, the working version's are inherited before the body is bound; a DIFFERENT dialect is refused `400 template.validation.dialect_invalid` (`details.dialect`, `established_dialect`, `template_id`) before validation or any write (the endpoint used to write a changed dialect silently — the defect 135 §C fixed on MCP only); a different `type` stays `type_immutable`. |
| 2026-09-13 | v2.11 | 123 §A table resolution | §9.7: the columns endpoint is table-ADDRESSED — a table absent from the namespace's listing is now **`404 datasource.table_not_found`** (was an empty list), with a "Did you mean …?" line naming the nearest listed table; the message asserts non-existence on complete-catalog dialects and "does not exist, or the credentials cannot see it" on privilege-filtered ones ([Datasources §7A](/docs/datasources.md#7a-schema-introspection)). The empty-list rule survives for unknown `schema`/namespace filters on the listing endpoints; an existing table with zero readable columns stays an empty list. |
| 2026-09-11 | v2.10 | 118 learned semantic layer | New **§9.7A**: the three introspection endpoints and `GET /datasources/{name}` carry the learned-fact block (`facts[]`, additive, omitted-when-none on the listings) — the same shape and code as the MCP twins; the drift check runs at read on `/columns` and on a complete `/tables` listing. No REST write for facts in round 1. |
| 2026-09-10 | v2.9 | T187 release audit | §5.10 and the template release: both verbs are now **audited** (`pipeline.version.released` / `template.version.released`, with `version` and `via` = `session` or `api_key`) and §5.10 states who may release over REST (a promoter's session, or a key whose issuer is a promoter — D4's human decision, expressed either way). |
| 2026-09-08 | v2.8 | 101 version lifecycle | §5.11 is the draft **purge** (row + executions deleted; sole-draft ⇒ the entity goes); new §5.12–§5.15 — `POST /pipelines/{id}/versions/{v}/discard|restore`, `DELETE /pipelines/{id}/versions/{v}` (drafts only), `POST /pipelines/{id}/current` (the manual switch) — and §5.6 becomes the **entity purge** (`include_exclusive_draft_templates`, response carrying the offered set). All session-only (`403 auth.session.required` for keys) and audited (enums.md §15's lifecycle table). §8.10/§8.11/§8.6 mirror for templates by name. The sticky `current_version` (D60) moves only on release / discard-of-current / restore-above-current / switch / purge-of-current-draft; imports never move an existing pointer ([Versioning §3.4](/docs/versioning.md#34-current_version-is-sticky-and-event-driven-d60)). |
| 2026-09-08 | v2.7 | 094 pool settings and retirement | §9.3's response gains **`pool`** (additive): every tunable HikariCP key's effective value, unit and source (`configured` / `dialect_default` / `application_default` / `hikari_default`) — what the pool RUNS with, beside the `properties.hikari` map of what the row STORES. §9.1/§9.4: out-of-range pool values are now REFUSED (`400 datasource.validation.properties_invalid`) instead of being silently rewritten by HikariCP ([Datasources §5](/docs/datasources.md#5-connection-pool-configuration)). §9.4/§9.5: an update or delete RETIRES the pool rather than closing it — new leases miss it at once, statements already running finish on the connection they hold, and a per-instance reaper closes it when drained or at the configured ceiling. All additive; no request shape changed. |