# Pipeline Contract Specification **Status:** v1.69 (revised — see Change Log) **Owner:** datapipelines.co core **Depends on:** [Type System spec](/docs/type-system.md) **Last updated:** 2026-11-01 --- ## 1. Purpose This spec defines the **structure and lifecycle of a Pipeline** — the central artifact of datapipelines.co. A Pipeline is a versioned, declarative DAG of nodes that: - Declares an **input contract** (parameters) - References **templates** (versioned SQL generators) for each node - References **datasources** by name (resolved per-environment) - Declares **where each node's output goes** (tempdb staging, external datasource write-back, or caller return) - Declares the **dependency graph** (which nodes wait for which) - Declares **execution settings** (tempdb engine, etc.) - Is **fully portable across environments** (no env-specific values embedded) A Pipeline is authored by a human via the UI editor, or generated by an LLM via MCP, or imported from a JSON file. Once authored, it is **immutable per version** — editing creates a new version. --- ## 2. Design Principles 1. **Environment-portable by construction.** No env-specific values (hostnames, credentials, IDs, UUIDs) in the Pipeline JSON. Datasources are referenced by name and resolved per-environment. Node IDs and table names are stable, human-readable strings. 2. **Declarative, not imperative.** The Pipeline declares *what* runs and *what depends on what*, not *how* it runs. The executor decides parallelism, ordering, and lifecycle based on DAG topology. 3. **Templates are first-class, separately versioned.** SQL/FTL lives in template entities, not inline in the Pipeline. Pipelines reference templates by `{id, version}`. 4. **Single shared Context at runtime.** All nodes render their templates against the same Context map, initialized from the pipeline's input parameters. Future "calculators" extend the Context before/around node execution. 5. **Node `type` drives executor behavior.** `DQL`, `DML`, `DDL` declare what kind of SQL the template generates. The executor branches: stage ResultSet (DQL), record row count (DML), record success (DDL). 6. **Output target is explicit per node — with one default.** A DQL node's output can go to tempdb (staging), an external datasource (write-back), or the caller (result return). If the `output` block is omitted, the target is `caller`. 7. **Stable, human-readable identifiers.** Node IDs, table names, datasource references — all readable strings (`fetch_orders`, `stg_orders`, `pg-prod`). UUIDs are forbidden in the Pipeline JSON body. 8. **Validate-on-write, universally.** Nothing invalid ever reaches the database — pipelines, templates, datasources, every saved contract validates fully at create/update time. The executor never receives an invalid pipeline. (This principle is cross-cutting: [Templates §7](/docs/templates.md#7-validation-rules), [Datasources §9](/docs/datasources.md#9-validation-rules) apply it to their own entities.) 9. **The caller node is the result node — declared, not topology-derived.** At most one node per pipeline resolves to `output.target: "caller"` (explicitly or by omission); that node's ResultSet is the pipeline's result. Zero caller nodes is legal: a pure write-back/ETL pipeline returns only execution stats. There is no `terminal_node_id` field and no topology-based auto-detection. --- ## 3. Top-Level Pipeline Schema ### 3.1 JSON structure ```json { "schema_version": 1, "id": "a1b2c3d4-...", "name": "acme/finance/monthly_revenue_report", "display_name": "Monthly Revenue Report", "description": "Joins PG orders with MySQL customers; aggregates by customer.", "version": 3, "owner": "user-uuid", "created_at": "2026-08-01T10:00:00Z", "updated_at": "2026-08-05T14:30:00Z", "settings": { "tempdb": { "engine": "H2", "config": { "max_memory_mb": 1024 } } }, "parameters": { "start_date": { "type": "DATE", "required": true, "description": "Inclusive start of the reporting period." }, "end_date": { "type": "DATE", "required": true, "description": "Inclusive end of the reporting period." }, "min_total": { "type": "BIGDECIMAL", "precision": 12, "scale": 2, "required": false, "default": "0.00", "description": "Filter customers with lifetime value below this." }, "include_cancelled": { "type": "BOOLEAN", "required": false, "default": false, "description": "Whether to include cancelled orders." } }, "nodes": [ { "id": "fetch_orders", "description": "Pull orders in date range from PG.", "type": "DQL", "source": "pg-prod", "template": {"id": "acme/finance/fetch_orders.sql", "version": 2}, "output": {"target": "tempdb", "table": "stg_orders"}, "depends_on": [] }, { "id": "fetch_customers", "description": "Pull active customers from MySQL.", "type": "DQL", "source": "mysql-prod", "template": {"id": "acme/finance/fetch_customers.sql", "version": 1}, "output": {"target": "tempdb", "table": "stg_customers"}, "depends_on": [] }, { "id": "revenue_by_customer", "description": "Join orders + customers; aggregate revenue.", "type": "DQL", "source": "tempdb", "template": {"id": "acme/finance/join_revenue.sql", "version": 1}, "output": {"target": "tempdb", "table": "int_revenue"}, "depends_on": ["fetch_orders", "fetch_customers"] }, { "id": "cache_to_warehouse", "description": "Materialize the revenue table to the reporting warehouse.", "type": "DQL", "source": "tempdb", "template": {"id": "acme/finance/select_revenue.sql", "version": 1}, "output": { "target": "datasource", "datasource": "pg-warehouse", "table": "monthly_revenue_cache", "mode": "replace" }, "depends_on": ["revenue_by_customer"] }, { "id": "final_report", "description": "Final projection and filter for caller.", "type": "DQL", "source": "tempdb", "template": {"id": "acme/finance/final_report.sql", "version": 1}, "output": {"target": "caller"}, "depends_on": ["revenue_by_customer"] } ] } ``` ### 3.2 Field reference | Field | Type | Required | Description | |---|---|---|---| | `schema_version` | integer | yes | Pipeline schema version. Currently `1`. | | `id` | string (UUID) | yes | Pipeline identifier. Stable across versions. | | `name` | string | yes | Machine name, and a **folder path**: **2–10** `/`-separated segments, each `[a-z0-9][a-z0-9_.-]{0,63}`, ≤ 200 chars total — the same grammar template names take ([Template Hierarchy §4.1](https://github.com/msabiransari/datapipelines/blob/main/docs/template-hierarchy-design.md#41-grammar), §14). **A folder is required**: `test/scratch` is a name, `scratch` is refused. The full path IS the name; folders are virtual and have no identity. Stable identifier for MCP references and cross-pipeline calls; there is no rename ([Template Hierarchy §4.5](https://github.com/msabiransari/datapipelines/blob/main/docs/template-hierarchy-design.md#45-no-rename-no-move)). | | `display_name` | string | yes | Human-readable name. Shown in UI. | | `description` | string | yes | Long-form description. Shown in UI and MCP tool descriptions. | | `version` | integer | yes | Pipeline version. Monotonically increasing per pipeline. | | `owner` | string (UUID) | yes | User ID of the pipeline owner. | | `created_at` | ISO 8601 timestamp | yes | Creation time. | | `updated_at` | ISO 8601 timestamp | yes | Last modification time. | | `settings` | object | optional | Pipeline-level execution settings. See §5. | | `parameters` | object | yes | Input parameter declarations. See §6. | | `nodes` | array of Node | yes | The DAG node list. See §4. Must be non-empty. | | `checks` | array of Check | optional | The release checks — cross-checks the SERVER runs before a release is allowed. See §3.3. Omitted means none; `"checks": []` canonicalizes to omitted. | > **Composed read shape vs portable body (2026-08-08).** The JSON above is the shape a client READS. The portable pipeline body — what authors submit, what `pipeline_versions.body_json` stores, what exports carry — is the author-owned fields only: `schema_version`, `name`, `display_name`, `description`, `settings`, `parameters`, `nodes`, and the optional `checks` (§3.3, added 2026-09-14 — additive per §15.2). The server-assigned fields (`id`, `version`, `owner`, `created_at`, `updated_at`) live on the pipeline *record* and are composed into read responses; on input they are **ignored entirely** (absent from the inbound model — the strongest form of the write-protection rule; they cannot be smuggled in any casing). | ### 3.3 Release checks (`checks[]`) A pipeline body MAY carry `checks`: an optional list of read-only statements the **server** runs against a datasource the workspace can read, each paired with the value the author expects it to produce. Release of the version is gated on every check passing — the run outcomes and the refusal code are §13.17; the save-time rules are §12.12. ```json "checks": [ { "id": "row_count", "name": "Orders row count matches the raw rollup.", "datasource": "pg-prod", "sql": "SELECT COUNT(*) FROM orders WHERE created_at >= :start_date", "expected": { "kind": "value", "value": 1023412, "tolerance": 0 } }, { "id": "null_share", "name": "Null customer ids stay below one percent.", "datasource": "pg-warehouse", "sql": "SELECT COUNT(*) FILTER (WHERE customer_id IS NULL)::float / COUNT(*) FROM orders", "expected": { "kind": "range", "min": 0, "max": 0.01 } }, { "id": "calendar_complete", "name": "No gap rows in the calendar spine.", "datasource": "pg-warehouse", "sql": "SELECT d FROM calendar d WHERE NOT EXISTS (SELECT 1 FROM orders o WHERE o.day = d)", "expected": { "kind": "rows", "rows": 0 } } ] ``` | Field | Type | Required | Description | |---|---|---|---| | `id` | string | yes | Identifier, `[a-z0-9_]{1,63}`, unique within the body; the run rows key on it. | | `name` | string | yes | The human sentence a release dialog shows, 1–200 characters. | | `datasource` | string | yes | A registered datasource name, resolved per-environment like a node `source`. `tempdb` is refused: the staging database exists only inside an execution, and a check runs with no execution context. | | `sql` | string | yes | One read-only statement. A check has **no rendering**: `${}` interpolation is refused; values bind as `:name` SQL parameters only. | | `expected` | object | yes | `{ "kind": "value" \| "range" \| "rows", ... }`. `value` needs `value` (numeric compare, absolute `tolerance`, default 0); `range` needs `min` ≤ `max` (inclusive); `rows` needs a non-negative `rows`. | Normative notes: - `checks` is versioned with the body exactly like `nodes`: additive per §15.2, and body-hash neutral — an existing body (no `checks` key) deserializes to an empty list and serializes back byte-identically; an explicit `"checks": []` canonicalizes to the same absent form. - Every `:name` bind in `sql` must name a **declared pipeline parameter** (§6). The calculator Context (§7.2) is NOT available to a check — a check runs outside any execution. A run binds the declared parameters exactly as an execute does, and the release gate's run supplies **no parameters**: for any parameter a check binds, the declared DEFAULTS are what the gate proves — not every combination a caller can pass — while a check written against fixed literals proves exactly the baseline those literals name. Expectations are static — a fixed value, range or row count — so an expectation that is only true for one window is a baseline-specific statement, and the baseline belongs in the check's `name`. - The author — agent or human — supplies the query and the expectation, **never an observed value**. `observed` exists only on the server's own run rows (`pipeline_check_runs`, metadata-db §4.20); the body's shape has no field that could carry one in. ### 3.3.1 The release's caller-output record (#328) A release records **the caller node's result columns** — what the dashboard save-time input-contract check (§12.12's cousin on the dashboard side, dashboards.md §4.3) judges a SQL-sourced source against. This subsection is the source of truth for that record. - **What is recorded (D1).** At release, the flip copies the `result_schema_json` of the most recent `pipeline_executions` row for THIS `(pipeline_id, pipeline_version)` with `status = 'SUCCESS'`, a non-null schema, and `started_at > COALESCE(v.updated_at, v.created_at)` of the version row — an execution that ran THIS body (`updated_at` moves on every draft write and never at release). Child executions (`parent_execution_id` non-null) are never candidates. **No probe and no execution runs at release**: a release with no qualifying execution still releases and records NULL (`caller_output = not_observed` in the response); a release is never refused for a missing run. - **Where it lives (D2).** Two nullable JSONB columns (metadata-db §4.5/§4.6, migration V47): `pipeline_executions.result_schema_json` — every execution's caller-output schema, written by the same `recordResult` write that fills `result_row_count`; and `pipeline_versions.caller_output_json` — the release's copy, written in the flip's own statement. The JSON is an **array of `{name, type, nullable}`** objects: `name` is the column name as the driver reported it, `type` a §7 `LogicalType` wire value (`DECIMAL` entries may also carry `precision`/`scale`), `nullable` boolean — an absent or JSON-null `nullable` reads as `true` (unknown admits). A record past the bounds (256 columns, 128 characters per name — `ExecutionRepository`'s constants), or one with a blank column name, is recorded as NULL with a `warn`: never a truncated array, never a failed execution. The record is OUTSIDE `body_hash` (like `implements`): recording changes no hash and opens no draft. - **The one word (D3).** The release response (`caller_output`, rest-api §5.10) and the `pipeline.version.released` audit row carry exactly one word — `recorded` (the D1 read found the version's latest run), `declared` (a SCRIPT caller: the pinned contract is the answer), `none` (no caller node), `not_observed` (a SQL caller, nothing observed — today's validator behaviour continues for that release). Ids, names, versions and this word are all an audit row says; never the column list. - **Precedence in the reader (D4).** The dashboard-save reader (`PipelineReleaseFacts.releaseOf`) answers a release's `outputColumns` in this order: no caller node → EMPTY; a SCRIPT caller → its contract's table columns (the record never overrides a declared contract); any other caller node → the version's `caller_output_json` parsed to columns, null when there is none. The promotion payload carries the record beside `version` and `body_hash`, validated at the receiver (shape, wire types, the bounds) and stored on the imported release row (§9.2/§10.4) — a malformed record refuses the batch and nothing unchecked is stored. --- ## 4. Node Schema ### 4.1 JSON structure (DQL node, staging to tempdb) ```json { "id": "fetch_orders", "description": "Pull orders in date range from PG.", "type": "DQL", "source": "pg-prod", "template": {"id": "acme/finance/fetch_orders.sql", "version": 2}, "output": {"target": "tempdb", "table": "stg_orders"}, "depends_on": [] } ``` ### 4.2 JSON structure (DQL caller node, returns to caller) ```json { "id": "final_report", "description": "Final projection.", "type": "DQL", "source": "tempdb", "template": {"id": "acme/finance/final_report.sql", "version": 1}, "output": {"target": "caller"}, "depends_on": ["revenue_by_customer"] } ``` ### 4.3 JSON structure (DQL node, write-back to external datasource) ```json { "id": "cache_to_warehouse", "description": "Materialize revenue to reporting warehouse.", "type": "DQL", "source": "tempdb", "template": {"id": "acme/finance/select_revenue.sql", "version": 1}, "output": { "target": "datasource", "datasource": "pg-warehouse", "table": "monthly_revenue_cache", "mode": "replace" }, "depends_on": ["revenue_by_customer"] } ``` ### 4.4 JSON structure (DML node, side-effect only) ```json { "id": "update_last_run", "description": "Update last_run timestamp in metadata.", "type": "DML", "source": "pg-meta", "template": {"id": "acme/finance/update_last_run.sql", "version": 1}, "depends_on": ["final_report"] } ``` Note: no `output` block. DML's side effect IS the output. ### 4.5 JSON structure (DDL node, schema setup) ```json { "id": "create_index", "description": "Create index on staging table for join performance.", "type": "DDL", "source": "tempdb", "template": {"id": "acme/finance/create_idx_revenue.sql", "version": 1}, "depends_on": ["revenue_by_customer"] } ``` ### 4.6 Field reference | Field | Type | Required | Description | |---|---|---|---| | `id` | string | yes | Node identifier. `[a-z0-9_]+`. Unique within the pipeline. Stable across versions and environments. | | `description` | string | yes | Human-readable. Shown in UI editor. | | `type` | string (enum) | yes | One of `DQL`, `DML`, `DDL`, `PIPELINE`, `CALCULATOR`, `SCRIPT`. Drives executor behavior. | | `source` | string | yes, except PIPELINE, CALCULATOR and SCRIPT | Datasource name, OR `"tempdb"` for in-memory staging. Must be a registered datasource name in the env, or `"tempdb"`. Forbidden on `PIPELINE` nodes (§12.9), `CALCULATOR` nodes (§12.10) and `SCRIPT` nodes (§12.13 — a SCRIPT is tempdb-only). | | `template` | object | yes, except PIPELINE and CALCULATOR | Template reference: `{id, version}`. Immutable. See [Templates spec](/docs/templates.md). Forbidden on `PIPELINE` nodes (§12.9) and `CALCULATOR` nodes (§12.10). On `SCRIPT` nodes it pins a `jsonata`/`javascript` template (§12.13). | | `pipeline` | object | PIPELINE nodes only | Child pipeline reference: `{name, version}` — the pinned pipeline version the node executes. Required on `PIPELINE` nodes; absent on SQL node types. See §4.9. | | `parameters` | object | no | Child input bindings on a `PIPELINE` node: each key names a child declared parameter or a child calculator `context_key`; each value is a typed literal in the target's §6.3 wire encoding, or `"${ref}"` resolving against the parent's Context tiers (a parent parameter, a parent calculator `context_key`, an org/platform key) at the identical type. See §4.9. | | `kind` | string | CALCULATOR nodes only | The catalog calculator this node evaluates ([Calculators §2](/docs/calculators.md)). Required on `CALCULATOR` nodes; forbidden on every other type. See §4.10. | | `inputs` | object | CALCULATOR and SCRIPT nodes only | On a `CALCULATOR` node: the kind's inputs, by input name — a `"$name"` string is a **reference** to a Context key; every other JSON value is a literal typed against the kind's declared input type (§4.10). On a `SCRIPT` node: the pinned contract's inputs, by input name — a `"$name"` string names a Context key, anything else names a tempdb table an ancestor stages (§4.12). Required on both; forbidden on every other type. | | `context_key` | string | CALCULATOR nodes on a single-output kind; value-mode SCRIPT nodes | The Context key this node writes, per §6.1's `[a-z_][a-z0-9_]*`. Deliberately not called `output`: it names a value downstream nodes bind as `:context_key`, never a table. XOR with `context_keys` on a `CALCULATOR` node (§4.10); required on a value-mode `SCRIPT` node (§4.12); forbidden on every other type. | | `context_keys` | object | CALCULATOR nodes on a multi-output kind | The Context key each of the kind's named outputs is written to, as `{output name: context key}` — every declared output mapped, each key per §6.1. XOR with `context_key`; forbidden on every other type. See §4.10. | | `output` | object | conditional | Optional for `DQL` nodes — omitted means `{"target": "caller"}`. Forbidden for `DML` / `DDL` / `CALCULATOR` nodes. On `PIPELINE` nodes, permitted only when the pinned child has a caller node (§12.9). On `SCRIPT` nodes: required in `row`/`table` mode (`tempdb` or `caller`), forbidden in `value` mode (§4.12). See §4.7. | | `depends_on` | array of string | yes | Parent node IDs. Empty array for source nodes. Must reference existing node IDs. No cycles. **Data flow only** — never an edge added to avoid contention between nodes; the executor owns scheduling (§4.11). | | `settings` | object | no | Per-node execution settings. v1 holds one key, `timeout_seconds` — this node's wall-clock deadline. See §4.11. | | `strict` | boolean | no | SCRIPT nodes only: fail the node when the rejects table is non-empty (`pipeline.script.rejects_strict`). Default `false` (partition — the rejects are written and the run continues). Legal only when the pinned contract declares rejects (§12.13). See §4.12. | ### 4.7 `output` block reference The `output` block declares where the node's ResultSet goes. Optional for DQL nodes — **omitted means `{"target": "caller"}`**; forbidden for DML/DDL. On a PIPELINE node the block declares where the child execution's caller result lands, and is permitted only when the pinned child has a caller node (§12.9 `pipeline_output_on_sideeffect_child`). | `target` value | Additional fields | Description | |---|---|---| | `"tempdb"` | `table: string`; SCRIPT only: `rejects: string` | Stage the ResultSet into the in-memory tempdb under the given table name. Downstream nodes can query it via `source: "tempdb"` SQL referencing the table. A node whose data downstream nodes consume must declare this explicitly. On a `SCRIPT` node `rejects` names the tempdb table the function's rejected rows go to, required exactly when the pinned contract declares `rejects` (§12.13); the rejects table joins the tempdb namespace's uniqueness rule (§12.1). | | `"caller"` | (none — `rejects` is refused, §12.13 R5) | Return the ResultSet as the pipeline's result. **Default when `output` is omitted.** At most one node per pipeline may resolve to this target; zero is legal (write-back pipelines return stats only). A `row`/`table`-mode SCRIPT may be the caller node only when its contract declares no rejects (§12.13). | | `"datasource"` | `datasource: string`, `table: string`, `mode: "replace" \| "append"` | Stream the ResultSet to an external datasource's table. `replace` = TRUNCATE+INSERT in one transaction; `append` = INSERT only. The target table must already exist in the datasource (or be created by a preceding DDL node in the pipeline). | ### 4.8 `source` field rules - `source` is a string. Either: - A registered datasource name in this environment (validated against the datasource registry at write time), OR - The reserved literal `"tempdb"`, meaning "run this SQL against the in-memory staging database for this execution." - The reserved `"tempdb"` literal cannot also be a registered datasource name (validation rejects registering a datasource with that name). ### 4.9 JSON structure (PIPELINE node) ```json { "id": "revenue", "description": "Monthly revenue component.", "type": "PIPELINE", "pipeline": {"name": "acme/finance/monthly_revenue", "version": 4}, "parameters": {"start_date": "${start_date}", "region": "EU"}, "output": {"target": "tempdb", "table": "stg_revenue"}, "depends_on": [] } ``` A PIPELINE node executes another pipeline — the version pinned by `pipeline` — as a real, separate **child execution** (§8.5) and consumes its result. Composition is by invocation, not inlining: the child keeps its own execution record, tempdb, stats, and SSE stream. Field rules: - `pipeline` — required: `{name, version}`. `name` per §3.2's path grammar (a child may live under any folder); `version` a positive integer pinning an existing live pipeline version (mutable DRAFT during authoring, immutable RELEASED for release/import). Self-reference (`name` = the containing pipeline's own name) is invalid. There is no "latest". - `source` and `template` — **forbidden** (mirrors "output forbidden on DML/DDL"): the node runs a pipeline, not SQL. - `parameters` — optional map filling the child's inputs: each key names a child **declared parameter**, a child **CALCULATOR `context_key`** or a child **value-mode SCRIPT `context_key`** (078 A5-composition — supplying the key skips the child's node exactly as a direct execute-time supply does, §4.10; child calculator and SCRIPT keys are optional, never required, so `pipeline_parameter_unmapped` still reads declared parameters only). Each value is either a typed literal obeying the target's §6.3 wire encoding, or the string form `"${ref}"` resolving against the PARENT's Context tiers: a parent declared parameter, a parent CALCULATOR `context_key` (typed by its kind's output), a scalar parent SCRIPT `context_key` (ANY at save), or an org/platform key (org always STRING, platform per §0.2's canonical types) — at the identical type. An ANY-output key on either side skips the type check: the value is typed only by the run. A child SCRIPT key is ANY at save time; a referenced scalar is encoded at run time and checked against the pinned script contract by the child's node. Object/list references do not cross this boundary. No expressions, no concatenation — a value is a literal or a reference, nothing in between (v1). **No auto-passthrough:** a parent and child calculator key spelled the same are NOT implicitly mapped — the mapping is always an explicit entry here, or the child's node computes its own value. - `output` — standard §4.7 block, permitted only when the pinned child has a caller node. Zero-caller child ⇒ `output` must be absent; the node is side-effect-only and downstream `depends_on` gives ordering. - `depends_on` — unchanged. A pipeline whose entity is DISCARDED (every version discarded) still resolves existing pinned references — the pinned version keeps resolving — but blocks NEW references at save time (`pipeline_reference_deleted`, §12.9). This mirrors template deletion exactly. Draft authoring may pin a DRAFT or RELEASED child version (#459, owner requirement superseding 101/D58). Release and import require a RELEASED child (`pipeline_reference_not_released`); missing or DISCARDED child references remain refused. --- ### 4.10 JSON structure (CALCULATOR node) ```json { "id": "fiscal_q", "description": "The fiscal quarter this run reports on.", "type": "CALCULATOR", "kind": "fiscal_quarter", "inputs": {"date": "$current_date", "fiscal_start": "$org_fiscal_start_date"}, "context_key": "run_fiscal_quarter", "depends_on": [] } ``` A CALCULATOR node evaluates one **pure catalog function** ([Calculators](/docs/calculators.md)) and writes typed values into the execution Context. It runs no SQL, touches no database, and produces no table. Downstream nodes read a value the way they read any Context key — `:run_fiscal_quarter` in a template, `"$run_fiscal_quarter"` in another calculator's `inputs`. The node writes **one value or a named set**, decided by the kind, never by the author. A **single-output** kind (most of the catalog) writes its one value under `context_key`, as above. A **multi-output** kind declares a set of named outputs (`period_bounds` declares `start` and `end`), and the node maps EVERY output to a Context key through `context_keys`: ```json { "id": "window", "description": "The quarter this run reports on.", "type": "CALCULATOR", "kind": "period_bounds", "inputs": {"date": "$current_date", "unit": "quarter"}, "context_keys": {"start": "window_start", "end": "window_end"}, "depends_on": [] } ``` Field rules: - `kind` — required: a name in the registry. The catalog is additive and a `kind` never changes meaning, because a `kind` is written into bodies that are versioned, exported and promoted. The catalog entry says whether the kind is single- or multi-output. - `inputs` — required (an empty object is legal for a kind with only optional inputs). `"$name"` is a **reference** to a Context key; anything else is a literal typed against the kind's declared input type, which is what makes `"fiscal_start": "09-15"` a per-pipeline override with no config edit. An input the kind declares optional may be omitted, and the kind then applies its documented default. - `context_key` — required on a single-output kind, per §6.1. It may shadow an org or platform key; it may **never** shadow a declared parameter (§12.10 `calculator_output_collision`), and no two nodes may write the same key. - `context_keys` — required on a multi-output kind: an object mapping **every** declared output name to its Context key (`{"start": "window_start", "end": "window_end"}`). No partial mapping — a caller who needs one value still maps both, so no reader can bind a key the node never writes. Each mapped key obeys every rule `context_key` does: §6.1's name shape, no shadowing a declared parameter, one writer per key. - `context_key` XOR `context_keys` — never both, never neither, and the field must fit the kind's shape (`context_key` on a multi-output kind or `context_keys` on a single-output one is refused, §12.10 `calculator_output_shape_mismatch`). - `source`, `template`, `output` — **forbidden**, for the same reason `source`/`template` are forbidden on a PIPELINE node: this node is not the kind of thing they describe. - `depends_on` — unchanged, and load-bearing in a way it is not elsewhere: **sequencing is topology**. A reference to another node's key, and a SQL node binding `:that_key`, are valid only from a node that depends on the producer, directly or transitively (§12.10 `calculator_input_unordered`). Binding only ONE of a multi node's keys still requires the edge. Array order means nothing. At run time the node evaluates at its DAG position, writes its value (once, every key on a multi-output kind), and reports through SSE and history like any other node — `rows_out: 0`, plus `context_key` and `context_value` on a single node's stats (`context_values`, every key, on a multi node's) so the run detail page and `executions_get` show what it produced. A failure is the standard node failure record with `pipeline.node.calculator_failed` (§13.4). **Every key a calculator node writes is also an implicit optional execute input** (078, owner ruling 2026-09-05; extended to the named set 121). A caller may supply it in the execute request's `parameters` object, typed by the kind's output — an ANY-output kind (`coalesce`, `if_null`, `map`) accepts any JSON scalar. Supplied, the node is **skipped**: it does not evaluate, the supplied value is what downstream nodes bind, and the node's stats carry `provided_by: "caller"` so a run record shows where the value came from. Unsupplied (an explicit JSON `null` reads as unsupplied), the node runs and computes the value exactly as before. A supplied value that fails coercion is refused with `pipeline.execution.invalid_parameter_type` (§13.3), exactly like a declared parameter — to the caller there is no second kind of execute input. For a multi-output node the override is **all-or-nothing**: every key supplied and the node is skipped, none and it computes; a proper subset is refused before any node runs with `pipeline.execution.calculator_keys_partial` (§13.3). ### 4.11 `settings.timeout_seconds` / `settings.query_timeout_seconds` — a node's own deadline and statement budget ```json { "id": "scan_trips", "type": "DQL", "source": "lake", "template": { "id": "trips/scan", "version": 3 }, "output": { "target": "tempdb", "table": "trips" }, "depends_on": [], "settings": { "timeout_seconds": 600, "query_timeout_seconds": 300 } } ``` | Field | Type | Required | Description | |---|---|---|---| | `timeout_seconds` | integer | no | This node's WALL-CLOCK deadline, in seconds, overriding `datapipelines.executor.node-timeout-seconds` (default 300) for this node alone. Must be a positive integer no greater than `datapipelines.executor.node-timeout-max-seconds` (default 900), or the save is refused with `pipeline.validation.node_timeout_invalid` (§12.8). | | `query_timeout_seconds` | integer | no | This node's own SQL **statement** timeout (156, #2), in seconds — overrides the pipeline's `settings.query_timeout_seconds` (§5.3), the datasource's `query_timeout_seconds` and the operator's per-dialect/application default, for this node alone. Legal only on a node type that runs a statement (`DQL`, `DML`, `DDL`); declared on `PIPELINE` or `CALCULATOR` it is refused. Must be a positive integer no greater than `datapipelines.executor.node-query-timeout-max-seconds` (default 900) **and** no greater than this node's own effective `timeout_seconds` (above) — refused, not clamped, with `pipeline.validation.node_query_timeout_invalid` (§12.8) naming both numbers. | **Three budgets, one precedence** (the same table appears in [configuration.md §3.2](/docs/configuration.md) and [dag-executor.md §5.3](/docs/dag-executor.md)): | Bound | Setting | Scope | Enforced by | |---|---|---|---| | Execution | `datapipelines.executor.execution-timeout-seconds` (600) | the whole execution | the executor | | Node | `node.settings.timeout_seconds`, else `datapipelines.executor.node-timeout-seconds` (300) | one node: RENDER → CONNECT → EXECUTE → STAGE → MATERIALIZE | the executor | | Statement | `node.settings.query_timeout_seconds` (156, §5.3), else pipeline `settings.query_timeout_seconds`, else the datasource's `query_timeout_seconds`, else the dialect's operator default ([configuration.md §3.2](/docs/configuration.md)), else `datapipelines.executor.node-query-timeout-seconds` (60) | one `execute*` call | the JDBC driver | Read it downward: the execution deadline is the outermost, the statement timeout the innermost, and the node deadline is what closes the gap between them. The statement bound is the **driver's**, and drivers honour it unevenly — a node whose driver ignores it is stopped by the node deadline anyway, which is exactly why the middle row exists. A pipeline-level `settings.execution.timeout_seconds` is still §5.4 future work: the outermost bound is the operator's setting, not the pipeline's. **Statement-timeout precedence in full (156, #2).** The statement row above compresses five tiers into one cell; read individually: a node may override the pipeline's statement timeout, which may override the datasource's, which may override the operator's per-dialect default, which falls back to the flat application default. `settings.query_timeout_seconds` — at node level or pipeline level (§5.3) — is bounded by the SAME operator ceiling as `node.settings.timeout_seconds`: `datapipelines.executor.node-query-timeout-max-seconds` (default 900), refused rather than clamped at save (`pipeline.validation.node_query_timeout_invalid` / `pipeline.validation.pipeline_query_timeout_invalid`, §12.8). A node's own `query_timeout_seconds` must also not exceed that SAME node's effective wall-clock deadline (the Node row above) — a statement budget longer than the node's own lifecycle could ever reach is refused at save, naming both numbers. `pipeline.node.query_timeout`'s failure detail (§13.4) names which tier resolved the effective value (`source: node | pipeline | datasource | dialect | application`), so an author can tell whether their own override took effect or an operator default did. A node that exceeds its deadline fails with `pipeline.node.timeout` (§13.4, HTTP 504), whose `details` carry `timeout_seconds`, `elapsed_ms` and the `phase` the budget went in. **A timeout is a signal, not a wall to route around.** The first response to a node that will not fit its budget is to make the work smaller — pre-aggregate, push the filter down, prune on the partition column. Slicing one scan into four quarterly nodes to dodge the deadline produces a pipeline that is slower, four times as likely to fail, and lies about what it does. When the scan is legitimately long, raise THIS node's `timeout_seconds` and say why. A PIPELINE node that declares no `timeout_seconds` is not bounded by the node deadline: its work is a child execution, already bounded by `execution-timeout-seconds` one level down. One that declares it, is. --- ### 4.12 JSON structure (SCRIPT node) ```json { "id": "shape_orders", "description": "Order lines at reporting grain; cents to currency; refunds preserved.", "type": "SCRIPT", "template": { "id": "acme/shape/order_lines.jsonata", "version": 3 }, "inputs": { "orders": "stg_orders", "tz": "$org_timezone", "min_total": "$min_total" }, "output": { "target": "tempdb", "table": "order_lines", "rejects": "order_lines_rejected" }, "strict": false, "depends_on": ["stage_orders"] } ``` A SCRIPT node evaluates the pinned script template as a **pure function** over staged data and the Context (the transform-nodes design record §3). The language is the pinned template's `type` (`jsonata` or `javascript`, when its isolate is installed); the **mode** (`row`, `table`, `value`) and the input/output shapes are the pinned version's **contract** — never repeated on the node, which conforms to it at save time (§12.13). The function has no handle to tempdb, the network, the filesystem, the clock or a random source: iteration is the executor's job (`row` mode reads the single table input in `result-batch-size` batches), `$now()`/`$millis()` return the execution's `current_timestamp`, and `$random()`, `$shuffle()` and `Math.random()` are refused (`pipeline.script.evaluation_failed`). The output is a function of the template version and its inputs — every run is reproducible. | Field | Rule | |---|---| | `template` | Pinned like every pin; its type must be `jsonata` or `javascript` (`pipeline.validation.script_template_type`). | | `source` | **Forbidden** (`script_source_forbidden`). A SCRIPT is tempdb-only: it reads what upstream nodes staged plus the Context, and writes tempdb, the Context or the caller. It never touches a datasource — the read-only rule holds trivially, and every run is reproducible from staged inputs. | | `inputs` | The CALCULATOR convention. `"$name"` is a Context key of any tier; anything else is a tempdb table a node this one depends on stages. The key set must equal the contract's input names (`script_input_contract`); a table no upstream stages or a `$key` nobody writes is `script_input_unknown`; a value input whose Context type does not fit is `script_input_contract`. An object-valued key (R4) binds only here. | | `output` | **Table output** (`row`/`table` mode): `{ "target": "tempdb", "table", "rejects"? }` or `{ "target": "caller" }` (the caller-node rules of §9 apply unchanged). `rejects` on a `caller` output, or a rejects-declaring contract pinned by a caller node, is `script_rejects_on_caller` (R5 — rejects are never silently dropped). **Value/object output** (`value` mode): no `output` block — the node writes `context_key` only (R3: no JSON result bodies), obeying every calculator rule: §6.1 name shape, one writer per key, never shadowing a declared parameter (`calculator_output_collision` / `calculator_output_name_invalid` reused), topology-ordered. Any other output/mode pairing is `script_output_shape`. `rejects` present without the contract declaring it — or absent when it does — is `script_rejects_undeclared` / `script_rejects_missing`. | | `strict` | Boolean, default `false`. Legal only when the contract declares rejects (`script_strict_without_rejects`); then a non-empty rejects table fails the node with `pipeline.script.rejects_strict` (count + first ten reasons). Default is partition: the rejects are written and the run continues. | | `depends_on` | Sequencing is topology, as for calculators: a `$key` written by another node is bindable only from a node that depends on its writer (`calculator_input_unordered` reused). | **Every key a `value`-mode SCRIPT writes is an implicit optional execute input**, exactly like a calculator's (§4.10): supplied by the caller, the node is skipped and its stats carry `provided_by: "caller"`; the supplied value is checked against the contract's declared output at the node (an `object` output accepts any JSON object under the `max-value-bytes` cap). **Object keys have one reader (R4).** A key whose writer declares `output.kind: object` may be bound only by a SCRIPT `inputs` entry; a SQL node's `:key`, a calculator input or a `PIPELINE` node's `parameters` reference to it is refused at pipeline save with `pipeline.validation.script_object_key_bound`, naming the key, its writer and the offending node. A caller may supply it at execute time as any JSON object under the cap; it persists as jsonb in the execution's Context snapshot. ## 5. Settings ### 5.1 `settings.tempdb` — staging engine configuration ```json "settings": { "tempdb": { "engine": "H2", "config": { "max_memory_mb": 1024 } } } ``` | Field | Type | Required | Description | |---|---|---|---| | `engine` | string | optional (default `"H2"`) | Staging engine. v1 supports only `"H2"`. Future: `"DUCKDB"`, etc. | | `config` | object | optional | Engine-specific configuration. Keys depend on `engine`. For `H2`: `max_memory_mb` (default 1024). | If `settings.tempdb` is omitted entirely, defaults to H2 with default config. ### 5.2 Why pipeline-level (not global) - Different pipelines have different needs: a small lookup-heavy pipeline fits in 256 MB; a large aggregation pipeline needs 4 GB. - Pipeline authors declare the resource need upfront; operators budget accordingly. - Settings travel with the pipeline across environments — same engine choice in dev and prod (the actual `pg-prod` connection differs, but the staging engine choice doesn't). ### 5.3 `settings.query_timeout_seconds` — the pipeline-wide SQL statement timeout ```json "settings": { "query_timeout_seconds": 300 } ``` | Field | Type | Required | Description | |---|---|---|---| | `query_timeout_seconds` | integer | no | The default SQL **statement** timeout, in seconds, for every `DQL`/`DML`/`DDL` node in this pipeline that does not declare its own `node.settings.query_timeout_seconds` (§4.11). Overrides the datasource's `query_timeout_seconds` and the operator's per-dialect/application default; overridden itself by a node's own setting. Must be a positive integer no greater than `datapipelines.executor.node-query-timeout-max-seconds` (default 900), or the save is refused with `pipeline.validation.pipeline_query_timeout_invalid` (§12.8). | Pipeline-level, not global, for the same reason §5.2 gives `settings.tempdb`: different pipelines scan different volumes against different engines, authors declare the resource need once for the whole pipeline instead of repeating it on every node, and the setting travels with the pipeline across environments. It does not raise or lower any node's own **wall-clock** deadline (`node.settings.timeout_seconds`) — the two settings are independent, and §12.8's invariant is checked per node against whichever statement timeout applies to it. ### 5.4 Future settings (out of scope for v1) - `settings.execution.parallelism` — per-pipeline concurrency limit override (different from the global default). - `settings.execution.timeout_seconds` — per-pipeline overall timeout. - `settings.templates.cache_size` — override template cache size for this pipeline's executions. - `settings.output.default_format` — default wire format for `/execute` (JSON / Arrow / CSV). --- ## 6. Parameters (Input Map Declaration) ### 6.1 Schema `parameters` is an object whose keys are parameter names (`[a-z_][a-z0-9_]*`, length 1–63 — no leading digit, since a name Freemarker cannot reference is undeclarable; the 63-char cap matches the identifier caps elsewhere and stops an unbounded key becoming an unbounded error string) and whose values are parameter descriptors. ```json "parameters": { "param_name": { "type": "INTEGER" | "BIGINTEGER" | "DECIMAL" | "BIGDECIMAL" | "BOOLEAN" | "STRING" | "BINARY" | "DATE" | "TIME" | "TIMESTAMP", "required": true | false, "default": , // optional; only honored if required = false "precision": , // required for DECIMAL, BIGDECIMAL "scale": , // required for BIGDECIMAL; required for DECIMAL with exact-numeric semantics "description": "...", // optional but recommended "constraints": { // optional (#194) — value rules; a value that breaks one is REFUSED, never adjusted "min": , "max": , // ordered types; wire-encoded in the parameter's own type; inclusive "min_length": , "max_length": , // STRING (characters) and BINARY (decoded bytes); non-negative "pattern": "" // STRING; anchored (the whole value); at most 256 characters }, "cardinality": "SINGLE" // optional (#194); SINGLE is the only value a pipeline accepts today (§6.2) } } ``` `constraints` and `cardinality` are the parameter engine's declaration model ([parameter-engine record](https://github.com/msabiransari/datapipelines/blob/main/docs/superpowers/specs/2026-09-21-parameter-engine-design.md) §3.5, P28), shared so that a pipeline and a parameter set judge a value by the same `ParameterValueValidator` (`typesystem`). Both are omitted from a body that does not declare them, so bodies saved before 2026-09-26 serialise — and hash — unchanged. ### 6.2 Rules - `type` is one of the canonical types from [Type System §3](/docs/type-system.md#3-canonical-types-v1), excluding `NULL`. - `required: true` and `default` present is **invalid** (validation error: `pipeline.validation.conflicting_required_default`). - `precision` / `scale` rules match the [Type System §7.3 column descriptor rules](/docs/type-system.md#73-field-by-field-rules). - `default` must be valid JSON matching the wire encoding for the type: - `INTEGER`, `DECIMAL` → JSON number - `BIGINTEGER`, `BIGDECIMAL`, `STRING`, `BINARY`, `DATE`, `TIME`, `TIMESTAMP` → JSON string - `BOOLEAN` → JSON boolean - Parameter values supplied at execution time must satisfy the schema. Type mismatch → `pipeline.execution.invalid_parameter_type`. - **One validator, one null policy (#194).** Every value — supplied, or a default being applied — is judged by the shared `ParameterValueValidator`: the §6.3 coercion, then the declared precision and scale, then `constraints`. A value that breaks a rule is `pipeline.execution.parameter_constraint_violation` (§13.3) with `details.reason` naming it (`min`, `max`, `min_length`, `max_length`, `pattern`, `pattern_budget`, `scale`, `precision`). A STORED declaration save would refuse today — only a body saved before #194, when `Parameter` ignored unknown keys, can hold one — is never trusted where it is used: judging a value or a default against it is `pipeline.execution.parameter_declaration_invalid` (409, §13.3, #268) with `details.reasons` naming the save-time codes, never a 500; a stored `cardinality: MULTI` is refused the same way rather than bound as a list. That refusal is the SERVER's defect, so it outranks the caller's own (#298): when one binding carries both — a caller's bad value on one parameter and a stored declaration refused on another — the response is the 409 with both failures in `details.failures`, in the order found, whichever came first. Each judged declaration is compiled once per bind. A JSON `null` or an absent key is **unsupplied** — neither judged nor refused — and resolves as it always has: the `default`, then `parameter_required`, then `null` for an optional parameter. A published endpoint reaches the same binder, and a URL has no null: an absent query key is its only unsupplied form (`?x=` is an empty-string value). - **Precision and scale are enforced on values** (since 2026-09-26, #194 — a deliberate break, REST API change log v2.37): a `DECIMAL(12,2)` / scale-2 `BIGDECIMAL` refuses `12.345` (`reason: scale`, nothing is rounded) and a `DECIMAL(6,2)` refuses `99999.5` (`reason: precision` — at most `precision − scale` integer digits). Trailing zeros carry no digits (`12.500` fits scale 2 and binds as sent); a `DECIMAL` with no `scale` is approximate and has no places to check; a `BIGDECIMAL` with no `precision` is unbounded in its integer part. Before, both were descriptive only. - `constraints` (#194) — `min`/`max` apply to `INTEGER`, `BIGINTEGER`, `DECIMAL`, `BIGDECIMAL`, `DATE`, `TIME`, `TIMESTAMP`, wire-encoded in the parameter's own type (a `BIGDECIMAL` bound is a JSON string) with `min ≤ max`; `min_length`/`max_length` apply to `STRING` (characters, counted as code points) and `BINARY` (decoded bytes), non-negative with `min_length ≤ max_length` — an undeclared `max_length` leaves the value unbounded, as before; `pattern` applies to `STRING`: a `java.util.regex` pattern matched against the WHOLE value, at most 256 characters, refused at save if it uses a backreference, lookahead/lookbehind, a possessive quantifier, an atomic group or the COMMENTS flag, and matched over a read-counting input with a budget of 100,000 reads (past it: `reason: pattern_budget`). A key outside the catalogue, a key on a type it does not apply to, a malformed bound or length, and an unsafe pattern are refused at save (§12.7). The `default` must satisfy the parameter's own rules (`default_invalid`). - `cardinality` (#194) — `SINGLE` (the default when omitted) or `MULTI`. **`MULTI` is refused (`pipeline.validation.cardinality_unsupported`) until the dashboard round adopts list binding**: the field is the contract, shared with the parameter engine; the semantics land later. Any other value is refused with the same code. ### 6.3 Wire encoding of input parameter values Input parameter values are submitted by clients (via REST API or MCP tool) as JSON. Their wire encoding follows the same rules as Type System §3.1 — `BIGINTEGER` and `BIGDECIMAL` parameters are sent as strings; `INTEGER` and `DECIMAL` (precision ≤ 15) as numbers; etc. This is the **symmetric contract**: data flows in and out of the pipeline using the same type rules. **Coercion is strict — wrong wire encoding is rejected, never silently converted:** - A JSON number supplied where `BIGDECIMAL`/`BIGINTEGER` (string-on-wire) is declared → `pipeline.execution.invalid_parameter_type`. (Accepting it would silently lose precision for values beyond IEEE 754 safe range.) - A JSON string supplied where `INTEGER`/`DECIMAL`/`BOOLEAN` (number/boolean-on-wire) is declared → `pipeline.execution.invalid_parameter_type`. - `TIMESTAMP` parameter values MUST carry an explicit offset or `Z`; a zone-less timestamp string is rejected — the server never guesses the client's timezone. - `DATE`/`TIME` values must be exact ISO 8601 (`YYYY-MM-DD`, `HH:MM:SS[.ffffff]`). The fractional part, when present, is 1–6 digits — sub-microsecond input is rejected, not silently truncated (2026-08-08: strictness applies on ingress exactly as on egress; leniency here would make §3.5's exact egress a silent transformation). - `BINARY` parameter values must be PADDED standard base64 (RFC 4648 §4, length ≡ 0 mod 4) — the same alphabet and padding §3.5 mandates on egress; unpadded input is rejected (2026-08-08). - Precision and scale bind values: a `DECIMAL`/`BIGDECIMAL` value with more decimal places than the declared `scale`, or more integer digits than `precision − scale`, is `pipeline.execution.parameter_constraint_violation` (`details.reason`: `scale` / `precision`) — never rounded. Until 2026-09-26 a declared precision/scale was not checked against supplied values (`12.345` bound unchanged into a `DECIMAL(12,2)` parameter); enforcing it is the second deliberate break of #194 ([REST API change log v2.37](/docs/rest-api.md#appendix-a-change-log)). - Nothing is trimmed: a value with surrounding whitespace — `" 12.50 "` for `BIGDECIMAL`, `" 12 "` for `BIGINTEGER` — is `pipeline.execution.invalid_parameter_type`. Until 2026-09-26 the two BIG types were trimmed before parsing (and a published endpoint trimmed its `INTEGER`/`DECIMAL`/`BOOLEAN` query values too); that tolerance was retired as a deliberate break ([REST API change log v2.36](/docs/rest-api.md#appendix-a-change-log), #194). The coercion lives in `typesystem` since then, shared with the parameter engine ([Type System §3.1](/docs/type-system.md#31-wire-encoding-summary)). - A textual `BIGINTEGER`/`BIGDECIMAL` value longer than **1024 digits** is refused UNPARSED (`too_many_digits` under `invalid_parameter_type`) — on JDK 21 the parse of an n-digit string is O(n²), so a megabyte of digits is seconds of CPU from one value (#278, 2026-09-28). The length check is O(1) and runs before `BigInteger`/`BigDecimal` ever see the text; every caller that coerces through the shared paths — the execute API, `pipelines_execute`, a parameter set's textual `default_value`, an expression literal — inherits the same cap. --- ## 7. Execution Context (Runtime Construct) The Context is a **runtime in-memory map** — never serialized in the Pipeline JSON. It is constructed per execution. ### 7.1 Lifecycle 1. Pipeline execution starts. The executor seeds the Context with the **org tier** — the deployment's `datapipelines.org.*` values (Configuration §3.21) — and then the **platform tier**: `current_date`, `current_timestamp`, `execution_id`. 2. Executor reads pipeline input parameters (from REST/MCP call). 3. Executor validates each parameter against the pipeline's `parameters` schema. Defaults applied for missing optional params. 4. Executor overlays the resolved parameters onto the Context: `Map` where keys are context keys and values are typed Kotlin objects (Date, BigDecimal, Boolean, String, etc.). A declared parameter that spells an org or platform key the same way **is** the override. 5. For each node, in topological order: a. The template engine renders the node's template against the **current Context**, and the rendered SQL's `:key` binds resolve against it too. b. A `CALCULATOR` node (§4.10) evaluates its kind once and writes every key it declares — its `context_key`, or each mapped `context_keys` value — into the Context; every node that `depends_on` it, directly or transitively, sees them. c. The rendered SQL executes against the node's `source`. d. Behavior depends on `type` and `output.target` (see §8). 6. The caller node's ResultSet (the node resolving to `output.target: "caller"`, if any) is the pipeline's result. Pipelines with no caller node return execution stats only. 7. The **fully resolved Context** — every tier, calculator outputs included — is persisted with the execution ([DAG Executor §8](/docs/dag-executor.md)). ### 7.2 What's in the Context — one namespace, five tiers Every value a node can bind is a typed Context key matching §6.1's `[a-z_][a-z0-9_]*`. A node binds `:key` without knowing which tier supplied it; the tier only decides who wins when two of them spell a key the same way. Lowest precedence first: | # | Tier | Keys | Who sets it | |---|---|---|---| | 1 | **org config** | `org_currency_name`, `org_currency_symbol`, `org_fiscal_start_date`, `org_week_start`, `org_timezone` — the yml path minus the `datapipelines.org.` prefix, dots and dashes as `_`. All typed `STRING`; `org_fiscal_start_date` is an `MM-DD` string the calculator kinds parse | the deployment's `application.yml` (Configuration §3.21) | | 2 | **platform** | `current_date` (`DATE`, evaluated in `org_timezone`), `current_timestamp` (`TIMESTAMP`), `execution_id` (`STRING`) | the executor, at execution start | | 3 | **declared `parameters`** | whatever §6.2 declares, after defaulting | the pipeline body — declaring a key an org or platform value also provides IS the override, and it is visible in the body | | 4 | **execute-time inputs** | declared parameters (§6.3), and every key a `CALCULATOR` or value-mode `SCRIPT` node writes as an implicit **optional** input (§4.10, §4.12 — supplied → the node is skipped; unsupplied → the node runs; a multi-output calculator's keys are all-or-nothing) | the caller's `parameters` object | | 5 | **calculator outputs** | every key a `CALCULATOR` node writes (§4.10 — one, or a multi-output kind's whole mapped set), and every key a value-mode `SCRIPT` node writes (§4.12 — including an `object`-shaped value, persisted as jsonb under the `datapipelines.script.max-value-bytes` cap) | the node, at its DAG position | A calculator output may shadow an org or platform key; it may **never** shadow a declared parameter, and one is refused at save time with `pipeline.validation.calculator_output_collision` (§12.10) — the same rule binds a value-mode SCRIPT's key (§12.13). An **object-valued** key (a SCRIPT whose contract declares `output.kind: object`) has ONE reader: another SCRIPT's `inputs` — a SQL bind, a calculator input or a composition parameter naming it is refused at save with `pipeline.validation.script_object_key_bound` (R4, §4.12). A calculator that RUNS writes over everything below tier 5 — including a caller-supplied value for a *different* key — but a key the caller supplied skips its node (§4.10), so the two never contest the same key in one run. Org and platform keys are deployment constants, so the save-time dry render knows them: a template binding `:org_currency_symbol` validates without the pipeline declaring anything. None of them is a secret. ### 7.3 What's NOT in the Context - Upstream node outputs (those are tempdb tables OR external-datasource tables, referenced by name in SQL). - Connection credentials or env-specific values. - Execution metadata beyond the platform tier's three keys — timings, node stats and the result's location travel on the execution record, not in a namespace templates render against. ### 7.4 Template variable resolution Templates use Freemarker syntax to reference Context keys. A key declared in `parameters` is a **value** and is referenced as a bind parameter — `:name` — which the executor binds on a prepared statement, so a caller-supplied `STRING` is never parsed as SQL (Templates §4.5). `${}` interpolation is for **structure** (table names, dynamic fragments) and is refused for declared parameter names at save time (`template.validation.parameter_interpolated`, §12.6): ```ftl SELECT order_id, customer_id, total_amount FROM orders WHERE order_date BETWEEN :start_date AND :end_date AND total_amount >= :min_total ``` The template engine enforces that all referenced variables exist in the Context — referencing an undefined variable is a render failure (`pipeline.node.template_render_failed`). This is also enforced at **save time**: pipeline validation dry-renders every referenced template against the pipeline's declared `parameters` (using defaults where present, type-appropriate sample values otherwise). A template variable with no corresponding pipeline parameter fails validation with `pipeline.validation.template_parameter_undeclared` — the error never waits for execution. Pipeline `parameters` is the **single declaration point** for template variables; templates do not declare their own parameter schemas ([Templates spec](/docs/templates.md)). --- ## 8. Node Execution Behavior (per `type` and `output.target`) > **Consistency:** a write-back node commits on its own connection when it finishes, and nothing > rolls it back if a later node fails — each node is atomic on its own database, the pipeline as a > whole is not. The full model, the seams between databases, and how to design a write-back around > it are [DAG Executor §16](/docs/dag-executor.md#16-consistency-model). ### 8.1 DQL nodes A DQL node runs a `SELECT` query and produces a ResultSet. Behavior depends on `output.target`: | `output.target` | Executor behavior | |---|---| | `tempdb` | Stream the ResultSet into the tempdb table named by `output.table`. Downstream nodes reference this table by name in their SQL. | | `caller` (default when `output` omitted) | Materialize the ResultSet into the Redis result store and emit `data_ready` with the schema, the inline first page, and the result cursor ([REST API §7](/docs/rest-api.md#7-result-delivery)). At most one node per pipeline resolves to this target. | | `datasource` | Stream the ResultSet to the external datasource + table named in `output`. Apply `mode`: `replace` does `TRUNCATE` (or `DELETE`) + `INSERT` in one transaction; `append` does `INSERT` only. The target table must exist (created by a preceding DDL node or pre-existing). | ### 8.2 DML nodes A DML node runs `INSERT`, `UPDATE`, `DELETE`, or `MERGE` against its `source`. The executor: 1. Renders the template, executes the SQL. 2. Captures the affected row count. 3. Records row count in `node_stats`. No staging, no output block. DML nodes are side-effect sinks. They typically appear as the last node(s) in a pipeline that needs to write back results — but write-back is more naturally modeled as a DQL node with `output.target: "datasource"` (let the framework handle the INSERT). DML is for cases where the SQL is too complex for the framework's auto-streaming (conditional logic, MERGE with custom conflict handling, etc.). ### 8.3 DDL nodes A DDL node runs `CREATE`, `ALTER`, `DROP`, or `TRUNCATE` against its `source`. The executor: 1. Renders the template, executes the SQL. 2. Records success/failure. 3. No staging, no output block. Use cases: create indexes on tempdb tables for join performance, create target tables in external datasources before a write-back DQL node, prepare schema state. ### 8.4 Type-based execution dispatch ```kotlin when (node.type) { DQL -> executeDql(node, context, staging) DML -> executeDml(node, context) DDL -> executeDdl(node, context) PIPELINE -> executeSubPipeline(node, context) // §8.5 — before render/source resolution } fun executeDql(node, context, staging) { val sql = render(node.template, context) val source = resolveSource(node.source, staging) val rs = source.executeQuery(sql) // omitted output block resolves to Caller at deserialization time when (val output = node.output ?: NodeOutput.Caller) { is NodeOutput.Tempdb -> staging.stage(rs, output.table) is NodeOutput.Caller -> materializeCallerResult(rs) // → Redis result store, REST API §7 is NodeOutput.Datasource -> streamToExternal(rs, output.datasource, output.table, output.mode) } } ``` ### 8.5 PIPELINE nodes A PIPELINE node executes the pipeline pinned by its `pipeline` reference as a **child execution**: a real, separate execution with its own execution record, own tempdb, own stats, and own SSE stream, started through the internal execution service (never HTTP). The child runs under the parent's principal; authorization is checked on the parent only. The `{name, version}` reference resolves **within the active workspace** — cross-workspace references do not exist in v1. Dispatch happens before render/source resolution — a PIPELINE node carries neither a `template` nor a `source`. - **Parameters.** The node's `parameters` map becomes the child's execution parameters: literals pass through as supplied; each `"${ref}"` reference resolves against the parent's live Context tiers, including scalar SCRIPT outputs. A child value-mode SCRIPT key takes a scalar reference as ANY, with output typing enforced by the child's pinned script contract. - **Result.** The child's caller-node ResultSet streams **directly** to the parent executor (`direct` delivery — nothing is materialized to the result store, and the result is not re-fetchable afterwards; re-running is the recovery path) and lands per the node's `output` block, exactly like a DQL node's ResultSet (§8.1). A zero-caller child produces no stream: the parent waits for child completion, and success/failure is the node's outcome. - **Failure.** A failed child fails the PIPELINE node fail-fast with `pipeline.node.child_execution_failed`; the detail carries the child's error code and execution id, so the debugging trail leads to a real execution record. - **Cancellation.** Cancelling an ancestor cancels the whole family: the cancellation flag is honored for every execution sharing the family's `root_execution_id` ([Metadata DB §4.6](/docs/metadata-db.md#46-pipeline_executions)). A descendant stopped because an ancestor was cancelled — or because an ancestor's `execution-timeout-seconds` expired — ends `ABORTED`, never `FAILED`: it did not fail, and its own deadline did not expire. Only the execution whose own deadline fired reports `pipeline.execution.timeout`. - **Guards.** Composition depth is bounded by `datapipelines.pipelines.max-composition-depth` (default 5), checked statically at save time (`composition_too_deep`, §12.9 — the current pinned bodies define the reference tree) and again at run time (`pipeline.node.composition_depth_exceeded` — a backstop; reaching it means save-time validation was bypassed). **Both count the same unit: pipelines.** A pipeline with no PIPELINE nodes is depth 1 and the bound is inclusive, so the default admits a chain of 5 pipelines (4 parent→child hops) and refuses the 6th — identically at save time and at run time. Mutable draft references can form cycles; the static walk and runtime depth bound terminate them. Release revalidates the current graph. Child executions do **not** take per-user concurrency slots — only root executions do; a waiting parent holding a slot while its children queue would deadlock. --- ## 9. The Caller Node (Result Node) There is **no `terminal_node_id` field** and no topology-based auto-detection. The result node is simply the node that resolves to `output.target: "caller"` — explicitly, or by omitting its `output` block (§4.7). ### 9.1 Resolution 1. For each DQL or SCRIPT node, resolve its effective target: the declared `output.target`, or `caller` if `output` is omitted (DQL only — a SCRIPT's output is always explicit, §4.12). 2. At most one node may resolve to `caller` — this is the **caller node**, and its ResultSet is the pipeline's result. 3. Zero caller nodes is legal: the pipeline is a pure write-back/ETL pipeline. Execution returns stats only and emits no `data_ready` event. DAG position is irrelevant: the caller node may be a sink, or sit mid-DAG (though mid-DAG caller nodes are unusual — a node whose data downstream nodes consume must stage to `tempdb`, so a caller node's data is consumed by nobody else). ### 9.2 Validation rules | Code | Check | |---|---| | `pipeline.validation.multiple_caller_nodes` | More than one node resolves to `output.target: "caller"` (explicitly or by omission). | | `pipeline.validation.non_dql_caller_target` | A `DML` or `DDL` node has an `output` block (`caller` or otherwise). Only DQL nodes have outputs. (Same check as `dml_has_output` / `ddl_has_output` — listed here for the caller-specific case.) | Any combination of `tempdb` / `datasource` / `caller` targets across DQL nodes is otherwise legal — a DQL sink may write to a datasource, stage to tempdb for a later DML node, or return to the caller. ### 9.3 Multiple-sink example (legitimate) A pipeline that returns data to the caller AND writes a side effect: ``` fetch_orders ─┐ ├─→ revenue_by_customer ─┬─→ cache_to_warehouse (DQL, output: datasource pg-warehouse) fetch_customers ┘ └─→ final_report (DQL, output omitted → caller) ``` Two sinks: `cache_to_warehouse` (write-back) and `final_report` (caller return). One node resolves to `caller` → valid. ✓ ### 9.4 Zero-caller example (pure ETL) ``` fetch_orders ─→ script ─→ write_to_warehouse (DQL, output: datasource pg-warehouse) ``` No node resolves to `caller`. The pipeline returns execution stats only; no `data_ready` event is emitted. ✓ --- ## 10. Output Table Naming Rules ### 10.1 Rules - Format: `[a-z0-9_]+`, length 1–63 (H2 identifier limits). - Uniqueness is **per namespace**: all `tempdb` `output.table` values must be unique among themselves (they share one staging database), and `datasource` `output.table` values must be unique per target datasource. A tempdb table and a write-back table may share a name; two write-backs to different datasources may share a name. - Must not be `tempdb` or any name starting and ending with `__` (reserved namespace). - **Recommended convention** (not enforced, but LLM prompts and UI defaults should follow): - `stg_*` for tables staging raw source data - `int_*` for intermediate scripted tables - Cache tables written to external datasources: name them per the target's convention (often `monthly_revenue_cache`, `daily_orders_materialized`, etc.) - Stable across versions and environments. Renaming is a breaking change requiring downstream template SQL updates — bump pipeline version. ### 10.2 Why stable names Downstream SQL in templates references these names directly: `SELECT * FROM stg_orders JOIN stg_customers ON ...`. The names are part of the pipeline's vocabulary. They must be: 1. **Predictable** for template authors (humans and LLMs). 2. **Stable** so a pipeline promoted from dev to prod doesn't break. 3. **Traceable** in logs — `pipeline.node.fetch_orders.stg_orders` is readable, `pipeline.node.fetch_orders.tbl_a1b2c3d4` is not. --- ## 11. Environment Portability Rules ### 11.1 What's portable (in the Pipeline JSON) - Pipeline `id`, `name`, `display_name`, `description`, `version`. - All node IDs, descriptions, types. - All `output.table` names. - All `template` references (`{id, version}`). - All `source` *names* (not connection details). - All `parameters` declarations. - All `settings`. ### 11.2 What's NOT portable (resolved per-environment) - **Datasource connections** — the name `pg-prod` resolves to different JDBC URLs in dev vs. prod. The mapping is environment-specific config, stored separately (see [Datasources spec](/docs/datasources.md)). - **Template bodies** — portable, but live in their own registry with their own version history. - **Pipeline UUID** — stable across envs if exported and re-imported. ### 11.3 Promotion workflow 1. Export pipeline from dev: `GET /pipelines/{id}/versions/{version}` returns the JSON (see §14). 2. Optionally export referenced templates (separate call or bundled export). 3. Import to prod: `POST /pipelines/import` with the JSON. The import: - Checks if pipeline `id` already exists in prod. - If yes, increments `version` and stores the new version. - If no, creates new with `version: 1`. - Validates all `source` names (excluding `tempdb`) against the prod datasource registry — fails with `pipeline.import.missing_datasource` if any name is unresolvable. - Validates all `output.datasource` references (for write-back nodes) against the prod datasource registry. - Validates all `template` references against the prod template registry — fails with `pipeline.import.missing_template` if any referenced template version is missing. ### 11.4 Forbidden in Pipeline JSON Hard validation errors at write-time. Validation code: `pipeline.validation.forbidden_env_specific_value` (listed in §12.1). **Scanned fields:** every string value in `nodes[].source`, `nodes[].output.datasource`, `nodes[].output.table`, `nodes[].id`, and all keys/values under `settings`. (Template bodies are validated separately by the template registry; parameter *values* are runtime data and are not scanned.) **Detection heuristics applied to each scanned string:** - JDBC URL pattern: `^jdbc:` prefix. - Hostname pattern: contains a dot-separated name with a known TLD-like tail or matches `[a-z0-9-]+\.[a-z0-9-.]+:[0-9]+` (host:port). - IP address pattern: IPv4 dotted-quad or bracketed IPv6. - UUID pattern: `[0-9a-f]{8}-[0-9a-f]{4}-...` (the pipeline-level `id` and `owner` fields are the only allowed UUIDs). - Credential-shaped: `password=`, `pwd=`, `token=`, `secret=` key fragments; `dpk_` API-key prefix. - Absolute path: leading `/` or drive-letter pattern. - Environment literal: exact-match `dev`, `staging`, `prod`, `production` as a whole value. These heuristics are deliberately conservative (whole-value or prefix matches) — `stg_orders` containing "stg" is fine; a `source` of `pg-prod` is fine (it is a datasource *name*; the check applies to values that are not references into the datasource registry). --- ## 12. Validation Rules All checks run at pipeline create/update time. A pipeline that fails any check is rejected with HTTP 400 and an error response. ### 12.1 Structural validations | Code | Check | |---|---| | `pipeline.validation.schema_version_unsupported` | `schema_version` is supported (currently only `1`); also the family's malformed-body stand-in — a JSON value of the wrong type for its declared field (a number or boolean for a string, a string or float for an integer) answers it with `details.reason: "wrong_type"`, `details.expected`, `path` naming the field, and never the value (#333) | | `pipeline.validation.name_invalid` | `name` is a path of 2–10 `/`-separated segments, each `[a-z0-9][a-z0-9_.-]{0,63}`, ≤ 200 chars total — a folder is required (§3.2). `details.reason` is `folder_required` when a folder is the only thing missing, `grammar` otherwise. Also raised on a PIPELINE node's `pipeline.name` (§4.9), at path `nodes[i].pipeline.name` | | `pipeline.validation.new_root_requires_confirmation` | **The AGENT surface only** (094): `pipelines_create` refuses a `name` whose ROOT segment has no pipelines under it yet, unless the call carries `confirm_new_root: true`. `details.root` is the segment in question and `details.existing_roots` lists the roots that DO exist, so the agent can reuse one or go and ask. `test/` is always allowed. The name is perfectly LEGAL — this is a confirmation, not a grammar failure, which is why it is a distinct code from `name_invalid`. REST, the UI and `pipelines_update` are unaffected: a person choosing a folder has already decided, and an update cannot change a name | | `pipeline.validation.duplicate_node_id` | All node `id` values are unique | | `pipeline.validation.duplicate_name` | Pipeline `name` not already taken — HTTP 409, mapped from the `pipelines` UNIQUE constraint. Uniqueness is PER WORKSPACE (`uq_pipelines_workspace_name`, since the V4 re-key — two workspaces may each hold a `report`; the demo's same-name pipelines depend on it) and includes soft-deleted rows: execution history references the name, so a deleted pipeline's name is not reusable in its workspace until hard-deleted. (Corrected twice: an early revision said "live pipelines only"; a 2026-08-08 correction then said GLOBAL, which was the V3 truth and stopped being true at V4.) See [Metadata DB §4.4](/docs/metadata-db.md#44-pipelines) | | `pipeline.validation.duplicate_output_table` | `output.table` values are unique per namespace: among `tempdb` targets, and per target datasource among `datasource` targets (§10.1) | | `pipeline.validation.invalid_identifier` | `output.table` and node `id` match `[a-z0-9_]+` | | `pipeline.validation.reserved_identifier` | No node id or table name is `tempdb` or matches `__.*__` | | `pipeline.validation.forbidden_env_specific_value` | No scanned field contains env-specific values (hostnames, IPs, JDBC URLs, credentials, UUIDs, paths — §11.4) | ### 12.2 DAG validations | Code | Check | |---|---| | `pipeline.validation.dangling_dependency` | Every id in every node's `depends_on` exists in `nodes` | | `pipeline.validation.cycle_detected` | The dependency graph is acyclic | | `pipeline.validation.empty_pipeline` | `nodes` is non-empty | | `pipeline.validation.pipeline_too_large` | `nodes` count ≤ 1000 (added 2026-08-08). A hard bound so save-time validation cannot be turned into a denial of service — the validator must be crash-proof against hostile input regardless (see the crash-safety rule below), and this bound is defence in depth on top of that. | > **Crash-safety (normative, 2026-08-08).** Save-time validation runs on attacker-influenced JSON from an `author`-scoped principal and MUST NOT be crashable by input. Two specific requirements: (1) graph traversal (cycle detection, reachability) is **iterative**, never recursive in graph depth — a deep `depends_on` chain must not exhaust the JVM stack; (2) any value scanned by the §11.4 portability heuristics is length-bounded before regex evaluation (a scanned value longer than 512 chars cannot be a hostname/URL/UUID/credential and is passed through un-flagged), and the heuristic patterns are bounded in repetition depth. A `StackOverflowError` reaching the request thread from a 2 KB payload is a defect, not a medium. ### 12.3 Caller-node validations (see §9.2) | Code | Check | |---|---| | `pipeline.validation.multiple_caller_nodes` | At most one node resolves to `output.target: "caller"` (explicitly or by omitting `output`) | ### 12.4 Node-type validations | Code | Check | |---|---| | `pipeline.validation.type_invalid` | Each node `type` is one of `DQL`, `DML`, `DDL`, `PIPELINE` | | `pipeline.validation.dml_has_output` | DML nodes must NOT have an `output` block | | `pipeline.validation.ddl_has_output` | DDL nodes must NOT have an `output` block | | `pipeline.validation.output_target_invalid` | `output.target` (when the block is present) is one of `tempdb`, `caller`, `datasource` | | `pipeline.validation.output_table_missing` | `output.target: "tempdb"` requires `output.table` | | `pipeline.validation.output_datasource_missing` | `output.target: "datasource"` requires `output.datasource` and `output.table` | | `pipeline.validation.output_mode_invalid` | `output.target: "datasource"` requires `mode`, and it must be `replace` or `append` — a MISSING mode is rejected, never defaulted (2026-08-08: one of the two values TRUNCATEs the target table; guessing is not acceptable) | ### 12.5 Datasource validations | Code | Check | |---|---| | `pipeline.validation.unknown_datasource` | Every node `source` (except `tempdb`) AND every `output.datasource` exists in the env's datasource registry | | `pipeline.validation.datasource_readonly` | No write-shaped use names a readonly datasource: no `DML`/`DDL` node `source` and no `output.datasource` resolves to one flagged `is_readonly` (workspaces design 2026-08-16 §6, D6). The three and only three write shapes, each a distinct runtime raise site, checked as one rule; `details` carries the node id, the datasource name, and which shape fired (`dml_source` / `ddl_source` / `output_target`). DQL reads and everything `tempdb` are untouched — the check is on the use, never on the datasource alone | ### 12.6 Template validations | Code | Check | |---|---| | `pipeline.validation.template_not_found` | Every `template.id` exists in the template registry. A RESTORE re-judges it against today's registry, under the workspace writer guard (#517, [Versioning §3.5.3](/docs/versioning.md#353-the-reverse-arrows-into-other-families-320)) | | `pipeline.validation.template_version_not_found` | Every `template.version` exists for that template id. A RESTORE re-judges it against today's registry, under the workspace writer guard (#517, [Versioning §3.5.3](/docs/versioning.md#353-the-reverse-arrows-into-other-families-320)); a pinned version DISCARDED since refuses this code with `details.reason: discarded` | | `pipeline.validation.template_dialect_mismatch` | Template's `dialect` matches the node's `source` dialect (for `source != "tempdb"`). For `source: "tempdb"`, template `dialect` must match the dialect of the engine declared in `settings.tempdb.engine` (H2 in v1; DuckDB templates become valid when that engine lands). Applies only to `type='sql'` templates — the only kind a node can legally reference — each of which carries a non-null dialect by the schema ([Templates](/docs/templates.md); template-hierarchy-design §7, 046) | | `pipeline.validation.template_type_mismatch` | A DQL/DML/DDL node references a `type='html'` template — refused at pipeline save. Every template a node can legally reference is `sql`; `details` carries `template_type` (template-hierarchy-design §7, 046) | | `pipeline.validation.template_parameter_undeclared` | Dry-render check: every referenced template renders successfully against the pipeline's declared `parameters` (defaults where present, type-appropriate sample values otherwise). An undefined variable fails here, at save time (§7.4). | | `pipeline.validation.template_render_failed` | Dry-render failed for a reason OTHER than an undeclared variable — a type-mismatched built-in, an unresolvable imported macro, an expression error (added 2026-08-08: a render failure at save time is a validation outcome, never an exception escaping the validator, and never mislabeled as `template_parameter_undeclared`) | 042 B2, not a `pipeline.validation.*` row: beside the dry render, save also scans each resolved template's parse tree, and a **declared** parameter name found inside a `${}` interpolation is refused with `template.validation.parameter_interpolated` (§13.9, HTTP 400) — declared parameters are values and bind as `:name` (Templates §4.5); the message names both forms. The scan is AST-based (Templates §4.2 reasoning), honours macro-parameter and loop-variable shadowing, and no spelling hides a live interpolation from it (pinned against Freemarker 2.3.34). The declared set is the pipeline's `parameters` block **including every calculator output key** (078 A1): every key a CALCULATOR node writes joins the set — a single node's `context_key` typed by its kind's output, each of a multi-output node's mapped keys typed by its own output (121) — and is refused in one more position a plain parameter is not — a conditional's test (`<#if x??>`, `<#elseif x>`) — because a derived value gating SQL structure is the same hole as an interpolated one, one directive earlier. ### 12.7 Parameter validations | Code | Check | |---|---| | `pipeline.validation.parameter_type_invalid` | Each parameter `type` is one of the 10 allowed canonical types (NULL excluded) | | `pipeline.validation.parameter_precision_missing` | `precision` is present when type is `DECIMAL`. For `BIGDECIMAL` precision is OPTIONAL — omitted means unbounded, exactly as in the type system's §4 encoding (adjudicated 2026-08-08; a declared parameter follows the same semantics as a derived column) | | `pipeline.validation.parameter_name_invalid` | Every parameter key matches `[a-z_][a-z0-9_]*`, length 1–63 (added/anchored/bounded 2026-08-08 — no leading digit since `${1st_date}` is not legal Freemarker; 63-char cap per §6.1) | | `pipeline.validation.parameter_scale_missing` | `scale` is present when type is `BIGDECIMAL`, or `DECIMAL` with exact semantics | | `pipeline.validation.conflicting_required_default` | `required: true` and `default` are not both set | | `pipeline.validation.default_type_mismatch` | `default` survives the FULL §6.3 coercion for its declared type, not merely the JSON-type check — a default that would fail at execution fails at save (D2, adjudicated 2026-08-08) | | `pipeline.validation.default_invalid` | `default` coerces but breaks the parameter's own rules — a `constraints` bound, length or pattern, or its declared precision/scale (`min: 0` with `default: -1`; `1.234` for `DECIMAL(12,2)`). `details.reason` names the rule, as `parameter_constraint_violation` does (#194) | | `pipeline.validation.constraint_not_applicable` | A `constraints` key on a type it does not apply to (§6.2: `min`/`max` on the ordered types, the lengths on `STRING`/`BINARY`, `pattern` on `STRING`); `details.constraint` names it (#194) | | `pipeline.validation.constraint_invalid` | A malformed `constraints` block (`details.reason`): `not_an_object`, `unknown_key`, `length_not_an_integer`, `pattern_not_a_string` (the wire pre-scan), `bound_type` (a bound not wire-encoded in the parameter's type), `min_greater_than_max`, `negative_length`, `min_length_greater_than_max_length` (#194) | | `pipeline.validation.pattern_invalid` | A `pattern` over 256 characters (`too_long`), one that does not compile (`syntax`), or one using a construct the regex budget refuses — a backreference, lookahead/lookbehind, a possessive quantifier, an atomic group, the COMMENTS flag (`unsafe_construct`) (#194) | | `pipeline.validation.cardinality_unsupported` | `cardinality` other than `SINGLE`: `MULTI` until the dashboard round adopts list binding, and any value that is not a cardinality at all; `details.supported` is `["SINGLE"]` (#194) | ### 12.8 Settings validations | Code | Check | |---|---| | `pipeline.validation.tempdb_engine_unsupported` | `settings.tempdb.engine` is `H2` (v1) | | `pipeline.validation.tempdb_config_invalid` | `settings.tempdb.config` keys are valid for the chosen engine | | `pipeline.validation.node_timeout_invalid` | A node's `settings.timeout_seconds` is a positive integer no greater than `datapipelines.executor.node-timeout-max-seconds` (default 900). Refused rather than clamped: an author who writes 14 400 and silently runs at 900 debugs a timeout that says nothing about what they asked for. `details` carries the node id, the requested value and the ceiling (§4.11) | | `pipeline.validation.pipeline_query_timeout_invalid` | `settings.query_timeout_seconds` (the pipeline-wide SQL statement timeout) is a positive integer no greater than `datapipelines.executor.node-query-timeout-max-seconds` (default 900). `details` carries the requested value and the ceiling (§5.3) | | `pipeline.validation.node_query_timeout_invalid` | A node's `settings.query_timeout_seconds` (this node's own SQL statement timeout) is a positive integer no greater than `datapipelines.executor.node-query-timeout-max-seconds` (default 900); is not declared on a node type that runs no SQL statement (`PIPELINE`, `CALCULATOR`); and does not exceed that node's own effective wall-clock deadline (`settings.timeout_seconds`, else `datapipelines.executor.node-timeout-seconds`) — a statement budget the node's own lifecycle could never reach is never what an author meant. `details` names which check failed, the node id, the requested value, and the number it was compared against (§4.11) | ### 12.9 Composition validations The PIPELINE-node rules (§4.9). Everything here is computed against the exact pinned child bodies. A draft child is mutable, so its verdict describes its current content and release revalidates the graph. Released child bodies are immutable. | Code | Check | |---|---| | `pipeline.validation.pipeline_not_found` | `pipeline.name` exists in the registry. A RESTORE re-judges it against today's registry, under the workspace writer guard (#517, [Versioning §3.5.3](/docs/versioning.md#353-the-reverse-arrows-into-other-families-320)) | | `pipeline.validation.pipeline_version_not_found` | Pinned `version` exists for that name. A RESTORE re-judges it against today's registry, under the workspace writer guard (#517, [Versioning §3.5.3](/docs/versioning.md#353-the-reverse-arrows-into-other-families-320)) | | `pipeline.validation.pipeline_self_reference` | Node does not reference its containing pipeline | | `pipeline.validation.pipeline_reference_deleted` | Referenced pipeline's entity is DISCARDED — every version discarded, derived since V19 (blocks NEW references only — discarding never breaks an existing pinned reference, mirroring template deletion) | | `pipeline.validation.pipeline_reference_not_released` | A PIPELINE node pins a lifecycle-ineligible child: release/import require RELEASED; draft authoring admits DRAFT or RELEASED (#459 supersedes 101/D58 for authoring). **A release that flips the child (#517):** the release's preparation, outside the metadata transaction, validates each flipping pipeline in the AUTHORING mode (a DRAFT child admitted — it flips first); inside the transaction, after its dependencies flipped, the STRICT rule runs again on the same body, bound to the consented body hash and resolved entity UUID; the exact-version CAS releases those checked bytes, so nothing commits RELEASED over a child that is not. A DRAFT child the request does not flip is `pipeline.release.dependency_not_released` before validation ever sees it ([Versioning §5.3](/docs/versioning.md#53-release-lock)). **A restore (#517):** a child pin DISCARDED since the parent was discarded refuses the parent's restore with this code, `status: DISCARDED`; repair is bottom-up ([Versioning §3.5.3](/docs/versioning.md#353-the-reverse-arrows-into-other-families-320)) | | `pipeline.validation.pipeline_node_has_source` | PIPELINE node has no `source` | | `pipeline.validation.pipeline_node_has_template` | PIPELINE node has no `template` | | `pipeline.validation.pipeline_parameter_unmapped` | Every required-without-default child parameter is supplied | | `pipeline.validation.pipeline_parameter_unknown` | Every supplied key names a child declared parameter, a child CALCULATOR `context_key` or a child value-mode SCRIPT `context_key` (ANY at save time, typed by the child's pinned contract at run time) | | `pipeline.validation.pipeline_parameter_type_mismatch` | Literals obey the child target's wire encoding; `${ref}` resolves against the parent's Context tiers — a parent parameter, a parent calculator or scalar SCRIPT `context_key`, an org/platform key — to a value of the identical type (an ANY-output key on either side skips the check, typed only by the run) | | `pipeline.validation.pipeline_parameter_invalid` | A literal that coerces but breaks the child parameter's own declaration — a `constraints` bound, length or pattern, or the child's declared precision/scale (#264; the composition twin of `default_invalid`, judged by the same shared validator the child's binder applies at run, so a value that would fail at execution fails at save — §12.7's D2 promise; a child declaration today's own rules refuse falls back to the type check, its defects being the child's verdict); and a same-type `${ref}` whose parent PARAMETER descriptor does not widen losslessly into the child's (`details.reason: narrowing`, parameter-engine record §6.4 — a calculator, org or platform tier carries no descriptor and stays type-only). `details.reason` names the rule; the value text never travels | | `pipeline.validation.pipeline_output_on_sideeffect_child` | `output` absent when the pinned child has zero caller nodes | | `pipeline.validation.composition_too_deep` | Static reference-tree depth ≤ configured max (`datapipelines.pipelines.max-composition-depth`, default 5). Computed iteratively, never recursively in graph depth — §12.2's crash-safety rule applies here too. | ### 12.10 Calculator-node validations The `CALCULATOR`-node rules (§4.10, calculators design §0.3). Every one of them is decided from the body alone — the registry is a deployment constant and the Context's org and platform tiers are configuration — so an author gets the whole verdict at save time, never at 3am. | Code | Check | |---|---| | `pipeline.validation.calculator_node_incomplete` | A `CALCULATOR` node declares `kind` and `inputs` — a missing key mapping is reported by the shape verdicts below, which name both legal fields | | `pipeline.validation.calculator_fields_on_non_calculator` | No other node type carries `kind`, `inputs`, `context_key` or `context_keys` | | `pipeline.validation.calculator_node_has_sql_fields` | A `CALCULATOR` node carries no `template`, `source` or `output` — it runs no SQL and writes Context keys, not a table | | `pipeline.validation.calculator_unknown` | `kind` names a kind in the registry ([Calculators §2](/docs/calculators.md)) | | `pipeline.validation.calculator_input_missing` | Every input the kind declares `required` is present | | `pipeline.validation.calculator_input_unknown` | Every supplied input name is one the kind declares, and every `$reference` names a Context key something provides — an org or platform key, a declared parameter, or another node's key | | `pipeline.validation.calculator_input_type_mismatch` | Literal inputs obey the kind's declared input type and §6.3's wire encoding, and a `$reference` whose type the body decides (org/platform key, declared parameter, another calculator's output — a multi-output node's key typed by its own output) must match it as well; a `LIST` input takes a JSON array | | `pipeline.validation.calculator_input_unordered` | A `$reference` to another node's key — and a SQL node binding `:that_key` — comes from a node that `depends_on` the producer, directly or transitively. Binding only one of a multi-output node's keys still requires the edge. Sequencing is topology, never array order | | `pipeline.validation.calculator_output_collision` | Every key a node writes collides with nothing: not a declared parameter (a calculator may shadow an org or platform key, never a parameter), and not another node's key — one writer per key per pipeline | | `pipeline.validation.calculator_output_name_invalid` | Every key a node writes — `context_key`, or each `context_keys` value — matches §6.1's `[a-z_][a-z0-9_]*` | | `pipeline.validation.calculator_output_shape_mismatch` | `context_key` XOR `context_keys`, and the field fits the kind: `context_keys` on a single-output kind, `context_key` on a multi-output kind, BOTH fields present, or neither — all refused. `details.reason` names which (`single_output_kind` / `multi_output_kind` / `both_fields` / `neither_field`) | | `pipeline.validation.calculator_output_unknown` | A `context_keys` entry names an output the kind does not declare. `details.known_outputs` lists the names it does | | `pipeline.validation.calculator_outputs_incomplete` | A declared output of a multi-output kind is not mapped in `context_keys` — no partial mapping, so no reader can bind a key the node never writes. `details.missing` lists the unmapped outputs | ### 12.11 MCP-surface entry-point gates (139) Two save-time refusals that live only on the AGENT surface (`pipelines_create` / `pipelines_update` over MCP), beside §12.1's `new_root_requires_confirmation`: each reads something only the agent loop has (the caller's own `mcp.tool.called` audit rows), which is why REST, the UI and every non-MCP caller are unaffected. Both check BEFORE any write; both refuse with the fix named. | Code | Check | |---|---| | `pipeline.validation.table_not_learned` | A saved body's template names a table the CALLING KEY never read the columns of: every `[a-z0-9_]{4,}` token of each pinned template body (the semantics fact check's tokeniser; every `${…}` interpolation span stripped first — a dynamic name is exempt because it is unknowable) that case-fold matches the node's source datasource's catalog listing must appear as a successful `datasources_get_columns` audit row with that `target` + `table` for this key — any time the audit log still holds the row (a key learns a table once per `datapipelines.audit.retention-days`, default a year: the rows age out with the log, [Auth §10.3](/docs/auth.md#103-retention); rows written before 139 carry no `table` and simply do not count). `details.tables` lists each `{datasource, table, clearing_call}`. `tempdb` sources are exempt; a name the catalog does not list is not this check's business (the probe and §12.5 say so); `_get_table_stats` is NOT required — columns are correctness, stats are performance | | `pipeline.validation.door_unacknowledged` | The body's door is a RAW_DATE_PAIR — two or more DATE parameters, no INTEGER parameter named `year`/`quarter`/`month`/`*_year`, and no CALCULATOR node whose kind declares a DATE output (the window writers: `period_start`, `period_end`, `prior_period`, `date_trunc`, `period_bounds`, `trailing_periods`) — and the call does not carry `door_acknowledged: true`. `details.parameters` names the pair; the refusal states rule 13's alternatives (a period parameter, or an anchor date with a window calculator). `NONE` when there are no DATE parameters; `PERIOD` otherwise. The flag is the `confirm_new_root` shape: a decision forced, not a copy | ### 12.12 Release checks The `checks[]` rules (§3.3, 140). One code covers every defect, because to the author they are one category — this check cannot stand as written; `path` (`checks[i].`) names the offender. Run-time outcomes — a statement that returns two columns, a datasource that is down — are NOT validation: they are the `fail` / `error` verdicts of the server's own check run, reported under §13.17. The read-only / single-statement discipline is enforced at RUN time by the bounded probe the runner rides; save time has no SQL grammar to consult, exactly as for node bodies. | Code | Check | |---|---| | `pipeline.validation.check_invalid` | A body carries at most **20** checks. Each check's `id` matches `[a-z0-9_]{1,63}`, is not `tempdb` or the reserved `__…__` namespace, and is unique within the body (the run rows key on it). `name` is 1–200 characters. `datasource` is not `tempdb` and resolves in this environment's registry (the §12.5 rule as for nodes). `sql` is non-blank, carries no `${}` interpolation (a check has no rendering), and every `:name` bind names a DECLARED pipeline parameter — the calculator Context is not available to a check; a cast (`amount::numeric`) is not a bind. `expected.kind` is the closed list `value` \| `range` \| `rows`, with the kind's members present and coherent: `value` needs `value` and `tolerance` ≥ 0; `range` needs `min` ≤ `max`; `rows` needs `rows` ≥ 0 | ### 12.13 Script validation rules The `SCRIPT`-node rules (§4.12, transform-nodes design §3.1, R3–R5). Each is decidable from the body plus the pinned version's contract, so an author gets all of them at save time — the dry-render analogue for scripts, where "the pipeline and the template agree" is proven, not hoped. Run-time refusals are §13.18's family. | Code | Check | |---|---| | `pipeline.validation.script_template_type` | A SCRIPT node's pinned template is `jsonata` or `javascript`. | | `pipeline.validation.script_source_forbidden` | A SCRIPT node carries no `source` — it is tempdb-only. | | `pipeline.validation.script_input_unknown` | An `inputs` entry names a tempdb table no ancestor stages, or a `$key` no tier and no node writes. | | `pipeline.validation.script_input_contract` | The `inputs` key set differs from the contract's input names (`details.missing` / `details.extra`), a table input is bound to a `$key` or vice versa, or a value input's Context type does not fit the contract's declared type. | | `pipeline.validation.script_output_shape` | The `output` block does not fit the contract's mode: a `target` on `value` mode (R3 — a value-mode SCRIPT writes its `context_key` only), a missing `context_key` on `value` mode, or a `row`/`table` mode without `{ "target": "tempdb", "table", "rejects"? }` or `{ "target": "caller" }`. | | `pipeline.validation.script_rejects_undeclared` | `output.rejects` is present but the pinned contract declares no rejects (or the node is not a SCRIPT at all). | | `pipeline.validation.script_rejects_missing` | The pinned contract declares rejects and the node's `output` names no rejects table. | | `pipeline.validation.script_rejects_on_caller` | `rejects` on a `caller` output, or a rejects-declaring contract pinned by a caller node (R5). | | `pipeline.validation.script_strict_without_rejects` | `strict` on a node whose pinned contract declares no rejects (or on a non-SCRIPT node). | | `pipeline.validation.script_object_key_bound` | A SQL `:bind`, a calculator `inputs` `$reference` or a PIPELINE `parameters` `${ref}` names a Context key whose writer's contract output is `kind: object` (R4 — an object key's one reader is another SCRIPT's `inputs`). | `calculator_input_unordered`, `calculator_output_collision` and `calculator_output_name_invalid` (§12.10) apply unchanged to a value-mode SCRIPT's `context_key` and to `$key` bindings whose writer is another node. --- ## 13. Error Code Catalog Error codes follow the format `{domain}.{entity}.{failure}`. Codes are lowercase, dot-separated, ASCII. Codes are **additive** — never reused, never renamed. ### 13.1 Pipeline validation (write-time) (See §12 for the full validation catalog. All validation errors use HTTP 400.) ### 13.2 Pipeline import | Code | HTTP | Description | |---|---|---| | `pipeline.import.missing_datasource` | 400 | Imported pipeline references a datasource name not registered in this env (as `source` or `output.datasource`) | | `pipeline.import.missing_template` | 400 | Imported pipeline references a template version not present in this env | | `pipeline.import.version_conflict` | 409 | Pipeline id+version already exists **with different content** (or the id collides with a soft-deleted pipeline's retained id); a same-hash re-import of an existing released version is an idempotent no-op (preserved-version import rules: [Versioning §9.2](/docs/versioning.md#92-import-with-preserved-versions)) | | `pipeline.import.context_key_missing` | 409 | The imported body binds a Context key this deployment does not provide — an `org_*` key the target's `datapipelines.org.*` block does not define, most often because the body was authored on a deployment with a key this one lacks (Configuration §3.21). Refused rather than silently defaulted: a promoted pipeline reading a made-up currency or fiscal start produces wrong numbers with no error anywhere | | `pipeline.import.hash_mismatch` | 400 | Import payload's declared `body_hash` doesn't match the hash recomputed from its body — transfer corruption or canonicalization drift between app versions ([Versioning §9.2](/docs/versioning.md#92-import-with-preserved-versions)) | ### 13.3 Pipeline execution (run-time) | Code | HTTP | Description | |---|---|---| | `pipeline.execution.not_found` | 404 | Pipeline id or version not found | | `pipeline.execution.parameter_required` | 400 | Required parameter missing from execution request | | `pipeline.execution.invalid_parameter_type` | 400 | Parameter value doesn't match declared type — also a calculator `context_key` supplied at execute time that fails coercion against its kind's output type (§4.10; an ANY-output key accepts any JSON scalar, a container is refused with this same code) | | `pipeline.execution.parameter_constraint_violation` | 400 | A supplied value — or a default being applied — breaks a declared rule of its parameter: a `constraints` bound, length or pattern, the pattern's read budget, or the declared precision/scale (§6.2, #194). Nothing is rounded or clamped. `details.reason` names the rule (`min`, `max`, `min_length`, `max_length`, `pattern`, `pattern_budget`, `scale`, `precision`); same shape as `invalid_parameter_type` otherwise. At a published endpoint it is one entry of `endpoint.request.invalid` | | `pipeline.execution.parameter_declaration_invalid` | 409 | The STORED declaration of a parameter being judged — a supplied value, or a default being applied — is one save refuses today: a `constraints` problem (`constraint_invalid`, `constraint_not_applicable`, `pattern_invalid`) or a `cardinality` other than `SINGLE` (`cardinality_unsupported`). Only a body saved before #194 carries one (`Parameter` ignored unknown keys then, so the block rode in unchecked). `details.reasons` lists those §12.7 save-time codes and the fix is to re-save the pipeline; a broken declaration nothing judges (unsupplied, no default) binds as before (#268) | | `pipeline.execution.calculator_keys_partial` | 400 | The caller supplied a PROPER SUBSET of a multi-output node's keys (121, §4.10). Override is all-or-nothing per node: every key supplied and the node is skipped (`provided_by: "caller"` on its stats), none and it computes; some is refused before any node runs, with the same shape as `invalid_parameter_type`. `details` carries `supplied` and `missing` | | `pipeline.execution.aborted` | 500 | Execution aborted unexpectedly (executor error) | | `pipeline.execution.timeout` | 504 | Execution exceeded timeout | | `pipeline.execution.concurrency_limit` | 429 | Too many concurrent executions for this user | | `pipeline.execution.not_running` | 409 | Cancel requested for an execution that is already terminal | | `pipeline.execution.template_unrendered` | 400 | **MCP-only (139)** — `pipelines_execute`/`pipelines_execute_node` of a DRAFT pipeline version whose pinned template version is itself a DRAFT written after the calling key's last successful `templates_render` of it. `details.templates` carries each `{id, version, updated_at, last_render}`. RELEASED pins are exempt (they cannot change); a `templates_update` makes the next execute refuse again until a render. Reads the caller's own audit rows, so REST and the UI are unaffected (§12.11) | | `pipeline.execution.instance_lost` | — | Recorded (never returned live) by the crash sweep when a RUNNING execution's instance died ([Metadata DB §8](/docs/metadata-db.md#8-operational-jobs)) | ### 13.4 Node execution | Code | HTTP | Description | |---|---|---| | `pipeline.node.template_not_found` | 500 | Template reference resolved at write-time but missing at run-time | | `pipeline.node.template_render_failed` | 500 | Render error (undefined variable, syntax, etc.) | | `pipeline.node.datasource_not_found` | 500 | Datasource name resolved at write-time but missing at run-time | | `pipeline.node.datasource_readonly` | 500 | Datasource resolved at write-time but its **live** registry entry is readonly at run-time: a write-shaped use (a `DML`/`DDL` node `source`, or any node's `output.target: "datasource"`) of a datasource flagged `is_readonly` after this pipeline version was saved — the workspaces D10 flip window. The executor re-checks the live registry entry (past the metadata cache) at node execution time, so the flip fails HERE instead of shipping the write; a PIPELINE node's child nodes pass the same backstop in their own execution | | `pipeline.node.datasource_connection_failed` | 502 | Could not acquire connection to datasource | | `pipeline.node.query_execution_failed` | 502 | SQL executed but failed (syntax, permission, etc.) | | `pipeline.node.query_timeout` | 504 | The node's statement outlived its JDBC query timeout — resolved in precedence order: the node's own `settings.query_timeout_seconds`, else the pipeline's `settings.query_timeout_seconds`, else the datasource's `query_timeout_seconds`, else the dialect's operator default (156, §3.2), else `datapipelines.executor.node-query-timeout-seconds` — and the driver cancelled it. The detail carries `timeout_seconds`, `elapsed_ms` and `source` (`node` / `pipeline` / `datasource` / `dialect` / `application` — which tier resolved the effective value). Sibling of `pipeline.execution.timeout`; distinct from `query_execution_failed` because "too slow for the budget" and "wrong SQL" want different fixes | | `pipeline.node.timeout` | 504 | The node outlived its WALL-CLOCK deadline (`node.settings.timeout_seconds`, else `datapipelines.executor.node-timeout-seconds`) and the executor stopped it — RENDER through MATERIALIZE, staging included. Distinct from `query_timeout`, which is ONE statement's budget enforced by the driver: this one is the executor's own and fires whatever the driver does, so a driver that honours neither `queryTimeout` nor `cancel()` still cannot hold a node past its budget. A statement that has not returned `datapipelines.executor.cancel-grace-seconds` after being cancelled is abandoned and logged once with the execution id; the node fails on schedule. The detail carries `timeout_seconds`, `elapsed_ms` and `phase` — the phase is what says whether to make the query cheaper or the staged result smaller | | `pipeline.node.staging_failed` | 500 | Could not stage ResultSet into tempdb | | `pipeline.node.writeback_failed` | 500 | Could not write ResultSet to external datasource (output.target: "datasource") | | `pipeline.node.writeback_target_missing` | 500 | Target table for write-back doesn't exist (preceding DDL node didn't run, or table not pre-created) | | `pipeline.node.child_execution_failed` | 500 | A PIPELINE node's child execution failed; the detail carries the child's error code and execution id | | `pipeline.node.composition_depth_exceeded` | 500 | Runtime composition-depth backstop hit; indicates a save-time validation gap, since static depth (§12.9) should catch it first | | `pipeline.node.commit_failed` | 502 | A streamed DQL node's finalizing commit was refused (#336): the source connection's autocommit-equivalent net effect (dag-executor §6.4.2) did not happen, so the node's success was a lie and the run fails instead. The detail carries `sql_state` (the driver's, via `FailureShape`); the driver message rides in the envelope's `message` like its `query_execution_failed` sibling. When the node failed for its own reasons the refusal is `addSuppressed` to that failure — the primary stays the result | | `pipeline.node.calculator_failed` | 500 | A `CALCULATOR` node's evaluation failed. The detail carries the node id, the `kind`, the input at fault and the reason — an unknown `unit`, a `format` that does not compile, text that does not match its pattern, a zero denominator. Save-time validation (§12.10) has already refused everything decidable from the body, so this is a value the run itself produced | | `pipeline.node.sql_parameter_missing` | 500 | The rendered SQL references a `:name` bind parameter the execution context does not declare. Raised before anything executes (042: a missing value bound as null would return wrong data instead of an error); the message names the parameter | | `pipeline.node.not_found` | 404 | A node-run debug query ([MCP §6.2.20](/docs/mcp-server.md#6220-pipelines_execute_node)) named a node id the resolved pipeline version does not hold (037 E2). `details` carries the node id and the version searched — after versioning, "no such node" usually means a typo, since the tool's E5 default already prefers the DRAFT body where authoring happens | | `pipeline.node.standalone_execution_refused` | 400 | A node-run debug query refused because the node has no standalone SQL to run (037 §A/E2): its `source` is `tempdb` — the staging database exists only inside a full execution, so use `pipelines_execute` — or it is a PIPELINE node, which runs a child pipeline, not SQL. `details.reason` names which (`tempdb_source` / `pipeline_node`) | ### 13.5 Staging | Code | HTTP | Description | |---|---|---| | `pipeline.staging.value_overflow` | 500 | Source value exceeds staging column capacity | | `pipeline.staging.precision_overflow` | 500 | Source precision exceeds tempdb DECIMAL max | | `pipeline.staging.engine_unavailable` | 500 | Configured staging engine (e.g., DuckDB) not on classpath | | `pipeline.staging.creation_failed` | 500 | Could not create tempdb instance | | `pipeline.staging.cleanup_failed` | 500 | Could not clean up tempdb instance (leaked; logged) | | `pipeline.staging.memory_limit_exceeded` | 500 | Per-execution memory limit hit | | `pipeline.staging.invalid_column_name` | 500 | Source column label fails identifier validation or duplicates another in the same result set ([Staging §4.5](/docs/staging.md#45-identifier-safety-normative)) | | `pipeline.staging.table_already_exists` | 500 | Staged CREATE TABLE targets a name already staged in this execution (defensive; save-time uniqueness is the primary guard) | ### 13.6 Type mapping | Code | HTTP | Description | |---|---|---| | `type_mapping.unknown_source_type` | — | Not an error. Warning returned in response. See [Type System §8.2](/docs/type-system.md#82-unknown--unmappable-types). | | `type_mapping.sql_variant` | — | Not an error. Warning for MSSQL sql_variant fallback. | ### 13.7 Authentication / authorization Defined and described in [Auth §9](/docs/auth.md#9-auth-errors); cataloged here as the single code registry. | Code | HTTP | Description | |---|---|---| | `auth.api_key.missing` | 401 | No credentials provided | | `auth.api_key.invalid` | 401 | API key not recognized, revoked, or the user it acts as is gone. A deactivated member or identity is `auth.principal_deactivated` since 180 | | `auth.api_key.expired` | 401 | API key past expiration | | `auth.session.invalid` | 401 | Session JWT malformed or signature invalid | | `auth.session.expired` | 401 | Session JWT past its expiry | | `auth.permission.undeclared` | 403 | The route or tool called declares no catalog permission, so nothing can judge it and it is refused (#215 slice (b)). `details.route` (`METHOD /path`) or `details.tool` names the surface; a Spring access-denied with no requirement answers it with empty details. A build defect the coverage guards (`RequiredScopeCoverageTest`, `MatrixRowReachabilityTest`) already fail, kept as the runtime defence. Replaced `auth.scope.insufficient`, retired with scopes ([permissions record PK8](https://github.com/msabiransari/datapipelines/blob/main/docs/superpowers/specs/2026-09-23-permissions-and-keys-design.md)) | | `auth.csrf.invalid` | 403 | CSRF token missing or mismatched (browser flows) | | `auth.login.domain_not_allowed` | 403 | Login rejected: email domain not in allowlist (OIDC or local). The SAME code is raised at INVITE time (113, [Auth §4.6](/docs/auth.md#46-invitations)) with a 400 — an invitation that could never be honoured at login is refused when it is created; the row's status is the login refusal's, which is a redirect rather than an envelope | | `auth.login.user_inactive` | 403 | Login rejected: account deactivated (OIDC or local) | | `auth.login.bad_credentials` | 401 | Local login rejected: email unknown or password incorrect — deliberately identical ([Auth §5A.5](/docs/auth.md#5a5-enumeration-resistance-and-the-password-policy)) | | `auth.login.locked` | 403 | Local login rejected: account locked after consecutive failures ([Auth §5A.3](/docs/auth.md#5a3-lockout)) | | `auth.password.change_required` | 403 | Session principal must change password before any other operation ([Auth §5A.4](/docs/auth.md#5a4-forced-password-change)) | | `auth.session.required` | 403 | An API-key principal reached a credential-minting operation (admin local-account create, password reset, `disable-local`, `unlock`, self-service password change); a key that mints an interactive credential escalates itself into an unpinned session that outlives its own revocation ([Auth §5A.7](/docs/auth.md#5a7-credential-minting-is-session-only)) | | `auth.role_required` | 403 | The principal's ROLE in the active workspace does not hold the permission the action needs ([RBAC design §2](https://github.com/msabiransari/datapipelines/blob/main/docs/superpowers/specs/2026-09-10-rbac-design.md); the catalog, [Auth §7.6](/docs/auth.md#76-operation-matrix--the-permission-catalog-authoritative)). `details.required` names the catalog permission (`pipeline.update`), `details.held` the role the principal was judged as (`viewer` … `workspace_admin`, `super_admin`; an MCP key's OWN role chosen at creation; a key's own `api_caller` / `promotion_receiver`) — informative only (#215). THE one authorization refusal for every principal since scopes were removed (#215 slice (b)): the MCP ownership rules ride it too — `templates_purge_draft` on another person's draft (`details.reason = not_creator`), `executions_cancel` on a run another credential started (`started_outside_mcp` \| `different_credential`) ([Auth §11A](/docs/auth.md#11a-roles)) | | `auth.key_issuer_role_lost` | 403 | A key principal judged through a workspace CONTEXT (the promotion config-value peer) lacks the permission ([Auth §9](/docs/auth.md#9-auth-errors); kept by the permissions record §4 as a code that still means something). Since keys v2 (A13) every MCP key carries its OWN role chosen at creation and refuses with `auth.role_required` — `details.held` names that role — so this code never answers an MCP call, and the retired cap/freshness reading (PK4, C2) is gone ([Auth §7.5](/docs/auth.md#75-key-roles)). Retrying with the same credential will not help | | `auth.key_kind_not_mintable` | 400 | Keys v2 (§7.4): issuance named NO kind — there is no default kind since the login mint retired (A15); name `mcp`, `endpoint` or `server`. A request naming the retired `user` wire word is an unknown kind (`endpoint.key_kind_refused`) | | `auth.key_name_taken` | 409 | Keys v2 A18: the create path named a key that already EXISTS (live) in the workspace — the A18 unique index speaking through the service's clean check ahead of it, so the answer is a catalogued conflict instead of the constraint's raw failure. `details.reason` is `key_name_taken`, `details.field` is `name`, `details.value` the offending name truncated ([Auth §7.4](/docs/auth.md#74-issuance)). Revoking the existing key frees the name | | `auth.key_workspace_inactive` | 404 | The workspace this key is pinned to has been deactivated (D-R10). The same 404 a member gets — the owner ruled (2026-09-14) that a deactivated workspace answers not-found on every surface, keys included, so deactivation never becomes a signal. The code stays distinct from `workspace.not_found` because the holder is a member of that workspace by construction (keys are issued by members and pinned there), so it reveals nothing the holder did not already know — and it is what an operator greps the audit log and this catalogue for. Reactivating the workspace restores the key | | `auth.principal_deactivated` | 401 | The principal's USER is deactivated (`users.is_active = false`) — a session's or MCP key's member, or an `endpoint` / `server` key's own identity (#215; revoking such a key deactivates its identity) (180, [roles design D15/§3.5](https://github.com/msabiransari/datapipelines/blob/main/docs/superpowers/specs/2026-09-20-roles-permissions-design.md)). Judged by one predicate (`PrincipalLiveness`) where each credential becomes a principal, through the same ~60 s liveness cache as revocation, so deactivation kills every credential within one TTL and reactivation restores them; a session additionally has its cookie cleared and an HTML navigation is sent to `/login?error=inactive`. The promotion peer never answers this code — every refusal there is `auth.promotion.key_invalid`. A deactivated WORKSPACE is not this code: it keeps the 404 rule (`auth.key_workspace_inactive`, `workspace.not_found`, an unknown endpoint) | | `auth.api_key.expiry_invalid` | 400 | Key issuance named an expiry this surface cannot use: an unknown preset, a custom date that does not parse, a custom date already in the past, or `custom` with no date. `details.reason` is `unknown_preset` \| `date_unparseable` \| `date_in_past` \| `date_missing`, and `details.value` echoes the offending value truncated. A 400 rather than a 401/403: the credential is fine, the request body is not ([Auth §7.4](/docs/auth.md#74-issuance)) | | `auth.promotion.key_invalid` | 401 | The promotion peer credential was absent, malformed, or did not match the receiver's configured promotion server key. The SAME code answers a receiver that has no key configured at all — promotion is disabled there and fail-closed, and one code keeps the response from telling a wrong key apart from a disabled receiver ([Versioning §10.6](/docs/versioning.md#106-the-promotion-peer-credential--a-shared-server-key-ratified-2026-09-01)) | ### 13.8 Datasource Defined and described in [Datasources §9–10](/docs/datasources.md#9-validation-rules). | Code | HTTP | Description | |---|---|---| | `datasource.validation.name_invalid` | 400 | `name` fails the identifier rules | | `datasource.validation.dialect_invalid` | 400 | `dialect` not in the supported set | | `datasource.validation.jdbc_url_malformed` | 400 | URL fails the dialect adapter's parse | | `datasource.validation.jdbc_url_scheme_invalid` | 400 | URL scheme doesn't match the dialect | | `datasource.validation.password_missing` | 400 | `password` required on create | | `datasource.validation.properties_invalid` | 400 | Test pool build rejected a `hikari`/`jdbc` property | | `datasource.validation.property_empty` | 400 | A DECLARED dialect property carries an empty, whitespace-only or null value — refused at register/update (and bootstrap) rather than stored as `""` (109 §B); the field names the key | | `datasource.validation.query_timeout_invalid` | 400 | `query_timeout_seconds` present but < 1 | | `datasource.validation.duplicate_name` | 409 | Name already exists — the namespace is global across workspaces too (workspaces design §3: `name` stays the PK/AAD anchor) | | `datasource.validation.workspace_forbidden` | 400 | Workspaces D8 refusal: non-admin attempted the `global` flag (or any mutation of a global datasource), a `readonly` flip on a global datasource, or a workspace binding the caller is not in — including member CUD while `member-datasources-enabled` is off (workspaces design §8) | | `datasource.in_use` | 409 | Delete blocked: pipelines reference this datasource | | `datasource.not_found` | 404 | Datasource name unknown on a read/mutate/test path (added 2026-08-11, gate C) | | `datasource.grant_required` | 404 | The datasource exists on the instance but is not GRANTED to the caller's workspace ([RBAC design §4](https://github.com/msabiransari/datapipelines/blob/main/docs/superpowers/specs/2026-09-10-rbac-design.md), D-R7). A 404 and not a 403 because the 404 rule has no exception for datasources: an ungranted datasource is INVISIBLE, and "this exists, you may not see it" turns the flat, global datasource namespace into an enumeration oracle — every name on the instance probeable one request at a time | | `datasource.driver_not_loaded` | 400 | JDBC driver JAR for the dialect is not on the classpath | | `datasource.lease_in_transaction` | 500 | A customer-database connection was requested while a metadata transaction was open on the thread — refused by design (one transaction, one database; see [dag-executor §16](/docs/dag-executor.md#16-consistency-model)) | | `datasource.validation.lake_dialect_required` | 400 | A lake-table operation (register / import / unregister / list) was attempted on a datasource whose dialect is not `LAKE` (089 §A) | | `datasource.validation.lake_namespace_invalid` | 400 | A lake table's `namespace` fails the segment grammar (the pipeline/template §4.1 segment, minus `.`, 1–9 segments) | | `datasource.validation.lake_name_invalid` | 400 | A lake table's `name` fails the segment grammar (the pipeline/template §4.1 segment, minus `.`) | | `datasource.validation.lake_format_invalid` | 400 | A lake table's `format` is not `parquet` or `iceberg` (metadata-db §4.15's closed set) | | `datasource.validation.lake_location_invalid` | 400 | A lake table's `location` is not an `s3://` or `file://` URI, or carries a character the SQL-injection boundary refuses (quote, backslash, control character, whitespace) | | `datasource.validation.lake_manifest_url_forbidden` | 400 | A lake-table import's manifest URL is not under the datasource's own endpoint/bucket — no arbitrary URL fetch (SSRF boundary, 089 §A) | | `datasource.lake_table_duplicate` | 409 | The (datasource, namespace, name) triple is already registered (mapped from `uq_lake_tables_datasource_namespace_name`, metadata-db §4.15) | | `datasource.lake_table_not_found` | 404 | Unregister named a lake table that is not registered — a REGISTRY fact ("not registered"), deliberately distinct from `datasource.table_not_found` ("not in the database's catalog"): the lake's registry, not the introspector, answers for its tables (123 §A) | | `datasource.table_not_found` | 404 | A table-addressed read (columns / table stats / preview rows) named a table the namespace's catalog listing does not contain (123 §A, datasources.md §7A). Where the dialect's catalog is complete the table does not exist; where it is privilege-filtered the table may only be invisible to the datasource's credentials — the message says which, and names the nearest listed table when one is close | | `datasource.table_forbidden` | 403 | The table IS in the catalog listing but the read failed with a permission SQLSTATE — the datasource's credentials cannot read it (123 §A). A 403, not the 404 `grant_required` argues for: a table on a GRANTED datasource is not invisible (the listing already told the caller it exists), so saying so leaks nothing | | `datasource.validation.lake_table_unreadable` | 400 | A lake-table registration/import named a table the pre-flight could not read — the view did not create or a one-row scan through it failed (109 §A, datasources.md §8C.1). Refused before storing; the message carries the bounded engine error | | `datasource.validation.lake_no_healthy_tables` | 400 | A LAKE pool build whose EVERY registered table was refused at the SQL-emission boundary (158, #129, datasources.md §8C.2): nothing would be queryable, so the build is refused instead of serving an empty pool. The message names every refused table and its reason; each refusal is also recorded on its registry row. The fix is the registered content — repair or unregister the named tables | | `datasource.lake.table_unavailable` | 502 | A pipeline node referenced a registered lake table whose connect-time view creation is recorded as failed (`lake_tables.last_error`, V20) — the table's view is skipped on every connection (109 §A, datasources.md §8C.2). `details` carry `table` and the recorded `last_error` | | `pipeline.execution.datasource_unreachable` | 502 | Pre-execution reachability check failed for a referenced datasource | ### 13.9 Template Defined and described in [Templates §7](/docs/templates.md#7-validation-rules). | Code | HTTP | Description | |---|---|---| | `template.validation.syntax_error` | 400 | Freemarker parse failure | | `template.validation.dangerous_construct` | 400 | Forbidden Freemarker construct (SSTI hardening, Templates §4.2) | | `template.validation.id_invalid` | 400 | Template `id` fails the identifier rules | | `template.validation.new_root_requires_confirmation` | 400 | **The AGENT surface only** (094): `templates_create` refuses an `id` whose ROOT segment has no templates under it yet, unless the call carries `confirm_new_root: true`. Same `details.root` / `details.existing_roots` shape, same `test/` exemption and same REST/UI exemption as `pipeline.validation.new_root_requires_confirmation` | | `template.validation.dialect_invalid` | 400 | `dialect` not in the supported enum — or absent on a `sql` template (a dialect is required on `sql` and forbidden on every other type; 046, extended in 7b) | | `template.validation.type_invalid` | 400 | `type` not one of `sql`, `html`, `jsonata`, `javascript` (046, template-hierarchy-design §5.4; the script types since 7b) | | `template.validation.dialect_not_allowed` | 400 | `dialect` is present on a template whose type declares none — `html`, `jsonata` or `javascript` (046, template-hierarchy-design §7; extended to the script types in 7b). Deliberately distinct from `dialect_invalid`: presence on the wrong type and an unknown value are different failures | | `template.validation.type_immutable` | 400 | A payload attempted to change a template's `type`, which is chosen at create and identical on every version (046, template-hierarchy-design §5.3) | | `template.validation.html_entity` | 400 | The body contains an HTML entity where a SQL operator belongs (`<`, `>`, `&`, `"`, `'`) — the body was HTML-escaped between the author and the server (an agent client, a copy from a rendered page). Refused at save with `details.entity` and the first line it appears on, because at execution it surfaces as an opaque driver syntax error far from its cause (2026-09-11). | | `template.validation.engine_unsupported` | 400 | `engine` does not match the template's type — `freemarker` iff `sql`/`html`, `none` iff `jsonata`/`javascript` (type-conditional since 7b, transform-nodes design §2.1) | | `template.validation.freemarker_forbidden` | 400 | A script-typed template carried `imports` or `is_library: true`, or its body contains a Freemarker construct (`${`, `<#`, `<@`) — a script body is evaluated, never rendered (7b, D-T9); `details.rule` names which (`imports` / `is_library` / `body`) | | `template.validation.schema_version_unsupported` | 400 | `schema_version` not a value v1 supports (only `1`) | | `template.validation.is_library_without_macros` | 400 | `is_library: true` but body has no macro definitions or has output outside them | | `template.validation.import_not_found` | 400 | `imports` entry references a missing template id/version | | `template.validation.import_not_library` | 400 | `imports` entry references a template with `is_library: false` | | `template.validation.import_cycle` | 400 | Import graph contains a cycle | | `template.validation.import_depth_exceeded` | 400 | Transitive import depth > 10 | | `template.validation.duplicate_alias` | 400 | Two `imports` entries share an alias | | `template.validation.parameter_interpolated` | 400 | A declared pipeline parameter name appears inside a `${}` interpolation: declared parameters are values and must be referenced as `:name`, bound as SQL parameters (042; Templates §7.2). The declared set includes every calculator output key (078 A1), which is refused in a conditional's test (`<#if x??>`, `<#elseif x>`) too. The message names the parameter and shows the `:name` form | | `template.validation.duplicate_name` | 409 | Template name already exists in this workspace — `UNIQUE(workspace_id, name)`, soft-deleted included (the `pipeline.validation.duplicate_name` shape, for templates; added 2026-08-28, T23) | | `template.not_found` | 404 | Template id (or id+version) unknown — or soft-deleted on a read/mutate path (added 2026-08-11, gate C) | | `template.in_use` | 409 | A template's removal refused — the discard, the draft purge, the version purge or the entity purge — while a version of a pipeline, a parameter set or (since #320) a visualization pins it (the exact version for a discard or a draft purge that keeps other versions, LIVE dependents only; ANY version, discarded included, for an entity purge and a draft purge that takes the entity — [Versioning §3.5.3](/docs/versioning.md#353-the-reverse-arrows-into-other-families-320)). `details`: `template_id`; `pinned_by` the pinning pipeline names (always present, possibly empty); `referencing_parameter_sets` and `referencing_visualizations` the set and visualization names, each present only when non-empty. The MCP `templates_purge_draft` narrows the echo to the pins the caller's view admits and sets `pins_hidden: true` when others exist. Deleting is otherwise soft and existing pins keep resolving — the refusal makes "who still uses this?" unmissable, it does not change delete semantics | | `template.version.conflict` | 409 | Content-hash precondition failed on a template draft write/release/discard — another writer changed it after the caller loaded it ([Versioning §4](/docs/versioning.md#4-content-hash-body_hash)) | | `template.version.not_draft` | 409 | Template release or discard requested but no DRAFT version exists ([Versioning §3](/docs/versioning.md#3-version-lifecycle)) | | `template.version.not_released` | 409 | Discard targeted a template version that is not RELEASED — a DRAFT is purged, never discarded (101, [Versioning §3.1](/docs/versioning.md#31-statuses-and-verbs)) | | `template.version.not_discarded` | 409 | Restore targeted a template version that is not DISCARDED (101, [Versioning §3.1](/docs/versioning.md#31-statuses-and-verbs)) | | `template.version.last_release` | 409 | The purge path refused: a RELEASED template version is never purged, and a template holding any non-draft version is never purged (101, [Versioning §3.5](/docs/versioning.md#35-the-lifecycle-table) D57) | | `template.version.not_eligible` | 409 | Manual switch targeted a template version that is not live and posture-eligible (101, [Versioning §3.4](/docs/versioning.md#34-current_version-is-sticky-and-event-driven-d60)) | | `template.version.confirm_mismatch` | 400 | The typed-confirm guard on the irreversible template purge dialogs (102): the `confirm` form field did not name what the dialog asked the user to type — the version (`v4`) on a version purge, the template's name on the entity purge. Checked BEFORE the lifecycle service runs ([UI §5.1](/docs/ui-screens.md#51-standard-states) typed-confirm) | | `template.authoring.disabled` | 403 | An authoring write (create, update/draft, release, discard, delete) on a server with `datapipelines.deployment.authoring-enabled=false` — the template mirror of `pipeline.authoring.disabled` ([Versioning §5.5](/docs/versioning.md#55-drafts-are-a-deployment-capability-039)); reads, execution and import are unaffected | | `template.contract_invalid` | 400 | A script template's `contract` block failed a §2.2 rule (transform-nodes design); `details.rule` names which (`row_mode_inputs`, `type_unsupported`, `mode_output_mismatch`, `empty_case_missing`, `row_case_lists_table`, `unknown_field`, `wrong_type`, …) (7b) | | `template.invariant_invalid` | 400 | A script template's invariant does not compile as JSONata, or produced a non-boolean on a test case; `details` names the invariant (and the case) (7b) | | `template.test_failed` | 400 | A script template's test case failed at save or at release (the suite runs on both — §8.1); `details` names the case, the assertion, and a diff bounded to 2 000 chars (7b) | | `template.blocks_not_allowed` | 400 | `contract` / `invariants` / `tests` on an `sql` / `html` template — the blocks belong to the script types only (7b); `implements` likewise (7e, transform-nodes design §2.3) | | `template.render_not_applicable` | 400 | `templates_render` / `POST /api/v1/templates/render` on a script type — there is nothing to render; `details.use` points at `templates_evaluate` (7b) | | `template.implements_unresolved` | 400 | An `implements` entry on a template write does not resolve to a fact the writing workspace may cite — absent, not visible from the workspace, a DATASOURCE fact, or a WORKSPACE fact of a kind other than `definition` / `exclusion` / `preference` ([Templates §3.4](/docs/templates.md#34-implements-and-drift)). One answer for every case — not-found semantics, never "it exists but you cannot see it"; `details.fact_id` names the first offending entry. Refused before anything is written (7e) | | `template.release.dependency_not_released` | 409 | Reserved — raised once the template release participant lands (#518). Template release blocked: a template version its imports pin is not RELEASED and this request does not release it. `details` is the shared release-blockers shape `{root, blockers:[{node_id, family, name, version, status, reason, paths}], blockers_truncated}` ([REST API §5.10](/docs/rest-api.md#510-release-pipeline)), `reason` one of `not_included`, `missing`, `discarded`, `hidden`, `cycle`, `limit`, `not_released`; at most `datapipelines.release.dependencies.max-blockers` rows, the rest set `blockers_truncated`; the code carries the ROOT's family and the blocker's family is `blockers[].family` (#517) | | `template.release.plan_conflict` | 409 | Reserved — raised once the template release participant lands (#518). A consented snapshot fact changed between the dependency preview and the submit — an identity, status, body hash, current pointer, pin path or edge no longer matches the submitted `tree_hash`, even when someone else released the node meanwhile. `details` `{reason:"snapshot_changed"}` plus `node_id`/`path` only when visible; nothing flipped; reopen the preview. A root hash mismatch keeps `template.version.conflict` ([Versioning §5.3](/docs/versioning.md#53-release-lock), #517) | | `template.release.plan_required` | 428 | Reserved — raised once the template release participant lands (#518). Cascade consent without a preview `tree_hash` — dependencies selected, or a legacy alias combined with a plan field, with no digest to compare. No dependency names in `details`; obtain a fresh dependency preview ([Versioning §5.3](/docs/versioning.md#53-release-lock), #517) | | `template.release.plan_invalid` | 400 | Reserved — raised once the template release participant lands (#518). The plan is malformed: a `tree_hash` that is not a digest, unknown or duplicate node ids, the root or a RELEASED node selected, an unsupported selection, or a legacy alias set to true together with explicit `include_dependencies` (two authorities). `details.field` is a bounded locator; no submitted value is reflected (#517) | | `template.release.dependency_cycle` | 409 | Reserved — raised once the template release participant lands (#518). The release closure repeats an exact node on its active path; `details.path` is the visible bounded node trail of that one cycle, never every expanded path (#517) | | `template.release.dependency_limit` | 422 | Reserved — raised once the template release participant lands (#518). The complete closure exceeds a configured bound ([Configuration §3.35](/docs/configuration.md#335-release-dependencies)); `details` `{limit, maximum}` with `limit` one of `depth`, `versions`, `edges`, `wire_bytes` — never a hidden total. Enforced while the closure is enumerated, so a truncated closure never authorises a release (#517) | | `template.release.busy` | 409 | The workspace mutation guard's lock wait (`datapipelines.release.dependencies.lock-wait-seconds`) ran out — another pin-affecting template write or release in the same workspace holds it. Nothing was written; retry with a current plan (#517) | | `template.release.bundle_unsupported` | 409 | Reserved — raised once promotion carries full closures (#523). The promotion peer lacks the full-closure envelope capability; no bodies were transmitted. The family is the promoted root's (#517) | ### 13.10 Result retrieval Defined and described in [REST API §7](/docs/rest-api.md#7-result-delivery). | Code | HTTP | Description | |---|---|---| | `result.execution_not_found` | 404 | Execution ID unknown | | `result.execution_incomplete` | 409 | Execution not yet completed | | `result.execution_failed` | 410 | Execution failed; no result exists | | `result.expired` | 410 | Result TTL elapsed; re-run the pipeline | | `result.format_unsupported` | 400 | Unknown `format` parameter | | `result.too_large` | 500 | Caller result exceeded `datapipelines.result.max-size-bytes`; execution failed | | `result.storage_unavailable` | 503 | Redis unavailable. At result-write time the execution fails (the code in its `pipeline_failed`); on a read — [REST API §10.3](/docs/rest-api.md#103-replay-sse-stream)'s replay log, `details.reason: event_log_unavailable` — the request is refused and may be retried. Was 500 until v1.56 (#487) | ### 13.11 Rate limiting / idempotency | Code | HTTP | Description | |---|---|---| | `rate_limit.exceeded` | 429 | Per-user rate limit hit (single code for all layers — REST, MCP, login). The catalog's default `user_message` is the API sentence — "You're sending requests faster than we allow. Wait a moment and try again." — and every API surface answers with it (REST, `/mcp`, the endpoint-key budget, `ApiErrors.rateLimitExceeded`). The LOGIN damper (`LoginRateLimitFilter`, auth.md §9) is the one override: "Too many sign-in attempts. Wait a minute and try again." (#232) | | `rate_limit.unavailable` | 429 | The limiter could not decide (Redis unreachable); the request is refused, not admitted — the limiter fails **closed** | | `idempotency.key_reused_for_different_request` | 409 | Same `Idempotency-Key` submitted with a different request body | ### 13.12 Workspace resolution Raised by the workspace resolution layer ([Auth §5](/docs/auth.md#5-oidc-login-flow)) — the per-request `DP-Workspace` switch and API-key pinning — and by the workspace CRUD and membership surfaces. **The 404 rule governs (RBAC design D-R5).** A workspace the caller cannot reach — unknown name, not a member, or deactivated — is ONE answer: `workspace.not_found`, the same status and body a genuinely missing row produces. Never a 403. `workspace.membership_required` survives in exactly one place, where no workspace was ADDRESSED at all (a principal with zero memberships), because there is no name there whose existence a 403 could leak. | Code | HTTP | Description | |---|---|---| | `workspace.membership_required` | 403 | The principal has no active workspace at all (zero memberships). Not used for an addressed workspace — that is `workspace.not_found` | | `workspace.header_forbidden` | 400 | `DP-Workspace` sent on an API-key request; a key's workspace is pinned at issuance and cannot switch | | `workspace.session_required` | 403 | An API-key principal reached a session-only workspace action (the UI's create/members/delete/switch); the key's workspace is pinned at issuance and `switch` mints a session, so the credential class is refused outright | | `workspace.not_found` | 404 | The addressed workspace does not exist FOR THIS CALLER: unknown name, non-member, or deactivated — one answer for all three (D-R5), so nothing about a workspace is probeable. Also refuses a promotion batch naming a workspace the receiver does not have (O-4); nothing is auto-created | | `workspace.last_admin` | 409 | The membership change would leave the workspace with no admin. Its own code rather than `workspace.in_use`: the caller's next step is "give someone else the admin role first", which is a different instruction from "empty the workspace first" | | `workspace.self_membership` | 409 | The caller addressed their OWN membership — changing their role, revoking their login-minted key or removing themselves (#208, owner ruling 2026-09-22). Its own code rather than `workspace.last_admin`: the remedy is "ask another workspace admin", not "promote somebody first" | | `workspace.inactive` | 404 | A super admin addressed a deactivated workspace on a path that must refuse it (D-R10). Members never see this code — for them a deactivated workspace is `workspace.not_found`, because deactivation must not become a signal — and API keys get `auth.key_workspace_inactive` | | `workspace.invitation.not_found` | 404 | The INVITATION addressed does not exist (113): revoked already, never created, or created for another workspace. Its own code rather than `workspace.not_found` because the workspace itself resolved fine — the not-found thing is the invitation — and one answer for all three causes keeps an admin from probing which emails hold pending invitations off the status code ([Auth §4.6](/docs/auth.md#46-invitations)) | | `workspace.validation.name_invalid` | 400 | Workspace name fails `[a-z0-9_-]+`, 1–63 | | `workspace.validation.duplicate_name` | 409 | Workspace name exists (global namespace, soft-deleted included — house rule) | | `workspace.in_use` | 409 | Delete blocked: workspace still owns non-deleted pipelines/templates/datasources. Deactivation (D-R10) is the operation that needs no such check — it purges nothing | ### 13.13 Versioning / draft-release lifecycle / promotion Defined and described in [Versioning](/docs/versioning.md). Codes are additive; the draft/release lifecycle and the preserved-version import rules live there (this table is the catalog entry, Versioning is the semantics). | Code | HTTP | Description | |---|---|---| | `pipeline.version.conflict` | 409 | Content-hash precondition failed on a draft create/write, release, or discard — another writer changed the pipeline after the caller loaded it; `details` carries `current_body_hash`, `current_status`, `updated_by`, `updated_at` (Versioning §4.2) | | `pipeline.version.not_draft` | 409 | Pipeline release or discard requested but no DRAFT version exists (Versioning §5.3/§5.4) | | `pipeline.version.not_released` | 409 | Discard targeted a version that is not RELEASED — a DRAFT is purged, never discarded; a DISCARDED version is already retired (101, [Versioning §3.1](/docs/versioning.md#31-statuses-and-verbs)) | | `pipeline.version.not_discarded` | 409 | Restore targeted a version that is not DISCARDED — there is nothing to restore (101, [Versioning §3.1](/docs/versioning.md#31-statuses-and-verbs)) | | `pipeline.version.pinned` | 409 | Discard or purge refused: a LIVE version of another pipeline — or, since #320, of a dashboard whose source pins it — exact-pins this version. A sole-draft purge uses live/exact parent pins plus stored any-version dashboard pins (DISCARDED included for a dashboard with a live version), before any destructive write (#462). `details` names the pinning entities: `pinned_by` the pipelines (`{pipeline, version, node}`), `referencing_dashboards` the dashboards (`{dashboard, version, status}`), each present only when non-empty besides `pinned_by`, which is always present (101, [Versioning §3.5](/docs/versioning.md#35-the-lifecycle-table) graph rule 1, [§3.5.3](/docs/versioning.md#353-the-reverse-arrows-into-other-families-320)) | | `pipeline.version.last_release` | 409 | The purge path refused: a RELEASED version is never purged, and an entity holding any non-draft version is never purged — discard is per version, the entity stays; restore or release something first (101, [Versioning §3.5](/docs/versioning.md#35-the-lifecycle-table) D57) | | `pipeline.version.not_eligible` | 409 | Manual switch targeted a version that is not a live, posture-eligible version — DISCARDED, missing, or a DRAFT under a hardened posture (101, [Versioning §3.4](/docs/versioning.md#34-current_version-is-sticky-and-event-driven-d60) D60) | | `pipeline.version.confirm_mismatch` | 400 | The typed-confirm guard on the irreversible purge dialogs (102): the `confirm` form field did not name the version the dialog asked the user to type (`v4`). Checked BEFORE the lifecycle service runs — a mismatch never reaches the verb ([UI §5.1](/docs/ui-screens.md#51-standard-states) typed-confirm, the second use after the CLI's `--clean`) | | `pipeline.release.dependency_not_released` | 409 | Pipeline release blocked: a dependency in the release closure (a pinned template version or child pipeline version) is not RELEASED and this request does not release it — a visible DRAFT the plan did not include, a MISSING or a DISCARDED pin. `details` is the shared release-blockers shape `{root, blockers:[{node_id, family, name, version, status, reason, paths}], blockers_truncated}` ([REST API §5.10](/docs/rest-api.md#510-release-pipeline)), `reason` one of `not_included`, `missing`, `discarded`, `hidden`, `cycle`, `limit`, `not_released`; at most `datapipelines.release.dependencies.max-blockers` rows, the rest set `blockers_truncated`; the code carries the ROOT's family and the blocker's family is `blockers[].family`. **A restore (#517)** whose stored body pins a present-but-DRAFT template or child version refuses this code in the same shape, `reason: not_released`, before anything is written (defensive: no save rule lets a once-released body pin a draft). A bare legacy `release_pinned_templates=true` still cascades the direct template pins until the dialog posts a plan (#522); a DISCARDED or MISSING pin is never releasable ([Versioning §5.3](/docs/versioning.md#53-release-lock), #517) | | `pipeline.release.plan_conflict` | 409 | A consented snapshot fact changed between the dependency preview and the submit — an identity, status, body hash, current pointer, pin path or edge no longer matches the submitted `tree_hash`, even when someone else released the node meanwhile. `details` `{reason:"snapshot_changed"}` plus `node_id`/`path` only when visible; nothing flipped; reopen the preview. A root hash mismatch keeps `pipeline.version.conflict` ([Versioning §5.3](/docs/versioning.md#53-release-lock), #517) | | `pipeline.release.plan_required` | 428 | Cascade consent without a preview `tree_hash` — dependencies selected, or a legacy alias combined with a plan field, with no digest to compare. No dependency names in `details`; obtain a fresh dependency preview ([Versioning §5.3](/docs/versioning.md#53-release-lock), #517) | | `pipeline.release.plan_invalid` | 400 | The plan is malformed: a `tree_hash` that is not a digest, unknown or duplicate node ids, the root or a RELEASED node selected, an unsupported selection, or a legacy alias set to true together with explicit `include_dependencies` (two authorities). `details.field` is a bounded locator; no submitted value is reflected (#517) | | `pipeline.release.dependency_cycle` | 409 | The release closure repeats an exact node on its active path; `details.path` is the visible bounded node trail of that one cycle, never every expanded path (#517) | | `pipeline.release.dependency_limit` | 422 | The complete closure exceeds a configured bound ([Configuration §3.35](/docs/configuration.md#335-release-dependencies)); `details` `{limit, maximum}` with `limit` one of `depth`, `versions`, `edges`, `wire_bytes` — never a hidden total. Enforced while the closure is enumerated, so a truncated closure never authorises a release (#517) | | `pipeline.release.busy` | 409 | The workspace mutation guard's lock wait (`datapipelines.release.dependencies.lock-wait-seconds`) ran out — another pin-affecting pipeline write or release in the same workspace holds it. Nothing was written; retry with a current plan (#517) | | `pipeline.release.bundle_unsupported` | 409 | Reserved — raised once promotion carries full closures (#523). The promotion peer lacks the full-closure envelope capability; no bodies were transmitted. The family is the promoted root's (#517) | | `pipeline.release.template_needs_review` | — | Not an error — a WARNING in the release response's `warnings` array (§14) and on the release dialog's pin row: the released pipeline pins a template version that reads `needs_review` because it cites a retired learned fact ([Templates §3.4](/docs/templates.md#34-implements-and-drift)). The release proceeds — a fact edit never blocks a release on its own; one warning per such pin, its `message` naming each retired fact and, when superseded, its successor (7e) | | `pipeline.promotion.not_released` | 409 | Promotion selected a pipeline whose candidate version is not RELEASED — drafts are never promoted (Versioning §10.3 guard 1) | | `pipeline.promotion.not_newer` | 409 | Promotion push of a version not greater than the target environment's current version for that pipeline — same-version pushes are a bug, not a no-op (Versioning §10.3 guard 2) | | `pipeline.promotion.missing_datasources` | 409 | Promotion pre-validation (Versioning §10.5): one or more datasource names the batch references do not exist on the target. Reported ONCE for the whole batch, `details.missing_datasources` naming every absent one, before anything is pushed — the target is left byte-unchanged rather than failing mid-batch | | `pipeline.promotion.target_is_authoring` | 409 | Promotion into a target whose `authoring-enabled` is true — dev is where drafts live, and a receiver never authors (Versioning D7/§10.1). Raised by the receiver, so a misconfigured sender cannot push into an authoring deployment | | `pipeline.promotion.target_unreachable` | 502 | The promotion target did not answer, timed out, or answered something that is not this API — a transport failure on the SENDER, never a refusal by the receiver (a receiver's refusal is re-raised with the receiver's OWN code). `details` carries `target` and, when there was one, `target_status`. Logged WARN without a stack: the operator's own peer deployment being down is not a defect in this one | | `pipeline.authoring.disabled` | 403 | An authoring write (create, update/draft, release, discard, delete) on a server with `datapipelines.deployment.authoring-enabled=false` — a promotion receiver never authors (Versioning D7/§5.5). Reads, execution and import are unaffected. The template mirror is `template.authoring.disabled` (§13.9) | ### 13.14 Published endpoints The contract and the complete status table are [REST API §19](/docs/rest-api.md#19-published-endpoints); the key kinds and the hierarchical bindings are [Auth §7.7](/docs/auth.md#77-key-kinds-and-published-endpoint-bindings). This table is the catalog entry — it lands with the constants (a catalogued code split from its constant leaves the build red between the two), and the two sections above land with the surfaces that raise them. The bare `endpoint.*` codes are two-segment: the domain has no entity dimension, the same shape as `datasource.in_use` and `template.not_found`. The `endpoint.request.*` family is the request validator's, and it is the only family a caller of a published endpoint can provoke by changing their query string — every one of its codes is reported inside one `endpoint.request.invalid` response whose `details.errors[]` names every defect at once. | Code | HTTP | Description | |---|---|---| | `endpoint.path_invalid` | 400 | `path_pattern` is not a legal path ([REST API §19.1](/docs/rest-api.md#191-the-path-grammar)): wrong segment alphabet, fewer than three or more than 10 segments, over 200 characters, a trailing slash, a `**`, a malformed `{variable}`, or a variable in the category or version segment | | `endpoint.path_reserved` | 400 | The pattern's category segment is reserved (172, R-EP5): it matches `v[0-9]+` — the product's own API namespace — or it is the literal `api`. `details.segment` names it. Re-checked when the serve registry is built, so a row written around the publish path is never served | | `endpoint.path_conflict` | 409 | The pattern could match the same URL as an already-published one (`/a/{x}` against `/a/b`); `details.conflicting_path` names the other. Ambiguity is refused, never resolved by precedence — at request time at most one pattern may match | | `endpoint.path_variable_unknown` | 400 | A `{variable}` in the path names no declared parameter of the pipeline's released version | | `endpoint.pipeline_not_readonly` | 409 | The pipeline is not side-effect-free: a `DML`/`DDL` node whose `source` is a registered datasource (`tempdb` is exempt — #171), or a DQL node writing back to a datasource, transitively through `PIPELINE` children; `details.node_id` names the offender. Raised at publish. The SAME code is returned as `503` at serve time ([REST API §19.4](/docs/rest-api.md#194-the-response)) when a later release breaks the rule under a live endpoint — the endpoint is not run | | `endpoint.pipeline_not_released` | 503 | The published pipeline has no servable version: nothing its pointer names is eligible — RELEASED always, a DRAFT only under development posture ([Versioning D63](/docs/versioning.md#2-decision-log)) | | `endpoint.not_found` | 404 | No endpoint matches the request path. Deliberately identical for an unknown path and a disabled one — distinguishing them would let an unauthenticated caller enumerate the registry | | `endpoint.method_not_allowed` | 405 | Any method but `GET` on a published path; the response carries `Allow: GET` | | `endpoint.not_acceptable` | 406 | An `Accept` this surface cannot satisfy (v1 serves `application/json`) | | `endpoint.key_not_bound` | 403 | The presented key is not among those bound at the first ancestor of the request path carrying any binding. A deeper binding REPLACES an inherited one, so a key bound higher up is refused rather than inherited ([Auth §7.7](/docs/auth.md#77-key-kinds-and-published-endpoint-bindings)) | | `endpoint.key_kind_refused` | 403 | The key's kind is wrong for what it is doing: an `endpoint` key outside its bound endpoints and the cursor of executions it started; an `endpoint` key presented to an endpoint with no binding on any ancestor (where only `user` keys with `execute` are accepted — an unbound endpoint key authorises nothing); a `server` key anywhere but the promotion routes it opens as `DP-Promotion-Key` (091); or an issuance whose `kind` contradicts its `scopes`/`bindings`. `details.reason` names which | | `endpoint.promotion.key_missing` | 409 | A promotion batch carries an endpoint binding naming an API key by name that the target deployment does not have. Reported before anything is pushed, like every other promotion pre-validation, so the target is left byte-unchanged | | `endpoint.request.invalid` | 400 | One or more request defects; `details.errors[]` carries `{parameter, code, message}` for every one of them ([REST API §19.3](/docs/rest-api.md#193-the-request)) | | `endpoint.request.parameter_unknown` | 400 | A query parameter the version does not declare. Strict on purpose: a typo that silently ran the default would return plausible, wrong rows | | `endpoint.request.parameter_repeated` | 400 | The same query key appeared more than once; an endpoint takes one value per parameter | | `endpoint.request.value_too_large` | 400 | A single parameter value over 4 KB | ### 13.15 Learned semantics The learned semantic layer ([design record](https://github.com/msabiransari/datapipelines/blob/main/docs/superpowers/specs/2026-09-11-learned-semantic-layer-design.md) §3.1, §7.1, O-4): what `semantics_record` / `semantics_retire` refuse. Two-segment codes, like `datasource.not_found` — the domain has no entity dimension; every code is about one fact. Landed with the constants (`PipelineErrorCodes.Semantics`, mirrored in `modules/datasources`'s `SemanticsErrorCodes` because the recorder lives below `pipeline-contract`) and their `ApiErrorCatalog` rows. | Code | HTTP | Description | |---|---|---| | `semantics.kind_invalid` | 400 | `kind` is not in the closed list ([Enums §19](/docs/enums.md#19-learnedfactkind--what-a-learned-fact-is-about)), or is not a kind of the requested `scope` — the nine data kinds are `DATASOURCE`, the three business kinds `WORKSPACE`; `details.kind` and `details.scope` | | `semantics.fact_invalid` | 400 | `fact` is outside its 8–1000 character window, `refs` is empty on a DATASOURCE-scope kind (a `unit` without a column is meaningless; a WORKSPACE-scope `definition`/`exclusion`/`preference` may carry none — a rule that spans datasources, 136 §B), or `evidence_summary` exceeds 300 characters; `details.field` names which | | `semantics.ref_unresolved` | 400 | A ref does not resolve against the datasource's LIVE introspection — the table is not there, or the column is not one of its columns. The store never starts stale (§3.1); `details.ref` carries the offending `{schema, table, column}` | | `semantics.ref_mismatch` | 400 | The `fact` (or `evidence_summary`) text names a table the `refs` do not, and the name is a NEAR-MISS: an identifier-shaped token within an edit distance of 2 of a listed table without being one (the same nearest-name rule as `datasource.table_not_found`). `details.token` names the offending word and `details.suggestion` the nearest listed table. A token that spells a listed table exactly is not refused: the tool appends that table to the stored `refs` and reports it in `refs_added` (129, owner ruling 2026-09-14). A datasource with no catalog listing (an empty registry) skips the check | | `semantics.evidence_refused` | 400 | `evidence_sql` is not a single read-only `SELECT`/`WITH` (the `sql_probe` classifier's rule), or names a `:parameter` — evidence binds nothing. Refused before any connection opens | | `semantics.evidence_failed` | 400 | `evidence_sql` ran once through the probe path and the database refused it, or it exceeded the probe's timeout; the fact is NOT recorded — evidence that does not run is not evidence. `details.reason` is `execution_failed` or `timeout`; an unreachable datasource stays `pipeline.execution.datasource_unreachable` (502) | | `semantics.duplicate` | 409 | An identical LIVE fact — same `(scope, workspace, datasource, kind, refs, fact)` — already exists; `details.existing_id` names it. Refused rather than rate-limited (O-4): an agent looping on the same fact learns nothing from a second row. A retired fact is not a duplicate | | `semantics.not_found` | 404 | The fact addressed by `id` (or by `supersedes`) does not exist, or is a WORKSPACE fact of another workspace — the D-R5 answer, identical for both | ### 13.16 MCP surface The MCP server's own refusals (120). Two-segment, like `semantics.not_found` — the domain has no entity dimension. The resource surface's unknown-URI answer is deliberately absent here: it is the JSON-RPC protocol's RESOURCE_NOT_FOUND ([MCP §9.1](/docs/mcp-server.md#91-mcp-level-errors)), which never travels in a §9.2 content envelope. Landed with the constant (`PipelineErrorCodes.Mcp`) and its `ApiErrorCatalog` row. | Code | HTTP | Description | |---|---|---| | `mcp.doc_not_found` | 404 | `docs_get` named a document the shipped skill does not carry ([MCP §6.2.41](/docs/mcp-server.md#6241-docs_get)); `details.name` and `details.known_docs` — the tool-surface answer to a resource read's not-found | ### 13.17 Release checks The release-check run's refusals (140, §3.3). Checks are opt-in: a version with no `checks[]` can never produce one of these — the gate runs only where the body declares them. Save-time shape defects are §12.12's `pipeline.validation.check_invalid`; this section is the RUN-time family, raised on the release path against the server's own fresh run — the author never supplies an observed value, so there is nothing here a caller could fake. | Code | HTTP | Description | |---|---|---| | `pipeline.check.failed` | 409 | release refused — a check on the version failed or errored; details.checks lists expected/observed/message per failing check; override requires override_checks_reason (≥ 10 chars) | ### 13.18 Script The SCRIPT node's execution-time refusals (7c, #7; the transform-nodes design record's §7). The HTTP column is the class the node failure record carries, as for every execution-time family in §13.4; through `templates_evaluate` and the save-time test suite these surface inside `template.test_failed` (400) or as the tool's own status. The save-time twins of these checks are §12.13's ten `pipeline.validation.script_…` rows. | Code | HTTP | Description | |---|---|---| | `pipeline.script.input_contract_violation` | 500 | an input violates the contract's declared inputs (record §5.1): the staged schema does not cover a contract column, a `nullable: false` column carried null, a value input's type does not fit. | | `pipeline.script.input_too_large` | 500 | a table input exceeds `datapipelines.script.max-input-rows` — refused BEFORE loading (record §4.3). | | `pipeline.script.evaluation_failed` | 500 | the script threw; the detail carries the engine message bounded to 2 000 chars (record §7). | | `pipeline.script.timeout` | 504 | the engine's own wall-clock bound fired (`pipeline.node.timeout` is the outer backstop — the engine reports first, record §4.3). | | `pipeline.script.resource_limit` | 500 | a declared resource limit fired — depth (JSONata); heap/statements/stack/output/input bytes/result bytes (JavaScript); `details.kind` names which. | | `pipeline.script.pool_exhausted` | 503 | the evaluation pool's admission bound is full (record §4.3 — a node failure in a run, 503 on the tool/route). | | `pipeline.script.row_shape_mismatch` | 500 | a returned row's key set is not exactly the declared column set, with batch and row number (record §5.3). | | `pipeline.script.value_type_mismatch` | 500 | a returned value does not fit its column's wire form, with batch and row number (record §5.3). | | `pipeline.script.precision_lost` | 500 | a numeric value does not fit its declared precision/scale (record §5.3, R1 — DECIMAL rounds half-even; BIGDECIMAL is exact). | | `pipeline.script.value_too_large` | 500 | a value/object output or a single string above the byte caps (record §4.3: `max-value-bytes`, `max-string-bytes`). | | `pipeline.script.invariant_failed` | 500 | an invariant evaluated to other than `true`; names the invariant and its message (record §5.4). | | `pipeline.script.invariants_too_large` | 500 | the invariants' read-back exceeded `max-input-rows` (record §5.4 — an assertion over an unbounded table is a SQL node or a release check, not an invariant). | | `pipeline.script.rejects_strict` | 500 | `strict: true` and the rejects table is non-empty; carries the count and the first ten reasons (record §5.4). | | `script.js.unavailable` | 400 | the JavaScript isolate is not installed (D-T16): disabled and reported once at INFO at boot, refused at save/evaluate; never a 503. | ### 13.19 Schedules The scheduler's refusals (#9; [Scheduler](/docs/scheduler.md), the scheduler design revision's §5). Two families, like the rest of §13: `schedule.validation.*` is the caller's request (400 — `details.field` names the field), and the bare `schedule.*` codes are a state the schedule is in (409), except the three not-found rows. Nothing here is raised while a schedule FIRES: a run the scheduler could not start is a `not_started` run with a reason on its trail ([Scheduler §5](/docs/scheduler.md#5-runs-states-and-reasons)), never an error response. A Run-now replay with a different body answers §13.11's `idempotency.key_reused_for_different_request`, and a pipeline's parameter refusals are exactly an interactive run's (§13.4). Landed with the constants — the catalog's `PipelineErrorCodes.Schedule` and the scheduler's own `ScheduleErrorCodes`, pinned equal by `SchedulerUnitTest` — and their `ApiErrorCatalog` rows. | Code | HTTP | Description | |---|---|---| | `schedule.not_found` | 404 | no live schedule with that id in the caller's workspace — a deleted schedule is absent, never "deleted" | | `schedule.run.not_found` | 404 | no run with that id under that schedule | | `schedule.notification.not_found` | 404 | no notification delivery with that id on that run (#442b, [REST API §20.13](/docs/rest-api.md#2013-resend-a-notification)) | | `schedule.name_taken` | 409 | another live schedule of the workspace holds the name | | `schedule.revision_conflict` | 409 | `If-Match` named a revision that is no longer current — someone changed the schedule since it was read; `details.current_revision` | | `schedule.run.overlap` | 409 | Run now while a run of the schedule is queued, starting or running (one run in flight per schedule) | | `schedule.blocked` | 409 | Run now against a blocked schedule; `details.blocked_reason` — unblock first (resume never clears a block) | | `schedule.not_blocked` | 409 | unblock on a schedule that is not blocked | | `schedule.notification.not_resendable` | 409 | resend of a notification that is `pending` or `accepted` — only a `failed` or `uncertain` one can be queued again (#442b); `details.state` names the state it is in | | `schedule.notification.backlog_full` | 409 | resend refused at the strict deployment pending-delivery cap; `details.pending` and `details.backlog_max`; the delivery stays unchanged | | `schedule.limit.per_workspace` | 409 | the workspace already holds `datapipelines.scheduler.max-schedules-per-workspace` live schedules | | `schedule.validation.request_invalid` | 400 | a missing or ill-typed request field | | `schedule.validation.name_invalid` | 400 | the name breaks the folder-path grammar pipelines and templates use | | `schedule.validation.cron_invalid` | 400 | not a five-field Unix cron — a seconds field, the disabled pattern `-` and an out-of-range field are all this | | `schedule.validation.timezone_invalid` | 400 | not an IANA region id (`Europe/Berlin`); a fixed offset such as `+02:00` is refused too | | `schedule.validation.interval_too_short` | 400 | two consecutive occurrences closer than `datapipelines.scheduler.min-interval-seconds` | | `schedule.validation.executor_unknown` | 400 | no executor is registered under that id (v1 registers `pipeline`) | | `schedule.validation.payload_invalid` | 400 | the executor refused the payload — for `pipeline`: an unknown field, no pipeline name, or a `version` other than `"current"` (`latest` is refused by name); `details.reason` names which — or the payload or parameters exceed 16 KiB or 8 levels | | `schedule.validation.target_not_found` | 400 | the payload names a pipeline the workspace does not hold, or its current version has no stored body; `details.pipeline` | | `schedule.validation.target_not_released` | 400 | the payload names a pipeline with no current version to follow (#280) — release it, or switch its current version, then save again; `details.pipeline` | | `schedule.validation.binding_invalid` | 400 | a `parameter_bindings` entry cannot be resolved (#9 slice 3) — an unknown parameter, a non-`DATE` type, an unknown keyword (`TODAY`/`YESTERDAY` only), an unknown source, a literal binding without a value, or a keyword with no reference at run time; `details.reason` (`unknown_parameter` / `type_mismatch` / `unknown_keyword` / `unknown_source` / `literal_invalid` / `no_reference`) and `details.parameter` name why | | `schedule.validation.binding_conflict` | 400 | the same parameter is supplied in `parameters` and in `parameter_bindings` — give it one source; the resolver rejects ambiguity instead of applying a precedence rule (`details.parameter`) | | `schedule.validation.notifications_invalid` | 400 | Notification settings refused (#442a). `details` contains only `{field, reason}`: `notifications`, `notifications.recipients`, an indexed recipient or event field; `not_an_object`, `not_a_list`, `not_a_string`, `syntax`, `too_long`, `too_many`, `unknown_event`, `domain_not_allowed` (#442b: outside a non-empty recipient-domain allowlist). No address is echoed | ### 13.20 Parameter sets The parameter engine's refusals (#194; the [parameter-engine design record](https://github.com/msabiransari/datapipelines/blob/main/docs/superpowers/specs/2026-09-21-parameter-engine-design.md) §10). `parameter.validation.*` is the author's document at save — the record's §4 order: shape, the §3 field rules, the graph, the expressions, the template pins, then the dry render and the metadata execution — every failure collected (§17.2's rule), `details.path` naming the field. `parameter.evaluate.*` is the runtime (the record's §5): a whole-request refusal carries its status; the rows marked "—" are never an error RESPONSE — they are reported per parameter in `state.errors[]` of a 200 evaluate, whose `valid` is then `false`. A parameter's `state.errors[]` can also carry another family's code when its selector failed — the selector runtime reports the owning subsystem's own code (the record's §17, C20): `pipeline.execution.datasource_unreachable`, `pipeline.node.query_execution_failed` (a refused read-only gate included, `details.reason = read_only_gate`), `pipeline.node.query_timeout`, `pipeline.node.sql_parameter_missing`, `pipeline.node.template_render_failed` / `template_not_found`, `datasource.table_forbidden`, `datasource.not_found` (a datasource no longer visible from the workspace). The `parameter.version.*` / `release` / `import` / `authoring` rows are the `template.*` twins of the lifecycle ([Versioning §3.5](/docs/versioning.md#35-the-lifecycle-table)). A selector template that interpolates a parent is refused with §13.9's existing `template.validation.parameter_interpolated`. `parameter.in_use` (409) — the consumer binding the record's §13 reserved — landed with #320: a dashboard that pins a set version refuses its purge and discard. Landed with the constants — `ParameterErrorCodes` in `modules/parameters` and its mirror `PipelineErrorCodes.Parameters`, pinned equal by reflection (`ParameterErrorCodesTest`), held to this table by `ParameterErrorCodesSpecDriftTest` — and their `ApiErrorCatalog` rows. | Code | HTTP | Description | |---|---|---| | `parameter.validation.name_invalid` | 400 | the set name fails the folder grammar pipelines and templates use (`details.reason`: `folder_required` / `grammar` / `missing`, or `immutable` when a write names a different set than the one it addresses — a set is never renamed), or a parameter name fails §6.1's grammar | | `parameter.validation.name_reserved` | 400 | a parameter name ending in `_count` or `__` — the generated bind namespace of a `MULTI` (the record's P29) | | `parameter.validation.new_root_requires_confirmation` | 400 | agent surface only — a set under a top-level folder that holds nothing yet, without `confirm_new_root: true` (the pipelines/templates shape); `details.existing_roots` | | `parameter.validation.duplicate_name` | 409 | the set name exists in the workspace — discarded sets included, names are unique forever ([Versioning §3.2](/docs/versioning.md#32-entity-status-is-derived-names-are-unique-forever)) | | `parameter.validation.duplicate_parameter` | 400 | two parameters of one set share a name | | `parameter.validation.too_many_parameters` | 400 | more parameters than `datapipelines.parameters.max-parameters-per-set` — refused before any parameter is read | | `parameter.validation.body_invalid` | 400 | the document's shape (owner ruling 2026-09-26): an unknown key, a wrong JSON type, a missing structural key or a `depends_on` longer than the set can hold, at any level — `details.path` names it, `details.reason` is `unknown_key` / `wrong_type` / `missing` / `too_long`; every one is reported, and nothing binds until there are none | | `parameter.validation.label_invalid` | 400 | a parameter's `label` (or the set's `display_name`) is missing, blank or over 120 characters | | `parameter.validation.description_too_long` | 400 | a description (a parameter's or the set's) over 2000 characters | | `parameter.validation.type_invalid` | 400 | a `type` outside the ten declarable logical types (`NULL` is a column fact, never a declaration) | | `parameter.validation.precision_missing` | 400 | a `DECIMAL` parameter with no `precision` (§6.2's rule, §12.7's twin) | | `parameter.validation.scale_missing` | 400 | a `BIGDECIMAL` parameter with no `scale` (§12.7's twin) | | `parameter.validation.kind_invalid` | 400 | a `kind` other than `INPUT` / `SELECT`, or none | | `parameter.validation.cardinality_invalid` | 400 | a `cardinality` other than `SINGLE` / `MULTI`, or `MULTI` on an `INPUT` (an input is always `SINGLE`) | | `parameter.validation.default_type_mismatch` | 400 | `default_value` is not a wire value of the parameter's type — or not an array for a `MULTI` | | `parameter.validation.default_not_an_option` | 400 | a `constants` select's `default_value` (or one of a `MULTI`'s members) is not among its option values | | `parameter.validation.default_invalid` | 400 | a default — hard-coded, a marked option, a sourced row at save's dry run — breaks the parameter's own constraints, precision or scale (`min: 0` with `default_value: -1`); `details.reason` names the rule | | `parameter.validation.source_missing` | 400 | a `SELECT` with no `source` | | `parameter.validation.source_ambiguous` | 400 | a `source` stating both `constants` and `template`, neither, or a `template` without its `datasource` (or a `datasource` without a `template`) | | `parameter.validation.source_not_allowed` | 400 | `constants` on an `INPUT` — an input's hard-coded value is its `default_value` | | `parameter.validation.constraints_on_select` | 400 | `constraints` on a `SELECT` — constraints are the `INPUT`'s | | `parameter.validation.constraint_not_applicable` | 400 | a constraint on a type it does not apply to (`pattern` on an `INTEGER`) | | `parameter.validation.constraint_invalid` | 400 | a malformed constraint: a bound not in the parameter's type, `min > max`, a negative length, `min_length > max_length` | | `parameter.validation.pattern_invalid` | 400 | a `pattern` over 256 characters, not compiling, or using a construct the regex budget refuses (backreferences, lookaround, possessive or atomic groups) | | `parameter.validation.format_invalid` | 400 | a `format` whose shape does not fit the type family — any `format` on `STRING`/`BOOLEAN`/`BINARY`, a `kind` on a temporal type, a `pattern` on a numeric one, or an unknown numeric `kind` | | `parameter.validation.presentation_invalid` | 400 | a `presentation.control` outside the catalogue ([enums §27](/docs/enums.md)) | | `parameter.validation.control_not_applicable` | 400 | a catalogued `control` outside its kind × cardinality × type row (the record's §3.7) | | `parameter.validation.format_pattern_invalid` | 400 | a temporal `format.pattern` over 64 characters, not compiling as a `DateTimeFormatter` pattern, or naming time fields on a `DATE` (date fields on a `TIME`) | | `parameter.validation.option_invalid` | 400 | a `constants` option with a null or mistyped `value`, a missing or empty `display_value`, or a value/label over `max-option-value-chars` / `max-option-label-chars`; `details.reason` | | `parameter.validation.option_duplicate` | 400 | two `constants` options share a value (equality is the canonical value's — `1.0` equals `1.00`) | | `parameter.validation.multiple_defaults` | 400 | more than one `constants` option marked `is_default` (P7) | | `parameter.validation.too_many_options` | 400 | an empty `constants` list, or one longer than `max-options-per-selector` — refused before any option is read | | `parameter.validation.dependency_unknown` | 400 | a `depends_on` entry names no parameter of the set | | `parameter.validation.dependency_self` | 400 | a `depends_on` entry names the parameter itself | | `parameter.validation.dependency_cycle` | 400 | the set's dependency graph (`depends_on`, direct or indirect) has a cycle; `details.cycle` is the path (`a -> b -> a`) | | `parameter.validation.bind_undeclared` | 400 | a selector template binds `:name` that is neither a parameter in the carrier's `depends_on` nor an org/platform tier key (§7.2; `execution_id` is absent), or a hand-written `__` bind | | `parameter.validation.ref_undeclared` | 400 | an expression `ref` names a parameter outside the carrier's `depends_on`, or the carrier itself | | `parameter.validation.expression_invalid` | 400 | a `hidden_expression`/`disabled_expression` that is not a §7 AST — an unknown `op` or key, a missing operand, an empty `and`/`or`; `details.reason` | | `parameter.validation.expression_depth_exceeded` | 400 | an expression deeper than `max-expression-depth` | | `parameter.validation.expression_too_large` | 400 | an expression with more nodes than `max-expression-nodes`, or an `in` list over 256 literals | | `parameter.validation.expression_cardinality` | 400 | `eq`/`neq`/`in`/`is_null` on a `MULTI` ref, or `contains`/`is_empty` on a `SINGLE` one | | `parameter.validation.expression_literal_type` | 400 | a literal not coercible to its ref's type | | `parameter.validation.expression_type_unsupported` | 400 | a `BINARY` ref tested by anything but `is_null` | | `parameter.validation.template_not_found` | 400 | the pinned template id does not exist in the workspace (§12.6's twin) | | `parameter.validation.template_version_not_found` | 400 | the pinned template exists, but not at that version — or only DISCARDED there (`details.reason: discarded`); a set's RESTORE re-judges its selector pins with this code and `template_not_found` under the workspace writer guard (#517) | | `parameter.validation.template_type_mismatch` | 400 | the pinned template is not `type = 'sql'` | | `parameter.validation.template_dialect_mismatch` | 400 | the pinned template's dialect differs from the source datasource's | | `parameter.validation.template_render_failed` | 400 | the dry render of a selector template against its parents' defaults failed (the record's §4 step 5) | | `parameter.validation.datasource_not_found` | 400 | a source's datasource is not visible from the workspace (bound or global) | | `parameter.validation.datasource_unreachable` | 400 | the metadata execution could not reach the source's datasource (the record's §4 step 6) — a set whose sources cannot be proven is not saved | | `parameter.validation.selector_query_failed` | 400 | the metadata execution reached the datasource and its statement failed — a column, a permission, the read-only gate (owner ruling 2026-09-26); `details.datasource_code` is the datasource's own code | | `parameter.validation.selector_columns_invalid` | 400 | a selector's columns are not exactly `value`, `display_value`, `is_default` — or `value` alone for an `INPUT`; `details` name the extra or missing column | | `parameter.validation.selector_value_type_mismatch` | 400 | the selector's `value` column type does not pass the record's §6.4 against the declared type (decided from metadata, never rows); `details.hint` | | `parameter.validation.input_source_multiple_rows` | 400 | a database-fed `INPUT`'s template returned two or more rows at save's dry run against its parents' defaults (owner ruling 2026-09-27; the record's §4 step 6 — the save-time twin of `parameter.evaluate.input_source_multiple_rows`) | | `parameter.validation.selector_order_by_missing` | 400 | a `SELECT`'s rendered SQL carries no `ORDER BY` (P7 — the check proves a clause is present, not that the order is total) | | `parameter.evaluate.unknown_parameter` | 400 | a `selections` key names no parameter of the set — the whole request is refused (the caller's error) | | `parameter.evaluate.too_many_values` | — | not an error response — reported per parameter in `state.errors[]` of a 200 evaluate: a `MULTI` longer than `max-multi-bind-values` | | `parameter.evaluate.too_many_binds` | — | not an error response — per parameter in `state.errors[]`: a rendered statement over `max-binds-per-statement` placeholders, every expanded one counted | | `parameter.evaluate.selector_rows_invalid` | — | not an error response — per parameter in `state.errors[]`: a selector row violating the option invariants (`details.reason`: `duplicate_value` / `null_value` / `empty_label` / `multiple_defaults`, `value_too_long` / `label_too_long` over `max-option-value-chars` / `max-option-label-chars`, or `columns` when the source's shape drifted since save — `details.row` names the row); the parameter then offers no options | | `parameter.evaluate.response_too_large` | 413 | the response would exceed `max-evaluate-response-bytes` — the whole request is refused, options are never truncated | | `parameter.evaluate.invalid_value_type` | — | not an error response — per parameter in `state.errors[]`: a submitted value in the wrong wire form, not coercible, or a duplicate `MULTI` member | | `parameter.evaluate.constraint_violation` | — | not an error response — per parameter in `state.errors[]`: a submitted value breaks a constraint, the precision or the scale; `details.reason` (`min`, `max`, `min_length`, `max_length`, `pattern`, `pattern_budget`, `scale`, `precision`) — or a database-fed `INPUT`'s row breaks the input's own rules (`details.source = row`; the row is not used) | | `parameter.evaluate.required_missing` | — | not an error response — per parameter in `state.errors[]`: required and nothing resolves (`details.reason`: `no_options` / `no_default` / `no_row`) | | `parameter.evaluate.too_many_options` | — | not an error response — per parameter in `state.errors[]`: a selector returned more rows than `max-options-per-selector` | | `parameter.evaluate.input_source_multiple_rows` | — | not an error response — per parameter in `state.errors[]`: an `INPUT`'s source template returned two or more rows; `default_value` is used | | `parameter.evaluate.selector_value_type_mismatch` | — | not an error response — per parameter in `state.errors[]`: the selector's `value` column no longer passes the record's §6.4 (schemas drift), or a row's value fails the shared validator for the declared type (a driver under-reporting its metadata — `details.reason`, `details.row`) | | `parameter.evaluate.template_unrendered` | 400 | MCP only — an evaluate of a draft set whose pinned DRAFT template changed after the key's last `templates_render` of it (the 139 gate's twin) | | `parameter.evaluate.timeout` | 504 | the evaluate's deadline (`evaluate-timeout-seconds`) — the whole request, never a half-form; the house mapping of an executor deadline (`pipeline.execution.timeout`, §13.3) | | `parameter.evaluate.selectors_saturated` | — | not an error response — per parameter in `state.errors[]`: the selector bulkhead's queue was already full at admission (decided at once; a deadline reached while queued or running is `parameter.evaluate.timeout` instead); a consumer that surfaces it uses rest-api §4's concurrency-limit shape | | `parameter.not_found` | 404 | no such set (or version) in the workspace, hidden by the promoter lens, or discarded on a read/mutate path — indistinguishable by design | | `parameter.version.conflict` | 409 | the precondition hash is stale — draft write, release, discard, purge; `details` carry the current hash, status and author ([Versioning §4.2](/docs/versioning.md#42-the-precondition-protocol)); an import onto a taken number with a different hash (`details.reason`) | | `parameter.version.not_draft` | 409 | release or a draft verb on a set with no DRAFT | | `parameter.version.not_released` | 409 | discard of a version that is not RELEASED (drafts are purged, never discarded) | | `parameter.version.not_discarded` | 409 | restore of a version that is not DISCARDED | | `parameter.version.last_release` | 409 | purge of a released version, or of an entity holding one | | `parameter.in_use` | 409 | purge or discard refused because a LIVE dashboard version pins the set (#320, [Versioning §3.5.3](/docs/versioning.md#353-the-reverse-arrows-into-other-families-320): the exact version for a discard and a draft purge that keeps other versions; every stored dashboard version, discarded included, for an entity purge and a draft purge that takes the entity). `details`: `id`, `version` (absent for an entity purge), `pinned_by` — one `{dashboard, version, status}` per pinning dashboard version | | `parameter.version.not_eligible` | 409 | switch to a version that is not live and posture-eligible | | `parameter.version.confirm_mismatch` | 400 | a destructive verb's typed confirmation did not match (the caller's input was wrong; nothing about the version refused) | | `parameter.release.dependency_not_released` | 409 | release with a pinned template version that is not RELEASED; `details.pins_not_released` lists every one until the parameter-set participant moves the details to the shared release-blockers shape (#519) — `release_pinned_templates=true` cascades them in the same transaction ([Versioning §5.3](/docs/versioning.md#53-release-lock), 142, #517). A set's RESTORE whose stored body pins a present-but-DRAFT template version refuses this code in the shared release-blockers shape, `reason: not_released` (#517, defensive) | | `parameter.release.plan_conflict` | 409 | Reserved — raised once the parameter release participant lands (#519). A consented snapshot fact changed between the dependency preview and the submit — an identity, status, body hash, current pointer, pin path or edge no longer matches the submitted `tree_hash`, even when someone else released the node meanwhile. `details` `{reason:"snapshot_changed"}` plus `node_id`/`path` only when visible; nothing flipped; reopen the preview. A root hash mismatch keeps `parameter.version.conflict` ([Versioning §5.3](/docs/versioning.md#53-release-lock), #517) | | `parameter.release.plan_required` | 428 | Reserved — raised once the parameter release participant lands (#519). Cascade consent without a preview `tree_hash` — dependencies selected, or a legacy alias combined with a plan field, with no digest to compare. No dependency names in `details`; obtain a fresh dependency preview ([Versioning §5.3](/docs/versioning.md#53-release-lock), #517) | | `parameter.release.plan_invalid` | 400 | Reserved — raised once the parameter release participant lands (#519). The plan is malformed: a `tree_hash` that is not a digest, unknown or duplicate node ids, the root or a RELEASED node selected, an unsupported selection, or a legacy alias set to true together with explicit `include_dependencies` (two authorities). `details.field` is a bounded locator; no submitted value is reflected (#517) | | `parameter.release.dependency_cycle` | 409 | Reserved — raised once the parameter release participant lands (#519). The release closure repeats an exact node on its active path; `details.path` is the visible bounded node trail of that one cycle, never every expanded path (#517) | | `parameter.release.dependency_limit` | 422 | Reserved — raised once the parameter release participant lands (#519). The complete closure exceeds a configured bound ([Configuration §3.35](/docs/configuration.md#335-release-dependencies)); `details` `{limit, maximum}` with `limit` one of `depth`, `versions`, `edges`, `wire_bytes` — never a hidden total. Enforced while the closure is enumerated, so a truncated closure never authorises a release (#517) | | `parameter.release.busy` | 409 | The workspace mutation guard's lock wait (`datapipelines.release.dependencies.lock-wait-seconds`) ran out — another pin-affecting parameter write or release in the same workspace holds it. Nothing was written; retry with a current plan (#517) | | `parameter.release.bundle_unsupported` | 409 | Reserved — raised once promotion carries full closures (#523). The promotion peer lacks the full-closure envelope capability; no bodies were transmitted. The family is the promoted root's (#517) | | `parameter.import.missing_template` | 400 | an import whose pinned template version is absent on the target — promotion order is templates before sets | | `parameter.authoring.disabled` | 403 | `datapipelines.deployment.authoring-enabled=false` refuses every authoring write ([Versioning §5.5](/docs/versioning.md#55-drafts-are-a-deployment-capability-039)) | ### 13.21 Request limits The platform's transport-level refusals (#279; [Configuration §3.31](/docs/configuration.md#331-web-request-limits)). A body over `datapipelines.web.max-request-bytes` is refused at a servlet filter on BOTH JSON surfaces — REST `/api/v1` and MCP `/mcp` — before authentication and before any parser reads a byte of it, so no handler, tool or permission is involved and the refusal carries no principal data (the envelope's `correlation_id` and this code only). The cap is a byte count on the REQUEST body; a response's budget is a different key (`parameter.evaluate.response_too_large` above). Jackson's `StreamReadConstraints` (nesting depth 100, string length 4M chars, number length 1000 digits) are stated explicitly on every request-body mapper rather than inherited, so a body WITHIN the cap that is adversarial for the parser is refused by the parser as a 400 (`details.reason: malformed_json`), not answered from an inherited default that could change under a version bump. The filter is mapped to `/api/v1/*` and `/mcp` only — a form-encoded UI post is Tomcat's `max-http-form-post-size` to bound, and a body sent on a body-less method (GET/HEAD/OPTIONS) is not counted, only drained within `server.tomcat.max-swallow-size`. The stated Jackson constraints hold on the auto-configured REST mapper, the MCP transport's, and — since #291 — every route that parses its own `@RequestBody String` (templates create/update/lifecycle/render/evaluate/import, pipelines create/update/import, execute), each of which reads its body through a REQUEST copy of its domain mapper (`RequestLimits.requestMapper`) and answers a read failure with its family's malformed-body 400, never the 500 backstop. The domain mappers themselves are left alone: they also read stored rows, script values and JSON columns, where a document deeper than a request may be is legitimate data — a request limit is a statement about request bytes. One set of routes is not yet covered: the parameter-set routes under `web/parameters` parse with the parameters module's strict mapper under the body cap and Jackson's inherited depth (#300). **Scalar shapes (#333).** Jackson binds a JSON number or boolean into a declared `String` as its text unless the mapper's `Textual` coercion config fails it, and `ALLOW_COERCION_OF_SCALARS` off never reaches that bind (the same mappers also parsed `"1"` into an integer and `1.5` into `1`). The four strict contract readers carry the same lines `ParameterSetJson` and `ArtifactJson` do — `ALLOW_COERCION_OF_SCALARS` and `ACCEPT_FLOAT_AS_INT` off, `Textual` failing the Integer/Float/Boolean shapes, `Boolean` failing Integer — so the value is refused with its PATH and the expected shape, never the value (Jackson's own message quotes it): the pipeline reader (`PipelineJson`, `PipelineDeserializer`) as `pipeline.validation.schema_version_unsupported` with `details.reason: "wrong_type"`; the template reader and the script blocks (`TemplateJson`, `ScriptBlocks`) as `template.contract_invalid` with `details.rule: "wrong_type"`; and the five routes that bind a typed request DTO (`POST /auth/api-keys`, `POST /endpoints`, `POST /endpoints/bindings`, `POST /promotion/push`, `POST /dashboards/bindings`) as `pipeline.execution.invalid_parameter_type` with `details.reason` `wrong_type` / `unknown_key` / `missing` and `details.path`, because they take the body as a tree and bind it through one strict request mapper (`StrictRequestBodies`, built over `RequestLimits.requestMapper`). Unknown keys stay tolerated on the pipeline and template readers (a re-import carries lifecycle keys) and on the promotion batch (a cross-version wire, `ignoreUnknown`), and are refused on the two request DTOs. Every echoed path is clipped and its control characters replaced. | Code | HTTP | Description | |---|---|---| | `request.body_too_large` | 413 | the request body exceeds `datapipelines.web.max-request-bytes` — refused before any handler; `details.limit_bytes` names the cap in effect | | `request.query_too_long` | 400 | a search term `q` over 200 characters after the trim, on EVERY surface that refuses rather than truncates (#504b): the pipelines, templates, parameter-set and user listings, and the five `/tree/search` routes on REST; both MCP list tools as `-32602 invalid params`. Refused before any read by the services that own a search (`SearchNeedle.of`), `details.limit` / `details.length`; the needle is never echoed. The web UI's search boxes truncate to the same bound and never meet it (the #490 rule, shared — its `parameter.validation.query_too_long` row retired here) | ### 13.22 Visualizations The visualization artifact's refusals (#10; the [dashboard implementation spec](https://github.com/msabiransari/datapipelines/blob/main/docs/superpowers/specs/2026-09-28-dashboard-implementation-spec.md) §3.1, §11, §14; [Dashboards §2.1](/docs/dashboards.md)). `visualization.validation.*` is the author's document at save — the strict reader's shape first, then the spec's §3.1 rules — every failure collected (§17.2's rule), `details.path` naming the field. The `visualization.version.*` / `import` / `authoring` rows are the `parameter.*` twins of the lifecycle ([Versioning §3.5](/docs/versioning.md#35-the-lifecycle-table)); `release.*` is the gate (D56) — `tests_*` and `mechanical_failed` are raised by the evidence gate the tests lane (L4) installs, and until then the module's default gate refuses every visualization release with `tests_missing`. `test.*` belongs to the test-session routes (L4). The spec's list gained eight rows at L1a — `name_invalid`, `name_taken`, the five `version.*` lifecycle refusals and `version.pinned` (the design record's §4.2: purge and discard guards keep referenced releases) — and `authoring.disabled`; `name_taken` takes the place of the mould's `duplicate_name` because the dashboard family's `duplicate_name` names its object namespace. Landed with the constants — `VisualizationErrorCodes` in `modules/visualization` and its mirror `PipelineErrorCodes.Visualization`, pinned equal by reflection (`VisualizationErrorCodesTest`), held to this table by `VisualizationErrorCodesSpecDriftTest` — and their `ApiErrorCatalog` rows. | Code | HTTP | Description | |---|---|---| | `visualization.validation.body_invalid` | 400 | the document's shape ([Dashboards §2.1](/docs/dashboards.md)): an unknown key at any level, a wrong JSON type, a missing structural key, an out-of-vocabulary literal, or a collection over its `datapipelines.visualization.*` bound (refused before its members are read) — `details.reason` `unknown_key` / `wrong_type` / `missing` / `not_allowed` / `too_many` / `too_large` / `too_long` / `blank`, `details.path` naming the field; a refusal echoes a key or a path, never a value | | `visualization.validation.name_invalid` | 400 | the visualization name fails the folder grammar pipelines, templates and parameter sets use (`details.reason`: `folder_required` / `grammar` / `missing`, or `immutable` when a write names a different visualization than the one it addresses — a visualization is never renamed) | | `visualization.validation.new_root_requires_confirmation` | 400 | agent surface only — a visualization under a top-level folder that holds nothing yet, without `confirm_new_root: true` | | `visualization.validation.name_taken` | 409 | the name exists in the workspace — discarded visualizations included, names are unique forever ([Versioning §3.2](/docs/versioning.md)) | | `visualization.validation.renderer_unsupported` | 400 | `renderer.kind` is a reserved kind round one does not render (`html`, `svg` — D13, D46), or a `renderer.version` the kind does not accept | | `visualization.validation.input_contract_invalid` | 400 | an input contract that cannot be satisfied: no inputs, an input name outside `[a-z][a-z0-9_]{0,63}`, an input with no columns, a column name outside the column grammar, or two columns of one input sharing a name | | `visualization.validation.transform_binding_invalid` | 400 | `transform` does not bind: the pinned template is missing or DISCARDED at that version (`details.reason` `template_not_found` / `template_version_not_found`), not a transform (`not_a_transform`), its contract's input names differ from `transform.inputs`' keys, a `transform.inputs` value names no visualization input, or a declared input column differs from the named input's column by name, type or nullability (`details.column`); a transform-less visualization with more than one input (`transform_required`) | | `visualization.validation.config_schema_invalid` | 400 | `config` fails the renderer's schema: for `plotly` an object whose `data` is an array of objects each with a `type` from the vendored bundle's list and whose `layout` is an object; for `table` and `kpi` their own schemas ([Dashboards §2.1](/docs/dashboards.md)); `details.path` names the offending path inside `config` | | `visualization.validation.binding_unbound` | 400 | a `bindings` entry does not bind: its path is not the binding grammar or does not resolve inside `config` (`details.reason` `path_invalid` / `path_unresolved`), or its column is not in the transform's OUTPUT contract (or, with no transform, the one input's columns) — `details.column` | | `visualization.validation.test_case_invalid` | 400 | a `tests.cases[]` entry that is not runnable: a missing or duplicate case name, fixtures missing an input or naming an unknown one, a fixture row with a column outside the input's contract or a value outside its type, an assertion kind outside the closed vocabulary or missing its argument | | `visualization.not_found` | 404 | no such visualization (or version) in the workspace, hidden by the promoter lens, or discarded on a read/mutate path — indistinguishable by design | | `visualization.version.conflict` | 409 | the precondition hash is stale — draft write, release, discard, purge; `details` carry the current hash, status and last writer | | `visualization.version.not_draft` | 409 | release or a draft verb on a visualization with no DRAFT | | `visualization.version.not_released` | 409 | discard of a version that is not RELEASED (drafts are purged, never discarded) | | `visualization.version.not_discarded` | 409 | restore of a version that is not DISCARDED | | `visualization.version.last_release` | 409 | purge of a released version, or of an entity holding one | | `visualization.version.not_eligible` | 409 | switch to a version that is not live and posture-eligible | | `visualization.version.pinned` | 409 | discard or purge of a version a LIVE dashboard version pins (graph rule 1 of [Versioning §3.5](/docs/versioning.md)); `details.pinned_by` names the dashboards | | `visualization.release.tests_missing` | 409 | release without evidence: no test case (`details.reason: no_cases`), no run of the candidate version (`no_runs`), or its latest run still open (`run_open`) — the evidence gate's ladder is [REST API §22.2](/docs/rest-api.md#222-routes) | | `visualization.release.tests_stale` | 409 | release whose latest run's `body_hash` is not the candidate's — the content changed after the run | | `visualization.release.tests_red` | 409 | release whose latest run for the candidate is not GREEN; `details` name the case | | `visualization.release.mechanical_failed` | 409 | release whose server-run mechanical test (the spec's §11.3) fails now; `details` name the step, case and path | | `visualization.release.dependency_not_released` | 409 | release with a pinned transform template version that is not RELEASED, without `release_pinned_templates` consent (or MISSING); `details.pins_not_released` lists every one until the visualization participant moves the details to the shared release-blockers shape (#520) | | `visualization.release.plan_conflict` | 409 | Reserved — raised once the visualization release participant lands (#520). A consented snapshot fact changed between the dependency preview and the submit — an identity, status, body hash, current pointer, pin path or edge no longer matches the submitted `tree_hash`, even when someone else released the node meanwhile. `details` `{reason:"snapshot_changed"}` plus `node_id`/`path` only when visible; nothing flipped; reopen the preview. A root hash mismatch keeps `visualization.version.conflict` ([Versioning §5.3](/docs/versioning.md#53-release-lock), #517) | | `visualization.release.plan_required` | 428 | Reserved — raised once the visualization release participant lands (#520). Cascade consent without a preview `tree_hash` — dependencies selected, or a legacy alias combined with a plan field, with no digest to compare. No dependency names in `details`; obtain a fresh dependency preview ([Versioning §5.3](/docs/versioning.md#53-release-lock), #517) | | `visualization.release.plan_invalid` | 400 | Reserved — raised once the visualization release participant lands (#520). The plan is malformed: a `tree_hash` that is not a digest, unknown or duplicate node ids, the root or a RELEASED node selected, an unsupported selection, or a legacy alias set to true together with explicit `include_dependencies` (two authorities). `details.field` is a bounded locator; no submitted value is reflected (#517) | | `visualization.release.dependency_cycle` | 409 | Reserved — raised once the visualization release participant lands (#520). The release closure repeats an exact node on its active path; `details.path` is the visible bounded node trail of that one cycle, never every expanded path (#517) | | `visualization.release.dependency_limit` | 422 | Reserved — raised once the visualization release participant lands (#520). The complete closure exceeds a configured bound ([Configuration §3.35](/docs/configuration.md#335-release-dependencies)); `details` `{limit, maximum}` with `limit` one of `depth`, `versions`, `edges`, `wire_bytes` — never a hidden total. Enforced while the closure is enumerated, so a truncated closure never authorises a release (#517) | | `visualization.release.busy` | 409 | The workspace mutation guard's lock wait (`datapipelines.release.dependencies.lock-wait-seconds`) ran out — another pin-affecting visualization write or release in the same workspace holds it. Nothing was written; retry with a current plan (#517) | | `visualization.release.bundle_unsupported` | 409 | Reserved — raised once promotion carries full closures (#523). The promotion peer lacks the full-closure envelope capability; no bodies were transmitted. The family is the promoted root's (#517) | | `visualization.import.id_taken` | 409 | an import whose exported id is held by another visualization on this server — ids are globally unique and never re-issued (C29) | | `visualization.import.missing_template` | 400 | an import whose pinned template version is absent on the target — promotion order is templates first | | `visualization.authoring.disabled` | 403 | `datapipelines.deployment.authoring-enabled=false` refuses every authoring write ([Versioning §5.5](/docs/versioning.md#55-drafts-are-a-deployment-capability-039)) | | `visualization.test.session_not_found` | 404 | no such test session for the visualization, or its token does not match | | `visualization.test.session_expired` | 410 | the session outlived `datapipelines.visualization.session-ttl-minutes`, was already submitted, or its content moved on (`details.reason`) | | `visualization.test.screenshot_too_large` | 413 | a screenshot over 4 MiB — the upload route's own cap, independent of the platform's 2 MiB request-body cap. A declared body of up to the cap + 2 MiB is drained after the refusal so the connection closes only after the client's last byte and the 413 is observable by any client (#467); a larger declaration is closed after Tomcat's swallow | | `visualization.test.screenshot_invalid` | 400 | a screenshot that is not a PNG or WebP image the server can read (media type, magic bytes, dimensions) | ### 13.23 Dashboards The dashboard artifact's refusals (#10; the implementation spec §3.2, §8, §9, §14; [Dashboards §2.2](/docs/dashboards.md)). `dashboard.validation.*` is the author's document at save, again at release and — against the pinned dependencies' CURRENT state — at every runtime configuration read; `details.path` names the field and a refusal names the object, never a value. The `version.*` / `import.*` / `authoring.*` rows are the lifecycle twins; `release.dependency_not_released` names every dependency a release needs released (D61). `runtime.*` and `refresh.*` belong to the server runtime (L2) and `key.*` to the `dashboard` key kind (L5); they are catalogued now so the families are complete. The spec's list gained ten rows at L1a — `name_invalid`, `name_taken`, `dependency_not_found`, the five `version.*` lifecycle refusals, `import.missing_dependency` and `authoring.disabled`. Landed with `DashboardErrorCodes` (`modules/visualization`) and its mirror `PipelineErrorCodes.Dashboard`, pinned by `DashboardErrorCodesTest` and `DashboardErrorCodesSpecDriftTest`. | Code | HTTP | Description | |---|---|---| | `dashboard.validation.body_invalid` | 400 | the document's shape ([Dashboards §2.2](/docs/dashboards.md)): an unknown key at any level, a wrong JSON type, a missing structural key, an out-of-vocabulary literal (`scope`, a `parameter_state` setting, a position), an object whose `type` does not match the list it sits in (`details.reason` `wrong_object_type`), a timeout outside its bounds (`out_of_range`) or a collection over its bound (`too_many`) — `details.path` naming the field | | `dashboard.validation.name_invalid` | 400 | the dashboard name fails the folder grammar (`details.reason`: `folder_required` / `grammar` / `missing` / `immutable`) | | `dashboard.validation.new_root_requires_confirmation` | 400 | agent surface only — a dashboard under a top-level folder that holds nothing yet, without `confirm_new_root: true` | | `dashboard.validation.name_taken` | 409 | the name exists in the workspace — discarded dashboards included, names are unique forever | | `dashboard.validation.duplicate_name` | 400 | two objects of one dashboard — visualization occurrences, groups, actions, action controls, sources and the pinned set's parameters — share a name, or a name is outside `[a-z][a-z0-9_]{0,63}` (`details.reason` `grammar`); ONE namespace per dashboard, across types | | `dashboard.validation.unknown_object` | 400 | a reference to nothing: a group member, action target, action control's action, layout placement, grid item, parameter scope or override naming no object, source or pipeline parameter of this dashboard | | `dashboard.validation.target_not_visualization` | 400 | an action's target names an object that is not a visualization occurrence | | `dashboard.validation.empty_targets` | 400 | an action with `scope: targets` and an empty or missing `targets` list, or `scope: all` carrying targets (D55) | | `dashboard.validation.too_many_invocations` | 400 | the dashboard's sources resolve to more DISTINCT executions (a pinned pipeline release with its resolved bindings, outgoing overrides applied — the refresh's sharing identity) than one refresh may run (`datapipelines.dashboards.admission.max-executions-per-refresh`, default 16); `details.invocations` and `details.max`. Refused at save and at release so no valid document is permanently `dashboard.refresh.saturated` | | `dashboard.validation.parent_action_binding` | 400 | an action control bound to a PARENT parameter's control — a parameter whose `dependents` is non-empty in the pinned set (R2, D42) | | `dashboard.validation.scope_omits_consumer` | 400 | a declared `parameter_scopes` entry that omits a group consuming the parameter, directly or through a dependent (D41) | | `dashboard.validation.source_not_released` | 400 | a source pins a pipeline version that is not RELEASED (D1) | | `dashboard.validation.source_not_read_only` | 400 | a source pins a pipeline release that fails the read-only rule, transitively through its child pipelines (D38) | | `dashboard.validation.parameter_unbound` | 400 | a REQUIRED parameter of a source's pinned pipeline release is bound neither to a set parameter nor to a literal; or a `{parameter}` binding names no parameter of the pinned set | | `dashboard.validation.input_unbound` | 400 | a named input of an occurrence's pinned visualization is mapped to no source, or a mapping names an input the visualization does not declare | | `dashboard.validation.input_contract_mismatch` | 400 | a source's caller output columns do not satisfy the mapped input's columns by name and type; `details.column` names the first | | `dashboard.validation.layout_invalid` | 400 | the layout does not place every visualization occurrence, group and action control exactly once, places one outside the 12-column grid, or declares `columns` other than 12 | | `dashboard.validation.dependency_not_found` | 400 | a pinned visualization, pipeline or parameter-set version does not exist in the workspace (or only DISCARDED there) | | `dashboard.not_found` | 404 | no such dashboard (or version) in the workspace, hidden by the promoter lens, or discarded on a read/mutate path — indistinguishable by design | | `dashboard.version.conflict` | 409 | the precondition hash is stale — draft write, release, discard, purge; `details` carry the current hash, status and last writer | | `dashboard.version.not_draft` | 409 | release or a draft verb on a dashboard with no DRAFT | | `dashboard.version.not_released` | 409 | discard of a version that is not RELEASED | | `dashboard.version.not_discarded` | 409 | restore of a version that is not DISCARDED | | `dashboard.version.last_release` | 409 | purge of a released version, or of an entity holding one | | `dashboard.version.not_eligible` | 409 | switch to a version that is not live and posture-eligible | | `dashboard.release.dependency_not_released` | 409 | release with a pinned visualization version that is not RELEASED (without `release_pinned_visualizations` consent), a pinned pipeline version that is not RELEASED and read-only, or a pinned set version that is not RELEASED; `details.dependencies_not_released` lists every one until the dashboard participant moves the details to the shared release-blockers shape (#521) | | `dashboard.release.plan_conflict` | 409 | Reserved — raised once the dashboard release participant lands (#521). A consented snapshot fact changed between the dependency preview and the submit — an identity, status, body hash, current pointer, pin path or edge no longer matches the submitted `tree_hash`, even when someone else released the node meanwhile. `details` `{reason:"snapshot_changed"}` plus `node_id`/`path` only when visible; nothing flipped; reopen the preview. A root hash mismatch keeps `dashboard.version.conflict` ([Versioning §5.3](/docs/versioning.md#53-release-lock), #517) | | `dashboard.release.plan_required` | 428 | Reserved — raised once the dashboard release participant lands (#521). Cascade consent without a preview `tree_hash` — dependencies selected, or a legacy alias combined with a plan field, with no digest to compare. No dependency names in `details`; obtain a fresh dependency preview ([Versioning §5.3](/docs/versioning.md#53-release-lock), #517) | | `dashboard.release.plan_invalid` | 400 | Reserved — raised once the dashboard release participant lands (#521). The plan is malformed: a `tree_hash` that is not a digest, unknown or duplicate node ids, the root or a RELEASED node selected, an unsupported selection, or a legacy alias set to true together with explicit `include_dependencies` (two authorities). `details.field` is a bounded locator; no submitted value is reflected (#517) | | `dashboard.release.dependency_cycle` | 409 | Reserved — raised once the dashboard release participant lands (#521). The release closure repeats an exact node on its active path; `details.path` is the visible bounded node trail of that one cycle, never every expanded path (#517) | | `dashboard.release.dependency_limit` | 422 | Reserved — raised once the dashboard release participant lands (#521). The complete closure exceeds a configured bound ([Configuration §3.35](/docs/configuration.md#335-release-dependencies)); `details` `{limit, maximum}` with `limit` one of `depth`, `versions`, `edges`, `wire_bytes` — never a hidden total. Enforced while the closure is enumerated, so a truncated closure never authorises a release (#517) | | `dashboard.release.busy` | 409 | The workspace mutation guard's lock wait (`datapipelines.release.dependencies.lock-wait-seconds`) ran out — another pin-affecting dashboard write or release in the same workspace holds it. Nothing was written; retry with a current plan (#517) | | `dashboard.release.bundle_unsupported` | 409 | Reserved — raised once promotion carries full closures (#523). The promotion peer lacks the full-closure envelope capability; no bodies were transmitted. The family is the promoted root's (#517) | | `dashboard.import.id_taken` | 409 | an import whose exported id is held by another dashboard on this server — never re-issued (C29) | | `dashboard.import.missing_dependency` | 400 | an import whose pinned pipeline, parameter set or visualization version is absent on the target and not in the bundle — promotion order is templates → parameter sets → pipelines → visualizations → dashboards (D61) | | `dashboard.authoring.disabled` | 403 | `datapipelines.deployment.authoring-enabled=false` refuses every authoring write | | `dashboard.runtime.configuration_stale` | 409 | a runtime operation's `configuration_id` is not the current one — the client reloads (the spec's §8.1) | | `dashboard.runtime.dependency_missing` | 409 | a pinned dependency of the served version no longer resolves at config time (a purged or discarded source release) — never a 500 | | `dashboard.refresh.saturated` | 429 | a refresh could not reserve its execution slots within `admission.max-wait-seconds`; `Retry-After` set (D53). `details.reason` says which (#762): `busy` — the instance was full for the whole wait, retrying later may succeed; `never_admissible` — the refresh needs more distinct executions than this instance can ever admit, refused at once without waiting, with `details.invocations` (the count), `details.ceiling` and `details.key` (the smaller of `dashboards.admission.max-executions-per-refresh` and `executor.max-concurrent-executions-per-instance`, the configuration key to raise) — retrying cannot succeed until an operator raises it | | `dashboard.refresh.result_too_large` | 422 | a source's result exceeded `results.max-bytes-per-source` (or the refresh's budget); inside the stream it is a `visualization_status` reason (D54) | | `dashboard.refresh.not_found` | 404 | an abort or read of a refresh that does not exist, is not the caller's, or already finished | | `dashboard.key.kind_refused` | 403 | a `dashboard` API key presented outside the runtime and refresh routes, or on `/mcp` | | `dashboard.binding.path_invalid` | 400 | a dashboard binding's `name_prefix` is not a legal folder of the name grammar (1–9 segments, or the root `/`), or names no folder at or above a dashboard the CALLER's workspace has — the #191 rule: whether another workspace has dashboards at the prefix is exactly what the refusal must not reveal; `details.name_prefix`. Asked on the binding routes and the Keys page's binding editor (L5) | ### 13.24 Navigation trees The session-only navigation trees' refusals (#460, #464; [REST API §24](/docs/rest-api.md#24-first-party-navigation-trees)). The ten routes — `GET /api/v1//tree` and `/tree/search` for pipelines, templates, dashboards, visualizations and parameter sets — answered every refusal through Spring's `ResponseStatusException` until #464, so a client met the stand-in `pipeline.execution.aborted` with `details.reason: response_status` and could not tell a stale continuation (retry from the beginning) from an outage of the promotion target (retry later). Each refusal now carries its own code. Every message is static: no cursor, `root`, `parent` or `q` is echoed, and the 503 for an unavailable view carries no `details` (the target's address and the transport reason stay in the server's log). That view 503 answers `Retry-After: 5` (a constant) and is logged once at WARN without a stack: since #554 mounts the tree on every family page, a promoter whose target is down meets it on every page load, and that is someone else's outage, not a defect in this process. The budget 503 stays at ERROR with the stack, because one node over 1 MiB of metadata is a server-side defect. A search term over the shared bound remains `request.query_too_long` (§13.21). Listing liveness is the §3.2 derivation, not a soft-delete column (V19 dropped `pipelines.is_deleted` and `templates.is_deleted`): an artifact is listed through `current_version`, or through its newest DRAFT when no version is current, and only while that listed version is DRAFT or RELEASED. No pointer can name a DISCARDED version. In every family the one statement that discards recomputes the pointer when it named the discarded version, and every other pointer write checks the version it names in the same statement, all under the workspace writer guard (#517). An entity whose every version is DISCARDED is therefore absent from browse and search, exactly as the legacy `ENTITY_LIVE` rule leaves it. | Code | HTTP | Description | |---|---|---| | `navigation.request.invalid` | 400 | a malformed, tampered, foreign or stale continuation (a cursor is bound to family, mode, root, parent, query, workspace, caller and admitted view, and a process restart expires it), a `parent` outside its `root`, a `root` or `parent` with a control character, an empty or dot segment, or over 512 characters, or a search term with a control character. Static message, nothing echoed. The client retries from the beginning | | `navigation.view.unavailable` | 503 | the caller's promotion view cannot be computed (its promotion target's inventory could not be read), so the tree refuses rather than answer a completed empty level. Retryable: `Retry-After: 5`. Logged at WARN without a stack; no `details` | | `navigation.metadata.too_large` | 503 | one node's serialized metadata alone exceeds the 1 MiB response budget, so no page can carry it. Logged at ERROR with the stack (a server-side defect) | --- ## 14. Pipeline Lifecycle Operations This section sketches the CRUD operations. Full HTTP details are in the [REST API spec](/docs/rest-api.md). | Operation | Method & Path | Notes | |---|---|---| | Create pipeline | `POST /pipelines` | Body: pipeline JSON without `id`, `version`, `created_at`, `updated_at` (server assigns). Version 1 lands **DRAFT** and `current_version` stays null — releasing is a human action ([Versioning §3.6](/docs/versioning.md#36-the-one-write-rule-copy-on-write), D55). | | Get pipeline (working version) | `GET /pipelines/{id}` | Returns the working version: the DRAFT when one exists, else the latest release. | | Get pipeline (specific version) | `GET /pipelines/{id}/versions/{version}` | | | List pipeline versions | `GET /pipelines/{id}/versions` | Returns version metadata. | | Update pipeline (writes the draft) | `PUT /pipelines/{id}` | Body: full pipeline JSON. Copy-on-write: first change after a release creates a DRAFT of the next version; further changes overwrite the draft in place. Hash-preconditioned. Never appends a released version. See [Versioning](/docs/versioning.md). | | Release pipeline (lock) | `POST /pipelines/{id}/release` | Human/UI action: promotes the DRAFT to RELEASED, bumps `current_version`. Hash-preconditioned; requires pinned template versions released. The response carries `warnings: [{code, message, template, version}]` — empty on a clean release; `pipeline.release.template_needs_review` (§13.13) for every pinned version that cites a retired fact (7e). A warning never refuses. | | Discard draft | `POST /pipelines/{id}/draft/discard` | Deletes the DRAFT (or flips it to DISCARDED if an execution references it). | | Delete pipeline | `DELETE /pipelines/{id}` | Soft delete. Executions of deleted pipelines fail with `pipeline.execution.not_found`. | | List pipelines | `GET /pipelines` | Filterable by owner, datasource, etc. | | Execute pipeline | `POST /pipelines/{id}/execute` | Body: `{parameters: {...}}`, optional `version`. With no `version` it runs the **working version** — the draft when one exists, else the latest release ([Versioning §7.2](/docs/versioning.md#72-execute-with-no-version-runs-the-working-version-d56-099), D56). Returns SSE stream. See [REST API spec](/docs/rest-api.md). | | Import pipeline | `POST /pipelines/import` | Body: full pipeline JSON (possibly with id). A carried `version` is honored (preserved-version import, [Versioning §9.2](/docs/versioning.md#92-import-with-preserved-versions)); absent, the next local version is allocated. | | Export pipeline | `GET /pipelines/{id}/export` | Returns pipeline JSON + manifest of referenced templates. Refused with `409 pipeline.promotion.not_released` when nothing has been released — an export feeds an import that lands RELEASED. | --- ## 15. Stability Promise The Pipeline Contract is **versioned, additive-only**. ### 15.1 What is frozen in v1.1 - The top-level Pipeline JSON shape (`schema_version`, `id`, `name`, `nodes`, `settings`, etc.). - The Node JSON shape (`id`, `type`, `source`, `template`, `output`, `depends_on`). Gained the optional `pipeline` / `parameters` fields for PIPELINE nodes 2026-08-17 — additive per §15.2; readers that predate them ignore the keys. - The `type` enum: `DQL`, `DML`, `DDL`. `PIPELINE` added 2026-08-17 as an additive value per §15.2 — old documents remain valid; a deserializer that predates it rejects it with `type_invalid`. - The `output.target` enum: `tempdb`, `caller`, `datasource`. - The `output.mode` enum: `replace`, `append`. - The parameter descriptor shape. - The reserved literal `"tempdb"`. - The error code format and all error codes defined in §13. - The naming rules: **pipeline `name`** is the §3.2 path grammar (widened from `[a-z0-9_]+`, 1–63 in 2026-09-05 — see [Template Hierarchy §14.2](https://github.com/msabiransari/datapipelines/blob/main/docs/template-hierarchy-design.md) for the one shape that stopped being legal and why it needs no migration); **node ids, `output.table` and parameter keys keep the identifier rules they always had** (`[a-z0-9_]+`, 1–63, and `[a-z_][a-z0-9_]*` respectively) — 067 widened the NAME rule only. - The caller-node resolution rules (§9): omitted `output` → `caller`, at most one caller node, zero legal. ### 15.2 What is NOT frozen - New optional fields may be added (e.g., `tags`, `metadata`) — non-breaking. - New error codes may be added — non-breaking. - New `output.target` values may be added (e.g., `kafka`, `s3`) — non-breaking for v1 clients. - New `type` values may be added (e.g., `EXPRESSION`, `HTTP`) — non-breaking for v1 clients. - New `settings.tempdb.engine` values may be added (e.g., `DUCKDB`) — non-breaking. ### 15.3 Evolution rules - `schema_version` bumps on any non-additive change. - New versions are documented with migration notes. - Old pipeline JSON continues to load and execute under its original `schema_version` (we maintain backward-compatible readers). ### 15.4 Which version counter governs what Three independent counters exist across the system — none of them is "the version" alone: | Counter | Lives on | Governs | Bumped when | |---|---|---|---| | **Doc revision** (`v1.2` in this doc's header) | Each spec's Status line + Change Log | The prose contract | The spec text changes | | **`schema_version`** (integer field) | Pipeline JSON, Template JSON, and the result schema envelope — three **independent** counters that happen to share a field name | The JSON shape of that entity family | A non-additive change to that family's JSON shape | | **Entity `version`** (integer field) | Each pipeline / template instance | The content of one entity | Every save (immutable-per-version) | A `schema_version` bump in the Type System's schema envelope says nothing about Pipeline JSON, and vice versa — clients must interpret `schema_version` in the context of the document that carries it. --- ## 16. Worked Examples ### 16.1 Minimal pipeline: single-source read ```json { "schema_version": 1, "name": "acme/reporting/active_users", "display_name": "Active Users", "description": "List all active users.", "parameters": {}, "nodes": [ { "id": "active_users", "description": "Read active users from PG.", "type": "DQL", "source": "pg-prod", "template": {"id": "acme/reporting/active_users.sql", "version": 1}, "depends_on": [] } ] } ``` Single DQL node, no `output` block → resolves to `caller`. This is the caller node. ✓ ### 16.2 Two-source join (most common pattern) See §3.1 for the full example. ### 16.3 Pipeline with write-back ```json { "name": "acme/finance/monthly_revenue_etl", ... "nodes": [ {fetch_orders, DQL, pg-prod → tempdb stg_orders}, {fetch_customers, DQL, mysql-prod → tempdb stg_customers}, {revenue_by_customer, DQL, tempdb → tempdb int_revenue}, { "id": "write_to_warehouse", "type": "DQL", "source": "tempdb", "template": {"id": "acme/finance/select_revenue.sql", "version": 1}, "output": { "target": "datasource", "datasource": "pg-warehouse", "table": "monthly_revenue_cache", "mode": "replace" }, "depends_on": ["revenue_by_customer"] }, { "id": "return_report", "type": "DQL", "source": "tempdb", "template": {"id": "acme/finance/report.sql", "version": 1}, "output": {"target": "caller"}, "depends_on": ["revenue_by_customer"] } ] } ``` Two sinks: write-back side effect + caller return. Validation passes (exactly one `caller` target). ✓ ### 16.4 Pipeline with DDL index creation ```json { "id": "create_idx", "type": "DDL", "source": "tempdb", "template": {"id": "acme/finance/create_idx_revenue.sql", "version": 1}, "depends_on": ["revenue_by_customer"] } ``` Where the template is: ```ftl CREATE INDEX IF NOT EXISTS idx_int_revenue_customer ON int_revenue (customer_id) ``` A downstream JOIN node then benefits from this index when querying `int_revenue`. ### 16.5 Pipeline with DML audit record ```json { "id": "record_execution", "type": "DML", "source": "pg-meta", "template": {"id": "acme/finance/record_execution.sql", "version": 1}, "depends_on": ["return_report"] } ``` Template: ```ftl INSERT INTO pipeline_executions (pipeline_id, executed_at, status) VALUES ('${pipeline_id}', CURRENT_TIMESTAMP, 'SUCCESS') ``` Side-effect only. No `output` block. No data returned. ✓ --- ## 17. Implementation Notes (Non-Normative) ### 17.1 Where this lives in the codebase The Pipeline Contract is implemented in the `pipeline-contract` Gradle module: - `Pipeline` data class (top-level) - `Node` data class - `NodeType` enum (`DQL`, `DML`, `DDL`) - `NodeOutput` sealed interface (`Tempdb`, `Caller`, `Datasource` implementations — matches [DAG Executor §4](/docs/dag-executor.md#4-executor-facing-model)) - `PipelineSettings` data class (with `TempdbSettings` nested) - `Parameter` data class - `TemplateRef` data class (`{id, version}`) - `PipelineValidator` — runs all §12 validations - `PipelineSerializer` / `PipelineDeserializer` — Jackson-based JSON ser/deser; the deserializer resolves an omitted `output` on DQL nodes to `NodeOutput.Caller` - `ExecutionContext` — runtime mutable map (see §7) - `CallerNodeResolver` — implements §9 resolution - `PipelineRepository` — persistence via `NamedParameterJdbcTemplate` ### 17.2 Validation pipeline ``` JSON input → PipelineDeserializer (syntactic parse; omitted output → Caller) → PipelineValidator (semantic validation, §12 — includes template dry-render) → CallerNodeResolver (resolve the caller node via §9) → Pipeline entity (validated, ready to persist or execute) ``` Validation is exhaustive — all checks run, all failures collected, returned together (not fail-fast). This gives authors the full picture on a broken pipeline. A handful of §12 codes are necessarily raised at the DESERIALIZATION step rather than by the validator — verdicts on wire values that have no typed representation (`type_invalid`, `output_target_invalid`, `output_mode_invalid`, `parameter_type_invalid`, `tempdb_engine_unsupported`). They are still collected exhaustively across the whole document before binding fails, preserving the full-picture guarantee (2026-08-08). ### 17.3 Persistence Pipelines are persisted in the app's own Postgres metadata DB. The DDL authority is [Metadata DB](/docs/metadata-db.md) — tables `pipelines`, `pipeline_versions`, `pipeline_executions`. This spec does not restate column lists. --- ## 18. Open Questions / Future Additions Out of scope for v1.1, tracked for future: - **Calculators** (v2): pre-execution transformers that read Context and write additional keys (`quarter` from `date`, etc.). - **Non-SQL node types** (v2): `EXPRESSION` nodes (no SQL, script data in/out via expression language), `HTTP` nodes (call external API, stage response). - **Conditional execution** (v2): skip nodes based on a Context expression. - **Retries** (v2): per-node retry policies. - **Streaming** (v2): pipe rows between nodes instead of full materialization. - **Pipeline-level scheduling** (v2): declare cron-style schedules in the pipeline. - **Additional `output.target` values** (v2): `kafka` (publish to topic), `s3` (write object), `email` (send report), `webhook` (POST result). - **Auto-create target table for write-back** (`output.auto_create: true`) — v1.1 candidate. - **Additional `settings.tempdb.engine` values** — DuckDB in v1.1 or v2. --- ## Appendix A: Change Log | Date | Version | Author | Change | |---|---|---|---| | 2026-11-01 | v1.69 | 779 (#779) | §4.9, §8.5 and §12.9: literal and scalar-reference mappings onto child value-mode SCRIPT keys share the existing caller-override contract; SCRIPT targets are ANY at save and checked by the child's pinned output contract at run. Object/list references remain refused. | | 2026-11-01 | v1.68 | 762 (#762) | §13.23's `dashboard.refresh.saturated` row: additive `details.reason` (`busy` / `never_admissible`) and, for `never_admissible`, `details.invocations`, `details.ceiling` and `details.key`. Code, status and `Retry-After` unchanged; the §13.23 row count is unchanged. | | 2026-10-27 | v1.67 | 464 (#464) | New **§13.24 Navigation trees**: `navigation.request.invalid` (400), `navigation.view.unavailable` (503, `Retry-After: 5`, logged at WARN without a stack) and `navigation.metadata.too_large` (503) replace the stand-in `pipeline.execution.aborted` on the ten session-only tree routes ([REST API §24](/docs/rest-api.md#24-first-party-navigation-trees)); the prose states the listing liveness rule and the pointer invariant it relies on. Landed in the SAME commit as the `PipelineErrorCodes.Navigation` constants and the `ApiErrorCatalog` rows. | | 2026-10-25 | v1.66 | 706 (#706) | §13.7's `auth.role_required` row: `details.held` for an MCP key reads "its OWN role chosen at creation" (was "member role capped at author"); the `auth.key_issuer_role_lost` row now states the code's remaining meaning — the context-judged promotion path ([Auth §9](/docs/auth.md#9-auth-errors)) — and drops the retired PK4 cap and the TTL-demotion sentence. Catalog text only: no code added, removed or renamed, so every count stands. | | 2026-10-24 | v1.65 | #517 release dependencies, core (S1) | **§13:** the release-plan codes in each family's subsection (record §14: `dependency_not_released` replacing the template-only refusal code for pipeline and parameter, `plan_conflict`, `plan_required`, `plan_invalid`, `dependency_cycle`, `dependency_limit`, `busy`, `bundle_unsupported`; the families whose participant lands later carry Reserved rows). **§12.9:** a release that flips a DRAFT child validates in authoring mode outside the transaction and strictly inside it. Round three (#719): **§12.6 / §12.9 / §13 (the template, §13.13 lifecycle and parameter rows)** — a pipeline's and a parameter set's restore re-judge their outbound pins (missing/DISCARDED → the validation code, `reason: discarded` on a discarded template version; DRAFT → `dependency_not_released`, the shared shape's new reason `not_released`). | | 2026-10-25 | v1.64 | #711 — renumbered at merge after #442's v1.63 | §4.12 names the random-source boundary: JSONata `$random()` / `$shuffle()` and JavaScript `Math.random()` refuse with `pipeline.script.evaluation_failed`. | | 2026-10-25 | v1.63 | 442c (#442) | §13.19: schedule.notification.backlog_full (409), with details.pending and details.backlog_max; refused resend leaves its delivery unchanged. | | 2026-10-24 | v1.62 | 442b (#442) | **§13.19 gains two notification codes**: `schedule.notification.not_found` (404) and `schedule.notification.not_resendable` (409, the `schedule.*` state family) — the explicit resend of a run's notification ([REST API §20.13](/docs/rest-api.md#2013-resend-a-notification)). Landed with their `ScheduleErrorCodes` constants, the `PipelineErrorCodes.Schedule` mirror and the 404 catalog row. | | 2026-10-22 | v1.61 | 8b (#8) | JavaScript is available through the shared seam; script.js.unavailable means an absent isolate, never boot 503; resource_limit covers the byte and output limits. | | 2026-10-21 | v1.60 | #640 the script-node rename (script-node-rename lane, V50, owner ruling 2026-10-20) | **The TRANSFORM node type is SCRIPT**: the `type` enum, §4.12 "JSON structure (SCRIPT node)", §12.13 "Script validation rules", §13.18 "Script" and its codes `pipeline.script.*`, the node fields' prose (`template`, `inputs`, `strict`, `rejects`) and every cross-reference anchor that carried the old word. A rename only: no behaviour, route, permission or key change beyond the names, and no alias (the ruling). The visualization block's `transform` key, `transform.template` / `transform.inputs` and the CSS/SVG sense keep their word. | | 2026-10-16 | v1.59 | 467 (#467) the over-cap screenshot refusal reaches the client | **§13's `visualization.test.screenshot_too_large` row** names the bounded drain: up to the cap + 2 MiB of a declared-over-cap body is read and discarded after the 413, so the close is a FIN after the client's last byte; beyond it Tomcat's swallow and close apply. No code or status change | | 2026-10-11 | v1.58 | 504b (#504b) one shared search-needle rule | **§13.21 gains `request.query_too_long`** (400); **§13.20 loses `parameter.validation.query_too_long`** — the #490 refusal becomes the shared `SearchNeedle` rule applied by every search-owning service (pipelines, templates, parameter sets, users, the five `/tree/search` reads) at the same 200-after-trim bound, `details.limit`/`details.length`, the needle never echoed; MCP answers it as `-32602` (§6.2.44 v1.74) and the UI boxes still truncate, never refuse. Landed in the same commit as its constant, the `ApiErrorCatalog` rows (400 + user message), and the doc re-points (§13 row count stays 377: one row out of §13.20, one into §13.21). | | 2026-10-09 | v1.57 | 487 (#487) a replay-log read fault is not an expiry — renumbered at merge after #490's v1.56 | **§13.10** `result.storage_unavailable`: **500 → 503** (owner's ruling, 2026-10-09) — a transient fault of the server's own store, like `pipeline.transform.pool_exhausted`. The row also names the read surface it gains: §10.3's replay answers a log read Redis did not answer with this code (`details.reason: event_log_unavailable`) instead of `410 result.expired`. A deliberate status change on a frozen contract: the result-write path's code and behaviour are unchanged (it fails the execution; no HTTP status), and every REST or UI read whose stored result header cannot be parsed — the result cursor, `GET /executions/{id}`, a published endpoint's serve, the execution detail page — now answers this code with 503 where it answered 500. | | 2026-10-09 | v1.56 | 490 (#490) the search needle's bound | §13.20 gains `parameter.validation.query_too_long` (400): the parameter-set listing search refuses a `q` over 200 characters after the trim on REST, and on MCP as `-32602`, `details.limit`/`details.length`, the needle never echoed; the bound is `ParameterSetService.MAX_QUERY_LENGTH`, the number the UI boxes already truncated to. Landed in the same commit as its constants and catalog row (§13 row count 376 → 377). | | 2026-10-06 | v1.55 | #462 draft purge pins | §13 pinned-error row states sole-draft dashboard history scope and refusal before destructive writes. | | 2026-10-06 | v1.54 | 459 (#459) + merge follow-up | **§4.9 / §12.9 / the error table**, written by the lane: a PIPELINE node may pin a live DRAFT child while the parent is a draft, and release and import still require RELEASED children (`pipeline_reference_not_released`). Composition verdicts describe a draft child's current body, and release revalidates them. The follow-up adds this row and the bump. The error row's old reason ("a DRAFT child can be purged out from under its parent") still holds for drafts: the pipeline draft purges have no pin guard yet (versioning §3 rule 3, #462). | | 2026-10-04 | v1.53 | #382 merge follow-up | **§13.21** — the strict-reader sentence names five routes that bind a typed request DTO, `POST /dashboards/bindings` added (#382 moved it onto the reader; #382 review F1). No code, status or error shape changed. | | 2026-10-03 | v1.52 | 442a (#442) | Saved schedule notification settings: recipients and five event choices, validated and shown by role; omitted PUT settings preserved; mail remains off pending part b. | | 2026-10-02 | v1.51 | 328 (#328) the release records its caller node's result columns | New **§3.3.1** — the source of truth for the caller-output record: the flip copies the version's latest qualifying execution's `result_schema_json` into `pipeline_versions.caller_output_json` (D1, in the flip's own statement); the D2 `{name, type, nullable}` shape with the 256-column / 128-character record bounds and the JSON-null-reads-`true` rule; the D3 word (`recorded` \| `declared` \| `none` \| `not_observed`) the response and audit row carry — never the column list; the D4 precedence (the record never overrides a transform caller's declared contract) and the promotion carry (validated at the receiver, stored on the imported release). No new error code; the body, the hash and §12.12 are untouched. | | 2026-10-02 | v1.50 | 333 (#333) the strict contract readers | **§13.21 gains the scalar-shapes paragraph**: the pipeline, template, transform-block and DTO readers refuse a JSON number or boolean where a string is declared (and a string or float where an integer is declared) with the path and the expected shape, never the value. `pipeline.validation.schema_version_unsupported` is also the pipeline reader's wrong-type answer (`details.reason: "wrong_type"`); `template.contract_invalid` gains `details.rule: "wrong_type"`. No new code, no status change. | | 2026-10-01 | v1.49 | L5 (#367, #10) the `dashboard` key kind | New **§13.23 row `dashboard.binding.path_invalid`** (400): a dashboard binding's `name_prefix` is not a legal folder of the name grammar (1–9 segments, or the root `/`), or names no folder at or above a dashboard the CALLER's workspace has (the #191 non-disclosure rule). The refusal the binding routes and the Keys page's dashboard binding editor answer; the reserved `dashboard.key.kind_refused` stays unused (the surfaces answer the one catalogued `endpoint.key_kind_refused`, whose `details.reason` names the kind). §13.23 is 36 rows. | | 2026-10-01 | v1.48 | L4b (#353) stale text after the gate landed | **§13.22, no new code:** `visualization.release.tests_missing` no longer says the default gate refuses every release (#352 installed the gate — it names `no_runs` / `run_open` / `no_cases`), and `visualization.test.session_expired` names the shipped key `datapipelines.visualization.session-ttl-minutes`, not the spec's unshippable `tests.session-ttl-minutes`. | | 2026-09-29 | v1.47 | L2 (#10) the dashboard runtime — renumbered at merge after 320's v1.45 and 264's v1.46 | New **§13.23 row `dashboard.validation.too_many_invocations`** (400): the validator counts a dashboard's DISTINCT invocations — the refresh's own sharing identity: a pinned pipeline release plus its resolved bindings, outgoing overrides applied — and refuses more than `max-executions-per-refresh` at save and at release. The reader bounded `sources[]` at 400 (50 × 8) while a refresh admits 16 executions, so a document valid at save could be permanently `dashboard.refresh.saturated` (429 says "retry"); the runtime's saturated answer stays for a genuinely full instance. §13.23 is 35 rows. | | 2026-09-29 | v1.46 | 264 (#264) parameter values at every entry — renumbered at merge after 320's v1.45 | **§12.9's save answers tighten (#264)** — REST API v2.58. A `PIPELINE` node's parameter literal is judged by the child parameter's WHOLE declaration (the shared validator, §12.7's `checkDefault` mould): a value that coerces but breaks the child's `constraints`, length, pattern or declared precision/scale is refused at save — and at release and import, which run the same composition rules — with the new `pipeline.validation.pipeline_parameter_invalid` (§12.9; `details.reason`, never the value text), the composition twin of `default_invalid` and of the run-time `parameter_constraint_violation`. A same-type `${ref}` is accepted only when the parent PARAMETER's descriptor widens losslessly into the child's (parameter-engine record §6.4; `reason: narrowing`); tiers without a descriptor (calculator `context_key`s, org/platform keys) stay type-only, and a cross-type pair stays `pipeline_parameter_type_mismatch` — widening across types is NOT adopted (the run's wire encoder re-encodes by the child's type). A child declaration today's own rules refuse falls back to the type-only check. Existing stored bodies keep running; a next save re-judges (the shipped demos and every §16 example replayed: 12 bodies, 3 composition mappings, 0 refused — `CompositionRulesReplayTest` pins the corpus). | | 2026-09-30 | v1.45 | 320 (#320) dependency guards | **§13.20 gains `parameter.in_use`** (409 — the consumer binding the record reserved): a dashboard that pins a set version refuses its purge and discard. `pipeline.version.pinned` names the dashboards whose sources pin the release (`details.referencing_dashboards`, beside `pinned_by`). `template.in_use`'s row is rewritten — it named `referencing_pipelines` and `references`, stale since C32 — to what the code answers: `pinned_by`, `referencing_parameter_sets` and, new, `referencing_visualizations`, on every verb that removes a version. | | 2026-09-29 | v1.44 | L1a (#10) the visualization module | New **§13.22 Visualizations** (30 rows) and **§13.23 Dashboards** (34 rows): the spec §14's 21 + 24 codes (its "18" and "22" miscount its own lists) plus the lifecycle rows the verbs need — `name_invalid`, `name_taken` (409; the mould's `duplicate_name`, renamed because the dashboard family's `duplicate_name` is its object namespace), `version.not_draft` / `not_released` / `not_discarded` / `last_release` / `not_eligible`, `authoring.disabled`, and `visualization.version.pinned` (a live dashboard pins it), `dashboard.validation.dependency_not_found`, `dashboard.import.missing_dependency`. Landed with their constants, catalog rows and `doc_url` anchors. | | 2026-09-29 | v1.43 | 310 (#310) audit-log retention | §12.11's `pipeline.validation.table_not_learned` row: the learning lasts as long as the audit row it reads — once per `datapipelines.audit.retention-days` (default a year), not "the key's lifetime", now that the audit log is retained (auth §10.3). The code, its status and its details are unchanged | | 2026-09-28 | v1.42 | 298 (#298) the server's fault names the status | §6.2: a binding that carries both a caller's bad value and a stored declaration refused today answers the stored declaration's 409 (`PipelineValidationException` ranks the server-owned code first) with both failures in `details.failures` in the order found — before, the first failure's code won, so a caller's error listed earlier made it a 400. Each judged declaration is compiled once per bind. No code or row added. | | 2026-09-28 | v1.41 | 298 (#291) request constraints on every body | §13.21: the stated Jackson constraints now hold on every route that parses its own `@RequestBody String` too — each reads through a REQUEST copy of its domain mapper (`RequestLimits.requestMapper`, `copy()` then the copy's own factory: the domain mappers keep the library's bounds for stored rows, transform values and JSON columns) and answers a read failure with its family's malformed-body 400; the parameter-set routes remain, tracked in #300. No code added. | | 2026-09-28 | v1.40 | 286 (#268) — renumbered at merge after 279's v1.38 and 278's v1.39 | **§13.3** gains `pipeline.execution.parameter_declaration_invalid` (409) and **§6.2**'s one-validator rule says when it answers: a stored declaration save refuses today — a body saved before #194 with a `constraints` block or a `cardinality` that `Parameter` then ignored — judged by a value or a default. It used to reach the validator's `IllegalArgumentException` (a 500); a stored `MULTI` used to bind as a list. Nothing changes for a valid declaration, or for a broken one nothing judges. The pre-deploy scan for such bodies is in deployment §8.3. | | 2026-09-28 | v1.39 | 278 (#278) the BIG-number digit cap | §6.3: a textual `BIGINTEGER`/`BIGDECIMAL` value over **1024 digits** is refused unparsed — `too_many_digits` under `invalid_parameter_type` — before the O(n²) JDK 21 parse can spend seconds of CPU on a megabyte of digits. The cap is `ParameterCoercion.MAX_NUMERIC_DIGITS`, injected through `ParameterValueLimits.maxNumericDigits` (the P34 shape), so the execute API, `pipelines_execute`, a parameter set's textual `default_value` and an expression literal all inherit it. No new code, no catalogue row. | | 2026-09-28 | v1.38 | 279 (#279) the request-body cap | New **§13.21 Request limits**: `request.body_too_large` (413) — the one transport-level code, raised by the platform's body-cap filter on `/api/v1` and `/mcp` before authentication and before any parser. Jackson's `StreamReadConstraints` are stated explicitly on every request-body mapper (nesting 100, string 4M chars, number 1000 digits). Landed in the SAME commit as the `PipelineErrorCodes.Request` constant, the `ApiErrorCatalog` rows and the `AuthErrors` anchor row. | | 2026-09-27 | v1.37 | #280 the schedule form says what is wrong | **§13.19** gains `schedule.validation.target_not_released` (400, the `schedule.validation.` family default) — the payload names a pipeline with no current version to follow, release it or switch its current version then save again (`details.pipeline`). It was a `payload_invalid` / `details.reason=pointer_null` refusal before, one code shared with the shape and size refusals, so a static per-code user message could not say "release it"; `payload_invalid`'s description loses that clause and keeps the shape reasons. The save's `message` keeps its sentence; the run side's `pointer_null` reasons are untouched. Landed in the SAME commit as the constants in `ScheduleErrorCodes` and `PipelineErrorCodes.Schedule` and the executor that raises it. | | 2026-09-27 | v1.36 | 194c (#194) parameter engine lane C — the selector runtime | **§13.20** loses `parameter.validation.selector_probe_unavailable`, the one code lane B documented as temporary: the selector runtime (`SelectorRunner`) implements the save-time probe and `ParameterSetValidator` now REQUIRES it, so a template-backed source is always proven by a real dry run. The constant, its `PipelineErrorCodes.Parameters` mirror and this row leave in one commit; the §13 row count moves 305 → 304 on this lane's base. The evaluate rows say what the runtime taught: `selector_rows_invalid`'s further reasons (`value_too_long`, `label_too_long`, `columns`), a sourced `INPUT` row under `constraint_violation` / `selector_value_type_mismatch`, and the section's intro names the other families' codes a failed selector reports on its parameter. No code or status changed beyond the retirement. | | 2026-09-26 | v1.35 | 194b (#194) parameter engine lane B — the frozen model | New **§13.20 Parameter sets**: the `parameter.*` family, 81 codes, one row each — the record's §10 plus the owner's three rulings of 2026-09-26/27 (`parameter.validation.body_invalid` for the document's shape at every level, `parameter.validation.selector_query_failed` when the save-time probe's statement fails, `parameter.validation.input_source_multiple_rows` when a database-fed input's dry run returns two rows) and the lane's temporary `parameter.validation.selector_probe_unavailable` (gone with lane C). The evaluate's per-parameter codes are "—" (never a response status; reported in `state.errors[]` of a 200); `parameter.evaluate.timeout` is 504 (the house deadline mapping, §13.3 — the record's §14 item 1). Landed with `ParameterErrorCodes`, its mirror `PipelineErrorCodes.Parameters` and the `ApiErrorCatalog` rows. | | 2026-09-26 | v1.34 | 194a (#194) parameter engine lane A — the shared validator | §6.1: a parameter gains optional **`constraints`** (`min`, `max`, `min_length`, `max_length`, `pattern`) and **`cardinality`** (`SINGLE`/`MULTI`), the parameter engine's declaration model; both omitted when absent, so no stored body's hash moves. §6.2: every value (and every applied default) is judged by the shared `ParameterValueValidator` with ONE null policy — `null` and absent are unsupplied and resolve as before; **precision and scale are enforced on values** (the second deliberate break, REST API v2.37); `MULTI` is refused until the dashboard round. §12.7 gains `default_invalid`, `constraint_not_applicable`, `constraint_invalid`, `pattern_invalid`, `cardinality_unsupported`; §13.3 gains `pipeline.execution.parameter_constraint_violation` (400, `details.reason`). Additive per §15.2 except the precision/scale enforcement, which is named as a break. | | 2026-09-26 | v1.33 | 194a (#194) parameter engine lane A | §6.3: **nothing is trimmed** — a `BIGINTEGER`/`BIGDECIMAL` value with surrounding whitespace is `pipeline.execution.invalid_parameter_type`; the two BIG types were trimmed before parsing until today, a deliberate break recorded in REST API v2.36. `ParameterCoercion` and `ParameterWireEncoder` moved to `typesystem` (made public) so the parameter engine shares the one implementation; the rejection wording is unchanged. | | 2026-09-26 | v1.32 | scheduler lane 1 (#9) — numbered after origin/main's v1.31 (232) | New **§13.19 Schedules**: sixteen `schedule.*` codes — `schedule.not_found` / `schedule.run.not_found` (404), the 409 state family (`name_taken`, `revision_conflict`, `run.overlap`, `blocked`, `not_blocked`, `limit.per_workspace`) and the 400 `schedule.validation.*` family (`request_invalid`, `name_invalid`, `cron_invalid`, `timezone_invalid`, `interval_too_short`, `executor_unknown`, `payload_invalid`, `target_not_found`). None is raised while a schedule fires. Landed with `ScheduleErrorCodes` and their `ApiErrorCatalog` rows. | | 2026-09-25 | v1.31 | 232 (#232) the limiter's own sentence | §13.11's `rate_limit.exceeded` row now states WHICH user message answers on which surface: the catalog default (the request-volume sentence) for every API surface, the sign-in sentence only from the login damper. Code, status and `details` unchanged; no new code, no catalog row count change. | | 2026-09-25 | v1.30 | 7e (#7) the semantic link | §13.9 gains `template.implements_unresolved` (400 — an `implements` entry that is not a WORKSPACE `definition`/`exclusion`/`preference` fact visible from the writing workspace; not-found semantics) and the `template.blocks_not_allowed` row names `implements` beside the three blocks. §13.13 gains `pipeline.release.template_needs_review` with HTTP `—`: a WARNING in the release response's new `warnings` array, never an error status (the §13.6 type-mapping shape; `ApiErrorCatalog.NEVER_RETURNED_LIVE`). §14's release row names the response's `warnings`. Landed in the same commit as the constants, the catalog rows and the drift counts (§13 row count 202 → 204). | | 2026-09-24 | v1.29 | 7c (#7) the TRANSFORM node | `NodeType` gains `TRANSFORM` (§4.6 lists six). §4.12: the node — a pinned `jsonata`/`javascript` template evaluated as a pure function over staged data, `inputs`/`output`/`strict` conforming to the pinned contract (R3–R5), the implicit-optional-input rule for a value-mode `context_key`, and R4's one-reader rule for object keys. §4.7: the `rejects` companion of a tempdb output; §7.2: object keys persist as jsonb and read as TRANSFORM-only; §9.1: a `row`/`table`-mode TRANSFORM may be the caller node. §12.13 gains the ten `pipeline.validation.transform_*` rows (the §13 row count and §12 code count move in the same commit as the constants) and §13.18 the `pipeline.transform.*` execution-time family plus `transform.js.unavailable` — 7b's privately declared constants unified here and deleted. | | 2026-09-24 | v1.28 | 215a (#215) the permission catalog | §13's `auth.role_required` row: `details.required` is a catalog permission (`pipeline.update`) and `details.held` the role the caller was judged as. No code added, removed or renamed — the catalogued set and every drift count are unchanged. Status caught up with the change log (it read v1.24). | | 2026-09-23 | v1.27 | 7b (#7) transform template types | §13.9 gains the transform rows: `template.validation.freemarker_forbidden` (D-T9 — `imports`/`is_library`/a Freemarker construct on a transform type), `template.contract_invalid` (the §2.2 model rules, `details.rule` naming which), `template.invariant_invalid`, `template.test_failed` (the save/release suite), `template.blocks_not_allowed` (the blocks on `sql`/`html`) and `template.render_not_applicable` (`templates_render` on a transform type). The existing `type_invalid` / `dialect_not_allowed` / `engine_unsupported` rows' RULES changed: the type vocabulary is now four values, `dialect` is forbidden outside `sql`, and `engine` is type-conditional (`none` iff a transform type). Landed in the same commit as the constants, the catalog rows and the drift counts (§13 row count 183 → 189). The §13.18 Transform section is 7c's; the codes the runner emits until then are declared privately (the 7b handback names them). | | 2026-09-19 | v1.26 | 172 (#172) | §13.14 gains `endpoint.path_reserved` (400): a published path's category segment matching `v[0-9]+` or equal to `api` is refused, `details.segment` naming it — the companion of the old fixed-letter root's retirement (REST API §19: endpoints now serve at `/api///`). Checked at publish and re-checked when the serve registry is built. Landed in the same commit as its `PipelineErrorCodes.Endpoint.PATH_RESERVED` constant. | | 2026-09-18 | v1.25 | 171 (#171) | `endpoint.pipeline_not_readonly` (§13.14) now judges a `DML`/`DDL` node by `source`, not type alone: `source: tempdb` is allowed (the execution's own in-memory H2, discarded with the run — a statement there cannot outlive the request), any registered datasource stays refused, named in the message. Applies transitively through `PIPELINE` children, at publish (409) and re-checked at serve (503), through the one rule (`ReadOnlyPipelineRule`). No new code — the existing one now covers a case it previously over-refused. | | 2026-09-17 | v1.24 | 158 (#129) | §13.8 gains `datasource.validation.lake_no_healthy_tables` (400): a LAKE pool build whose EVERY registered table was refused at the SQL-emission boundary is refused outright — nothing would be queryable, and the pre-fix shape failed the same build cryptically (a `SET search_path` naming a schema that was never created) on the first physical connection. The message names every refused table and its reason; each refusal is recorded on its registry row. Additive per §15.2; landed in the same commit as its `DatasourceErrorCodes` constant. | | 2026-09-17 | v1.23 | 156 query timeout settings (#2) | Pipeline and node SQL statement timeout overrides. New **§4.11** field `node.settings.query_timeout_seconds` and new **§5.3** `settings.query_timeout_seconds` (Future settings renumbered §5.3 → §5.4); precedence node > pipeline > datasource > per-dialect operator default ([configuration.md §3.2](/docs/configuration.md)) > `node-query-timeout-seconds`, bounded by the same ceiling as `node.settings.timeout_seconds` (`datapipelines.executor.node-query-timeout-max-seconds`, default 900). New §12.8 codes `pipeline.validation.pipeline_query_timeout_invalid` and `pipeline.validation.node_query_timeout_invalid` (refused on a non-SQL node type or when the statement budget exceeds the node's own wall-clock deadline, both refused rather than clamped). §13.4's `pipeline.node.query_timeout` detail gains `source` naming the resolved tier. Body-hash neutral: both keys are absent on every stored pipeline/node and serialize back absent. Additive per §15.2. | | 2026-09-15 | v1.21 | 139 the entry-point checks | **§12.11 (new)** — two MCP-surface save-time gates: `pipeline.validation.table_not_learned` (a saved template names a listed table the calling key never `datasources_get_columns`'d; audit-row read, key-lifetime window, `${…}` and `tempdb` exempt) and `pipeline.validation.door_unacknowledged` (a RAW_DATE_PAIR body refused until `door_acknowledged: true`, the `confirm_new_root` shape). **§13.3** — `pipeline.execution.template_unrendered` (400): execute of a DRAFT whose pinned DRAFT template postdates the key's last successful render. All three MCP-only: they read the caller's own `mcp.tool.called` audit rows. REST and the UI unaffected. Additive per §15.2; both validation codes live in §12.11, and `PipelineErrorCodesSpecDriftTest` pins both sides. | | 2026-09-15 | v1.22 | 140 | Release checks: optional §3.3 `checks[]` — server-run read-only statements with an expected value, versioned with the body like `nodes`, body-hash neutral; the agent supplies the query and expectation, never an observed value. §3.2 gains the field row; new §12.12 (`pipeline.validation.check_invalid`, one code for every declaration defect) and new §13.17 (`pipeline.check.failed`, 409 — release refused on a failing check, overridable with a reason). Additive per §15.2. | | 2026-09-14 | v1.20 | 136 §B refs optional on a WORKSPACE rule | §13.15 `semantics.fact_invalid`: the empty-`refs` arm is now DATASOURCE-scope only — a WORKSPACE-scope `definition`/`exclusion`/`preference` may carry no refs (a rule that spans datasources is recorded once, against the datasource the question is mostly about; V26 relaxes `chk_learned_facts_refs` the same way). No new code. | | 2026-09-13 | v1.19 | 123 table-not-found | §13.8 gains `datasource.table_not_found` (404) and `datasource.table_forbidden` (403) — the introspector's three-state table resolution (datasources.md §7A): every table-addressed read (columns / table stats / preview rows) resolves the table against the namespace's catalog listing first; absent is a 404 whose message says whether the dialect's catalog is complete or privilege-filtered and names the nearest listed table, present-but-unreadable (a permission SQLSTATE on the read) is a 403. An unknown table is no longer an empty answer; an existing table with zero readable columns stays one. `datasource.lake_table_not_found` is unchanged — a registry fact, not a catalog one. Additive per §15.2. | | 2026-09-13 | v1.18 | 121 calculator multi-output | A CALCULATOR kind may declare a **named output set** (§4.10, both JSON shapes): a single-output kind writes its value under `context_key` exactly as before; a multi-output kind (the catalog says which) maps EVERY declared output through `context_keys` — never both fields, never neither, no partial mapping. §12.10 gains `calculator_output_shape_mismatch`, `calculator_output_unknown` (`details.known_outputs`) and `calculator_outputs_incomplete` (`details.missing`); §13.3 gains `pipeline.execution.calculator_keys_partial` — caller override of a multi node's keys is all-or-nothing, a proper subset refused before any node runs with the same shape as `invalid_parameter_type`. Every mapped key obeys every rule a single key always has (name, collision, ordering, the §12.6 guarded set, derived inputs) and is typed by its own output. Body-hash neutral: `context_keys` is absent on every stored single-output body and serializes back absent; a single node's stats row is byte-identical, a multi node's carries `context_values`. Additive per §15.2. | | 2026-09-11 | v1.17 | 118 learned semantic layer | New **§13.15 Learned semantics** — seven two-segment `semantics.*` codes (`kind_invalid`, `fact_invalid`, `ref_unresolved`, `evidence_refused`, `evidence_failed` — all 400; `duplicate` 409; `not_found` 404), landed with `PipelineErrorCodes.Semantics`, the `datasources` mirror `SemanticsErrorCodes` (the recorder lives below this module) and their `ApiErrorCatalog` rows; §13 rows 163 → 170. | | 2026-09-10 | v1.16 | 109 §B empty dialect properties | §13.8 gains `datasource.validation.property_empty` (400): a DECLARED dialect property carrying an empty, whitespace-only or null value is refused at register/update — and so at bootstrap, which saves through the same validator — instead of being stored as `""` (the `catalog.ref: ""` incident). The field names the key; a bootstrap field whose whole value is one `${VAR}` that resolves set-but-empty is OMITTED (the operator's off-switch), so the shipped defaults still boot. Additive per §15.2. | | 2026-09-10 | v1.15 | 109 §A lake view isolation | §13.8 gains two rows: `datasource.lake.table_unavailable` (502) — a pipeline node referenced a registered lake table whose connect-time view creation is recorded as failed (`lake_tables.last_error`, V20); `details` carry `table` and the recorded `last_error` — and `datasource.validation.lake_table_unreadable` (400) — the registration/import pre-flight refusal: the candidate table's view did not create or a one-row scan through it failed, refused BEFORE storing with the bounded engine error as the message. Additive per §15.2. | | 2026-09-09 | v1.14 | 108 executor hardening | New §4.11: a node may declare its own WALL-CLOCK deadline, `settings.timeout_seconds` — the middle of three budgets whose precedence §4.11 now states as one table (execution ≥ node ≥ statement). §4.6 gains the `settings` row and states that `depends_on` is data flow only, never an edge added to avoid contention. §12.8 gains `pipeline.validation.node_timeout_invalid` (the ceiling is `datapipelines.executor.node-timeout-max-seconds`, refused rather than clamped) and §13.4 gains `pipeline.node.timeout` (504) — the executor's own bound, which fires whatever the driver does. Body-hash neutral: `settings` is absent on every stored node and serializes back absent. Additive per §15.2. | | 2026-09-09 | v1.13 | T202 node query timeout | §13.4 gains `pipeline.node.query_timeout` (504): a statement cancelled by its own JDBC query timeout reports the timeout, not `query_execution_failed` + driver text. | | 2026-09-08 | v1.12 | 099 draft-first (D55/D56) | §14's operation table: `POST /pipelines` lands v1 **DRAFT** with a null pointer, `GET /pipelines/{id}` is the working version, `POST …/execute` defaults to the working version, and `GET …/export` refuses a never-released pipeline with `pipeline.promotion.not_released`. No new error code and no §13 row: every refusal reuses a catalogued one. | | 2026-09-06 | v1.12 | 078 composition mapping | The parent→child mapping half of the calculator input ruling (v1.11): a PIPELINE node's `parameters` may now map onto a child CALCULATOR `context_key` as well as a declared parameter (supplied → the child's node is skipped, §4.10's rule composed), and a `"${ref}"` value resolves against all three parent Context tiers — a declared parameter, a parent calculator `context_key` (typed by its kind's output), an org/platform key (org STRING, platform canonical) — type-checked against the target with the unchanged codes, messages naming the tiers; an ANY-output key on either side skips the check (typed only by the run, A6's convention). **No auto-passthrough:** identically spelled parent/child calculator keys are not implicitly mapped — only explicit entries cross. The read surfaces list calculator keys under `parameters` as `{"type", "required": false, "derived": true}` (`"ANY"` for ANY-output kinds), derived on read, never stored. Additive per §15.2. | | 2026-09-06 | v1.11 | 078 calculator input ruling | A calculator `context_key` is an implicit **optional execute input** (owner ruling 2026-09-05): supplied in the execute request's `parameters` object → the node is skipped and the supplied value (coerced against the kind's output type; an ANY-output key takes any JSON scalar, refusal = `pipeline.execution.invalid_parameter_type`, §13.3) is what downstream nodes bind, marked `provided_by: "caller"` on the node's stats; unsupplied (JSON `null` included) → the node runs and computes it. §7.2's tier 4 widened accordingly (org < platform < parameters < execute-time inputs — now including calculator keys < calculator outputs). The parent→child mapping half of the ruling — a parent maps a calculator key into a child execution explicitly — ships as its own row with the composition change. Additive per §15.2. | | 2026-09-06 | v1.10 | 078 contract gaps | §12.6's interpolation refusal set gains every calculator output key (a CALCULATOR node's `context_key`, typed by its kind's output type): `${calc_key}` now fails save with `template.validation.parameter_interpolated` instead of the dry render's wrong code, the `${calc_key!}` shape no longer slips past the scan, and a calculator key is refused in a conditional's test (`<#if x??>`, `<#elseif x>`) as well — a derived value gating SQL structure is the same hole. Additive per §15.2. | | 2026-09-02 | v1.9 | 040 template used-by | §13.9 gains `template.in_use` (409) — template delete refused while any pipeline version pins any version of the template; the refusal carries the reverse scan's rows (pipeline, node, pipeline version, pinned version). Additive per §15.2. | | 2026-09-02 | v1.8 | 046 typed templates | §12.6 gains `pipeline.validation.template_type_mismatch` — a DQL/DML/DDL node referencing a `type='html'` template is refused at pipeline save (template-hierarchy-design §7). §13.9 gains `template.validation.type_invalid` (unknown `type` wire value), `template.validation.dialect_not_allowed` (a `dialect` present on an `html` template) and `template.validation.type_immutable` (a payload attempting to change a template's type). Additive per §15.2. | | 2026-09-01 | v1.7 | 037 agent data visibility | §13.4 gains `pipeline.node.not_found` (404) and `pipeline.node.standalone_execution_refused` (400) — the refusals of the `pipelines_execute_node` node-run debug query (MCP §6.2.20): an unknown node id, a `tempdb` source (staging exists only inside a full execution), or a PIPELINE node (it runs a child pipeline, not SQL). Node runs are agent debug queries, not executions — no history rows, no SSE, no idempotency (ratified, 037 §A). Additive per §15.2. | | 2026-09-01 | v1.6 | 042 bound parameters | §7.4's example and prose move declared parameters to the `:name` bind form (Templates §4.5 — a declared parameter is a value; `${}` is for structure). §12.6 gains the interpolation refusal: a declared name inside `${}` fails save with `template.validation.parameter_interpolated` (§13.9). §13.4 gains `pipeline.node.sql_parameter_missing` — a `:name` the execution context does not declare fails loudly before anything executes. Additive per §15.2. | | 2026-08-05 | v1.0 | initial draft | Initial pipeline contract: schema, nodes, parameters, validation, error codes, lifecycle | | 2026-08-05 | v1.1 | review feedback | Replaced `terminal_node_id` with auto-detection (§9). Added `type` enum (DQL/DML/DDL) — §4.6, §8. Added `output` block with `target` (tempdb/caller/datasource) — §4.7, §8.1. Added `settings.tempdb` — §5. Renamed `__staging__` → `tempdb` throughout. Removed `datasources_used` (redundant with node sources). Added `engine` field placeholder for templates. **Rejected `parallel_id`** — `depends_on` is mathematically complete for DAG parallelism (two nodes run in parallel iff neither is reachable from the other); a second parallelism source-of-truth would create reconciliation bugs. | | 2026-08-07 | v1.2 | consistency campaign | **D1:** omitted `output` defaults to `caller` (was `tempdb`); topology-based terminal auto-detection replaced by caller-node resolution (§9); any DQL target combination legal — deleted `dql_sink_missing_caller_target`, `no_dql_sink`, `disconnected_terminal`, `dql_missing_output`; `multiple_caller_targets` → `multiple_caller_nodes`; zero caller nodes legal. **D3:** template variables validated by save-time dry-render (§7.4, §12.6); pipeline `parameters` is the single declaration point. **D5:** §13 expanded with §13.7–13.11 (auth normalized to 3-segment codes, datasource, template, result, rate-limit/idempotency) — the single concrete code catalog. §6.3 strict coercion rules; §10.1 per-namespace table uniqueness; §11.4 scan fields + heuristics specified; §15.4 version-counter disambiguation; §17.3 defers DDL to metadata-db. See [SPEC-REVIEW-2026-08](https://github.com/msabiransari/datapipelines/blob/main/docs/SPEC-REVIEW-2026-08.md) | | 2026-08-11 | v1.3 | gate C review | Additive §13 rows: `template.not_found` (404) and `datasource.not_found` (404) — read/mutate-path misses previously borrowed `pipeline.validation.*` codes, which are 400s for write-time validation. Adjudicated answer to the catalog gap flagged by mcp-server's McpNotFound and web's ApiErrors. | | 2026-08-17 | v1.4 | pipeline composition | `NodeType` gains `PIPELINE` — a node that executes a version-pinned pipeline as a separate child execution and consumes its caller result (§4.9 node shape, §8.5 execution behavior). New §12.9 composition validations; §13.4 gains `pipeline.node.child_execution_failed` and `pipeline.node.composition_depth_exceeded`; §12.4 `type_invalid` enum widened. Additive per §15.2 (§15.1 records the additions). | | 2026-08-27 | v1.5 | workspaces readonly slice | §12.5 gains `pipeline.validation.datasource_readonly` and §13.4 gains `pipeline.node.datasource_readonly` (workspaces design 2026-08-16 §6, D6/D10): a datasource flagged `is_readonly` forbids the three write-shaped uses — save-time (400, `details` triple of node id + datasource name + shape) and at execution (500, the live-registry per-node backstop covering the D10 flip window and composed children). Additive; DQL reads and everything `tempdb` untouched. | | 2026-08-30 | v1.6 | local password auth | §13.7 gains three local-auth rows: `auth.login.bad_credentials` (401), `auth.login.locked` (403), `auth.password.change_required` (403) — defined in auth.md §9/§5A. The two existing `auth.login.*` descriptions reworded to name the method (OIDC or local). | | 2026-08-30 | v1.7 | credential-minting is session-only | §13.7 gains `auth.session.required` (403): an API-key principal reached an operation that mints or rotates an interactive credential. Closes a post-merge escalation found by adversarial verification of 026 — an admin-scoped key could create a local admin, read its one-time password from the response, and sign in, yielding an unpinned session that outlived revocation of the key. Defined in auth.md §5A.7. | | 2026-09-05 | v1.8 | pipeline folders (067) | §3.2 `name` becomes a **folder path** — character-for-character the template grammar ([Template Hierarchy §4.1](https://github.com/msabiransari/datapipelines/blob/main/docs/template-hierarchy-design.md#41-grammar)): 1–10 `/`-separated segments, each `[a-z0-9][a-z0-9_.-]{0,63}`, ≤ 200 chars. Folders are virtual (a name prefix, no table, no column, no id, no CRUD); workspaces stay the isolation boundary. §4.9's `pipeline.name` child reference takes the same grammar and reports `name_invalid` at `nodes[i].pipeline.name` when it does not. **Node ids, `output.table` and parameter keys are NOT widened** — §15.1 records exactly which naming rule moved. Relaxation per §15.2 with ONE narrowing: a name whose first character is `_` was legal and is not. It needs no migration and no deploy gate, because a pipeline name is validated at SAVE only (pipelines are UUID-addressed and nothing on the execute path re-checks the grammar), so a legacy `_scratch` pipeline keeps listing, opening and executing and only its next save is refused — the difference from templates' §4.6, which had to abort a deployment because their grammar is re-checked at render time. No new error codes. | | 2026-09-05 | v1.9 | published endpoints (074) | New **§13.14 Published endpoints** — the `endpoint.*` codes a publish, bind or serve can return (`pipeline_not_readonly`, path grammar/conflict, key-kind confinement, `202` on timeout). Codes only; the pipeline document itself is unchanged. | | 2026-09-05 | v1.10 | mandatory folders (077) | §3.2 `name` requires a **folder**: the path is now **2–10** `/`-separated segments, not 1–10 ([Template Hierarchy §4.1](https://github.com/msabiransari/datapipelines/blob/main/docs/template-hierarchy-design.md#41-grammar)). A bare `scratch` is refused; `test/scratch` is the sanctioned scratch form. §13.9's `pipeline.validation.name_invalid` row is unchanged as a CODE — no new code — but its `details` gain `reason`: `folder_required` when a folder is the only thing missing, `grammar` otherwise, so an agent can tell the two refusals apart without parsing the message. A **narrowing** of the frozen contract, taken pre-release under the owner ruling of 2026-09-05 and for §4.5's reason: there is no rename, so a root-level name minted after the tag is permanent. Still no migration on the pipelines side — validated at SAVE only, so a legacy flat pipeline keeps listing, opening and executing and only its next save is refused (§14.2). | | 2026-09-07 | v1.11 | dp-lake registry (089 §A) | §13.8 gains the lake-table codes: `datasource.validation.lake_dialect_required` (400 — a lake-table operation on a non-LAKE datasource), `lake_namespace_invalid` / `lake_name_invalid` (400 — the segment grammar), `lake_format_invalid` (400 — not `parquet` or `iceberg`), `lake_location_invalid` (400 — scheme allowlist plus the total injection refusal: locations are later interpolated into `CREATE VIEW` statements), `lake_manifest_url_forbidden` (400 — an import manifest URL outside the datasource's own endpoint/bucket; the SSRF boundary), `datasource.lake_table_duplicate` (409, mapped from metadata-db §4.15's named UNIQUE) and `datasource.lake_table_not_found` (404). Additive per §15.2. |