MCP Server Specification
Status: v1.41 (frozen contract — additive-only changes after this point) Owner: datapipelines.co core Depends on: Type System spec, Pipeline Contract spec, REST API spec, Auth spec, Templates spec Last updated: 2026-09-16
1. Purpose
datapipelines.co is MCP-native. Agentic tools (Claude Desktop, GLM, Copilot, custom LangChain/LlamaIndex agents, etc.) connect to a datapipelines.co instance via the Model Context Protocol to discover pipelines, execute them, and read results — without writing custom integration code.
This spec defines:
- The MCP transport (Streamable HTTP) and how clients connect.
- The authentication model (API key issued per-user-per-agent from the UI).
- The tool surface (functions the agent can call).
- The resource surface (entities the agent can read as files).
- The prompt surface (predefined workflows the agent can invoke).
- The error model (how datapipelines errors map to MCP errors).
2. Design Principles
- MCP is a thin adapter over REST. Every MCP tool maps to one or more REST endpoints defined in REST API spec. No business logic in the MCP layer — it's translation only.
- Tools for actions, resources for inspection. If an agent needs to do something (execute a pipeline, create a template), it calls a tool. If it needs to read something (look at a pipeline definition), it reads a resource. We avoid duplicating read operations as both.
- API key, not OAuth. Self-hosted, internal-users-only deployment model makes OAuth overkill. The user grabs an API key from the UI, passes it to their agent, the agent uses it — in either the
DP-API-Keyheader or anAuthorization: Bearer dpk_...header. See Auth §8.5. - MCP versioning follows the protocol. We commit to a specific MCP protocol version per datapipelines.co release, and document upgrade paths when the protocol evolves.
- Fail loudly, never silently. MCP-level errors (transport, auth) and application errors (pipeline validation, datasource unreachable) both surface as structured errors the agent can act on. No silent fallbacks.
- Workspace-scoped by the key (workspaces design §5.2/§9). Every tool and resource operates inside the workspace the API key is PINNED to at issuance —
DP-Workspaceis refused on MCP requests (400 workspace.header_forbidden), because a header-switchable agent key would make every leaked key a skeleton key across the user's workspaces. Pipelines, templates and executions of other workspaces are ABSENT (not hidden): their ids resolve as not-found. Datasources visible here are exactly the ones GRANTED to the pinned workspace (D-R7): there is no global datasource, and one that is not granted is ABSENT, not hidden. Theinitializeresult'sinstructionsfield states this so an agent does not reason about invisible siblings.
3. Transport
3.1 Transport choice: Streamable HTTP
We expose MCP via the protocol's Streamable HTTP transport:
- Single endpoint:
POST /mcp(andGET /mcpfor server-to-client notifications/SSE). - Content types:
application/jsonfor single requests/responses,text/event-streamfor streamed responses. - Works through standard HTTP infrastructure (proxies, load balancers, TLS terminators).
- No WebSocket requirement (which would need custom proxy config).
This is the protocol's network-native transport, appropriate for our self-hosted, network-resident deployment model. The stdio transport (used for local-tools) is not supported — our product is a server, not a local process.
Implementation gate — RESOLVED at P6b (2026-08-10). This spec is authored against the durable shape of MCP; the concrete protocol version string is a build-time input, not a frozen contract term. The checklist below was completed against the official specification and the shipped MCP SDK when the module was implemented:
- [x] Current protocol version string. Pinned to
2025-06-18, returned ininitialize.protocolVersion(§5.1) and accepted in theMCP-Protocol-Versionheader (§3.2); aPinnedTransportdecorator advertises it as the sole supported version.- [x] Version-negotiation rule. The server negotiates down to its pinned version — a client offering a newer version is served
2025-06-18(verified in-process).- [x] Streamable HTTP requirements. The v1 server is stateless:
POST /mcpacceptsapplication/json+text/event-stream;GET /mcpfor a server-initiated SSE stream is optional and NOT served (answered405) — so there are no server-to-client notifications in v1 (§5.1, §10), noMCP-Session-Idissuance, and no resumability headers. A stateful transport (session ids, the notification stream) is a v2 item (ROADMAP §3.7).- [x] SDK coordinates.
io.modelcontextprotocol:mcp-core+mcp-json-jackson2, bothmcp-sdk = 2.0.0, pinned in the version catalog (Module Structure §8).These resolutions were additive corrections to §3 and §5.1 only — the tool, resource, and prompt surfaces did not depend on them.
3.2 Endpoint structure
POST {host}/mcp
Headers:
Content-Type: application/json
Accept: application/json, text/event-stream
DP-API-Key: dpk_<id>.<secret> # OR: Authorization: Bearer dpk_<id>.<secret>
MCP-Protocol-Version: 2025-06-18 # placeholder — pinned by the §3.1 implementation gate
MCP-Session-Id: {session-uuid} # optional; server may issue for stateful sessions
Exactly one credential carrier is required, and the two are equivalent: DP-API-Key (the REST convention, REST API §3.6) or Authorization: Bearer dpk_... for MCP clients that can only set the standard Authorization header. Both route through the identical API-key validation path — see §4.1 and Auth §8.5. Session JWTs (dp_session cookie, or a Bearer token that is not a dpk_ key) are not accepted on /mcp.
Server response: JSON for single-message exchanges, text/event-stream for streamed responses.
3.3 Session lifecycle
- Stateless by default. Each request carries full auth context. Server does not require session continuity.
- Optional session. Server MAY issue an
MCP-Session-Idfor clients that want one. Session state = nothing important (cached auth, nothing else).
4. Authentication
4.1 Auth model
/mcp is API-key-only. Every MCP request must carry a datapipelines API key in one of two equivalent headers:
DP-API-Key: dpk_<id>.<secret>— the REST convention; the primary case.Authorization: Bearer dpk_<id>.<secret>— for MCP clients that can only set the standard Authorization header (Claude Desktop and several others). The filter recognizes thedpk_prefix and routes the token through the identical validation path.
Both are validated by Auth §7.3 — same lookup, same Argon2id verification, same 60s-TTL revocation/liveness re-check (Auth §11.4). A revoked key or a deactivated owner stops working within ~1 minute.
Session JWTs are not accepted on /mcp. There is no cookie auth and no non-dpk_ Bearer token path — a browser-embedded MCP client must use an API key like any other agent.
API keys are:
- Issued per-user-per-agent from the UI's API screen (e.g., "Claude Desktop key", "GLM key"); HTTP surface in REST API §16.1.
- Revocable, optionally expiring.
- Scoped
read/execute/author(hierarchical, Auth §7.5).adminis no longer issuable to a key — it was the only scope that ever bought a key an INSTANCE verb, and instance verbs are human. A key's scopes are a subset of what its issuer can do in the pinned workspace at issue time.
An agent's key is a user key — the same kind a program uses over REST. The UI labels it "Agent / API key" for exactly that reason: one credential kind, two surfaces. The other two kinds (Auth §7.7) do not reach /mcp at all — an endpoint key authorises published endpoints and a server key the promotion routes, and each is refused here with 403 endpoint.key_kind_refused by McpAuthFilter (the scope interceptor never sees /mcp, which is a servlet, so the refusal is made again at the transport). A scopeless key could otherwise read the whole tool catalogue through tools/list without being able to call any of it.
Enforcement is TWO axes, and a key must satisfy both (Auth §7.6, §11A). The minimum SCOPE and the minimum ROLE for every MCP tool are defined once in that matrix — this spec restates each tool's requirement in §6.2 for readability, but the matrix is authoritative on any conflict, and one function (ScopeMatrix.allowedTool) answers both here and at the REST interceptor.
A key can do at most what its ISSUER can do NOW. On every request the key's own scope is checked against the tool's minimum, AND the issuer's CURRENT membership in the pinned workspace is checked against the tool's minimum role — re-read inside the same 60s validation-cache TTL as revocation. So a key whose issuer was demoted or removed from the workspace stops working within about a minute, and the refusal is auth.key_issuer_role_lost rather than auth.role_required: retrying with that key will never work, and a new key from someone who still holds the role is the fix.
No MCP tool's minimum role is promote or super_admin, and none is ws_admin except datasources_test (it opens a live connection and writes the datasource's health down — an operational act). Release, promote, workspace creation and membership have no MCP tool at all: they are human verbs, which is the same reason no key holds admin scope. Registering, editing and deleting datasources remain UI/REST-only.
Datasource visibility is a GRANT (Auth §11A). A datasource not granted to the key's pinned workspace does not exist for it: datasources_list is the truth, and guessing a name gets the not-found envelope, never a "forbidden".
Security chain. /mcp (both POST and GET) is an explicit matcher in the Spring Security filter chain: CSRF-exempt (no cookie auth to forge against), no session cookies accepted, same scope enforcement as REST, same per-user rate limits (REST API §12). See Auth §8.5.
The principal travels WITH the call, never on the thread (134). McpAuthFilter resolves the key on the servlet thread and hands the principal to the MCP layer through the transport context (McpToolContext); the SDK then runs every tool handler on its own scheduler thread (boundedElastic), where Spring Security's thread-local context is empty. So a tool — and anything a tool calls — must take who is asking and which workspace as arguments from McpToolContext; a domain port or @Bean adapter that reads SecurityContextHolder answers correctly over REST and as "no principal" over MCP, with no error anywhere. The save-time datasource lookup was exactly that: a workspace-owned datasource validated as pipeline.validation.unknown_datasource over MCP while the same body was 201 over REST (measured 2026-09-14; the demo datasources are owner-less + granted, which is why the acceptance run never saw it). Since 134 the validator's datasource port takes the workspace explicitly, as the executor always did; McpSaveWorkspaceDatasourceE2eTest holds the line.
4.2 Unauthorized behavior
Missing credential (no DP-API-Key header and no Bearer dpk_ token):
- HTTP
401 Unauthorizedwith JSON body:{"error": {"code": "auth.api_key.missing", "message": "..."}} - MCP session is not established.
Invalid, revoked, expired, or deactivated-owner key:
- HTTP
401 Unauthorizedwithauth.api_key.invalid(orauth.api_key.expired).
Insufficient scope (e.g., a read key calling pipelines_create):
- The transport-level answer is HTTP
403 Forbiddenwithauth.scope.insufficientwhen the credential is rejected before dispatch. Once a session is established and a tool is dispatched, a scope failure is returned as a tool result withisError: truecarrying the sameauth.scope.insufficientcode (§9.2) — agents must handle both.
Codes follow the {domain}.{entity}.{failure} convention; the registry of record is Pipeline Contract §13.7.
4.3 Why not OAuth
OAuth adds:
- Authorization server (to build/maintain)
- Redirect flows (impossible for non-browser agents like Claude Desktop)
- Token refresh logic (per agent)
- Client registration (per agent)
For self-hosted, internal-users-only deployment, API keys are simpler and sufficient. Future multi-tenant SaaS deployment would revisit this.
5. Server Metadata & Capabilities
5.1 initialize response
{
"protocolVersion": "2025-06-18",
"serverInfo": {
"name": "datapipelines",
"version": "1.0.0"
},
"capabilities": {
"tools": {"listChanged": false},
"resources": {"listChanged": false, "subscribe": false},
"prompts": {"listChanged": false}
},
"instructions": "This server is workspace-scoped: every tool and resource operates inside the workspace the API key is pinned to. ..."
}
-
instructions(workspaces design §9) states the workspace context every agent reads first: content in other workspaces is absent (not hidden) — it resolves as not-found — and names are per-workspace for pipelines and templates while datasource names are globally unique. The full text ships asMcpServerFactory.SERVER_INSTRUCTIONS. -
tools.listChanged: false— the tool surface is static: the same 41 tools (§6.1) for every caller, for the lifetime of the server. Advertisingtruewould promisenotifications/tools/list_changedmessages the v1 server never sends. Dynamic per-pipeline tools (pipeline_execute_{name}, which would make the list genuinely mutable) are a v2 item — ROADMAP §3.7. When they land, this flips totruetogether with the notification implementation. -
resources.listChanged: false— the set of resource URIs does change as pipelines and executions are created, but the v1 server sends no change notifications; clients re-fetchresources/list(§7.3) when they need a current view. -
resources.subscribe: false— no live subscriptions in v1. Clients re-fetch resources as needed. -
prompts.listChanged: false— the prompt surface (§8) is static in v1. -
No
loggingcapability in v1. The v1 transport is stateless (§3.3): it answersGET /mcpwith405, so there is no server-to-client stream to carrynotifications/message. Advertisingloggingwould promise notifications no client can receive — the same reasoning aslistChanged: false. Live progress during a blockingpipelines_execute(§6.2.3) is therefore not available in v1; the authoritative per-node record is thenode_statsarray in the tool's final result. Logging/progress notifications return with the stateful transport in v2 (ROADMAP §3.7).
protocolVersion is the placeholder pending the §3.1 implementation-gate check. serverInfo.version is the datapipelines.co release version.
6. Tool Surface
6.1 Tool naming convention
Tools are named {domain}_{action}:
pipelines_listpipelines_getpipelines_executepipelines_execute_nodepipelines_createpipelines_updatetemplates_listtemplates_gettemplates_used_bytemplates_createtemplates_updatetemplates_rendertemplates_purge_draftdatasources_listdatasources_getdatasources_testdatasources_get_schemasdatasources_get_tablesdatasources_get_columnsdatasources_get_table_statsdatasources_preview_rowssql_probeexecutions_listexecutions_getexecutions_get_resultexecutions_cancelcalculators_listcalculators_getendpoints_createendpoints_listendpoints_getendpoints_deletelake_tables_registerlake_tables_importlake_tables_unregistersemantics_recordsemantics_listsemantics_retiredocs_listdocs_getpipelines_run_checks
A future enhancement: dynamically-generated per-pipeline tools (e.g., pipeline_execute_monthly_revenue_report) for pipelines the user wants to expose as named tools to agents. Marked for v2 (ROADMAP §3.7) — this is why tools.listChanged is false in v1 (§5.1).
6.2 Tool definitions
Every tool definition below carries a Scope row: the minimum scope the calling API key must hold. Those values are sourced from the Auth §7.6 operation matrix, which is authoritative — if this doc and the matrix ever disagree, the matrix wins. Scopes are hierarchical (author ⊃ execute ⊃ read; admin ⊃ all), so a listed scope is a floor, not an exact match. No v1 MCP tool requires admin (§4.1).
Every tool's result envelope, including its error shape, is §6.3.
6.2.1 pipelines_list
List pipelines the caller has access to.
{
"name": "pipelines_list",
"description": "List the pipelines of the key's pinned workspace, filtered by owner, datasource, or text search. Returns metadata (id, name, display_name, description, version, status, updated_at) — version is the WORKING version and status says DRAFT or RELEASED, so an unreleased pipeline is visible as such. Not the full body. Use pipelines_get for the body; pipelines in other workspaces are absent from this listing and resolve as not-found by id. Pipeline names are FOLDER PATHS (finance/payments/daily_settlement): pass prefix to BROWSE one level of that tree — prefix:\"\" lists the roots, prefix:\"finance\" lists what is directly under finance — and q to SEARCH across full paths. Start with prefix:\"\" to see which roots this workspace already uses before creating a pipeline under a new one.",
"inputSchema": {
"type": "object",
"properties": {
"owner": {"type": "string", "description": "Filter by owner user ID."},
"datasource": {"type": "string", "description": "Filter by datasource name."},
"q": {"type": "string", "description": "Full-text search on name and description. Searches across full paths; use prefix to browse instead."},
"prefix": {"type": "string", "description": "Browse ONE level of the folder tree instead of listing flat: returns that prefix's direct sub-folders (with counts) and its direct children. An empty string is the root. Use this to discover which roots and folders exist; use q to search across full paths."},
"limit": {"type": "integer", "default": 50, "maximum": 200}
}
}
}
Returns: array of pipeline metadata objects. Datasource references are per-node and are read from the body via pipelines_get — the listing does not aggregate them.
version is the working version, and status names it (D55/D56, 099): the draft's number when the pipeline has a draft, else the latest released one, with status = DRAFT or RELEASED. Since creation lands a DRAFT, a listing that reported the released pointer alone would show nothing at all for every freshly authored pipeline, and version: 1 on its own could not tell a reviewed release from a draft nobody has looked at. Both fields are also null in the one case where a pipeline has no version at all — its sole draft was purged, which deletes the entity with it (versioning §3.2).
Two presentations, chosen by prefix (067). Pipeline names are folder paths (Pipeline Contract §3.2, Template Hierarchy §14), so an agent needs to BROWSE as well as search:
-
prefixabsent — the flat listing above, underowner/datasource/q. Unchanged. -
prefixpresent (""is the ROOT) — one level of the tree, and the return shape is an object rather than an array:{ "prefix": "nyc", "folders": [{"path": "nyc/mobility", "segment": "mobility", "pipeline_count": 6}], "pipelines": [ /* the level's own leaves, same metadata objects */ ], "total": 0, "has_more": false }foldersare the prefix's DIRECT sub-folders with their whole-subtree counts;pipelinesare its direct children only. Never a subtree, never the whole list.owner,datasourceandqare ignored whileprefixis present — browse and search are different presentations. A prefix is a FOLDER PATH — 1 to 9 segments, not the 2-to-10 a NAME takes since 077 — sonycbrowses; one that is not a legal folder path answers an empty level, not an error.
When to use which: prefix to discover structure ("what roots exist? what is under finance?"), q to find something by name across full paths. Start a naming decision with prefix: "".
Scope: read.
6.2.2 pipelines_get
Fetch a full pipeline definition.
{
"name": "pipelines_get",
"description": "Get the full definition of a pipeline (the working version by default — the draft when unreleased edits exist, else the latest released version — or a specific version). Use this to read the pipeline body before executing or modifying it. The result carries the version, its status and body_hash — echo body_hash back as expected_hash on pipelines_update; a draft pointer is present when unreleased edits exist. When a node pins a template version that a newer released version outdates, an upgrade_available array names the node, the template and both versions — an offer to re-pin via pipelines_update, never an automatic change.",
"inputSchema": {
"type": "object",
"required": ["id"],
"properties": {
"id": {"type": "string", "format": "uuid", "description": "Pipeline ID."},
"version": {"type": "integer", "description": "Specific version. Defaults to the working version: the draft when one exists, else the latest released."}
}
}
}
Returns: full pipeline JSON body (per Pipeline Contract §3) merged with the fields the version-lifecycle protocol needs (versioning §4.2, since 035): the version's body_hash and status — echo body_hash back as expected_hash on pipelines_update — plus current_version (the latest RELEASED version, what execute-default runs) and a draft pointer when unreleased edits exist. Since 039 the DEFAULT body is the working version (versioning §7): the DRAFT when one exists, else the latest released — an agent that read released while a draft was open would rebase on stale content and quietly discard the draft with its next write. The response always states which version and status it returned; an explicit version argument still wins.
Since 078, the body's parameters also lists the pipeline's derived execute inputs: one entry per key a CALCULATOR node writes (121: every mapped context_keys value of a multi-output kind too) — {"type": <the key's output wire type, or "ANY">, "required": false, "derived": true} — because a calculator key is an implicit optional input of pipelines_execute (pipeline-contract §4.10: supply it and the node is skipped). Declared parameters carry no derived flag; absence is the false. Derived on read, never stored — the entries must not be echoed back on pipelines_update.
Since 040, the response also carries upgrade_available whenever a node's pinned template has a newer RELEASED version (040 D5): one {node, template_id, pinned, latest_released} row per outdating pin, computed from the very body being returned. Absent when no pin is outdated (omit-when-empty, the envelope convention). Surfaced, never applied — moving a pin is a pipeline edit (pipelines_update) and stays the caller's decision; a pin of a template DRAFT version is not an upgrade (the author is ahead of release, which is information, not a prompt). See Templates §5.4.
Scope: read.
6.2.3 pipelines_execute
Execute a pipeline.
{
"name": "pipelines_execute",
"description": "Execute a pipeline with the given input parameters. Returns execution events (node start/complete/fail) and the final result data. The result's schema describes column types; BIGINTEGER and BIGDECIMAL columns serialize as JSON strings — preserve them as strings when displaying or persisting to avoid precision loss. When the execution FAILS, the error result carries the full failure record (node context, rendered SQL, exception chain with the root cause last in caused_by) — the same object executions_get returns; quote its correlation_id when escalating. The server checks the loop: a DRAFT whose pinned draft template was updated after this key's last templates_render of it is refused pipeline.execution.template_unrendered — render, then run.",
"inputSchema": {
"type": "object",
"required": ["id", "parameters"],
"properties": {
"id": {"type": "string", "format": "uuid"},
"version": {"type": "integer", "description": "Specific version to run. Defaults to the WORKING version: the draft when one exists, else the latest released. Never clamped — an unknown version is refused, not rounded to the latest."},
"parameters": {
"type": "object",
"description": "Object whose keys match the pipeline's declared parameters. Values must match the declared types (BIGINTEGER and BIGDECIMAL as strings, others as JSON native types).",
"additionalProperties": true
}
}
}
}
Returns: an execution result object containing:
- Execution metadata (
execution_id,pipeline_id,status,duration_ms,node_stats). - Schema (array of column descriptors per Type System §7).
- The first page of rows inline (up to
datapipelines.result.page-size-rows), plustotal_rows,has_more,result_url, andexpires_at. - Warnings array (if any).
The result shape mirrors the REST data_ready event exactly — same fields (schema, the inline rows first page, row_count, total_rows, has_more, result_url, expires_at, and ttl_seconds so the agent knows its paging window without diffing timestamps), same uniform delivery model. There is no inline-vs-claim-check split: every caller result is materialized in Redis before the tool returns (REST API §7.1). For a result that fits in one page, the inline rows ARE the whole result and no follow-up call is needed; when has_more is true, page the remainder with executions_get_result (§6.2.15) within the TTL.
A pipeline with no caller node (Pipeline Contract §9) is legal — a pure write-back/ETL pipeline. Such an execution returns metadata, node_stats, and no schema/rows; this is success, not an error.
Long-running executions. The tool call is a single blocking request: it returns when the execution reaches a terminal state (SUCCESS, FAILED, ABORTED) or when datapipelines.executor.execution-timeout-seconds (default 600) elapses and the execution is aborted. For a 3-minute pipeline, the agent experiences one tool call that takes ~3 minutes; the HTTP response for that call stays open for the duration and the server writes nothing to it until the result is ready. (The REST SSE heartbeat, REST API §6.6, is an SSE-stream concept and does not apply here — an MCP tool call is not an event stream. Operators must therefore ensure proxy/load-balancer idle timeouts on /mcp exceed execution-timeout-seconds; see Deployment.)
MCP progress notifications for in-flight nodes are deliberately not implemented in v1 — the tool returns progress only as the final node_stats. Streaming execution events through the MCP transport is a v2 item (ROADMAP §3.7). The v1 stateless transport delivers no server-to-client notifications of any kind (§5.1), so node_stats in the tool's final result is the authoritative and only per-node record.
If the agent abandons the call (aborts the HTTP request, client crash): a blocking POST /mcp gives the servlet no disconnect callback, so the datapipelines.sse.disconnect-grace-seconds cancellation that a dropped REST SSE stream gets (REST API §6.8) does not apply to an abandoned tool call in v1 — the execution runs until it finishes or hits datapipelines.executor.execution-timeout-seconds. To stop an in-flight execution deterministically, cancel it: over MCP with executions_cancel (§6.2.35) when this key's own MCP calls started it, or out-of-band via DELETE /api/v1/executions/{id} (REST API §10.4) from any instance — in-flight statements are interrupted and the abandoned tool call returns an ABORTED result. There is no resumption path (a reconnecting agent must re-execute).
With no version, this runs the WORKING version (D56, 099): the draft when one exists, else the latest release — "always run the LAST version". On a development server that may well be a draft (drafts have been executable since 039); on a hardened server authoring-enabled=false refuses every draft-creating write, so the working version is a RELEASED version by construction (versioning §7.2). The execution record pins the version that actually ran and the executions screen marks a draft run, so a result is never ambiguous about what produced it. An explicit version is exact and never clamped.
Render before you run (139). Executing a DRAFT version whose pinned template version is itself a DRAFT written after this key's last successful templates_render of it is refused — pipeline.execution.template_unrendered, with details.templates carrying each {id, version, updated_at, last_render}. RELEASED pins are exempt (they rendered before release and cannot change), and so is any execute of a RELEASED version: the check reads the caller's own audit rows, so it is an AGENT-surface rule — REST, the UI and a human's key keep their freedom. A templates_update therefore makes the next execute refuse again until a fresh render: the render-then-run loop the SKILL's steps 3 and 5 describe, enforced at the entry point instead of asked of the model a third time. pipelines_execute_node (§6.2.20) runs the same check on the one node it resolves.
Scope: execute.
6.2.4 pipelines_create
Create a new pipeline.
{
"name": "pipelines_create",
"description": "Create a new pipeline. The body must satisfy the Pipeline Contract: nodes must form a DAG; at most one DQL node may resolve to output.target='caller' (a node that omits its output block resolves to 'caller' by default); zero caller nodes is legal for pure write-back pipelines; all datasource references must exist in this environment; all template references must exist and dry-render against the declared parameters. A node may also be type='CALCULATOR': it evaluates one catalog function and writes a typed value into the execution Context under context_key, which downstream nodes bind as :context_key — call calculators_list first for the kinds and their input names, and remember that a node referencing another node's context_key must depend_on it. A NEW top-level folder is refused until you confirm it: reuse an existing root, or ask the person first and then pass confirm_new_root: true. The server checks what you learned: a body whose template names a table this key never datasources_get_columns'd is refused pipeline.validation.table_not_learned with the clearing calls listed, and a raw-date door (two DATE parameters, no period parameter, no window calculator) is refused pipeline.validation.door_unacknowledged until you pass door_acknowledged: true. Returns the created pipeline with server-assigned id and version 1, which lands as a DRAFT: run it straight away, then STOP — a human releases it from the UI, and no tool releases anything.",
"inputSchema": {
"type": "object",
"required": ["name", "display_name", "nodes"],
"properties": {
"name": {"type": "string", "pattern": "^[a-z0-9][a-z0-9_.-]{0,63}(/[a-z0-9][a-z0-9_.-]{0,63}){1,9}$", "description": "Machine name, and a FOLDER PATH: 2-10 lower-case '/'-separated segments (finance/payments/daily_settlement). A FOLDER IS REQUIRED — a bare 'daily_settlement' is refused with pipeline.validation.name_invalid and details.reason='folder_required'; put experiments under test/. The root segment says who owns it — list the existing roots with pipelines_list {prefix: ''} and reuse one; ASK before minting a new root. Keep a pipeline under the same prefix as the templates it uses. There is no rename: the name is the pipeline's identity, so choose the folder now."},
"display_name": {"type": "string"},
"description": {"type": "string"},
"parameters": {"type": "object", "description": "Declared pipeline parameters (name -> {type, required, default, description}). This is the ONLY parameter declaration point: the full parameter map, defaults applied, is the render context for every template the pipeline references."},
"settings": {"type": "object", "description": "Pipeline-level execution settings (e.g., tempdb engine)."},
"nodes": {
"type": "array",
"description": "Pipeline nodes. Each node has type (DQL/DML/DDL/PIPELINE), source, template ref, depends_on array, and — for DQL only — an optional output block. Omitting output on a DQL node means output.target='caller'; at most one node per pipeline may resolve to 'caller'. A node whose data downstream nodes query must declare output.target='tempdb' with a table name explicitly. A PIPELINE node instead carries a pipeline ref {name, version} pinning an existing pipeline version to execute as a child execution, an optional parameters map (typed literals, or '${parent_param}' to pass a parent parameter through), and an optional output block allowed only when the pinned child has a caller node; it declares neither source nor template."
},
"checks": {
"type": "array",
"description": "Release checks (pipeline-contract §3.3): at most 20 objects, each {id, name, datasource, sql, expected} — id [a-z0-9_]{1,63} unique in the body, name 1-200 chars, sql ONE read-only statement. expected.kind is value (single numeric cell compared with absolute tolerance, default 0), range (the cell within min..max inclusive), or rows (the statement's row count equals rows). Every :name bind must name a DECLARED pipeline parameter (the calculator context is not available to a check); ${} interpolation is refused (a check has no rendering); tempdb is not a check datasource. You supply the query and the expectation, never an observed value — run them with pipelines_run_checks, and only the server's run produces observed."
},
"confirm_new_root": {"type": "boolean", "description": "Set true ONLY after a person has agreed to a new top-level folder. A name whose root segment has no pipelines or templates under it yet is refused with details.existing_roots listing the roots that do exist — reuse one of those, or ask the person first and then pass this. 'test/' never needs it."},
"door_acknowledged": {"type": "boolean", "description": "Set true ONLY when the question truly fixes two dates. A pipeline whose parameters are two raw DATE inputs with no period parameter (year, quarter, month, *_year) and no window CALCULATOR node is refused pipeline.validation.door_unacknowledged — the door is a decision: prefer the period vocabulary of rule 13, and never pass this to silence the refusal."}
},
"additionalProperties": false
}
}
Returns: created pipeline — id, version: 1, status: "DRAFT", body_hash (carry it into your next pipelines_update), current_version: null (nothing released yet) and the draft pointer.
Creation lands a DRAFT (D55, 099). POST /pipelines used to land version 1 RELEASED so that an MCP-authored pipeline was immediately executable; drafts have been executable since 039, so that justification bought nothing and cost a review — an agent following the old rule produced a released pipeline no human had looked at. The golden path for an agent is therefore: create → execute (no version, which runs your draft) → read the result → stop and tell the person it is ready to review. Releasing is a human action in the UI; there is no release tool and there will not be one (versioning D4).
A new ROOT folder needs the person's say-so (094). The root segment says who owns a thing and there is no rename, so this tool REFUSES a name whose first segment has nothing under it yet — pipeline.validation.new_root_requires_confirmation, with details.root and details.existing_roots (the same one-level query pipelines_list {"prefix": ""} serves). Reuse one of those roots, or ask the person and retry with confirm_new_root: true. test/ is always allowed. This is an AGENT-surface rule only: REST, the UI and pipelines_update are unaffected — a person choosing a folder in a form has already decided, and an update cannot change a name. It replaces an INSTRUCTION with a GUARANTEE: the schema and the SKILL already told an agent to list the roots and ask, and a model that did not, did not.
Learn before you write — the server now checks it (139). A body whose template names a table THIS API key never read the columns of is refused — pipeline.validation.table_not_learned, with details.tables listing each {datasource, table} and the one datasources_get_columns call that clears it. The check reads the caller's own mcp.tool.called audit rows (any time in the key's lifetime — a key learns once; rows written before 139 carry no table and simply do not count), tokenises each pinned template's body the way the semantics fact check does, and compares against the source datasource's catalog listing. tempdb sources, ${…} dynamic names and names the catalog does not list are exempt; _get_table_stats is NOT required — columns are correctness, stats are performance. Run it at pipelines_create and pipelines_update, because only there is the node's datasource known; the template tools themselves are unchecked (a template has a dialect, never a datasource). REST and the UI are unaffected.
The door is a decision (139). A body whose parameters contain two DATE parameters with no INTEGER period parameter (year, quarter, month, *_year) and no CALCULATOR node whose kind writes a DATE — a raw-date door — is refused pipeline.validation.door_unacknowledged, naming the two parameters and rule 13's alternatives (a period parameter, or an anchor date with a period_bounds / trailing_periods node), unless the call carries door_acknowledged: true. The flag is the confirm_new_root shape: it forces the decision instead of a copy, and passing it to silence the refusal is precisely the miss it exists to catch. REST and the UI are unaffected.
The whole pipeline is validated before it is stored — no invalid pipeline ever reaches the database (Pipeline Contract §2). Validation failures come back as a tool result with isError: true carrying the pipeline validation code (§9.2); the agent should fix and retry rather than assume partial creation.
Scope: author.
6.2.5 pipelines_update
Update an existing pipeline by writing its DRAFT (versioning §3.2/§7, since 035).
Same input as pipelines_create plus required id and required expected_hash — the body_hash of the version this edit is based on (pipelines_get or the previous update's result). The first update after a release creates the draft (copy-on-write); later updates overwrite that same draft in place. Same save-time validation applies — and the same 139 entry-point checks: a body naming a table this key never read the columns of is refused pipeline.validation.table_not_learned, and a raw-date door is refused pipeline.validation.door_unacknowledged until door_acknowledged: true (an update can change a body's parameters, unlike its name, so the door check belongs here too). That input includes the optional checks[] (§6.2.4's checks property, pipeline-contract §3.3) — the release checks the SERVER runs; run them with pipelines_run_checks (§6.2.42), and only the server's own run produces observed.
Returns: the draft version — version, status: "DRAFT", body_hash (carry this into the next write), current_version (the unmoved released pointer), and the draft pointer. The update does NOT release: an agent leaves the draft for a human to review and release from the UI (versioning D4). On pipeline.version.conflict someone else modified it after you loaded it — re-read, rebase, retry; never retry blindly.
Scope: author.
6.2.6 templates_list
List templates.
{
"name": "templates_list",
"description": "List the templates of the key's pinned workspace. Templates are reusable generators authored in Freemarker, referenced by id+version; each has a fixed type — 'sql' renders SQL for pipeline nodes (and carries a dialect), 'html' renders escaped output and declares none. Template ids are unique per workspace — another workspace's template resolves as not-found.",
"inputSchema": {
"type": "object",
"properties": {
"dialect": {"type": "string", "enum": ["POSTGRES", "ORACLE", "MSSQL", "MYSQL", "H2", "DUCKDB", "SQLITE", "LAKE"]},
"type": {"type": "string", "enum": ["sql", "html"], "description": "Filter by template kind: 'sql' (pipeline-referenced SQL) or 'html' (rendered output)."},
"q": {"type": "string"},
"prefix": {"type": "string", "description": "Browse ONE level of the folder tree instead of listing flat: returns that prefix's direct sub-folders (with counts) and its direct children. An empty string is the root. Use this to discover which roots and folders exist; use q to search across full paths."},
"is_library": {"type": "boolean", "description": "Filter to library templates (macro collections) or executable templates."},
"limit": {"type": "integer", "default": 50, "maximum": 200}
}
}
}
Returns: array of template metadata (id, version, type, dialect, display_name, description, is_library; dialect is null for html templates, since 046). A template's description is the only place it can hint at the parameters it expects — templates declare none (Templates §3.2).
Two presentations, chosen by prefix (067). Template ids have been paths since 043 and the templates browser has rendered them as a tree since 047, but this tool had no way to browse a folder at all until now:
-
prefixabsent — the flat listing above. Unchanged. -
prefixpresent (""is the ROOT) — one level, the same shapepipelines_listreturns withprefix, keyedtemplatesinstead ofpipelinesandtemplate_countinstead ofpipeline_count:{ "prefix": "nyc", "folders": [{"path": "nyc/mobility", "segment": "mobility", "template_count": 7}], "templates": [ /* the level's own leaves, same metadata objects */ ], "total": 0, "has_more": false }dialectandtypestill narrow both halves, so a folder whose whole subtree is filtered out is absent rather than empty.is_librarynarrows the level's leaves only — a folder count is over the whole subtree, and quietly subtracting library templates from it would make the tree disagree with what expanding the folder shows.qis ignored whileprefixis present.
When to use which: prefix to browse, q to search. A pipeline and the templates it uses should share a prefix — see Template Hierarchy §15.
Scope: read.
6.2.7 templates_get
Fetch a template body.
{
"name": "templates_get",
"description": "Get the body and metadata of a template version, including its imports array (the library macros it can call). Defaults to the working version — the draft when unreleased edits exist, else the latest released.",
"inputSchema": {
"type": "object",
"required": ["id"],
"properties": {
"id": {"type": "string"},
"version": {"type": "integer", "description": "Specific version. Defaults to the working version: the draft when one exists, else the latest released."}
}
}
}
Scope: read. The returned projection states its version and status — since 039 the default is the working version (versioning §7: the DRAFT when one exists, else the latest released), the template mirror of pipelines_get.
6.2.8 templates_create
Create a new template.
{
"name": "templates_create",
"description": "Create a new template. Templates use Freemarker syntax. A template declares NO parameters of its own: the variables its body may reference are exactly the parameters declared by the pipeline that calls it, with defaults applied. Describe the variables you expect in 'description' — that free text is how humans and agents discover them. Macros from library templates are made available by listing them in 'imports'; the body must NOT contain import or include directives, they are synthesized from the imports array. The 'type' is chosen here and never changes afterwards: 'sql' (default) requires a dialect and is what pipeline nodes reference; 'html' takes no dialect and renders through an auto-escaping engine. A NEW top-level folder is refused until you confirm it: reuse an existing root, or ask the person first and then pass confirm_new_root: true. Version 1 lands as a DRAFT: a pipeline draft may pin it and render against it while you iterate, and a human releases it from the UI — a RELEASED pipeline may only pin RELEASED template versions, so the template is released first.",
"inputSchema": {
"type": "object",
"required": ["display_name", "description", "body"],
"properties": {
"id": {"type": "string", "pattern": "^[a-z0-9][a-z0-9_.-]{0,63}(/[a-z0-9][a-z0-9_.-]{0,63}){1,9}$", "description": "Template id, and a FOLDER PATH: 2-10 lower-case '/'-separated segments (acme/finance/daily_orders.sql). A FOLDER IS REQUIRED — a bare 'daily_orders.sql' is refused with template.validation.id_invalid and details.reason='folder_required'; put experiments under test/, and shared macros under <owner>/lib/. Keep a template under the same prefix as the pipelines that read it. Optional; auto-generated if omitted. There is no rename, so choose the folder now."},
"engine": {"type": "string", "enum": ["freemarker"], "default": "freemarker", "description": "Template engine. v1 supports freemarker only."},
"type": {"type": "string", "enum": ["sql", "html"], "default": "sql", "description": "Template kind, fixed at creation and identical on every version: 'sql' renders SQL for pipeline nodes (requires 'dialect'); 'html' renders HTML through an auto-escaping engine (must have NO 'dialect')."},
"dialect": {"type": "string", "enum": ["POSTGRES", "ORACLE", "MSSQL", "MYSQL", "H2", "DUCKDB", "SQLITE", "LAKE"], "description": "SQL execution target. Required when type is 'sql' (the default); forbidden when type is 'html' — an html template declares no dialect."},
"display_name": {"type": "string"},
"description": {"type": "string", "description": "Free text. State the variables the body expects and their types — the template declares none."},
"imports": {
"type": "array",
"description": "Library templates whose macros this body calls. Aliases must be unique within the template; each referenced template must exist at that exact version and be is_library=true.",
"items": {
"type": "object",
"required": ["id", "version", "alias"],
"properties": {
"id": {"type": "string"},
"version": {"type": "integer"},
"alias": {"type": "string", "description": "Namespace the macros are bound to, e.g. 'dates' → <@dates.date_range .../>."}
},
"additionalProperties": false
}
},
"is_library": {"type": "boolean", "default": false, "description": "true if this template exists to be imported by others. A library body contains only <#macro>/<#function> definitions — no output outside macro definitions. body is still required."},
"body": {"type": "string", "description": "Template source. Must not contain <#import> or <#include>."},
"confirm_new_root": {"type": "boolean", "description": "Set true ONLY after a person has agreed to a new top-level folder. A name whose root segment has no pipelines or templates under it yet is refused with details.existing_roots listing the roots that do exist — reuse one of those, or ask the person first and then pass this. 'test/' never needs it."}
},
"additionalProperties": false
}
}
A new ROOT folder needs the person's say-so (094). Same rule as §6.2.4, same confirm_new_root argument, same details.root / details.existing_roots shape — the code is template.validation.new_root_requires_confirmation and the roots come from templates_list {"prefix": ""}. An OMITTED id is generated under test/ and needs no confirmation.
Save-time validation is parse-only — syntax, forbidden constructs, import resolution, and the type/dialect consistency rules (sql requires dialect, html forbids it; a payload trying to change an existing template's type is refused — Templates §7.1). A template is never rendered against a sample context at save time, because it does not know its callers' parameters; the dry-render check happens when a pipeline referencing it is saved (Templates §7.2). An agent authoring a template should therefore call templates_render (§6.2.9) with a representative context to confirm the output it produces.
Scope: author.
6.2.9 templates_render
Render a template against a supplied context (preview SQL).
{
"name": "templates_render",
"description": "Render a template against the provided context values and return the SQL it produces. Use this to preview generated SQL before creating a pipeline that references the template. The context is a free-form map: supply the same keys the calling pipeline would declare as parameters. Referencing a key absent from the context fails the render — that is the same failure a pipeline save would report.",
"inputSchema": {
"type": "object",
"required": ["id", "context"],
"properties": {
"id": {"type": "string"},
"version": {"type": "integer", "description": "Defaults to latest."},
"context": {
"type": "object",
"description": "Render context: the parameter map a calling pipeline would provide, defaults already applied. Values follow the wire conventions of the Type System (BIGINTEGER/BIGDECIMAL as strings, TIMESTAMP with Z or offset).",
"additionalProperties": true
}
},
"additionalProperties": false
}
}
Returns: rendered SQL string. This is a preview only — nothing is executed and nothing is stored.
Scope: author (it is the authoring loop's preview step; see the Auth §7.6 matrix).
6.2.10 datasources_list
List registered datasources (without credentials).
{
"name": "datasources_list",
"description": "List the datasources GRANTED to the key's pinned workspace. Visibility is the grant: a datasource registered elsewhere and not granted to this workspace is ABSENT, not hidden, and there is no such thing as a global datasource. This is the learn-first read: every entry carries name, dialect, description, the readonly flag, the workspace that REGISTERED it (omitted for an instance-level one), granted:true, connection metadata — never passwords — AND `facts`, the datasource-wide learned facts earlier sessions recorded (its time window, whether it is a sample), each with `trust` and evidence — read them before you assume coverage — AND `definitions`, this workspace's rules (definition, exclusion, preference) earlier pipelines chose on this datasource: a definition is a rule an earlier pipeline chose; read it before you choose yours, and reuse or supersede it, never re-choose. `datasources_get` is the same payload for ONE datasource — the refresh to call after recording facts with semantics_record.",
"inputSchema": {
"type": "object",
"properties": {
"dialect": {"type": "string"}
}
}
}
Scope: read. (Registering, editing and deleting a datasource are UI/REST-only — §6.2.22: no credential travels through an agent.) The listing is scoped to the key's pinned workspace exactly like REST §9.2.
Returns: one entry per granted datasource in the §6.2.11 per-datasource shape, facts included (126): the datasource-wide LEARNED FACTS (kinds window and sampling, the learned-semantic-layer design §7.2) in the §7A.5 fact shape — [] when none are recorded, an empty array never an absent key, so "nothing recorded" reads differently from "not served". Served as stored: the listing opens no connection, so there is nothing to recompute drift against (the 118 rule), and one enrichment read runs per listed datasource. definitions included (136 §A / T287): every WORKSPACE-scope fact (definition, exclusion, preference) visible to the reader that names this datasource — with refs on its tables or columns, or with none (§6.2.37, a rule that spans datasources) — in the same §7A.5 fact shape, newest last, [] when none; a definition is a rule an earlier pipeline chose; read it before you choose yours. facts keeps its datasource-wide meaning (window, sampling) — a rule is never in both. Both blocks come from the one enrichment read. The dialect filter behaves as before.
6.2.11 datasources_get
Fetch a single datasource (without password).
{
"name": "datasources_get",
"description": "Get metadata for a single datasource GRANTED to the key's pinned workspace: name, dialect, JDBC URL, the workspace that REGISTERED it (omitted for an instance-level one), granted:true, readonly flag, pool settings, `facts` — the datasource-wide learned facts agents recorded (its time window, whether it is a sample) — read them before you assume coverage — and `definitions`, this workspace's rules (definition, exclusion, preference) earlier pipelines chose on this datasource: a definition is a rule an earlier pipeline chose; read it before you choose yours. Credentials are never returned. A datasource that is not granted to this workspace resolves as not-found — the same answer a name that exists nowhere gets, so nothing about it can be probed.",
"inputSchema": {
"type": "object",
"required": ["name"],
"properties": {
"name": {"type": "string"}
}
}
}
Scope: read.
Returns: name, display_name, description, dialect, jdbc_url, username, query_timeout_seconds, pool (the hikari map), readonly (boolean — the §5.7 flag, machine-readable so an agent can see BEFORE authoring that DML/DDL/output-datasource uses will be refused), granted (always true — you are seeing this row because your workspace holds a grant on it, D-R7), workspace (the name of the workspace that REGISTERED it, omitted when a super admin registered it at the instance level — it is never null, because the retired null = global reading would be the wrong answer to "who can see this?"; visibility is granted) — plus introspection_include_schemas (Datasources §3.3) when the allowlist is non-empty (omitted when empty, the same envelope convention as REST §3.2), so an agent debugging why a schema is or isn't visible in the §6.2.16–18 introspection tools can see that an allowlist is active. Credentials are never returned. datasources_list (§6.2.10) emits the same per-datasource shape, facts included (126 — the listing is the learn-first read, so the facts ride the call every agent already makes): the datasource-wide LEARNED FACTS (kinds window and sampling, the learned-semantic-layer design §7.2), an array in the §7A.5 fact shape below, [] when none; served as stored, because this read opens no connection and has nothing to recompute drift against — and definitions (136 §A), the same block §6.2.10 describes: this workspace's rules (definition, exclusion, preference) on this datasource, newest last, [] when none; a definition is a rule an earlier pipeline chose; read it before you choose yours.
6.2.12 datasources_test
Test that a datasource connection can be established.
{
"name": "datasources_test",
"description": "Test connectivity to a datasource. Returns success/failure and server version on success. Useful for diagnosing pipeline connection errors.",
"inputSchema": {
"type": "object",
"required": ["name"],
"properties": {
"name": {"type": "string"}
}
}
}
Returns: {connected: bool, server_version: string?, error: string?}.
Scope: author — testing a connection opens a real pool against a production database, so it sits above plain read even though it mutates nothing.
6.2.13 executions_list
List recent executions.
{
"name": "executions_list",
"description": "List recent pipeline executions, optionally filtered by pipeline or status.",
"inputSchema": {
"type": "object",
"properties": {
"pipeline_id": {"type": "string", "format": "uuid"},
"status": {"type": "string", "enum": ["RUNNING", "SUCCESS", "FAILED", "ABORTED"]},
"limit": {"type": "integer", "default": 50, "maximum": 200}
}
}
}
Scope: read.
6.2.14 executions_get
Fetch metadata for a specific execution (no rows).
{
"name": "executions_get",
"description": "Get metadata for a specific execution: status, timing, node_stats, parameters used. On a FAILED execution, error carries the full failure record: code, message, correlation_id, node context (datasource, dialect, pinned template), the rendered SQL (:name form, no bound values) and the exception chain with stack frames — read error.code first, then error.exception.caused_by (root cause LAST), then error.sql; quote error.correlation_id when escalating. To get the result rows, use executions_get_result.",
"inputSchema": {
"type": "object",
"required": ["execution_id"],
"properties": {
"execution_id": {"type": "string", "format": "uuid"}
}
}
}
Scope: read.
Response (057, on a FAILED execution): error is the full failure record — the same object the SSE stream carried and error_json stores. (Plain fence: an example, not a tool definition — §6.2's json fences are exactly the input schemas McpToolSurfaceSpecDriftTest pins.)
{
"status": "FAILED",
"failed_node_id": "stage_daily_trips",
"error": {
"code": "pipeline.node.datasource_connection_failed",
"message": "Failed to initialize pool",
"details": {"phase": "connect", "node_id": "stage_daily_trips"},
"correlation_id": "1b0e6a52-…",
"node": {"id": "stage_daily_trips", "type": "DQL", "datasource": "sample-trips", "dialect": "POSTGRES",
"template": "nyc/mobility/sample_trips_daily.sql", "template_version": 1},
"sql": "SELECT * FROM trips WHERE borough = :borough",
"exception": {
"class": "java.lang.RuntimeException", "message": "Failed to initialize pool",
"frames": ["…capped at 40 per level…"],
"caused_by": [
{"class": "org.postgresql.util.PSQLException",
"message": "FATAL: password authentication failed for user \"dp_demo_ro\"", "frames": ["…"]}
]
}
}
}
Read error.code first, then error.exception.caused_by (outermost first — the root cause is the LAST entry), then error.sql. Quote error.correlation_id when escalating to a human; it is the field that joins the page to the server log. Under datapipelines.executions.error-detail=structured (Configuration §3.11) the exception and sql keys are absent; everything else stays. pipelines_execute's failure result (§6.3) carries the same record inline, so a failed run's error needs no second call.
6.2.15 executions_get_result
Fetch result rows (paginated) for a completed execution.
{
"name": "executions_get_result",
"description": "Fetch result rows for a completed execution, paginated via offset+limit. Returns schema + rows + pagination metadata. Works for ANY completed execution that produced a caller result, of any size, until its TTL expires (default 300s, set at execution time). Order is stable across pages. Reading pages does NOT extend the TTL — after expiry the result is gone and the pipeline must be re-run.",
"inputSchema": {
"type": "object",
"required": ["execution_id"],
"properties": {
"execution_id": {"type": "string", "format": "uuid"},
"offset": {"type": "integer", "default": 0, "minimum": 0},
"limit": {"type": "integer", "default": 1000, "minimum": 1, "maximum": 100000, "description": "Rows per page. Defaults to the server's result page size."},
"format": {"type": "string", "enum": ["json", "arrow", "csv"], "default": "json"}
},
"additionalProperties": false
}
}
This tool is a thin adapter over the REST cursor, REST API §7 — identical semantics, identical guarantees:
offset/limit/formatmap one-to-one onto the cursor's query parameters.offset+limitpaging over a result fully materialized in Redis before the cursor exists, so ordering is stable across pages.- Availability is uniform: every completed execution with a caller node has its result stored, regardless of size. There is no inline-vs-claim-check distinction to reason about (that split was removed in REST API v1.3).
- TTL is fixed at result-write time (
datapipelines.result.ttl-default-seconds, clamped between the min/max keys; a client may request one on the execute call viaDP-Result-TTL-Seconds). Page reads never extend it. - Auth:
readscope plus ownership of the execution —adminmay read any. Same rule as the REST cursor; theresult_urlis not a capability URL.
JSON format returns {schema, rows, row_count, offset, limit, total_rows, has_more, expires_at} — same body as REST §7.3.
Binary columns and non-JSON formats. BINARY column values in JSON results are base64 per the Type System's egress rules. For format: "arrow" or "csv", and for any result whose encoded payload would exceed 1 MB, the tool does not inline the bytes: it returns {"result_url": "...", "expires_at": "...", "format": "...", "total_rows": N, "reason": "payload_exceeds_inline_cap"} — the REST cursor URL, which the agent fetches with the same API key. Rationale: MCP tool results are model context; megabytes of base64 in a tool result poison an agent's window for no benefit. Payloads at or under the cap are inlined as base64 with their content type named.
Errors mirror REST §7.6 exactly, returned as tool results with isError: true (§9.2) — registry of record Pipeline Contract §13.10:
| Code | Meaning for the agent |
|---|---|
result.execution_not_found |
Unknown execution id — check executions_list. |
result.execution_incomplete |
Still running; wait or re-check with executions_get. |
result.execution_failed |
The execution failed; there is no result. Use executions_get for the failure. |
result.expired |
TTL elapsed. Re-run the pipeline — the result is unrecoverable. |
result.format_unsupported |
Unknown format value. |
Scope: read (+ ownership).
6.2.16 datasources_get_schemas
List a datasource's schemas — the entry point of the introspection flow.
{
"name": "datasources_get_schemas",
"description": "List the namespaces of a registered datasource by reading its live JDBC metadata, excluding the engine's own system schemas. The entry point of schema discovery: call this first, then get_tables(namespace), then get_columns for only the tables the SQL needs. Each entry carries an ordered `namespace` path and a `label`; on a two-level engine (catalog.schema, project.dataset) two entries can share a label and differ only by their outer segment, so pass the whole `namespace` back rather than the label. `schemas` repeats the labels for older clients. An empty list on a datasource with no namespaces is a valid answer. Read-only, for pipeline authoring.",
"inputSchema": {
"type": "object",
"required": ["name"],
"properties": {
"name": {"type": "string", "description": "Datasource name."}
}
}
}
Returns: {"schemas": ["label", ...], "entries": [{"namespace": [...], "label": "..."}], "truncated": bool} — the namespaces exactly as the driver reported them, as a page. Read entries: namespace is the ordered path (outermost first) to pass back to datasources_get_tables/_get_columns, and label is its last segment. On a two-level engine (catalog.schema, project.dataset) two entries can share a label and differ only by their outer segment, which is what the array form exists for; schemas repeats the labels for pre-087 clients and is kept for one release. truncated: true means the 2000-entry cap dropped some (on MySQL catalog routing the walk would otherwise span every database the server grants). On MySQL the databases arrive as JDBC catalogs (Connector/J defaults), so the listing reads them from getCatalogs() — the same vocabulary datasources_get_tables routes through; system schemas/databases (information_schema, mysql, performance_schema, sys on MySQL) are excluded on every dialect. An empty list is a valid result — a datasource with no namespace dimension (SQLite) has none to list. See Datasources §7A. A connection failure against the datasource is the catalogued pipeline.execution.datasource_unreachable isError envelope — the same rule applies to §6.2.17/§6.2.18.
Scope: author — introspection opens a live connection against the datasource, matching the datasources_test precedent.
6.2.17 datasources_get_tables
List a datasource's tables and views.
{
"name": "datasources_get_tables",
"description": "List the tables and views of a registered datasource by reading its live JDBC metadata. The listing spans namespaces — pass each table's reported `namespace` array to datasources_get_columns. A table carries `facts` when agents have recorded table-level learned facts on it (grain, sampling, window, a caveat) — read them before probing; a fact marked stale or needs_review is a warning, not a truth. For a LAKE datasource each table carries `partition_column` — a name means the table is hive-partitioned on it and a filter on that column prunes files; `null` means one unpartitioned file, every read scans it, and no filter prunes. Read-only, for pipeline authoring.",
"inputSchema": {
"type": "object",
"required": ["name"],
"properties": {
"name": {"type": "string", "description": "Datasource name."},
"namespace": {
"type": "array",
"items": {"type": "string"},
"description": "Optional namespace filter, outermost first, as returned by datasources_get_schemas. An unknown namespace matches nothing."
},
"schema": {"type": "string", "description": "Optional single-level filter; accepts the dotted 'catalog.schema' form. Superseded by namespace."}
}
}
}
Returns: {"tables": [{"namespace": [...], "schema", "name", "type", "remarks"?}], "truncated": bool} — namespace is the containing path outermost-first and schema its last segment (kept for one release so a pre-087 client reads what it always read); type is the driver's raw JDBC table type (TABLE, VIEW, BASE TABLE, ...); remarks is the engine-stored table comment, omitted when the driver/database has none; facts (118) is the table's TABLE-GRAIN learned facts — refs with no column: grain, window, sampling, a table-level caveat — in the §7A.5 shape, omitted when there are none. For a LAKE datasource each table entry also carries partition_column (126): the registered partition column's name, or null for a table with none — the same registry value §6.2.33's stats payload reports, so the listing and the stats read cannot disagree; a name means the table is hive-partitioned on it and a filter on that column prunes files, null means one unpartitioned file every read scans whole (filter pushdown inside a file is not pruning). Non-lake dialects carry no such key. Only a COMPLETE listing (no namespace/schema filter, not truncated) runs the §7A.5 drift mark for a fact whose table is no longer listed; a filtered listing proves nothing about what it did not list. The listing is capped at 2000 tables; truncated: true means the cap dropped some. The namespace and schema filters are exact-match, not LIKE patterns, and a namespace deeper than the dialect's own matches nothing. Without a filter the listing spans namespaces — pass each table's reported namespace to datasources_get_columns (there, no namespace argument means the connection's current one only, and a datasource reporting none fails with the catalogued pipeline.execution.parameter_required rather than merging same-named tables' columns — the merge hazard lives in datasources_get_columns alone; a tables listing carries each row's own namespace and cannot merge, so it deliberately has no such guard and works unfiltered on those datasources too).
Scope: author — introspection opens a live connection against the datasource, matching the datasources_test precedent.
6.2.18 datasources_get_columns
List one table's columns with canonical types.
{
"name": "datasources_get_columns",
"description": "List one table's columns with canonical types, read from the datasource's live JDBC metadata. Pass the table name exactly as datasources_get_tables returned it, and its `namespace` array with it. Without a namespace only the connection's current one is read; if the datasource reports none, an explicit namespace is required (list them with datasources_get_schemas). On a two-level engine an unqualified read can merge same-named tables from different catalogs, which is why the namespace is worth passing. Each column carries `facts` when agents have recorded learned facts on it — a unit, a time zone, what a coded value means, a join, a caveat — with trust and evidence: read them before probing, and treat stale or needs_review as a warning to re-verify, then record the superseding fact. An unknown table is refused as datasource.table_not_found, naming the nearest listed table when one is close; a LAKE datasource answers datasource.lake_table_not_found for a table its registry does not carry. Read-only, for pipeline authoring.",
"inputSchema": {
"type": "object",
"required": ["name", "table"],
"properties": {
"name": {"type": "string", "description": "Datasource name."},
"table": {"type": "string", "description": "Table name as returned by datasources_get_tables."},
"namespace": {
"type": "array",
"items": {"type": "string"},
"description": "The table's namespace, outermost first, as datasources_get_tables reported it. An unknown namespace matches nothing."
},
"schema": {"type": "string", "description": "Optional single-level filter; accepts the dotted 'catalog.schema' form. Superseded by namespace."}
}
}
}
Returns: array of {"name", "type", "precision", "scale", "nullable", "source_type", "warnings", "remarks"} — type is the canonical Type System type, source_type the driver's own type name, warnings the ingress mapper's warning messages (empty when the mapping was clean), remarks the engine-stored column comment (omitted when there is none); precision/scale/nullable/remarks are omitted when the metadata does not report them; facts (118) is the column's learned facts in the §7A.5 shape — every fact with a ref on this column, a two-column join fact on both — omitted when there are none. This is the read that runs the §7A.5 drift check: each fact's fingerprint is recomputed from the columns just listed, and a demotion (needs_review, stale) is written back to the row before it is served. An unknown table is refused as datasource.table_not_found (123 §A): the message asserts "does not exist" on complete-catalog dialects and "does not exist, or the datasource's credentials cannot see it" on privilege-filtered ones, and names the nearest listed table when one is close — an existing table with zero readable columns still returns an empty list. table and schema are exact-match identifiers — JDBC metadata name matching is case-sensitive, _/% are not wildcards; pass the name datasources_get_tables returned. System-schema rows are excluded; without a schema argument the read defaults to the connection's current schema (routed per dialect, Datasources §7A) so same-named tables in different schemas cannot merge their columns — and a datasource that reports no current schema makes that default impossible, so the call fails with the catalogued pipeline.execution.parameter_required instead of silently merging (datasources_get_schemas lists the schemas to pass; schemaless datasources such as SQLite are the exception — there is nothing to merge).
Scope: author — introspection opens a live connection against the datasource, matching the datasources_test precedent.
6.2.18a The learned-fact block on introspection responses (118)
The three introspection tools above, datasources_list (126) and datasources_get carry the facts agents recorded about the objects they return — and, since 136 §A, the workspace's rules under a second key, definitions, on the listing and on datasources_get (§6.2.10/§6.2.11) (learned-semantic-layer design D-S7, §7.2) — INLINE, so the fact is where the agent is already looking and there is no separate memory to forget to query. The REST twins (GET /api/v1/datasources/{name}, .../tables, .../tables/{t}/columns, REST API §9.7A) carry the identical block from the same code. Each fact renders as:
{
"id": "…",
"scope": "DATASOURCE",
"kind": "unit",
"fact": "value is already in the unit named by unit_col — never tenths",
"trust": "observed",
"drift": "column X no longer exists",
"evidence_summary": "unit=°C, value=21.4 | unit=mm, value=0.8",
"recorded_via": "mcp",
"recorded_at": "2026-09-11T10:15:00Z",
"from_this_workspace": true,
"source_pipeline": {"id": "…", "name": "finance/revenue"},
"conflict": true
}
drift, evidence_summary, source_pipeline and conflict are omitted-when-absent. trust is asserted (no evidence) / observed (evidence ran at record time) / verified (a human confirmed) / needs_review (the table's column set changed around a still-resolving ref) / stale (a referenced column or table no longer exists); retired facts are never served here (semantics_list with include_retired lists them). The drift check runs at read (design §6): datasources_get_columns recomputes each fact's per-table fingerprint from the columns it just read and a complete datasources_get_tables listing checks each fact's tables — a demotion is written back to the row (idempotent, one-way; the alternative is serving a mark the server computed and then forgot). Nothing re-maps: a rename and a "drop + unrelated add" are indistinguishable to a machine, so a stale fact stays beside the current columns until an agent records the superseding fact with evidence (semantics_record with supersedes). Conflicts coexist (D-S5): two live facts of one kind on the same refs are both served, each conflict: true; the reader decides. Provenance stops at the reader's visibility (D-S9): a DATASOURCE fact recorded from another workspace arrives with its evidence and trust and from_this_workspace: false; source_pipeline is present only when the reader's workspace can read that pipeline — the same findById(workspace, id) predicate every pipeline read uses.
6.2.19 datasources_preview_rows
Preview up to limit rows of one table's data — the counterpart to datasources_get_columns.
{
"name": "datasources_preview_rows",
"description": "Preview up to `limit` rows of one table's data, read live from the datasource. The counterpart to datasources_get_columns: this shows the DATA, that shows the shape. Without order_by the top-N is engine-arbitrary; pass order_by to see a chosen end of the data, e.g. direction DESC for the newest or largest rows. Read-only (SELECT); readonly datasources are valid targets. Values arrive wire-encoded: BIGINTEGER and BIGDECIMAL as strings, temporal as fixed-width ISO forms. An unknown table is refused as datasource.table_not_found naming the nearest listed table; a table the datasource's credentials cannot read is datasource.table_forbidden.",
"inputSchema": {
"type": "object",
"required": ["name", "table"],
"properties": {
"name": {"type": "string", "description": "Datasource name."},
"table": {"type": "string", "description": "Table name exactly as datasources_get_tables returned it."},
"schema": {"type": "string", "description": "Optional schema qualifier. Omitted means the connection's current schema."},
"order_by": {
"type": "array",
"description": "Sort terms applied in order. Each is an object with a column and a direction, never a free SQL string.",
"items": {
"type": "object",
"required": ["column"],
"properties": {
"column": {"type": "string", "description": "Column name exactly as datasources_get_columns returned it."},
"direction": {"type": "string", "enum": ["ASC", "DESC"], "default": "ASC"}
}
}
},
"limit": {"type": "integer", "default": 50, "minimum": 1, "maximum": 50}
}
}
}
Returns: {"datasource", "table", "schema"?, "columns": [{"name", "type"}], "rows": [{column: value, ...}], "row_count", "truncated"} — values wire-encoded per the result-cursor rules (BIGINTEGER/BIGDECIMAL as strings, temporal fixed-width ISO, BINARY base64). The server builds the ENTIRE statement and quotes every identifier with the dialect's quote character (backtick on MySQL, [...] on MSSQL, doubled " elsewhere) — the agent supplies identifiers only, and a blank identifier is -32602. order_by entries are {column, direction} objects; a free "col DESC" string is refused, not parsed. Without order_by the top-N is engine-arbitrary. The cap is applied in the dialect's own syntax (LIMIT, Oracle FETCH FIRST n ROWS ONLY, MSSQL TOP (n)) AND as JDBC maxRows/fetchSize. Readonly datasources are valid targets — the readonly refusal covers write-shaped node uses, and this is a SELECT. tempdb can never be a target: a datasource of that name cannot be registered (contract §4.8). The table is resolved before any statement runs (123 §A): an unknown table is datasource.table_not_found, naming the nearest listed table when one is close; a table the datasource's credentials cannot read — the SELECT refused with a permission SQLSTATE — is 403 datasource.table_forbidden. A connection failure is the catalogued pipeline.execution.datasource_unreachable; any other refused statement is the catalogued pipeline.node.query_execution_failed carrying the bounded driver message.
Scope: author — this returns arbitrary customer ROW DATA, not metadata; a read-scoped key does not acquire that reach (037 F).
6.2.20 pipelines_execute_node
Runs ONE pipeline node's rendered SQL against its own datasource — a debug query, not an execution.
{
"name": "pipelines_execute_node",
"description": "Runs ONE pipeline node's rendered SQL against its own datasource and returns up to 50 decoded rows — a debug query for testing a node in isolation, NOT a pipeline execution. DML and DDL nodes execute FOR REAL against the datasource, leaving no execution history or trace. No ancestors run and no tempdb exists: a node whose source is tempdb is refused. Parameters bind through the pipeline's declarations; unsupplied required parameters fall back to sample values and the response names them in sampled_parameters. Absent version runs the DRAFT if one exists, else the current released version; the response states which version and status ran. A draft whose pinned draft template was updated after this key's last templates_render of it is refused pipeline.execution.template_unrendered — render, then run.",
"inputSchema": {
"type": "object",
"required": ["pipeline_id", "node_id"],
"properties": {
"pipeline_id": {"type": "string", "format": "uuid"},
"node_id": {"type": "string", "description": "Node id within the pipeline body."},
"version": {"type": "integer", "minimum": 1, "description": "Pipeline version to read. Omitted: the DRAFT if one exists, else the current released version."},
"parameters": {
"type": "object",
"description": "Values for the pipeline's declared parameters, keyed by name. Types follow the declarations (BIGINTEGER and BIGDECIMAL as strings).",
"additionalProperties": true
}
}
}
}
Returns: {"node_id", "node_type", "datasource", "version", "status", "sql", "sampled_parameters"?, "elapsed_ms"} plus, for DQL nodes, {"columns": [{"name", "type"}], "rows": [{column: value, ...}], "row_count", "truncated"} (capped at 50 rows like datasources_preview_rows), or for DML/DDL nodes {"affected_rows"} — DML/DDL execute for real, with no execution row, no SSE and no idempotency record (ratified, 037 §A). sql is the rendered template output in its :name form; execution binds those names as statement parameters. status is the run version's lifecycle status (DRAFT/RELEASED), so the agent never infers which body it ran.
Refusals, all before anything runs: an unknown node id → pipeline.node.not_found (404 semantics); a node whose source is tempdb → pipeline.node.standalone_execution_refused with details.reason = tempdb_source (the staging database exists only inside a full execution — use pipelines_execute); a PIPELINE node → the same code with details.reason = pipeline_node (it runs a child pipeline, not SQL); a missing pinned template → pipeline.node.template_not_found; a render failure → pipeline.node.template_render_failed; a supplied parameter failing §6.3 coercion → pipeline.execution.invalid_parameter_type naming every failure; a rendered :name the pipeline does not declare → pipeline.node.sql_parameter_missing (042). A DML/DDL node against a readonly datasource is refused with pipeline.node.datasource_readonly; a datasource invisible to the key's workspace resolves as datasource.not_found, and an unreachable datasource as pipeline.execution.datasource_unreachable; a refused statement is pipeline.node.query_execution_failed.
Scope: author — this runs real SQL and returns arbitrary customer row data; the same 037 F reasoning as datasources_preview_rows.
6.2.21 templates_used_by
Which pipelines pin a given template version — the reverse arrow of a node's {id, version} template pin (040).
{
"name": "templates_used_by",
"description": "Which pipelines pin a given template version in their working version (the draft when unreleased edits exist, else the latest released). Returns one reference per node — pipeline name and id, node id, and the pipeline version carrying the pin — plus the distinct pipeline count. Use it before editing or retiring a template version to see who you would affect. It does not answer 'is it safe to delete' (that scan includes historical pipeline versions and lives in the delete refusal), and it never changes anything.",
"inputSchema": {
"type": "object",
"required": ["id", "version"],
"properties": {
"id": {"type": "string", "description": "Template id."},
"version": {"type": "integer", "description": "The pinned version to look for."}
},
"additionalProperties": false
}
}
Returns: {"template": {"id", "version"}, "scan": "working_version", "pipeline_count", "references": [{"pipeline", "pipeline_id", "node_id", "pipeline_version", "pipeline_version_status"}]} — one row per pinning NODE, so a pipeline with two nodes on the same version appears twice and pipeline_count stays the honest distinct count. The scan reads each pipeline's working version (draft-if-exists, versioning §7), so a draft that just adopted the pin is already counted — the "who do I notify" answer an author needs before editing a template (Templates §5.4). version is required and never clamped (the D2 rule: the question is per version). The delete-safety question — who pins ANY version, in ANY pipeline version ever — is a different scan and surfaces as the template.in_use delete refusal on the REST surface, not here.
An unknown template id is the catalogued template.not_found; a known id with no such version is the same code with a version detail. Scope: read (040 D7) — reference structure a workspace reader may already see by reading the pipelines themselves; no customer row data.
6.2.22 (removed) — no datasource writes on this surface
No credential travels through an agent. People add datasources in the UI; an operator uses POST /api/v1/datasources (Datasources §3.1) or the bootstrap file (Datasources §8A); agents use the datasources that already exist, by name. Creating, editing and deleting a datasource are UI/REST-only, without exception.
068 shipped a datasources_create tool here and wrote the hazard into its own description: a secret sent through a tool call transits the agent's context, its transcript and whatever logging the client does, so "prefer the UI for a real credential" was an instruction to the model. 094 (owner ruling 4) decided that hazard is not documentable away — the tool's own description is not a control — and removed it. The surface went 31 → 30 tools; this section number is kept so §6.2.23 onward do not shift.
What stays is every datasource tool that needs no credential: datasources_list, datasources_get, datasources_test, datasources_get_schemas, datasources_get_tables, datasources_get_columns, datasources_get_table_stats, datasources_preview_rows and sql_probe. An agent can therefore still discover a connection, confirm it works, read its shape and statistics, probe a SELECT and preview rows — everything authoring a pipeline needs — without one ever being handed a password. The same reasoning is why there is no api_keys_create: see the §6.2.23 preamble.
An agent that needs a datasource it cannot find should ask the person to add it in the UI, then read it back with datasources_list.
6.2.23 endpoints_create
Publish a released pipeline as a GET endpoint under /api/x (REST API §19). Calls the same service POST /api/v1/endpoints calls — the same read-only rule, the same ambiguity refusal, the same audit (049's rule: two entry points, one validated path).
{
"name": "endpoints_create",
"description": "Publish a released pipeline as a GET endpoint under /api/x. The pipeline must have a RELEASED version and must be side-effect-free: every node DQL into tempdb or the caller, transitively through PIPELINE nodes. A DML/DDL node, or a DQL node writing back to a datasource, is refused with endpoint.pipeline_not_readonly naming the node — that rule is what makes serving over GET safe, since GET is retried, preloaded and crawled. path is 1-10 segments, each a literal [a-z0-9][a-z0-9_.-]{0,63} or a {variable} naming a declared parameter; remaining parameters come from the query string. A path that could match the same URL as an existing one is refused (endpoint.path_conflict) rather than resolved by precedence. Calling the endpoint needs an API key bound to it — mint and bind one over REST or in the UI (auth.md §7.7); an unbound endpoint accepts user keys with the execute scope.",
"inputSchema": {
"type": "object",
"required": [
"path",
"pipeline"
],
"additionalProperties": false,
"properties": {
"path": {
"type": "string",
"description": "e.g. /finance/revenue/{region} — no /api/x prefix, no trailing slash."
},
"pipeline": {
"type": "string",
"description": "The pipeline NAME. It must have a released version."
},
"timeout_seconds": {
"type": "integer",
"description": "Clamped by datapipelines.endpoints.timeout-min-seconds/max-seconds. On timeout the endpoint answers 202 and the execution keeps running."
},
"description": {
"type": "string"
}
}
}
}
Returns the endpoint's wire shape: path, pipeline (by NAME), timeout_seconds, description, enabled, path_variables and the servable url.
6.2.24 endpoints_list
The published endpoints of the key's pinned workspace.
{
"name": "endpoints_list",
"description": "List the published endpoints of the key's workspace: path, pipeline name, timeout, whether it is enabled, and the path variables it binds. A disabled endpoint answers 404 exactly like an unpublished one, so this listing is the only way to see that it exists.",
"inputSchema": {
"type": "object",
"additionalProperties": false,
"properties": {}
}
}
Returns {endpoints: [...]} in the shape §6.2.23 returns.
6.2.25 endpoints_get
One published endpoint, by its path PATTERN.
{
"name": "endpoints_get",
"description": "One published endpoint by its path (the pattern, not a request URL — '/finance/revenue/{region}').",
"inputSchema": {
"type": "object",
"required": [
"path"
],
"additionalProperties": false,
"properties": {
"path": {
"type": "string",
"description": "The published path PATTERN, e.g. /finance/revenue/{region}."
}
}
}
}
Returns the §6.2.23 shape, or endpoint.not_found.
6.2.26 endpoints_delete
Unpublish an endpoint. The pipeline is untouched; key bindings on that node are not removed, since a node may still carry other endpoints beneath it.
{
"name": "endpoints_delete",
"description": "Unpublish an endpoint by its path. The pipeline is untouched — only the URL stops answering. Key bindings on that path are NOT removed: they describe a node of the tree, which may still carry other endpoints beneath it.",
"inputSchema": {
"type": "object",
"required": [
"path"
],
"additionalProperties": false,
"properties": {
"path": {
"type": "string",
"description": "The published path PATTERN, e.g. /finance/revenue/{region}."
}
}
}
}
Returns {path, deleted: true}, or endpoint.not_found.
Scope: author — the same floor datasources_test sits on: registering a connection opens a real pool against a production database at save time. global: true additionally requires admin and is refused with datasource.validation.workspace_forbidden, exactly as REST refuses it; admin-ness is a D8 rule, not a scope (Auth §7.6).
Mutating. Declared mutating in the tool catalog, so every call writes mcp.tool.called and mcp.tool.write at the dispatcher's single audit choke point (§6.3). The audit row carries the datasource NAME and never the credential.
The password caveat — a documented trade-off, not a bug. A password sent to this tool transits the agent's context window, the client's transcript, and whatever logging that client does. No server-side change can undo that; refusing the tool would not undo it either, it would only push operators to paste credentials somewhere worse. So the tool states it, in the description an agent reads before calling. Register a real production credential in the UI or over REST; use this tool with a credential the user is willing to have in that transcript — a read-only role, or a short-lived password they will rotate afterwards.
Suggested next call: datasources_test on the new name. Creation validates and builds a test pool, but the tool does not probe on your behalf.
6.2.23 calculators_list
The catalog of calculator kinds a CALCULATOR node can evaluate (Calculators §2, Pipeline Contract §4.10). The tool an agent calls before authoring a calculator node: a kind and its input names are the two things it cannot guess, and getting them from a 400 one at a time is a slow way to learn a fixed list.
{
"name": "calculators_list",
"description": "The catalog of calculator kinds a CALCULATOR node can evaluate: every kind with its typed inputs (name, type, required, whether it takes a JSON array, and its default when optional), its output type (or, for a multi-output kind, the named `outputs` set a node maps through `context_keys`), one worked example, and `phrases` — the everyday phrases the kind answers. Call this before authoring a CALCULATOR node — the kind names and input names are not guessable — and match the question's words against `phrases` before you pick a kind: a relative time phrase ('last quarter', 'month to date') is resolved by that lookup, never by interpreting it yourself. Also returns the Context keys every pipeline can reference without declaring anything: the deployment's org_* values and the platform keys current_date, current_timestamp and execution_id. Read-only.",
"inputSchema": {
"type": "object",
"properties": {},
"additionalProperties": false
}
}
Scope: read — and read in the strongest sense the surface has: the answer is a property of the BUILD, identical for every caller, every key and every workspace. No workspace scoping applies because there is no workspace data in it.
Response: kinds (each with kind, display_name, description, phrases, inputs, output, example), count, context_keys (org names and platform name/type pairs), and docs pointing at the catalog page. A single-output kind carries output (the wire type, or "ANY") and no outputs key at all; a multi-output kind (121) carries "output": null and outputs — the named set [{name, type, description}] a node maps through context_keys (Pipeline Contract §4.10), every name mapped or the save is refused. phrases lists the everyday phrases the kind answers — match the question's words against them before picking a kind (the same lookup the skill's calculator rule teaches). An input carries list: true only when it takes a JSON array, and default only when it is optional — the absent keys carry the same information as false/null would, without spending an agent's context window on eighty of them.
6.2.24 calculators_get
One kind's full definition — the same entry calculators_list returns, for a caller that already knows the name.
{
"name": "calculators_get",
"description": "One calculator kind's full definition: display name, description, typed inputs, output type (or the named `outputs` set of a multi-output kind), a worked example and `phrases` — the everyday phrases the kind answers, which you match the question's words against before picking a kind. Use it when you know the kind and need its exact input names and types. An unknown kind is refused with the catalogued names in the error detail. Read-only.",
"inputSchema": {
"type": "object",
"required": ["kind"],
"properties": {
"kind": {"type": "string", "description": "The kind name, e.g. fiscal_quarter."}
},
"additionalProperties": false
}
}
Scope: read.
Errors: an unknown kind is pipeline.validation.calculator_unknown with known_kinds in the detail — deliberately the SAME code a rejected pipelines_create returns for a bad kind, so an agent sees one fact about the world rather than two unrelated failures.
6.2.29 lake_tables_register
Register one table in a LAKE datasource's catalog — the dp-lake registry (metadata-db §4.15, Datasources §4.1). A LAKE datasource's tables are exactly the registered rows: the engine cannot list a bucket, so this catalog is what introspection and query resolution read. Mirrors POST /api/v1/datasources/{name}/tables (REST §9.8) — the SAME application service, so validation, the duplicate refusal and the pool invalidation are identical on both surfaces.
{
"name": "lake_tables_register",
"description": "Register one table in a LAKE datasource's catalog (the dp-lake registry). Mirrors POST /api/v1/datasources/{name}/tables: namespace (array of segments or the dotted 'acme.analytics' shorthand), table, format (parquet | iceberg) and location are required; partition_column is optional. The location is s3://bucket/prefix/ (parquet: a directory or glob; iceberg: the table's CURRENT metadata file, e.g. s3://bucket/table/metadata/00042-<uuid>.metadata.json — DuckDB 1.5.5 cannot scan a pyiceberg table by its root, so register the file, and re-register it when the table commits) or a file:// path — no other scheme, and no quotes, backslashes, whitespace or control characters (it is interpolated into the engine's CREATE VIEW, so the refusal is total). Segments follow the pipeline/template segment grammar without dots. Registering an already-registered (namespace, table) is the 409 datasource.lake_table_duplicate; a non-LAKE datasource is refused. Mutating.",
"inputSchema": {
"type": "object",
"required": [
"name",
"namespace",
"table",
"format",
"location"
],
"additionalProperties": false,
"properties": {
"name": {
"type": "string",
"description": "Datasource name. A LAKE datasource visible in the key's pinned workspace."
},
"namespace": {
"description": "Namespace — a segments array or the dotted shorthand. 1-9 segments, the segment grammar without dots.",
"anyOf": [
{
"type": "array",
"items": {
"type": "string"
}
},
{
"type": "string"
}
]
},
"table": {
"type": "string",
"description": "The table's name — one segment of the same grammar, e.g. events_by_day."
},
"format": {
"type": "string",
"enum": [
"parquet",
"iceberg"
]
},
"location": {
"type": "string",
"description": "s3://bucket/prefix/ (parquet dir/glob; iceberg: the current metadata file, not the table root) or file:// path. Nothing else; no injection chars."
},
"partition_column": {
"type": "string",
"description": "Optional. The hive-style partition column, e.g. event_date."
}
}
}
}
Scope: author — the datasource-mutation floor (Auth §7.6). Mutating a GLOBAL datasource's registry additionally requires admin, a workspaces D8 rule inside the shared service rather than a scope.
Mutating. Declared mutating in the tool catalog: every call writes mcp.tool.called and mcp.tool.write at the dispatcher's single audit choke point (§6.3).
Errors: datasource.not_found (unknown or invisible datasource), datasource.validation.lake_dialect_required (not a LAKE datasource), lake_namespace_invalid / lake_name_invalid / lake_format_invalid / lake_location_invalid (the grammar and the injection refusal), datasource.lake_table_duplicate (409 — the triple is taken).
Response: the stored row — namespace, name, qualified_name, format, location, partition_column, registered_at.
6.2.30 lake_tables_import
Bulk-register from a manifest.json tables[] block (the manifest shape documented in the description below), inline or by URL. Mirrors POST /api/v1/datasources/{name}/tables/import. A manifest URL is fetched server-side, and only from the datasource's own bucket/endpoint — derived from its declared dialect.endpoint / catalog.ref, or AWS S3 when neither is set; anything else is refused. There is no arbitrary URL fetch (SSRF).
{
"name": "lake_tables_import",
"description": "Bulk-register lake tables from a manifest's `tables[]` shape. Mirrors POST /api/v1/datasources/{name}/tables/import: pass EITHER tables (an array of {name, format, location|path, partition_column?, namespace?}, with publish_prefix for relative paths and an optional shared namespace) OR manifest_url. A manifest URL is fetched server-side ONLY from the datasource's own endpoint/bucket — derived from its declared dialect.endpoint / catalog.ref, or AWS S3 when neither is set; anything else is refused with datasource.validation.lake_manifest_url_forbidden (no arbitrary URL fetch — SSRF). Import is idempotent: already-registered tables are reported in already_registered, not errors. Mutating.",
"inputSchema": {
"type": "object",
"required": [
"name"
],
"additionalProperties": false,
"properties": {
"name": {
"type": "string",
"description": "Datasource name. A LAKE datasource visible in the key's pinned workspace."
},
"tables": {
"type": "array",
"description": "Inline form: manifest entries {name, format, location|path, partition_column?, namespace?}.",
"items": {
"type": "object"
}
},
"namespace": {
"description": "Shared namespace applied to entries that carry none — an array of segments or the dotted shorthand.",
"anyOf": [
{
"type": "array",
"items": {
"type": "string"
}
},
{
"type": "string"
}
]
},
"publish_prefix": {
"type": "string",
"description": "Base URI resolving relative entry paths, e.g. s3://bucket/lake/v1."
},
"manifest_url": {
"type": "string",
"description": "URL of a manifest.json. Fetched ONLY from the datasource's own endpoint/bucket or AWS S3; else refused."
}
}
}
}
Scope: author. Mutating — same audit pair as lake_tables_register.
Errors: the register set, plus datasource.validation.lake_manifest_url_forbidden (a URL outside the datasource's own roots) and pipeline.execution.datasource_unreachable (502 — the manifest could not be fetched from the datasource's own storage).
Response: registered (the stored rows), registered_count, already_registered (dotted qualified names skipped as already present — import is idempotent, so bootstrap can re-run it), already_registered_count.
6.2.31 lake_tables_unregister
Unregister one table. The objects in the bucket are untouched — the table stops being served by the datasource. Mirrors DELETE /api/v1/datasources/{name}/tables/{namespace}/{table}.
{
"name": "lake_tables_unregister",
"description": "Unregister one table from a LAKE datasource's catalog. Mirrors DELETE /api/v1/datasources/{name}/tables/{namespace}/{table}: the objects in the bucket are untouched — the table stops being served by the datasource. Unregistering a table that is not registered is the 404 datasource.lake_table_not_found, never a silent no-op. Mutating.",
"inputSchema": {
"type": "object",
"required": [
"name",
"namespace",
"table"
],
"additionalProperties": false,
"properties": {
"name": {
"type": "string",
"description": "Datasource name. A LAKE datasource visible in the key's pinned workspace."
},
"namespace": {
"description": "The table's namespace, outermost first — an array of segments or the dotted shorthand ('acme.analytics').",
"anyOf": [
{
"type": "array",
"items": {
"type": "string"
}
},
{
"type": "string"
}
]
},
"table": {
"type": "string",
"description": "The table to unregister."
}
}
}
}
Scope: author. Mutating — same audit pair as lake_tables_register.
Errors: datasource.not_found, datasource.validation.lake_dialect_required, the grammar codes for a malformed namespace/table, and datasource.lake_table_not_found (404 — an absent triple, never a silent no-op).
Response: {datasource, table, deleted: true} with table the dotted qualified name.
6.2.32 templates_purge_draft
Hard-delete a template that has NEVER been released — the one lifecycle verb an agent gets (101's bounded D61/D62 self-service fraction; humans release, and humans discard releases — D4/D57 are untouched).
{
"name": "templates_purge_draft",
"description": "Hard-delete a template that has NEVER been released: the only version is a DRAFT, created by this key's user, and pinned by nothing — no pipeline version anywhere, draft or released, may reference any version of it (the refusal names the pinning pipelines; templates_used_by answers the working-version scan if you need to inspect them). The sole-draft purge takes the entity row with it. A template holding any RELEASED or discarded version, another user's draft, or a pinned draft is refused — humans release and humans discard releases; an agent's own draft that should not exist is what this verb removes. Mutating.",
"inputSchema": {
"type": "object",
"required": ["id"],
"additionalProperties": false,
"properties": {
"id": {"type": "string", "description": "Template id."}
}
}
}
Returns: {"id", "purged": true}.
Admission is exact: the ONLY version is a DRAFT, created by THIS key's user (templates record a creator user, never a key — "this key created it" degrades to "this key's user"), and NOTHING pins any version of it in ANY pipeline version ever — the delete guard's every-version-ever scan, not the working-version scan templates_used_by (§6.2.21) answers. A pinned draft is refused with template.in_use and details.pinned_by naming the pipelines to change first; a template holding any RELEASED or discarded version is template.version.last_release; another user's draft is auth.scope.insufficient with details.reason: "not_creator"; an unknown id is template.not_found. The sole-draft purge takes the entity row with it, and a concurrent edit between the guard reads and the delete is template.version.conflict — never a silent delete of someone else's save.
Scope: author.
Mutating. Declared mutating in the tool catalog, so every call writes the §14 audit pair (mcp.tool.called and mcp.tool.write).
6.2.33 datasources_get_table_stats
One table's catalog statistics — row estimate, indexes, per-column bounds — always from the engine's OWN catalog, never a scan (Datasources §7C). The per-column min/max are the catalog's ESTIMATES (the planner's histogram ends from the last ANALYZE), not a scan — read them as bounds, and sql_probe for the exact bound. The fourth introspection tool, one step past the get_columns flow of §6.2.16–18.
{
"name": "datasources_get_table_stats",
"description": "One table's catalog statistics: a row estimate, the index list (a lake table's partition column reports as the pseudo-index it is), and per-column distinct / null-fraction / min-max bounds — the min/max are the catalog's estimates, not a scan — probe for the exact bound. Every number comes from the engine's own catalog (pg_class, information_schema, parquet footers) — never a scan of the table, so this is safe at any table size. When a dialect holds no catalog stats the stat fields are null and stats_source is \"none\" — probe an explicit count with sql_probe if you need one. An unknown table is refused as datasource.table_not_found, naming the nearest listed table when one is close; a LAKE datasource answers datasource.lake_table_not_found for a table its registry does not carry. Read this before writing a predicate — an unindexed filter on a large table is the timeout you will hit.",
"inputSchema": {
"type": "object",
"required": ["name", "table"],
"properties": {
"name": {"type": "string", "description": "Datasource name."},
"table": {"type": "string", "description": "Table name exactly as datasources_get_tables returned it."},
"namespace": {
"type": "array",
"items": {"type": "string"},
"description": "The table's namespace, outermost first, as datasources_get_tables reported it. An unknown namespace matches nothing."
}
}
}
}
Returns: {row_estimate?, stats_as_of?, stats_source, indexes: [{name, columns, unique, primary, kind}], columns: [{name, n_distinct?, distinct_is_ratio, null_fraction?, min?, max?}]} — the stat fields are omitted-when-null per the envelope convention (a missing row_estimate means "the catalog does not hold this", not zero), and stats_source names the catalog the numbers came from (pg_class, information_schema.tables, sys.partitions, parquet_metadata, …) or "none" when the dialect holds no catalog stats for the table. A LAKE table's registered partition column reports as the {kind: "partition"} pseudo-index — the lake has no indexes; the partition column IS the access structure the engine prunes on. An unknown TABLE is datasource.table_not_found (the §7A resolution, 123 §A — naming the nearest listed table when one is close), so empty stats now mean exactly one thing: the table EXISTS and the catalog holds nothing for it. An unknown datasource is datasource.not_found, and a connection failure is pipeline.execution.datasource_unreachable — the same translations as §6.2.16–18.
Scope: read — the engine's own stored ESTIMATES about shape, never customer row data (the §6.2.21 reasoning), which is why it sits below the sibling introspection tools' author floor.
6.2.34 sql_probe
Run ONE read-only SELECT/WITH against a datasource and get rows, the canonical schema, wall_ms of query time, and the EXPLAIN plan captured BEFORE the query ran — the bounded free-SQL probe (Datasources §7D). A debug probe, not an export.
{
"name": "sql_probe",
"description": "Run ONE read-only SELECT or WITH statement against a datasource and return up to `limit` wire-encoded rows, the canonical column schema, wall_ms of query time, and the EXPLAIN plan captured BEFORE the query ran — a bounded debug probe, not an export. The statement is classified before any connection opens: anything but a single SELECT/WITH, or a denylisted verb (INSERT, DROP, ATTACH, EXPLAIN, INTO, ...) anywhere in it, is refused without touching the datasource. Parameters bind as named :name placeholders through the same binder pipeline SQL uses; every referenced name must be supplied in `parameters` with its canonical type. `tempdb` is a scratch check, not an execution: the statement is prepared against an EMPTY scratch H2 in the staging mode (no staged tables, no data). Read `validation_status`: `executed` (`parsed: true`, rows) means a self-contained statement ran; `incomplete` (`parsed: null`, `missing_table`) means H2 stopped at the first staged table it could not find and NOTHING after that point — syntax or names — was checked; an error is a real H2 error (a syntax slip, a `VALUES` column named column1 where H2 says C1) found in milliseconds instead of a full run. To finish an incomplete check, restate the suspect construct over typed, aliased `VALUES` inputs so it executes here, or run the node with its real staged inputs. On a timeout the error details carry wall_ms and the plan, so the plan that explains the timeout survives it. A statement the database refuses for lack of privilege on an existing table is datasource.table_forbidden. The sql text never reaches the audit log — only its SHA-256 and length are recorded.",
"inputSchema": {
"type": "object",
"required": ["name", "sql"],
"additionalProperties": false,
"properties": {
"name": {"type": "string", "description": "Datasource name, or the reserved name tempdb to prepare a staging (H2) statement against an empty scratch engine. validation_status executed means it ran; incomplete with missing_table means H2 stopped at a staged table that exists only inside a full execution, and the rest of the statement is unverified."},
"sql": {"type": "string", "description": "ONE SELECT or WITH statement. A second statement or a denylisted verb is refused before any connection opens."},
"parameters": {
"type": "object",
"description": "Bind values for the statement's :name placeholders, keyed by name. type is the canonical logical type; value is its wire string (BIGINTEGER/BIGDECIMAL as decimal text, temporal in ISO forms, BINARY as padded base64). A null value binds SQL NULL.",
"additionalProperties": {
"type": "object",
"required": ["type"],
"properties": {
"type": {"type": "string", "enum": ["BOOLEAN","INTEGER","BIGINTEGER","DECIMAL","BIGDECIMAL","STRING","BINARY","DATE","TIME","TIMESTAMP"]},
"value": {"type": ["string", "null"]}
}
}
},
"limit": {"type": "integer", "default": 50, "minimum": 1, "maximum": 500},
"timeout_seconds": {"type": "integer", "default": 10, "minimum": 1, "maximum": 30}
}
}
}
Returns: {columns: [{name, type, precision?, scale?, nullable?}], warnings, rows, row_count_returned, truncated, wall_ms, plan?} — values wire-encoded per the result-cursor rules, exactly like datasources_preview_rows (§6.2.19). plan carries {scan?, estimated_rows?, raw, partitions_scanned?, partitions_total?}; on LAKE the partitions pair is DuckDB's Scanning Files: x/y marker, null when no static file filter pruned (never y/y — a filter selecting every file is optimized away entirely). limit defaults to 50 and clamps to 500; timeout_seconds defaults to 10 and clamps to 30.
Refusals split into two families. Argument faults (JSON-RPC -32602, nothing leased, nothing ran): the statement is not a single SELECT/WITH or carries a denylisted verb (the classifier is a conservative token scan — false positives err toward refusal, and a side-effecting function call it cannot see through is NOT caught; the datasource's own DB-user privileges remain the last line), or a parameters entry is missing/mistyped. Run failures (the catalogued pipeline.node.query_execution_failed isError envelope — a timeout under pipeline.node.query_timeout, Pipeline Contract §13.4): a timeout, whose details carry reason: "timeout", wall_ms and the pre-captured plan — the plan that explains the timeout survives it — or a driver refusal, carrying the bounded driver message. A driver refusal whose SQLSTATE is permission-denied (123 §A) is instead 403 datasource.table_forbidden — a referenced table exists but this datasource's credentials cannot read it. tempdb uses the scratch check below; the real staged tables exist only inside a full execution. The sql argument is audited as sql_sha256 + sql_length, never verbatim (§14).
The tempdb scratch check (2026-09-11; contract corrected 2026-09-16, #119). name: "tempdb" no longer refuses. The statement is PREPARED against a FRESH, EMPTY in-memory H2 opened in the staging engine's own MODE and lower-folding (the URL shape StagingFactory builds, minus the execution) — no registry, no lease, no visibility gate, nothing to read. The payload always carries check: "syntax_and_names" (the check attempted), engine, and validation_status, which says how far the engine got. "incomplete" — H2 reported a missing table (42102/42103/42104): {"check", "engine", "validation_status": "incomplete", "parsed": null, "missing_table": "<the first staged table H2 could not find>", "wall_ms", "note"}. H2 stops preparing at the first table it cannot resolve, so nothing after it has been checked — measured on the pinned 2.3.232: SELECT a.id FROM absent_a a FULL OUTER JOIN absent_b b ON a.id = b.id reports absent_a on the empty scratch and is a 42000 syntax error at OUTER once both tables exist (SqlProbeH2Test); an acceptance run had matched the SQL hash of a "passing" probe to its subsequently failing execution. parsed is deliberately null, not omitted: the key keeps its place and the value withdraws the affirmative the pre-#119 payload made (parsed: true, "the SQL is sound"). The note names the two ways to finish the check — restate the suspect construct over typed, aliased VALUES inputs so it executes on the scratch (validates the construct, not the real column types or data), or use pipelines_execute to run the DAG with its real staged inputs, which validates against the real schema at the cost of a run (pipelines_execute_node cannot read tempdb or reuse staged tables from an earlier execution); templates_render checks what the template emits, not whether it runs. "executed" — the statement was self-contained (a VALUES spine, a constant): parsed: true plus the ordinary probe payload above and a note that this validates the statement as written, not other parameter values or the real staged inputs. Anything else → the catalogued pipeline.node.query_execution_failed with H2's message — never disguised as incomplete. The case that motivated the check: an agent's caller node died on Column "column1" not found (H2 names VALUES columns C1…) after five source nodes ran green — a full DAG run to learn one H2 spelling. Parameters bind exactly as in a real probe; the classifier still admits only one SELECT/WITH; the scratch never creates a missing table from a guessed shape and never reads another execution's tempdb. Compatibility: no tool name, argument or count changed; parsed became nullable on the incomplete branch and validation_status was added — no typed consumer of the payload exists in this repository (the field is read by agents, not code), and this paragraph is the contract.
Scope: author — this returns arbitrary customer ROW DATA; the same 037 F reasoning as datasources_preview_rows.
6.2.35 executions_cancel
Request cancellation of a RUNNING execution — the MCP twin of DELETE /api/v1/executions/{id} (REST API §10.4) with one rule REST does not have: the same-credential rule.
{
"name": "executions_cancel",
"description": "Request cancellation of a RUNNING execution. This key can cancel ONLY an execution its own MCP calls started: the execution must have been triggered via MCP by this key's user, and an audit row must pair this key with the execution's correlation id — an execution started over REST, the UI, a pipeline node, a published endpoint, or another key of the same user is refused, and the refusal names which rule fired. A non-RUNNING execution is refused with its current status. Cancellation is requested, not awaited: the flag reaches the executing instance within about one poll interval (immediately when same-instance); poll executions_get for the terminal ABORTED status. Mutating.",
"inputSchema": {
"type": "object",
"required": ["execution_id"],
"additionalProperties": false,
"properties": {
"execution_id": {"type": "string", "format": "uuid"}
}
}
}
Returns: {execution_id, status: "cancellation_requested"}. Cancellation is REQUESTED, not awaited — the flag reaches the executing instance within about one poll interval (immediately when same-instance); poll executions_get (§6.2.14) for the terminal ABORTED.
The same-credential rule, in order: an execution the caller may not see is the same not-found every read path answers (ownership before anything else); a non-RUNNING execution is pipeline.execution.not_running with its current status in details; an execution started outside MCP (REST, the UI, a PIPELINE node, a published endpoint) is auth.scope.insufficient with details.reason: "started_outside_mcp"; and one started by a DIFFERENT credential — another key of the same user included — is the same code with details.reason: "different_credential". The join that proves "this key started it" is an audit row pairing this key id with the execution's correlation id, because pipeline_executions records the owner USER, never the key. pipelines_execute writes that row (mcp.execution.launched) BEFORE the blocking run begins — the dispatcher's end-of-call mcp.tool.called row cannot exist while the call is still blocking, and in-flight is exactly when a cancel matters.
Scope: execute — the REST twin's floor (the same-credential rule is a handler gate, not expressible as a scope).
Mutating. Declared mutating in the tool catalog — cancellation IS a write (it ends a running execution), and the mcp.tool.write row is the trace of WHOSE key stopped it (§14).
6.2.36 templates_update
Update an existing template by writing its DRAFT (versioning §3.2/§5.1/§5.2) — the MCP twin of REST PUT /templates (§8.4), and the template mirror of pipelines_update. Before 117 the only MCP path to change a template draft was templates_purge_draft + templates_create, and purge is refused the moment any pipeline version pins the template — exactly the state an authoring agent is in mid-build.
{
"name": "templates_update",
"description": "Update an existing template by writing its DRAFT — the first update after a release creates the draft (copy-on-write); later updates overwrite that same draft in place. Requires expected_hash: the body_hash you read (templates_get, or a previous templates_create/templates_update result) for the version you based your edit on. The result carries status='DRAFT' — your work is NOT released; a human releases it from the UI. On template.version.conflict someone modified it after you loaded it: re-read with templates_get, rebase, retry; never retry blindly. The body takes the same fields as templates_create, and the template's type is fixed at creation — an update naming a different type is refused with template.validation.type_immutable. type and dialect are optional: omitted, the working version's are inherited; a different dialect is refused. templates_purge_draft is for a template that was a mistake, not for editing one.",
"inputSchema": {
"type": "object",
"required": ["id", "expected_hash", "display_name", "description", "body"],
"properties": {
"id": {"type": "string", "pattern": "^[a-z0-9][a-z0-9_.-]{0,63}(/[a-z0-9][a-z0-9_.-]{0,63}){1,9}$", "description": "Template to update — the FOLDER PATH id it was created under (acme/finance/daily_orders.sql). Required here: §9.6, the name never travels in a path or anywhere else. There is no rename, so the id cannot change — an unknown id is the catalogued template.not_found."},
"expected_hash": {"type": "string", "description": "The body_hash of the version this edit is based on — templates_get, or a previous templates_create/templates_update result. A mismatch is a 409 template.version.conflict; re-read and rebase, never retry blindly."},
"engine": {"type": "string", "enum": ["freemarker"], "default": "freemarker", "description": "Template engine. v1 supports freemarker only."},
"type": {"type": "string", "enum": ["sql", "html"], "description": "Template kind — fixed at creation, so on an update it is OPTIONAL: omitted, the working version's is inherited; stated, it must equal it (template.validation.type_immutable otherwise)."},
"dialect": {"type": "string", "enum": ["POSTGRES", "ORACLE", "MSSQL", "MYSQL", "H2", "DUCKDB", "SQLITE", "LAKE"], "description": "Optional on update: omit it and the working version's dialect is inherited. When present it must be the dialect the template already has — a different one is refused with template.validation.dialect_invalid (a template pinned by pipeline nodes cannot change engine; create a new template instead). Never present for an html template."},
"display_name": {"type": "string"},
"description": {"type": "string", "description": "Free text. State the variables the body expects and their types — the template declares none."},
"imports": {
"type": "array",
"description": "Library templates whose macros this body calls. Aliases must be unique within the template; each referenced template must exist at that exact version and be is_library=true.",
"items": {
"type": "object",
"required": ["id", "version", "alias"],
"properties": {
"id": {"type": "string"},
"version": {"type": "integer"},
"alias": {"type": "string", "description": "Namespace the macros are bound to, e.g. 'dates' → <@dates.date_range .../>."}
},
"additionalProperties": false
}
},
"is_library": {"type": "boolean", "default": false, "description": "true if this template exists to be imported by others. A library body contains only <#macro>/<#function> definitions — no output outside macro definitions. body is still required."},
"body": {"type": "string", "description": "Template source. Must not contain <#import> or <#include>."}
},
"additionalProperties": false
}
}
The save-time validation is the SAME parse-only set templates_create runs (Templates §7.1); the write is the SAME TemplateDraftService.write call REST §8.4 makes — one write path, two surfaces. First write after a release copies the released version to a draft (copy-on-write); later writes overwrite that one draft in place.
Returns: the stored version's projection — id, version, status, body_hash (carry this into the next write), dialect, type, body, and the draft pointer (version, body_hash, updated_by, updated_at) when the write produced a draft. The update does NOT release (versioning D4): a human releases from the UI, and templates_render (§6.2.9) is the preview step before you hand the draft over. A write whose CONTENT equals the released content is the §5.1 no-op: status: "RELEASED", no draft opened, no version number burned. An unknown id is template.not_found; a stale expected_hash is template.version.conflict with the current state in details; an update naming a different type than the template's established one is template.validation.type_immutable (046 §5.3). type is optional on update too (136 §D, T289b): omitted, the working version's type is inherited — it used to default to sql, so an html template updated without type was refused type_immutable for a field it never changed; stated, it must equal the established one (template.validation.type_immutable, the draft service's own refusal). dialect is optional on update (135 §C, T276): omitted, the working version's dialect is inherited before validation — the draft already carries it, and an agent resending the value it read is the round trip this saves; present, it must equal the established one, and a different dialect is template.validation.dialect_invalid with details.dialect / details.established_dialect / details.template_id, refused before validation or any write (a template's pipeline nodes pin it against a source of that dialect; another engine is a new template). The REST PUT /templates (§8.4) is unchanged — it still requires the field and does not compare it; the inheritance is the MCP surface's, resolved before the shared write. There is deliberately no confirm_new_root argument: the update names a template that already exists and cannot mint a folder.
Scope: author. Mutating.
6.2.37 semantics_record
Record ONE fact you learned about a datasource that introspection could not tell you (learned-semantic-layer design §7.1) — the second half of the skill's learn before you assume step: what a session learned should not die with it.
{
"name": "semantics_record",
"description": "Record ONE fact you learned about a datasource that introspection could not tell you — a unit, a time zone, a sample rate, a grain, what a coded value means, a join that holds, a trap — so the next session reads it beside the columns instead of probing again. Never record what introspection already returns (types, keys, comments). refs name the table(s) and column(s) the fact is about, structurally; every ref is checked against the live schema and an unknown one is refused. The fact's text must agree with its refs: a catalog table the text names but refs omit is ADDED to the stored refs for you (refs_added in the result names what was added); a near-miss of a listed table is refused as semantics.ref_mismatch (the refusal names the nearest listed table). A lake table's ref takes schema as ONE dotted namespace string ({\"schema\": \"lake.mart\", \"table\": \"events\"}), never a namespace array. Pass evidence_sql (the SELECT that showed the fact): it runs once, its first rows become evidence_summary, and the fact is stored as observed — without it the fact is only asserted. scope DATASOURCE is about the data over the table's WHOLE window and is shared with every workspace the datasource is granted to — an observation tied to one window belongs in the pipeline description, not in a fact; scope WORKSPACE (definition, exclusion, preference) is this organisation's meaning and stays here — and a WORKSPACE rule may carry NO refs: a rule that spans datasources ('busiest day = A + B combined') is recorded once, against the datasource the question is mostly about, and names no table; a DATASOURCE fact always names at least one (a unit without a column is meaningless — semantics.fact_invalid). To correct a stale or wrong fact, record the replacement with supersedes: the old one is retired as superseded. An identical live fact is refused as semantics.duplicate. Mutating.",
"inputSchema": {
"type": "object",
"required": [
"scope",
"datasource",
"kind",
"fact"
],
"additionalProperties": false,
"properties": {
"scope": {
"type": "string",
"enum": [
"DATASOURCE",
"WORKSPACE"
],
"description": "DATASOURCE: a fact about the data, visible wherever the datasource is granted. WORKSPACE: this organisation's meaning, visible here only."
},
"datasource": {
"type": "string",
"description": "Datasource name (must be granted to this workspace)."
},
"kind": {
"type": "string",
"enum": [
"unit",
"time_zone",
"sampling",
"grain",
"window",
"enum_meaning",
"join",
"caveat",
"format",
"definition",
"exclusion",
"preference"
],
"description": "What the fact is about. unit, time_zone, sampling, grain, window, enum_meaning, join, caveat, format are DATASOURCE kinds; definition, exclusion, preference are WORKSPACE kinds."
},
"fact": {
"type": "string",
"minLength": 8,
"maxLength": 1000,
"description": "The fact, in one or two sentences, specific enough to act on: 'value is already in the unit named by unit_col', 'pickup_ts is naive local time (America/New_York)'."
},
"refs": {
"type": "array",
"items": {
"type": "object",
"required": [
"table"
],
"additionalProperties": false,
"properties": {
"schema": {
"type": "string",
"description": "Namespace as datasources_get_tables reported it (a label, or the dotted catalog.schema form). Omit for the connection's current schema."
},
"table": {
"type": "string",
"description": "Table name exactly as datasources_get_tables returned it."
},
"column": {
"type": "string",
"description": "Column name exactly as datasources_get_columns returned it. Omit for a table-grain fact (grain, window, sampling, a table-level caveat)."
}
}
},
"description": "The object(s) the fact is about. One ref for a column fact, two for a join, a column-less ref for a table-grain fact. Required (at least one) for a DATASOURCE kind; a WORKSPACE rule that spans datasources omits it."
},
"evidence_sql": {
"type": "string",
"description": "ONE read-only SELECT/WITH that shows the fact (no :parameters). Runs once at record time under the sql_probe rules; a statement that fails refuses the record."
},
"evidence_summary": {
"type": "string",
"maxLength": 300,
"description": "What the evidence showed, in your words. Defaults to the probe's first rows."
},
"source_pipeline_id": {
"type": "string",
"format": "uuid",
"description": "The pipeline you learned this while building, if any. Shown only to readers who can read that pipeline."
},
"source_version": {
"type": "integer",
"minimum": 1
},
"supersedes": {
"type": "string",
"format": "uuid",
"description": "The id of the fact this one replaces (a stale or wrong one); it is retired with reason superseded."
}
}
}
}
Returns: the stored fact in the FULL shape — the §6.2.18a block plus datasource, refs[] ({schema, table, column}, normalised), evidence_sql, recorded_by, source_version, supersedes, retired_at/retired_reason (omitted-when-absent) — plus refs_added when the tool appended a ref for you (a catalog table the text named exactly and refs omitted; the stored refs[] already carries it, each entry the same {schema, table, column} shape). Validation, in order, each a catalogued refusal before anything is written (Pipeline Contract §13.15): kind in the closed list AND of the requested scope (semantics.kind_invalid); the fact window, refs ≥ 1, the summary cap (semantics.fact_invalid); every ref resolves against the LIVE schema — one column read per referenced table, which is also the table's fingerprint (semantics.ref_unresolved; the store never starts stale); no identical live fact (semantics.duplicate, details.existing_id); evidence_sql runs ONCE through the sql_probe path — a statement the classifier refuses or that names a :parameter is semantics.evidence_refused, one the database refuses or that times out is semantics.evidence_failed, and the fact is NOT recorded. Trust follows the evidence: observed with it, asserted without. supersedes retires the named fact with reason superseded in the same call; an id the workspace cannot see — or a source_pipeline_id it cannot read — is the not-found answer (D-R5). A DATASOURCE-scope record on a datasource not granted to this workspace is the §5.3 not-found BEFORE anything runs — that gate IS the grant requirement (D-S8). Audited as semantics.recorded (kind, scope, datasource, refs, trust, via — never the fact text or the SQL) beside the dispatcher's mcp.tool.write.
refs is optional for a WORKSPACE rule (136 §B / T278). A definition, exclusion or preference that spans datasources ("busiest day = A + B combined") is recorded ONCE, against the datasource the question is mostly about, with refs omitted (or []): it stays bound to that datasource — for visibility, and for the definitions block §6.2.10/§6.2.11 carry — but names no table, so nothing resolves it and nothing marks it stale. The text/refs check still runs against that datasource's catalog (an exact table name in the text is added as a ref, a near-miss is semantics.ref_mismatch); a table of ANOTHER datasource in the text is prose. A DATASOURCE kind keeps requiring at least one ref — a unit without a column is meaningless — refused as semantics.fact_invalid (details.field: "refs").
Scope: author — recording is an authoring act (D-S8), the same bar as writing a pipeline that reads the datasource. Mutating.
6.2.38 semantics_list
The facts on a datasource this workspace can see, with trust, drift, refs, evidence and provenance.
{
"name": "semantics_list",
"description": "List the learned facts recorded on a datasource this workspace can see — every DATASOURCE fact (whoever recorded it) and this workspace's own WORKSPACE facts — with trust, drift, refs, the evidence SQL and who recorded it through what. The same facts also arrive inline on datasources_get / _get_tables / _get_columns, which is where to read them while authoring; use this to review, to find a fact's id to supersede or retire, or to answer 'what was recorded since <time>'. Retired facts are hidden unless include_retired.",
"inputSchema": {
"type": "object",
"required": [
"datasource"
],
"additionalProperties": false,
"properties": {
"datasource": {
"type": "string",
"description": "Datasource name."
},
"table": {
"type": "string",
"description": "Only facts with a ref on this table."
},
"scope": {
"type": "string",
"enum": [
"DATASOURCE",
"WORKSPACE"
]
},
"include_retired": {
"type": "boolean",
"default": false
},
"since": {
"type": "string",
"format": "date-time",
"description": "Only facts recorded at or after this ISO-8601 instant."
}
}
}
}
Returns: {datasource, facts: [FULL fact shape, oldest first], count}. Visibility is the store's one predicate: every DATASOURCE fact on the datasource (whoever recorded it, from_this_workspace says which) and this workspace's own WORKSPACE facts. since answers the §9 acceptance question "what was recorded since
Scope: read — facts ABOUT the data, never row data (the templates_used_by reasoning).
6.2.39 semantics_retire
Retire one fact with a reason — the D-S11 verb: a state, never a delete.
{
"name": "semantics_retire",
"description": "Retire one learned fact with a reason — it stops being served beside the columns but keeps its row (facts are never deleted; history is the audit). Prefer semantics_record with supersedes when you know the correct fact: that retires the old one and records the new in one step. A fact this workspace cannot see is not-found; a DATASOURCE fact another workspace established can only be retired by a workspace admin. Mutating.",
"inputSchema": {
"type": "object",
"required": [
"id",
"reason"
],
"additionalProperties": false,
"properties": {
"id": {
"type": "string",
"format": "uuid",
"description": "The fact's id, from semantics_list or an introspection response."
},
"reason": {
"type": "string",
"minLength": 3,
"maxLength": 300,
"description": "Why — one sentence, kept on the row and in the audit log."
}
}
}
}
Returns: the retired fact in the full shape (trust: "retired", retired_at, retired_reason). A fact this workspace cannot see is semantics.not_found (D-R5, identical for an id that exists nowhere); a DATASOURCE fact recorded from ANOTHER workspace needs ws_admin — auth.role_required otherwise (you do not silently retire what someone else established). Audited as semantics.retired. Prefer semantics_record with supersedes when the correct fact is known: one call retires the old and records the new.
Scope: author. Mutating.
6.2.40 docs_list
The skill's document catalog, as a tool (120, ruling R3). The skill has been served as resources since 095 (§7.2.4), but resources are a weak surface — several MCP clients fetch them reluctantly or never, while every client calls tools. This and docs_get serve EXACTLY what the resources serve, from the same SkillDocs loader, so the two surfaces cannot drift.
{
"name": "docs_list",
"description": "The datapipelines skill's document catalog: the operating core (`skill`) first, then every reference in the order the skill's own map lists them, each with its title and a one-line purpose (when to open it). The same documents the datapipelines://docs/skill resources serve, for clients that fetch resources reluctantly or never. Read-only.",
"inputSchema": {
"type": "object",
"properties": {},
"additionalProperties": false
}
}
Scope: read — the §6.2.23 reasoning: the manual this deployment ships is a property of the BUILD, identical for every caller, every key and every workspace.
Response: [{name, title, purpose}] — skill first (the operating core, SKILL.md), then every reference in the order the skill's own map lists them; title is the document's own H1, purpose the map's one-line "open this when…". A reference on disk with no map line, or a map line with no file, is a build failure (SkillDistributionTest), not a warning.
6.2.41 docs_get
One skill document's full markdown — the same bytes the datapipelines://docs/skill/<name> resource serves.
{
"name": "docs_get",
"description": "One datapipelines skill document's full markdown: `name` is `skill` for the operating core or a reference name from docs_list. Returns {name, title, markdown} — the same bytes the datapipelines://docs/skill/<name> resource serves. An unknown name is refused with the catalogued names in the error detail. Read-only.",
"inputSchema": {
"type": "object",
"required": ["name"],
"properties": {
"name": {"type": "string", "description": "The document name: `skill` for the operating core, or a reference name from docs_list, e.g. authoring-playbook."}
},
"additionalProperties": false
}
}
Scope: read.
Returns: {name, title, markdown}. name is skill for the operating core or a reference name from docs_list; the .md-suffixed form is accepted like the resource's own tolerance.
Errors: an unknown name is mcp.doc_not_found (pipeline-contract §13.16) with details.known_docs — the tool-surface answer to a resource read's RESOURCE_NOT_FOUND, which is a protocol-level error that cannot travel in a §9.2 content envelope.
6.2.42 pipelines_run_checks
Run a pipeline version's release checks (checks[], pipeline-contract §3.3) NOW (140). This is the only way an agent obtains an observed value: the SERVER runs every check through the same PipelineCheckRunner REST POST /pipelines/{id}/versions/{version}/checks/run (rest-api §5.16), the UI and the release gate ride, and persists one pipeline_check_runs row per check before returning. There is deliberately no pipelines_record_check tool — no tool records an observed value from a caller, because an observed value the caller supplied would be a claim, not a run.
{
"name": "pipelines_run_checks",
"description": "Run a pipeline version's release checks (checks[]) NOW, against their own datasources. The SERVER runs every check and persists one pipeline_check_runs row per check before returning — only the server's own run produces `observed`, and there is deliberately no pipelines_record_check tool: no tool records an observed value from a caller. Each run's verdict is pass (observed satisfied expected), fail (a value was produced and did not satisfy it), or error (no verdict could be formed: the datasource was unresolvable or unreachable, the statement was refused or returned a shape the expectation cannot compare, or the parameters did not bind — the truth recorded, never silently a fail). With no version the WORKING version's checks run (the draft when one exists, else the latest released). Returns {version, runs: [{check_id, name, expected, observed, verdict, message, ran_at}]}; an empty checks[] returns an empty runs array.",
"inputSchema": {
"type": "object",
"required": ["id"],
"properties": {
"id": {"type": "string", "format": "uuid"},
"version": {"type": "integer", "description": "Specific version whose checks to run. Defaults to the WORKING version: the draft when one exists, else the latest released. Never clamped — an unknown version is refused, not rounded to the latest."},
"parameters": {
"type": "object",
"description": "Object whose keys match the pipeline's declared parameters — the same binding pipelines_execute uses: undeclared keys are ignored, defaults fill the execute way, and the calculator context is NOT available to a check. Values must match the declared types (BIGINTEGER and BIGDECIMAL as strings, others as JSON native types).",
"additionalProperties": true
}
}
}
}
Scope: execute — the REST twin's floor (EXECUTE_PIPELINE, auth.md §7.6): a viewer runs what they can read, and a check run returns no row data beyond the one observed cell per check. The catalog declares the tool mutating (it writes the run rows), so the dispatcher's mcp.tool.write row is the trace of who commissioned them.
Returns: {version, runs: [{check_id, name, expected, observed, verdict, message, ran_at}]} — the REST checkRuns shape exactly: expected serialized from the CheckExpectation model (its non-null members only), verdict one of pass | fail | error, and observed/message/ran_at null exactly when the run has none. An error verdict means no verdict could be formed — the datasource was unresolvable or unreachable, the statement was refused, it returned a shape the expectation cannot compare (two columns for a value check), or the parameters did not bind; the reason is in message. It is the truth recorded, never silently a fail. A version with no checks[] returns an empty runs array.
Errors: an unknown pipeline or version is the §6.2.3 not-found refusal; {version: 0} is -32602, never clamped.
6.3 Tool result schema
All tool results follow this envelope:
{
"content": [
{
"type": "text",
"text": "..." // JSON-stringified payload for tools returning JSON
}
],
"isError": false
}
On error:
{
"content": [
{
"type": "text",
"text": "{\"error\": {\"code\": \"...\", \"message\": \"...\", ...}}"
}
],
"isError": true
}
The inner JSON matches the REST API error envelope's error object — same codes, same shape, so agents see consistent errors whether they come via REST or MCP.
Execution failures carry the full record (057). When the error is an execution failure — a pipelines_execute whose pipeline failed — the error object additionally carries node, sql and exception: the §6.2.14 failure record, so the agent that just ran the pipeline sees the root cause without a second call. Every other error keeps exactly the shape above.
Every tool result — success or error — carries the request's correlation_id in its _meta, echoing the DP-Correlation-Id of the underlying request so a user can hand an agent's output straight to an operator and have it traced (Observability §9).
7. Resource Surface
Resources are entities the agent can read as "files." Useful for agents that want to inspect definitions without calling tools.
7.1 Resource URI scheme
datapipelines://pipelines/{id} → latest version, full body
datapipelines://pipelines/{id}/versions/{version} → specific version
datapipelines://pipelines/{id}/parameters → the pipeline's parameter declarations only
datapipelines://templates/{id} → latest version
datapipelines://templates/{id}/versions/{version}
datapipelines://datasources/{name} → metadata, no password
datapipelines://datasources → list
datapipelines://executions/{execution_id} → execution metadata
datapipelines://executions/{execution_id}/events → SSE event replay as text
datapipelines://docs/skill → the agent skill's operating core (Markdown)
datapipelines://docs/skill/{reference} → one reference file of the skill
The docs kind is not an entity: it is the manual the server ships (§7.2.4). Every other
form addresses stored content and is workspace-scoped; docs/* is the same bytes for every
caller on a given build.
7.2 Resource examples
7.2.1 datapipelines://pipelines/{id}
Returns the pipeline JSON body, content-type application/json.
7.2.2 datapipelines://templates/{id}/versions/{version}
Returns the template body (Freemarker SQL), content-type text/x-freemarker-sql.
{id} contains slashes. A template id is a folder path (Template Hierarchy §4.1) and, since 077, always at least two segments — so the id is every segment after templates, not one. datapipelines://templates/acme/finance/daily_orders.sql is the latest version of acme/finance/daily_orders.sql. The versions/{version} suffix is recognised by the LAST two segments, never by position, so a folder named versions stays a folder: …/templates/acme/versions/report.sql is the template acme/versions/report.sql, and …/templates/acme/versions/report.sql/versions/2 is its version 2. (Corrected in 077: the parser had required exactly two segments since 043, so every hierarchical id read as not-found.)
7.2.3 datapipelines://datasources/{name}
Returns datasource metadata as JSON, with the password field redacted. Workspace-scoped like every datasource read (§2 principle 6): a name bound to another workspace resolves as not-found; datapipelines://datasources lists exactly the pinned workspace's visible set (bound + global).
7.2.4 datapipelines://docs/skill
Returns the agent skill's SKILL.md, content-type text/markdown — the same bytes the
deployment serves at GET /skill.md and the same file the repository holds at
.agents/skills/datapipelines/SKILL.md (§15). datapipelines://docs/skill/{reference}
returns one file of references/ by name, with or without the .md suffix
(…/skill/templates and …/skill/templates.md are the same resource); an unknown name is
RESOURCE_NOT_FOUND like any other unknown URI. {reference} is a NAME, never a path — it
is looked up in a map of packaged files, so no caller string reaches a file system.
7.3 Resource discovery
Agents use resources/list to discover URIs:
{
"method": "resources/list",
"params": {
"cursor": "eyJrIjoicGlwZWxpbmVzIiwibyI6MTAwfQ" // optional; omit for the first page
}
}
Returns a page of resource descriptors (URI, name, description, MIME type) plus nextCursor.
Pagination is mandatory and normative:
- Page size is fixed at 100 descriptors. It is not client-controllable — an agent asking for "everything" must page.
cursoris an opaque server-issued token. Clients MUST treat it as an opaque string: do not parse, construct, or persist it across server restarts. A cursor the server cannot decode → JSON-RPC-32602invalid params.- The response omits
nextCursoron the last page. Presence ofnextCursoris the only "there is more" signal. - Enumeration order is stable within a paging run (docs, then pipelines, then templates, then datasources, then executions; each by id). The
docsrows lead because they are the only constant-size kind — the skill plus one row per reference, identical on every server — so they cannot push an entity off a page, and an agent that lists resources at all meets the manual before it meets content. Entities created mid-run may be missed —resources/listis a discovery aid, not a consistent snapshot.
Scope filtering: the listing is filtered to what the calling key may read (read scope; ownership rules apply to executions) and to the key's pinned workspace (workspaces design §5.2/§5.3: its pipelines/templates/executions, its bound datasources plus global ones), so two agents see different resource sets on the same server.
Execution resources are windowed: only executions from the last 24 hours are enumerated. Older executions remain readable by direct URI (datapipelines://executions/{id}) as long as their metadata exists in the Metadata DB — they are simply not listed, because an unbounded execution history would make resources/list useless (and enormous) on any busy instance. Result rows are governed by the much shorter result TTL regardless (§6.2.15).
7.4 No subscriptions in v1
We do not support resources/subscribe in v1. Resources change rarely enough that re-fetch on demand is sufficient. Subscription support is a v2 candidate (would let agents react to new pipeline versions, etc.).
8. Prompt Surface
Predefined prompts the agent can invoke via prompts/get. Useful for steering agents toward common workflows.
Admission rule: a prompt ships only if every step it instructs the agent to take is achievable with the 41 tools in §6.1 and the resources in §7. A prompt that depends on a tool we have not built is a scripted failure — it reads as a supported capability and dead-ends the agent partway through. All three prompts meet the bar (§8.1, §8.2, §8.3); §8.2 returned in v1.1 together with the introspection tools it depends on.
8.1 analyze_pipeline
{
"name": "analyze_pipeline",
"description": "Guide the agent through analyzing a pipeline's structure, identifying potential issues, and suggesting improvements.",
"arguments": {
"type": "object",
"required": ["pipeline_id"],
"properties": {
"pipeline_id": {"type": "string", "format": "uuid"}
}
}
}
Returns a prompt instructing the agent to fetch the pipeline definition (pipelines_get), read each referenced template (templates_get), preview the generated SQL (templates_render) against representative parameter values, check the SQL against the node's target dialect, look for performance issues, and report findings. Read-only: the prompt never instructs the agent to modify anything. Every step uses a v1 tool.
8.2 create_pipeline_for_question
Shipped in v1.1 — returned together with the introspection tools it depends on (§6.2.16–18), which is what satisfies §8's admission rule: its schema-grounding step has an implementation, so the walkthrough cannot dead-end the agent or tempt it into hallucinating tables. (In v1 it was deliberately withheld for exactly that reason — a sequencing decision, not a rejection.)
{
"name": "create_pipeline_for_question",
"description": "Guide the agent through building a pipeline that answers a natural-language question: discover the datasource, introspect its real schema, author the SQL template, create and execute the pipeline.",
"arguments": {
"type": "object",
"required": ["question"],
"properties": {
"question": {"type": "string", "description": "The natural-language question to build a pipeline for (max 2000 characters)."}
}
}
}
Returns a prompt that walks the agent through:
datasources_listto pick the datasource holding the data the question needs.datasources_get_schemasto see the schemas, thendatasources_get_tables(schema)to list that schema's tables, thendatasources_get_columnsfor only the tables the SQL needs — never reference a table or column these tools did not return; if the data is not there, the agent stops and says so instead of guessing.templates_createfor the SQL template, describing its expected variables in its description.pipelines_createto assemble the pipeline.pipelines_executeto run it and report the result.
The question is embedded between sentinel lines — <<<QUESTION and QUESTION>>>, each on its own line — that the instructions tell the agent to treat as the question to answer, never as instructions to follow. Containment instead of prohibition: quotes and newlines in the question cannot close or extend the block (a question cannot smuggle a line the agent might read as a step, because the fence only ends at the exact sentinel line), and a question containing either sentinel is refused with -32602 — the fence cannot be forged from inside. The question argument is also length-capped at 2000 characters and refused when missing or blank; unlike §8.1/§8.3's UUID arguments it is free text by design (carrying the user's question is the feature), and the sentinel fence is the injection guard.
8.3 debug_failed_execution
{
"name": "debug_failed_execution",
"description": "Guide the agent through diagnosing why an execution failed.",
"arguments": {
"type": "object",
"required": ["execution_id"],
"properties": {
"execution_id": {"type": "string", "format": "uuid"}
}
}
}
Returns a prompt that walks the agent through reading the execution metadata and failed node's error (executions_get), comparing against recent executions of the same pipeline (executions_list), reading the pipeline and the failing node's template (pipelines_get, templates_get), re-rendering that template with the failed execution's parameters to see the exact SQL (templates_render), checking datasource reachability (datasources_test), and proposing a fix. Every step uses a v1 tool.
9. Error Handling
9.1 MCP-level errors
Protocol violations (malformed JSON-RPC, missing required fields, unsupported method):
{
"jsonrpc": "2.0",
"id": "...",
"error": {
"code": -32602,
"message": "Invalid params: missing required field 'id'."
}
}
Standard JSON-RPC error codes (-32700 parse error, -32600 invalid request, -32601 method not found, -32602 invalid params, -32603 internal error).
9.2 Application errors
All datapipelines.co application errors (auth failures, validation errors, execution failures) are returned as tool-call results with isError: true, not as JSON-RPC errors. This is the MCP convention — domain errors are content, not protocol errors.
The error payload inside the tool result matches the REST API error object exactly:
{
"content": [
{
"type": "text",
"text": "{\"error\":{\"code\":\"pipeline.node.datasource_connection_failed\",\"message\":\"Could not acquire connection to 'pg-prod'.\",\"user_message\":\"...\",\"details\":{...},\"doc_url\":\"...\"}}"
}
],
"isError": true
}
9.3 Transport errors
- HTTP 401 (
auth.api_key.missing/.invalid/.expired) → the key is absent, revoked, expired, or its owner was deactivated. Retrying does not help; the user must supply a new key. - HTTP 403 (
auth.scope.insufficient) → the key lacks the tool's minimum scope (§6.2, Auth §7.6). Retrying does not help; the user must mint a key with a higher scope. - HTTP 429 (
rate_limit.exceeded) → rate limited. Limits are per-user, shared across REST and MCP (REST API §12); honorRetry-Afterand back off. - HTTP 429 (
rate_limit.unavailable) → the limiter could not decide and refused the call (fail closed, REST API §12.3). Not your budget: honorRetry-Afterand retry, and do not treat it as a signal to reduce your request rate permanently. - HTTP 5xx → server error; agent should retry with backoff.
10. Logging
Not delivered in v1. The v1 transport is stateless (§3.3) and answers
GET /mcpwith405, so there is no server-to-client stream — the server advertises nologgingcapability (§5.1) and emits none of the notifications below. This section defines their shape for the stateful transport that lands with v2 (ROADMAP §3.7); until then the authoritative per-node record is thenode_statsarray in a tool's final result. The rest of this section is v2-forward.
When emitted (v2), the server sends notifications/message per the MCP logging spec:
{
"method": "notifications/message",
"params": {
"level": "info",
"logger": "datapipelines.executor",
"data": {
"message": "Node fetch_orders completed",
"execution_id": "exec-uuid",
"node_id": "fetch_orders",
"duration_ms": 1266
}
}
}
Levels: debug, info, notice, warning, error, critical, alert, emergency.
Log notifications carry the correlation_id of the originating request.
Agents can use these for visibility into an execution while a pipelines_execute call is still blocking (§6.2.3) — they are the v1 stand-in for MCP progress notifications, which are deferred to v2 (ROADMAP §3.7). They are advisory: delivery requires the client to have an open GET /mcp notification stream, and nothing in the execution contract depends on them. The authoritative per-node record is the node_stats array in the tool's final result.
11. Discovery
11.1 For end users
Users discover the MCP endpoint via the UI's "Connect an Agent" page, which exposes:
- The full MCP endpoint URL (
https://{host}/mcp). - API key creation/management (UI Screens; REST surface in REST API §16.1), including the scope picker — the page must state which scope an agent needs for what it will do (
readto browse,executeto run pipelines,authorto create them) and that a key's scopes cannot exceed the creator's. - A copy-pasteable configuration snippet for common agents, using whichever header that client supports (
DP-API-KeyorAuthorization: Bearer dpk_...— §3.2):- Claude Desktop:
mcpServersJSON forclaude_desktop_config.json. - Cursor: settings JSON.
- Generic HTTP MCP client: connection details.
- Claude Desktop:
11.2 For agents
Agents discover the server's capabilities via the standard MCP initialize handshake. No out-of-band config required beyond endpoint URL + API key.
12. Open Questions / Future Additions
Out of scope for v1, tracked for future (ROADMAP is the authoritative queue):
- Dynamic per-pipeline tools: register
pipeline_execute_{name}tools for pipelines flagged as "agent-exposed," so an agent sees them by name rather than discovering them by listing. Flipstools.listChangedtotrue(§5.1). v2, ROADMAP §3.7. - Result streaming / progress notifications via MCP: stream execution events through the MCP transport instead of returning them only in the final tool result — removes the blocking-call experience of §6.2.3. v2, ROADMAP §3.7.
- Resource subscriptions:
resources/subscribefor live updates when pipelines/templates change. v2, ROADMAP §3.7. - Datasource CREATE, UPDATE and DELETE tools: deliberate omission, not an oversight, and since 094 a standing rule rather than a case-by-case judgement — no credential travels through an agent (§6.2.22). Registration briefly existed as
datasources_create(068) and was removed; editing and deleting an established datasource are operator actions with a blast radius across every pipeline that references it. All three stay UI/REST-only (§4.1). - Sampling: support server-initiated LLM completions (rare for this product; agents do their own LLM work).
- OAuth support: when multi-tenant SaaS deployment materializes.
- MCP roots: not applicable (we are not a filesystem tool).
13. Security Review Checklist
(This section is normative for the implementation.)
- [ ] Every MCP endpoint requires auth (no unauthenticated access).
- [ ] API key validated on every request, not just session establishment — via
DP-API-KeyandAuthorization: Bearer dpk_..., both through the single Auth §7.3 path. No second, laxer code path for the Bearer form. - [ ] Session JWTs (
dp_sessioncookie, non-dpk_Bearer tokens) are rejected on/mcp— verify with a test that a valid browser session cannot call a tool. - [ ] Key revocation and owner deactivation take effect within the cache TTL (~60s) on
/mcp, not just on REST. - [ ] Scope enforced per tool against the Auth §7.6 matrix — one test per tool asserting the next-lower scope is refused with
auth.scope.insufficient. - [ ] Execution ownership enforced on
executions_get,executions_get_result, and execution resources — a validreadkey cannot read another user's results. - [ ]
resources/listfiltered by the caller's scope and ownership (§7.3), not just paginated. - [ ] Datasource passwords never included in tool results or resources;
datasources_testfailures do not echo credentials or JDBC URLs. - [ ] Error messages do not leak credentials or internal network topology.
- [ ] Rate limiting enforced at the MCP layer — the same per-user limits as REST, shared across both surfaces (a user cannot double their budget by splitting traffic).
- [ ]
/mcpis CSRF-exempt because it accepts no cookies — assert both halves; exemption without the cookie ban is a CSRF hole. - [ ] All MCP traffic over TLS (enforced by deployment, not just recommended).
- [ ] Audit log records every tool call (tool name, caller, target entity, timestamp, success/failure) with the
correlation_id. - [ ] Every call to a catalog-declared mutating tool writes exactly one
mcp.tool.writeaudit event — a node run that altered customer data must never be untraceable (§14).
14. Audit
Status: normative (052, ruling R4 on T65). The same
audit_logsink theauth.*events use — no separate table, no execution rows, no UI history.
The tool events are emitted at McpToolDispatcher — the single dispatch choke point, so no tool can forget its own trace. The resource event is emitted at McpResourceReader.read (120), the one place every read passes through, for the same reason:
| Event | When |
|---|---|
mcp.tool.called |
Every tool call, every outcome (success, domain error, invalid params, internal error, scope refusal) |
mcp.tool.write |
Exactly one per call to a tool the catalog declares mutating, emitted after the tool returns — on success and on failure alike. A §7.6 scope refusal never invoked the tool, so it writes no write event: the refusal is recorded by mcp.tool.called alone |
mcp.execution.launched |
One per pipelines_execute launch (107), emitted BEFORE the blocking run begins: key id + correlation id + pipeline/execution id. It exists so §6.2.35's same-credential rule can authorize cancelling an IN-FLIGHT execution — the end-of-call mcp.tool.called row only exists once the blocking call returns, which for this tool is after the execution is terminal |
mcp.resource.read |
Every resources/read, success or failure (120). Until this event the read side of the MCP surface left no trace at all — the audit of an agent's session could show 35 tool calls and silence about which documents it opened |
Fields (the two tool events identical; the resource event mirrors them): actor — user_id (the key's owner) and key_id; tool; target — the identifier-shaped argument only (execution/pipeline/template id or name); for version-aware tools the version the call named; for node runs the node_id; for the table-addressed schema tools (datasources_get_columns / _get_tables / _get_table_stats) the table (and the namespace segments when the caller passed one) beside the target datasource; for templates_render the template id beside the target; outcome (success | error + code | invalid_params | internal_error | scope_refused); elapsed_ms; correlation_id. A resource read records uri instead of tool/target, and its code names the failure (resource_not_found, forbidden, internal_error) — a name, never the JSON-RPC number. Since 139 the table/template identifiers are what the entry-point checks learn from (pipeline.validation.table_not_learned §6.2.4, pipeline.execution.template_unrendered §6.2.3); rows written before 139 carry neither key, and those simply do not count.
Which tools are mutating is a declared property of the catalog entry (McpToolCatalog.Entry.mutating), never a name pattern — McpToolCatalogBindingTest fails if a catalogued tool lacks the declaration or a known writer (pipelines_create, pipelines_update, pipelines_execute, pipelines_execute_node, templates_create, and since 140 pipelines_run_checks — it writes the pipeline_check_runs rows) is flagged read. The failure direction is asymmetric: a read tool declared mutating is a harmless over-audit; a mutating tool declared read is the hole.
Node runs are covered (the point of 052): pipelines_execute_node runs real DML/DDL with no execution row, no SSE, no idempotency record — §6.2.20's ratification covers "no execution history", not "no trace". The mcp.tool.write row naming pipeline, node and version IS the trace that the write happened and by whom.
Deliberately NOT recorded: SQL text, row data, parameter values. Those are customer data; the event records THAT a write happened and by whom, never what it contained. Only identifier-shaped arguments are read from the request — the parameters map is never touched. The one exception with a rule of its own (107): an argument named sql (sql_probe, §6.2.34) is recorded as sql_sha256 + sql_length, never verbatim — a probe statement is customer-authored text with the same transcript-hazard profile as a credential, and the hash is what an operator needs to pair an audit row with a reported statement without the log ever holding the text.
Failure discipline: emission happens after the tool returns and cannot change the tool's result or its error; an audit-sink failure is logged server-side (WARN + correlation id) and swallowed — the customer's call does not fail because bookkeeping did.
The event names are registered in Enums §15; the shared sink and its shape are Auth §10.
15. The skill: how an agent learns this server
Status: normative (095). One source, four deliveries.
An agent connecting from OUTSIDE a checkout used to get the tool descriptions, the prompts and eight lines of workspace context — and not one word about how to author. The skill it needed existed only as a file in the repository. It is now a shipped artifact of the server.
The source. .agents/skills/datapipelines/ — SKILL.md (the operating core: core
concepts, the naming grammar and confirm_new_root, the golden path, execution semantics,
promotion, error handling, best practices, and a map of the references) plus references/*.md,
which an agent opens only when it needs them. .claude/skills/datapipelines is a symlink to
that directory. There is exactly one source; everything below is derived from it and
drift-tested against it.
Delivery 1 — the handshake (push). initialize's instructions (§5.1) is the operating
core distilled: the introspection-first flow, the name grammar and confirm_new_root, "agents
describe datasources, humans register them", the three recoveries an agent gets wrong most
often, the draft rule, and a closing learning path that applies to creating AND updating:
read the core (docs_get/docs_list, the resource, or — with no MCP transport — plain-HTTP
GET /skill.md and GET /skill/{reference}.md), then the authoring playbook beyond two
nodes, then the task's reference. It lives in
modules/mcp-server/src/main/resources/mcp/server-instructions.txt — a file, so the diff is
readable and the bytes are assertable — and is capped at 4096 bytes, test-enforced: every
client injects it into every session, so a line that does not change what an agent DOES on its
first five calls belongs in the skill instead.
Delivery 2 — the resource (pull, MCP). datapipelines://docs/skill and
datapipelines://docs/skill/{reference} (§7.1, §7.2.4), read scope, listed by
resources/list ahead of the entity kinds. Since 120 the same bytes also answer as TOOLS —
docs_list / docs_get (§6.2.40–41), from the same SkillDocs loader — because resources
are a weak surface: several clients fetch them reluctantly or never, and every client calls
tools (R3).
Delivery 3 — the URL (pull, HTTP). GET /skill.md and GET /skill/{reference}.md,
text/markdown, unauthenticated — it is the manual, it holds no secret, and requiring a
key would mean an agent cannot learn to use its key correctly until after it has one. This is
the delivery for Cursor, Codex CLI, Copilot and anything else that speaks no MCP: one curl
into .agents/skills/datapipelines/ and the agent has the manual for the version this
deployment actually runs. An unknown reference is a 404 in the §4.2 envelope with
details.reason: "skill_reference_not_found".
The rendered /docs viewer does NOT list the skill, deliberately. That index is the
operator-facing spec set, grouped by DocsCatalog and link-rewritten to GitHub for anything it
does not package; the skill is agent-facing, carries YAML front matter that is meaningless as
HTML, and its reference map would render as dead links. The raw route is the one an agent
needs, and it is the one that exists.
Delivery 4 — the Claude Code plugin. .claude-plugin/marketplace.json at the repository
root and plugins/datapipelines/: /plugin marketplace add msabiransari/datapipelines then
/plugin install datapipelines@datapipelines. The plugin ships the skill and the MCP server
entry together; its skills/datapipelines/ is a build-time COPY, not a symlink, because a
marketplace is fetched with git and Claude Code skips a symlink pointing out of the plugin
directory. The deployment URL and the API key are plugin userConfig values substituted into
the server entry — ${user_config.url} / ${user_config.api_key} — because a plugin
.mcp.json expands only those and the ${CLAUDE_PLUGIN_*} path variables, never arbitrary
shell environment variables.
Packaging (why it cannot break main). mcp-server's processResources packages the
skill directory into the jar under its own classpath root, skill/ — never under docs/,
which web's DocsCatalog scans and where an ungrouped file fails the application context at
init. SkillDocs is the single reader of those bytes, so the resource and the URL cannot
answer differently.
The generated reference. references/tools.md is rendered from McpToolCatalog plus each
tool's definition — name, description, arguments with their descriptions, §7.6 scope,
mutating flag — by ./gradlew :modules:mcp-server:skillArtifacts, and a drift test fails when
the committed file is not what the catalog renders. The handwritten sections may NAME tools;
they may not list them. This is not a hypothetical: three hand-typed tool counts were stale
simultaneously when this was written.
The guards, each able to go red: SKILL.md ≤ 400 lines and its front matter unchanged;
instructions ≤ 4096 bytes and naming the resource URI; the packaged copy byte-identical to
the repo file for every file, both directions; the plugin copy likewise; tools.md equal to
the rendered catalog; the two resource URIs read, list and 404 correctly; GET /skill.md
200 text/markdown anonymous and GET /skill/nope.md 404 in the envelope.
Appendix A: Change Log
| Date | Version | Author | Change |
|---|---|---|---|
| 2026-09-16 | v1.41 | 147 incomplete tempdb validation (#119) | No new tools, no argument change. §6.2.34 sql_probe, name: "tempdb": a missing staged table is an INCOMPLETE validation, not a pass — the payload gains validation_status ("incomplete" | "executed") and parsed becomes null on the incomplete branch (present, not omitted; true only when the statement executed); the note says what was NOT checked and names the two ways to finish (a self-contained VALUES restatement, or a run with the real staged inputs). Measured on H2 2.3.232: a missing table stops preparation before a later syntax error, so the pre-#119 "parsed, every self-defined name resolved" claim was false (an acceptance run matched a passing probe's SQL hash to its failing execution). Tool and name descriptions updated (drift-pinned); guards SqlProbeH2Test (real engine, with the tables-present counterexample) and SqlProbeTempdbWireTest (real dispatcher, real JSON). Errors, binds, limits, cleanup, scope and the hash-only audit unchanged. |
| 2026-09-15 | v1.39 | 139 the entry-point checks | No new tools, one new input property. §6.2.4/§6.2.5 pipelines_create/pipelines_update: a body whose template names a table the key never datasources_get_columns'd is refused pipeline.validation.table_not_learned (§12.11; details.tables + the clearing calls); a raw-date door (two DATE parameters, no INTEGER period parameter, no window calculator) is refused pipeline.validation.door_unacknowledged unless the call carries the new door_acknowledged boolean — the confirm_new_root shape (094). §6.2.3 pipelines_execute / §6.2.20 pipelines_execute_node: a DRAFT version whose pinned DRAFT template was written after this key's last successful templates_render is refused pipeline.execution.template_unrendered (§13.3; details.templates). §14: the schema tools' rows gain table (+ namespace), templates_render rows gain template — the identifiers the checks learn from; pre-139 rows carry neither and do not count. All three checks are MCP-only (they read the caller's own audit rows); REST and the UI unaffected. |
| 2026-09-15 | v1.40 | 140 release checks over MCP | Tool surface 40 → 41: new §6.2.42 pipelines_run_checks (scope execute, mutating — it writes one pipeline_check_runs row per check, so the mcp.tool.write audit's business) — the server runs a version's §3.3 checks[] NOW through the same PipelineCheckRunner REST POST …/checks/run, the UI and the release gate ride, persisting the rows before returning; observed exists only because the server's own run produced it, and there is deliberately NO pipelines_record_check tool — no tool records an observed value from a caller. verdict is pass | fail | error (error = no verdict could be formed: datasource unresolvable/unreachable, statement refused or an uncomparable shape, parameters did not bind — the truth recorded, never silently a fail). §6.2.4 pipelines_create / §6.2.5 pipelines_update gain the optional checks array property — passed through to the §3 body verbatim, validation stays server-side (§12.11). §6.1 lists it after docs_get; §5.1's static-surface count, §8's admission-rule count, §14's known-writer list and auth.md §7.6's MCP table (the execute row) moved in the same commit; the numbering appends 6.2.42 after 6.2.41 like 120/117 appended theirs. |
| 2026-09-14 | v1.38 | 136 §D the three T289 follow-ups | No new tools, no new fields. §6.2.36 templates_update: type no longer defaults to sql — absent, the working version's is inherited (an html template updated without type was refused type_immutable, T289b); schema default dropped, description says so. §6.2.18 datasources_get_columns / §6.2.33 datasources_get_table_stats descriptions: a LAKE datasource answers datasource.lake_table_not_found for a table its registry does not carry (T289c). REST PUT /templates inherits dialect/type the same way (T289a, rest-api.md v2.12). |
| 2026-09-14 | v1.37 | 136 §A/§B definitions where the agent looks; refs optional on a rule | No new tools. §6.2.10 datasources_list / §6.2.11 datasources_get: each per-datasource entry gains definitions — every WORKSPACE-scope fact (definition, exclusion, preference) visible to the reader on that datasource, with refs or with none, §7A.5 shape, newest last, [] when none, from the same one enrichment read as facts (which keeps its datasource-wide meaning); the acceptance run's agent met the workspace's rule buried on a column and re-chose it (T287). Descriptions updated (drift-pinned). §6.2.37 semantics_record: refs is OPTIONAL — a WORKSPACE rule that spans datasources carries none (T278; V26 relaxes the CHECK; minItems dropped, refs out of required); a DATASOURCE kind still needs one (semantics.fact_invalid). |
| 2026-09-14 | v1.36 | 135 §C templates_update inherits the dialect | No new tools. §6.2.36 templates_update: dialect is optional — omitted, the working version's is inherited before validation; present and different from the established one, refused template.validation.dialect_invalid naming both in details (T276: three acceptance-run updates in one day were refused for omitting the field the draft already had, and resent with that same value). The property's description and the tool description say so; required unchanged (it never listed dialect — the refusal was the validator's). REST §8.4 unchanged. |
| 2026-09-14 | v1.35 | 134 MCP save sees workspace datasources | No new tools, no schema changes — a documented rule made true. §4.1: the principal travels with the call (transport context), never on the thread — the SDK runs tool handlers on its boundedElastic scheduler where Spring Security's thread-local is empty, so anything reachable from a tool takes its workspace as an argument. The save-time datasource port read the thread-local and every pipelines_create/pipelines_update over MCP resolved only owner-less datasources (unknown_datasource for a workspace-owned one; 201 over REST for the same body). Fixed at the port (DatasourceRegistry.describe(name, workspaceId)), the executor's existing shape; guard McpSaveWorkspaceDatasourceE2eTest. |
| 2026-09-13 | v1.34 | 126 facts where the agent looks | No new tools, no inputSchema changes — additive response fields only. §6.2.10 datasources_list: each entry gains facts, the SAME datasource-wide learned-facts array §6.2.11 serves ([] when none, never an absent key) — the listing is the call every audited agent made first, so the learn-first facts ride it instead of waiting for a datasources_get that was being skipped; datasources_get remains the single-datasource refresh after semantics_record. §6.2.17 datasources_get_tables: for a LAKE datasource each table entry gains partition_column — the registered column's name, or null for one unpartitioned file — the same registry value §6.2.33's stats payload reports; non-lake dialects carry no such key. Descriptions for both tools updated to match (drift-pinned text); §6.2.18a gains the listing as a facts carrier. |
| 2026-09-13 | v1.33 | 121 calculator multi-output | No new tools. §6.2.23/24 (calculators_list/calculators_get): a kind's entry gains the multi-output shape (pipeline-contract §4.10, 121) — a single-output kind keeps "output": "DATE" (byte-identical wire, no outputs key); a multi-output kind carries "output": null and outputs: [{name, type, description}], the named set a node maps through context_keys (every name mapped, or the save is refused). pipelines_execute's derived execute inputs now list every key a multi-output node writes, and its refusal set gains pipeline.execution.calculator_keys_partial (pipeline-contract §13.3) — a proper subset of one node's keys is refused before anything runs. executions_get's per-node stats gain context_values on multi-output nodes (single nodes byte-identical, provided_by unchanged). |
| 2026-09-12 | v1.32 | 120 docs as tools + catalog phrases | Tool surface 38 → 40: new §6.2.40 docs_list and §6.2.41 docs_get (both scope read, non-mutating) — the skill's documents as TOOLS (ruling R3), serving exactly the SkillDocs bytes the §7.2.4 resources serve, because several MCP clients fetch resources reluctantly or never while every client calls tools. New code mcp.doc_not_found (pipeline-contract §13.16) for an unknown docs_get name — the resource read's not-found is the protocol's RESOURCE_NOT_FOUND, which a §9.2 content envelope cannot carry. §6.2.23/24 (calculators_list/calculators_get): every kind's entry gains phrases — the everyday phrases the kind answers (ruling R2); the descriptions say to match the question's words against them before picking a kind (ruling R1: the skill teaches the lookup, never the interpretation). §14 gains mcp.resource.read — every resources/read audited at McpResourceReader.read (uri, outcome, code, elapsed_ms, correlation id, key id + owner); until now a resource read left no trace. auth.md §7.6's MCP table, the §5.1 static-surface count and §8's admission-rule count moved in the same commit; §6.1 lists the two; the numbering appends 6.2.40–41 after 6.2.39 like 117/107 appended theirs. |
| 2026-09-13 | v1.32 | 123 §A table resolution | No new tools. §6.2.18 datasources_get_columns and §6.2.33 datasources_get_table_stats: an unknown table is now refused as datasource.table_not_found (was an empty list / empty stats), naming the nearest listed table when one is close — the module resolves the table with the same catalog call the tables listing uses. §6.2.19 datasources_preview_rows: the table is resolved before any statement runs (unknown → table_not_found), and a present-but-unreadable table (permission SQLSTATE) is 403 datasource.table_forbidden — the same classification §6.2.34 sql_probe gains for a permission-refused statement. All four descriptions updated (inputSchemas unchanged); the LAKE registry branches are untouched. |
| 2026-09-11 | v1.30 | tempdb scratch probe + demo-free skill | §6.2.34 sql_probe: name: "tempdb" is a syntax-and-names check against an empty scratch H2 in the staging mode (was a refusal); pass = parsed: true + missing_table. Tool descriptions carry no sample-data names (acme/finance/…, /finance/revenue/{region}, events_by_day) — SkillHasNoDemoContentTest scans the skill and the rendered tools.md. New template.validation.html_entity (§13.9) on templates_create/_update. |
| 2026-09-11 | v1.29 | 117 templates_update | Tool surface 34 → 35: new §6.2.36 templates_update — the draft write REST PUT /templates (§8.4) makes, the template mirror of pipelines_update: same parse-only validation as templates_create, same TemplateDraftService.write one-write-path rule, required id (§9.6) and expected_hash, copy-on-write first write / in-place later writes / the §5.1 no-op (status: "RELEASED", no draft pointer), template.not_found / template.version.conflict / template.validation.type_immutable refusals. Scope author, mutating. No confirm_new_root argument: an update names a template that exists and cannot mint a folder (094's create-only rule). Before 117 the only MCP path to change a template was purge-and-recreate, and purge is refused once any pipeline pins the template — exactly the state an authoring agent is in mid-build (versioning D4 untouched: the agent writes the draft, a human releases). §6.1 lists it after templates_create; §5.1's static-surface count and §8's admission-rule count updated. The fence is drift-pinned to the shipped schema like every §6.2 block; the numbering APPENDS 6.2.36 for the same reason 107 appended 6.2.32–35 — 6.2.23/24 already appear twice and a mid-group insertion would renumber ~20 cross-doc anchors. auth.md §7.6's MCP table and the ScopeMatrixSpecDriftTest counts (34 → 35, both axes) moved in the same commit. |
| 2026-09-11 | v1.31 | 118 learned semantic layer | Tool surface 35 → 38: new §6.2.37 semantics_record (one learned fact — kind from the closed enums.md §19 list, structural refs validated against the LIVE schema, evidence_sql run once through the probe path with its first rows as evidence_summary, supersedes retiring the predecessor; scope author, mutating, audited as semantics.recorded), §6.2.38 semantics_list (the facts a workspace can see, with since; scope read) and §6.2.39 semantics_retire (retire with a reason — a state, never a delete; a DATASOURCE fact another workspace established needs ws_admin; scope author, mutating). New §6.2.18a: the learned-fact block facts[] on datasources_get (datasource-wide kinds), _get_tables (table-grain facts and every stale one) and _get_columns (per column) — the §6 drift check runs at read on /columns and on a complete /tables listing and writes its demotion back; conflicts coexist (D-S5); source_pipeline renders only where the reader can read the pipeline (D-S9). The three descriptions (§6.2.11/17/18) say so. §5.1's static-surface count and §8's admission-rule count updated. Design record: docs/superpowers/specs/2026-09-11-learned-semantic-layer-design.md. |
| 2026-09-09 | v1.28 | 107 agent probes | Tool surface 30 → 34: new §6.2.32 templates_purge_draft (the bounded D61/D62 self-service verb — hard-deletes a never-released, unpinned, author-owned draft template, the entity row with it; scope author, mutating), §6.2.33 datasources_get_table_stats (one table's catalog statistics — row estimate, indexes incl. the lake partition pseudo-index, per-column bounds — always from the engine's OWN catalog, never a scan; scope read: stored estimates about shape, never row data), §6.2.34 sql_probe (ONE classified read-only SELECT/WITH, row-capped at 500 and timeboxed at 30 s, answering rows + canonical schema + the EXPLAIN plan captured BEFORE the query — so the plan survives the timeout it explains; tempdb refused with the §6.2.20 code; scope author, the 037 F row-data rule) and §6.2.35 executions_cancel (cancel a RUNNING execution this key's OWN MCP calls started — the same-credential rule joins key id x correlation id on the audit log — pipelines_execute gained a launch-time mcp.execution.launched row for exactly this, because the dispatcher's row is written when the call ENDS and this call blocks until the execution is terminal; requested, not awaited; scope execute, mutating). §6.1 lists the four; §5.1's static-surface count and §8's admission-rule count updated. §6.2.3's abandoned-call paragraph now points at executions_cancel; §12's "an MCP cancel tool" future line removed (shipped). §14 records the one audit exception: a sql argument is logged as sql_sha256 + sql_length, never verbatim. The four entries' fences are drift-pinned to the shipped schemas like every §6.2 block; the numbering appends 6.2.32–35 because 6.2.23/24 already appear twice (endpoints and calculators) and 6.2.29–31 are the lake tools. |
| 2026-09-08 | v1.27 | T199 LAKE in the MCP enum | No new tools. §6.2.6 and §6.2.8 dialect enums gain LAKE — the server accepted it since 087/089, the ADVERTISED schema still listed seven values, so clients refused every LAKE template before the server saw it (an agent blamed its own typing and bypassed MCP over REST). The enum is now derived from Dialect.entries; DialectEnumSchemaTest and the §6.2 drift test pin it. |
| 2026-09-08 | v1.26 | 099 draft-first (D55/D56) | Additive: response VALUES and descriptions, no new tool and no new argument. §6.2.4 pipelines_create lands version 1 as a DRAFT — the response carries status: "DRAFT", current_version: null and the draft pointer, and the description tells an agent to run it and then STOP for a human to release (D4 without exception). §6.2.8 templates_create mirrors it. §6.2.5 pipelines_execute documents its default in the version property: with none given it runs the WORKING version — the draft when one exists, else the latest release (D56) — never clamped for an explicit one. §6.2.1 pipelines_list rows now state the working version and a new status (DRAFT/RELEASED), because a listing that reported the released pointer alone would show nothing for every freshly authored pipeline. The datapipelines://pipelines/{id} resource (no /versions/{n}) serves the working version too, so reading a body and running it cannot disagree. Tool count unchanged at 30. |
| 2026-09-04 | v1.19 | 072 calculators | Tool surface 22 → 24: new §6.2.23 calculators_list and §6.2.24 calculators_get (both scope read, non-mutating), projecting the CalculatorRegistry catalog — typed inputs, output type and a worked example per kind, plus the org and platform Context keys a body may reference without declaring anything. pipelines_create / pipelines_update need no new arguments (a CALCULATOR node is part of the body) but their descriptions now name the type and point at calculators_list. executions_get needed no change and gained two things anyway: parameters is now the fully resolved Context after the run (org keys, platform keys, parameters, calculator outputs — DAG Executor §7.3) and each CALCULATOR node's node_stats entry carries context_key/context_value. §5.1's static-surface count and §8's admission rule updated. |
| 2026-09-04 | v1.18 | 068 datasources_create | Tool surface 21 → 22: new §6.2.22 datasources_create (scope author, mutating), which calls the same DatasourceCreateService POST /api/v1/datasources does — one payload binder, one set of workspaces D8 rules, one duplicate-name refusal. global: true still requires admin, refused with datasource.validation.workspace_forbidden. The result is the datasources §3.2 shape with password_set: true and no password at any depth. The tool's description carries the accepted trade-off: a password passed through an agent transits its context, transcript and client logging — prefer the UI or REST for a real credential. §4.1, §5.1, §6.1 and §8's admission-rule counts updated, and §14's "no datasource management tools" omission narrowed to update/delete. |
| 2026-09-02 | v1.17 | 040 template used-by | Tool surface 20 → 21: new §6.2.21 templates_used_by (which pipelines pin a template version in their working version — one reference per node with the carrying pipeline version; scope read, 040 D7). §6.2.2 pipelines_get gains upgrade_available (omit-when-empty; node/template/pinned/latest-released rows; surfaced, never applied). §6.1, §5.1 and §8 admission-rule counts updated. |
| 2026-08-05 | v1.0 | initial draft | Initial MCP server spec: streamable HTTP transport, API key auth, 15 tools, 8 resource types, 3 prompts, error model |
| 2026-08-05 | v1.1 | propagation | Updated pipelines_create tool to v1.1 Pipeline Contract shape (no terminal_node_id, no datasources_used; nodes carry type, output, settings). |
| 2026-08-10 | v1.3 | P6b build (Gate C) | Aligned the frozen spec with the merged mcp-server module. Additive/corrective only. §3.1 implementation-gate RESOLVED: protocol version pinned 2025-06-18 (negotiate-down), the v1 transport is stateless — GET /mcp optional and NOT served (405), no session ids, no resumability; SDK mcp-sdk 2.0.0. §5.1: logging capability removed — a stateless transport has no stream to deliver notifications/message, so advertising it promised notifications no client can receive. §10: marked not-delivered-in-v1 (defines the v2 shape only); node_stats in a tool's final result is the authoritative per-node record. §6.2.3: corrected the abandoned-call paragraph — a blocking POST /mcp has no disconnect callback, so disconnect-grace cancellation does not apply to an abandoned MCP tool call (only out-of-band DELETE /executions/{id} + the execution timeout do); result-shape enumeration now lists ttl_seconds (mirrors REST data_ready). §6.2.9 (bare rendered-SQL string) and §6.2.10 (dialect free {"type":"string"}, no enum) unchanged — the code was aligned to them. Rate limiting on /mcp (§13), repository limit/offset push-down, execution-record persistence and the admin all-executions listing are cross-surface carry-forwards to web/app (P6a/P7), not defects in this module. |
| 2026-08-07 | v1.2 | consistency campaign | Per SPEC-REVIEW-2026-08 §2.11. [D11] §3.2/§4.1 auth rewritten: DP-API-Key or Authorization: Bearer dpk_... through one validation path; session JWTs explicitly rejected; security-chain note added (auth §8.5). [D15] Scope row on all 15 tools sourced from the auth §7.6 matrix; read-only → read; admin acknowledged as required by no v1 tool. [D9] §6.2.15 rewritten to the uniform REST §7 cursor (offset/limit/format, fixed TTL, stable order, ownership check), 1 MB inline cap with cursor-URL fallback, result.* error table; §6.2.3 returns first page + result_url (claim-check language gone). [D3/D12] templates_create drops params_schema, gains imports [{id,version,alias}], is_library, engine; templates_render context is a free-form parameter map. [D1] pipelines_create node description: omitted output → caller, at most one caller node, zero legal. [D7] §6.2.3 documents blocking-call semantics, execution timeout, abandoned-call cancellation after grace, and deferral of MCP progress notifications. [D10] X-API-Key → DP-API-Key. [M] §5.1 listChanged: false across capabilities; §6.2.1 drops datasources_used; §7.3 resources/list pagination specified (opaque cursor, page size 100, 24h execution window, scope filtering); §8.2 create_pipeline_for_question removed from the v1 surface (ROADMAP §2); §3.1 verification marker reframed as an implementation-gate checklist; §12 futures re-tiered against ROADMAP; §13 checklist expanded. |
| 2026-08-14 | v1.4 | v1.1 introspection build | Tool surface 15 → 18: new §6.2.16 datasources_get_schema, §6.2.17 datasources_get_tables, §6.2.18 datasources_get_columns — read-only JDBC metadata introspection (author scope, the datasources_test precedent), sourced from datasources §7A with canonical type mapping, 200-table snapshot cap, empty-list-for-unknown-filter. §6.1 lists the three; §5.1 static-surface count updated; §12 future-work bullet removed (shipped). |
| 2026-08-14 | v1.5 | v1.1 introspection build | §8 prompt surface 2 → 3: create_pipeline_for_question (§8.2) returns with the introspection tools it depends on. Admission-rule paragraph rewritten (18 tools; all three prompts meet the bar). question argument: free text by design, length-capped at 2000 chars (-32602 outside 1..2000), embedded in a delimited data-not-instructions block. |
| 2026-08-15 | v1.6 | surface restructure (part 1) | datasources_get_schema removed (§6.2.16 block deleted) together with its REST twin GET /datasources/{name}/schema: the bundled whole-schema snapshot bundled columns into the table listing; table listings stay lightweight so more tables fit in one response. Tool surface 18 → 17; §6.1, §5.1, §8 admission-rule counts updated; the introspection flow remains datasources_get_tables → datasources_get_columns until the schemas listing lands. |
| 2026-08-15 | v1.7 | surface restructure (part 2) | New §6.2.16 datasources_get_schemas — the introspection flow's entry point (schemas → tables → columns). Tool surface 17 → 18; §6.1, §5.1, §8 counts updated. §6.2.17 datasources_get_tables description + Returns now state the flow contract: the unfiltered listing spans schemas — pass each table's schema to datasources_get_columns; §6.2.18's description now states that without a schema argument only the connection's current schema is read. MySQL databases arrive as JDBC catalogs, so the schemas listing reads getCatalogs(); an empty list is valid on schemaless dialects. |
| 2026-08-15 | v1.8 | semantics via remarks | §6.2.17/§6.2.18: table and column descriptors gain remarks — the engine-stored comment from JDBC REMARKS, omitted when the driver/database has none. |
| 2026-08-15 | v1.9 | surface restructure (part 3) | §8.2 create_pipeline_for_question walkthrough rewritten to the three-step grounding flow: datasources_get_schemas → datasources_get_tables(schema) → datasources_get_columns for only the tables the SQL needs. The never-reference-unreturned-tables rule and the sentinel fence are unchanged. |
| 2026-08-15 | v1.10 | hardening round 3 (005 review fix-cycle) | §6.2.16: datasources_get_schemas returns a page {\"schemas\": [...], \"truncated\": bool} capped at 2000 (was a bare array). §6.2.17/§6.2.18: without a schema argument, a datasource reporting no current schema (database-less MySQL URL) fails with the catalogued pipeline.execution.parameter_required (recovered via datasources_get_schemas) instead of a merged/spanning answer — descriptions and Returns updated; blank remarks are omitted, never \"\". Input schemas unchanged (output-shape and error-behavior changes only). |
| 2026-08-16 | v1.11 | hardening round 4 (007 review fix-cycle) | §6.2.11: datasources_get (and datasources_list, which shares the projection) now returns introspection_include_schemas when the allowlist is non-empty — omitted when empty, the same envelope as REST §3.2 — so an agent debugging schema visibility can see an allowlist is active. Output-shape change only; inputSchema untouched. |
| 2026-08-16 | v1.12 | hardening round 4 (007 review fix-cycle) | §6.2.17: datasources_get_tables no longer fails on a datasource reporting no current schema — the parameter_required guard is scoped to datasources_get_columns (6.2.18), the only operation with a merge hazard; tool description updated (inputSchema unchanged). |
| 2026-08-17 | v1.13 | pipeline composition | §6.2.4/§6.2.5: the nodes inputSchema description now covers the PIPELINE node type (pipeline ref pinning, parameter literals and ${parent_param} references, output legality). Runtime behavior is unchanged — composition executes through the internal execution service, not a new tool. |
| 2026-08-28 | v1.14 | workspaces surfaces slice | §2 principle 6 + §5.1 instructions: the workspace context statement (key-pinned scope; other workspaces absent, not hidden). §6.2.10/§6.2.11 descriptions + Returns gain workspace/readonly; datasource listings/by-name reads are workspace-scoped (bound + global — the REST §9.2/§9.3 predicate). §7.2.3/§7.3: the same scoping for datasource resources and resources/list. No new tools, no inputSchema changes. |
| 2026-09-01 | v1.15 | agent data visibility (037) | Tool surface 18 → 20: new §6.2.19 datasources_preview_rows (≤50 wire-encoded rows of one table, order_by as {column, direction} objects, service-built + dialect-quoted statements, readonly datasources valid) and §6.2.20 pipelines_execute_node (ONE node's rendered SQL on its own datasource — a debug query, not an execution: no history/SSE/idempotency, DML/DDL for real, tempdb-source and PIPELINE nodes refused with pipeline.node.standalone_execution_refused, unknown node pipeline.node.not_found, E5 draft-if-exists version default with status always stated). §6.1, §5.1, §8 admission-rule counts updated. Both author: the first tools returning arbitrary customer row data (037 F). |
| 2026-09-02 | v1.16 | MCP audit (052) | New §14 Audit (normative, ruling R4): mcp.tool.called (every call, since the original build) registered + mcp.tool.write (NEW — exactly one per catalog-declared mutating call, node runs included, after the tool returns on success and failure; scope refusals excluded because the tool never ran). Emitted at the dispatcher, not per-tool. Mutating is a declared catalog-entry property guarded by McpToolCatalogBindingTest. Never SQL/row data/parameter values. §13 gains the mutating-call checklist line. Both events registered in Enums §15 the same commit (docs-audit check C). No tool surface change. |
| 2026-09-05 | v1.18 | pipeline folders (067) | Additive arguments only — no new tool names, the surface stays 21. pipelines_list and templates_list each gain prefix: absent = the flat listing (unchanged); present ("" = the root) = ONE level of the folder tree, returning `{prefix, folders[{path, segment, *_count}], pipelines |
| 2026-09-05 | v1.19 | published endpoints (074) | Tool surface 24 → 28: endpoints_create / endpoints_list / endpoints_get / endpoints_delete (§6.2) — publish a released, side-effect-free pipeline as GET /api/x/… and bind endpoint-kind keys to it. create/delete are author, the reads read (auth.md §7.6). An endpoint-kind key cannot reach /mcp at all (refused at McpAuthFilter; /mcp is a servlet outside ScopeInterceptor's reach — security pass). No api_keys_create tool: a credential must not transit an agent's transcript. |
| 2026-09-05 | v1.20 | mandatory folders (077) | No new tools; two patterns and two descriptions. §6.2.4/§6.2.5 pipelines_create/pipelines_update name narrows to the 2–10-segment grammar (Template Hierarchy §4.1) — rendered from PipelineNameGrammar.pattern itself, so it moved with the rule. §6.2.8 templates_create id gains a pattern for the first time and it is TemplateNameGrammar.pattern: the schema had been advertising the pre-043 flat [a-z0-9_.-]+ in prose, three grammar changes stale (audit T129, 2026-09-05). Both descriptions now state that a folder is required, that details.reason='folder_required' is how the refusal is recognised, and that experiments go under test/. An omitted templates_create id is generated under test/. |
| 2026-09-07 | v1.21 | 087 connector seams | No new tools. §6.2.22 datasources_create gains the credential object (Datasources §3.4) and drops username/password from required — either shape is accepted, both together are refused; the dialect enum gains LAKE; the result carries credential.kind and a derived password_set. §6.2.16 datasources_get_schemas returns entries: [{namespace, label}] beside the legacy schemas array — two catalogs' same-named schemas are two entries, which a list of bare labels could not express. §6.2.17/§6.2.18 gain a namespace array argument beside schema (which now also accepts the dotted form), and every table row carries namespace beside schema. |
| 2026-09-07 | v1.20 | dp-lake registry (089 §A) | Tool surface 28 → 31: lake_tables_register / lake_tables_import / lake_tables_unregister (§6.2.29–31) — the dp-lake catalog (metadata-db §4.15): register one table, bulk-import a manifest.json tables[] block (inline or fetched server-side from the datasource's OWN endpoint/bucket only — the SSRF boundary), unregister. All three are author (auth.md §7.6) and declared mutating. |
| 2026-09-08 | v1.22 | 089 Iceberg location correction | No surface change — two DESCRIPTION strings corrected to the measured rule (datasources.md §8C.7): §6.2.29's tool description and its location property now say an Iceberg table's location is the current metadata FILE (…/metadata/00042-<uuid>.metadata.json), not the table root — DuckDB 1.5.5 cannot scan a pyiceberg table by its root, so an agent following the old text registered a location whose view fails at connect. The code strings and the fence moved together (the §6.2 fences are drift-pinned to the shipped schemas). |
| 2026-09-08 | v1.23 | 094 the agent boundary | Tool surface 31 → 30: datasources_create REMOVED. §6.2.22 becomes the policy it was an exception to — no credential travels through an agent; the section number is kept so §6.2.23 onward do not shift. 068 accepted the hazard with a warning in the tool's own description; a description is not a control. Datasource create, update and delete are UI/REST-only; every read and probe tool is unchanged. §4.1's scope-enforcement paragraph, §5.1's listChanged count, §6.1's list, §6.2.10's scope note, §8's admission-rule count and §14's omission entry all follow. |
| 2026-09-08 | v1.24 | 094 new-root confirmation | §6.2.4 pipelines_create and §6.2.8 templates_create gain confirm_new_root (boolean) and REFUSE a name whose root segment has nothing under it yet — pipeline.validation.new_root_requires_confirmation / template.validation.new_root_requires_confirmation (Pipeline Contract §13), with details.root and details.existing_roots from the same one-level query pipelines_list/templates_list {prefix: ""} serve. test/ is exempt; an omitted templates_create id (generated under test/) is exempt. Agent surface only: REST, the UI and pipelines_update are unchanged, and no tool other than these two accepts the argument. 067/077 had already told an agent to list the roots and ask; this is the same sentence as a guarantee. No tool-count change. |
| 2026-09-08 | v1.25 | 095 skill distribution | Additive. New §15 The skill: how an agent learns this server — one source (.agents/skills/datapipelines/), four deliveries: the initialize handshake (§5.1, now the operating core distilled, capped at 4096 bytes and living in a resource file), the MCP resource, the unauthenticated GET /skill.md route, and the Claude Code plugin. §7.1 gains two URI forms — datapipelines://docs/skill and …/skill/{reference} (§7.2.4) — read-scope Markdown served from the packaged copy; §7.3's enumeration order gains docs at the FRONT (the only constant-size kind). No tool surface change: the count stays 30, and references/tools.md is now RENDERED from McpToolCatalog rather than typed, drift-tested against the committed file. |