# datapipelines.co — Specification Index

Spec-first project: these documents ARE the product definition; implementation follows them. Consistency is enforced mechanically — any change here must pass `../scripts/docs-audit.sh` (exit 0) before merging. The August 2026 consistency campaign and its ratified decisions are recorded in [SPEC-REVIEW-2026-08](https://github.com/msabiransari/datapipelines/blob/main/docs/SPEC-REVIEW-2026-08.md).

## What the product is

A self-hosted server that executes **declarative JSON pipelines** — DAGs of templated-SQL nodes — against heterogeneous databases, staging intermediate results in a per-execution in-memory H2, and returning results through a uniform Redis-backed cursor. It is **MCP-native**: LLM agents author and execute pipelines as first-class clients alongside the REST API and the browser UI.

```
                 ┌────────────── auth (OIDC users · API-key agents · scopes) ──────────────┐
  Browser UI ────┤                                                                          │
  REST client ───┼──► pipeline-contract ──► dag-executor ──► datasources (JDBC dialects)    │
  MCP agent ─────┤        │    │                 │      └──► staging (per-execution H2)     │
                 │        │    └── templates (Freemarker SQL, versioned, libraries)         │
                 │        └─────── type-system (canonical types, per-dialect mappers)       │
                 │                                                                          │
                 └── results/events → Redis (cursor, TTL) · metadata → Postgres (Flyway) ───┘
```

## Reading order

New to the project? Read in this order: **type-system → pipeline-contract → templates → datasources → staging → dag-executor** (the engine), then **rest-api → mcp-server → auth** (the surfaces), then the rest as needed.

## The documents

### Contract authorities (read these before anything that cites them)

| Doc | Status | One line |
|---|---|---|
| [type-system.md](/docs/type-system.md) | v1.1 frozen | 11 canonical logical types, wire encodings (`BIG*` = string), per-dialect JDBC mapping tables, UTC normalization |
| [pipeline-contract.md](/docs/pipeline-contract.md) | v1.3 frozen | Pipeline/Node JSON schema, caller-node result model, validation rules, **the single error-code catalog (§13)** |
| [enums.md](/docs/enums.md) | v1.2 living | Every enum's wire value + serialization convention; error-code *domain* registry |
| [configuration.md](/docs/configuration.md) | v1.70 | **Official configuration authority** — source-traced uses, exact environment mappings, defaults, validation, launcher inputs and build-time drift guards |
| [auth.md](/docs/auth.md) | v2.5 | Generic OIDC login, internal JWT, API keys, **the scope↔operation matrix (§7.6)**, Spring Security chain |
| [metadata-db.md](/docs/metadata-db.md) | v1.1 frozen | **The only doc that writes DDL** — full Postgres schema, Flyway V1 source, operational jobs |

### Engine

| Doc | Status | One line |
|---|---|---|
| [templates.md](/docs/templates.md) | v1.6 frozen | Freemarker SQL templates: versioning, library imports (`{id, version, alias}`), SSTI hardening, parse-only save validation |
| [datasources.md](/docs/datasources.md) | v1.8 | Named connections, 8 dialects (incl. `LAKE` — dp-lake, §8C), Hikari/JDBC property passthrough (§5.6 refusal sets), AES-GCM credential storage |
| [staging.md](/docs/staging.md) | v1.10 frozen | Per-execution in-memory H2: lifecycle, identifier safety, mutex-guarded connection behind `withConnection`, memory limits |
| [dag-executor.md](/docs/dag-executor.md) | v1.3 | Coroutine executor: topological execution, fail-fast, cancellation (Redis flag), result materialization |
| [scheduler.md](/docs/scheduler.md) | v1.0 | Schedules (#9): five-field cron in an IANA zone with a stated DST rule, missed/overlap policy, run states and what blocks, the system identity schedules fire under, per-instance capacity, one dispatcher across instances. Contributor doc — not packaged into the product yet |
| [calculators.md](/docs/calculators.md) | v1.0 | The calculator catalogue: 23 pure `kind`s a `CALCULATOR` node can run (calendar/fiscal periods, date arithmetic, rounding, percent change, coalesce/map), the eight always-present Context keys (`org_*`, `current_date`, …), `$name` references vs literals, the topology rule. `calculators_list` returns the same catalogue live; drift-guarded against the registry. |
| [template-hierarchy-design.md](https://github.com/msabiransari/datapipelines/blob/main/docs/template-hierarchy-design.md) | normative | The name-is-the-path grammar (2–10 segments since 077, the root holds folders only), virtual folders with no identity, typed templates (`sql` needs a dialect, `html` does not) and the explorer's tree contract |

### Surfaces

| Doc | Status | One line |
|---|---|---|
| [rest-api.md](/docs/rest-api.md) | v1.4 frozen | Endpoints, envelopes, SSE execution stream, **uniform result-delivery cursor (§7)**, auth/user-admin endpoints (§16) |
| [mcp-server.md](/docs/mcp-server.md) | v1.3 frozen | Streamable HTTP MCP: 35 tools, resources, prompts, the agent skill's four deliveries (§15) — a thin adapter over REST. There is deliberately **no** datasource-write tool (§6.2.22): no credential travels through an agent |
| [ui-screens.md](/docs/ui-screens.md) | v1.1 | 12 CRUD screens: Thymeleaf + htmx, `/partials/**` convention, standard states |
| [pipeline-editor.md](/docs/pipeline-editor.md) | v1.2 | Cytoscape execution/visualization surface: vendored assets, SSE wiring, canvas a11y model |
| [versioning.md](/docs/versioning.md) | v1.4 ratified | Draft/release lifecycle (copy-on-write drafts in the version tables, content-hash preconditions, UI-only release), version numbers as cross-env identities, UI-driven promotion with two-sided guards |

### Operations

| Doc | Status | One line |
|---|---|---|
| [deployment.md](/docs/deployment.md) | v1.2 | Docker image, compose/k8s, multi-instance model, Redis requirements, graceful shutdown, sizing |
| [environments.md](/docs/environments.md) | v1.0 | The two variables an org sets — `DATAPIPELINES_ENV` (their name) and `DATAPIPELINES_POSTURE` (`development` \| `hardened`) — the posture table, the env-var contract, loader recipes, demo as a flag, secrets, first login |
| [observability.md](/docs/observability.md) | v1.1 draft | JSON logs + correlation IDs, metric naming/cardinality rules, redaction (normative), health endpoints |
| [module-structure.md](/docs/module-structure.md) | v1.2 | Gradle modules, exhaustive dependency table, persistence ownership, version catalog + implementation gates |
| [key-providers.md](/docs/key-providers.md) | v1.0 normative | Where the AES keys that encrypt datasource credentials come from: the `KeyProvider` contract, the envelope design, the AWS-KMS recipe, and the provider status table (§7 — only `env` ships today) |

### Meta

| Doc | Purpose |
|---|---|
| [ROADMAP.md](/docs/roadmap.md) | Deferred (v1.1/v2), rejected-with-reasoning, operator responsibilities — prevents re-litigation |
| [SPEC-REVIEW-2026-08.md](https://github.com/msabiransari/datapipelines/blob/main/docs/SPEC-REVIEW-2026-08.md) | The 2026-08 consistency campaign: findings, ratified decisions D1–D15, per-doc resolutions |
| [ARCH-AUDIT-2026-08.md](https://github.com/msabiransari/datapipelines/blob/main/docs/ARCH-AUDIT-2026-08.md) | Service-layer gap + multi-instance readiness audit (2026-08-31): findings M1–M10, S1–S5, drift list — pending review |
| [TEST-GAP-2026-09.md](https://github.com/msabiransari/datapipelines/blob/main/docs/TEST-GAP-2026-09.md) | Behaviour-coverage and browser-suite gap audit (2026-09): backfill tiers and the golden-path backlog — findings record, proposed not ratified |
| [semantic-layer-research.md](https://github.com/msabiransari/datapipelines/blob/main/docs/semantic-layer-research.md) | Research brief (pre-spec) on semantic layers for agent-facing data — input to a future `semantic-layer.md`, not a commitment |
| [../DEVELOPMENT.md](https://github.com/msabiransari/datapipelines/blob/main/DEVELOPMENT.md) | Developer setup: local infra, OIDC setup, build/run/test, git workflow |
| `modules/mcp-server/src/main/resources/skill/` | The agent manual's narrative resources — flat-named documents with `area`/`layer`/`purpose` front matter, rendered at boot and served over MCP and `GET /skill.md` (mcp-server.md §15). The tools references, error codes and calculator catalog are GENERATED from the catalogs; a deliberate prose change regenerates the golden expectations. Not a spec: it tells an AGENT how to use the product, while these docs define what the product is |
| [../plugins/datapipelines/](https://github.com/msabiransari/datapipelines/blob/main/plugins/datapipelines) | The Claude Code plugin — the MCP server entry plus a under-20-line POINTER skill that says to read the deployment's own manual (`docs_get skill`); no content is mirrored. Installed with `/plugin marketplace add msabiransari/datapipelines` |

## House rules (cross-cutting)

- **Single authority per fact.** Config keys → configuration.md. DDL → metadata-db.md. Error codes → pipeline-contract §13. Enum wire values → enums.md. Scopes per operation → auth §7.6. Other docs link; they never restate.
- **Universal save-time validation.** Nothing invalid ever reaches the database (pipeline-contract §2.8).
- **Custom headers carry the `DP-` prefix** (rest-api §3.6).
- **Docs are the contract.** Frozen docs change additively only; every change lands with a Change Log row and a green `docs-audit.sh`.