datapipelines.co
Menu

Reference

datapipelines.co — Specification Index

Spec-first project: these documents ARE the product definition; implementation follows them. Consistency is enforced mechanically — any change here must pass ../scripts/docs-audit.sh (exit 0) before merging. The August 2026 consistency campaign and its ratified decisions are recorded in SPEC-REVIEW-2026-08.

What the product is

A self-hosted server that executes declarative JSON pipelines — DAGs of templated-SQL nodes — against heterogeneous databases, staging intermediate results in a per-execution in-memory H2, and returning results through a uniform Redis-backed cursor. It is MCP-native: LLM agents author and execute pipelines as first-class clients alongside the REST API and the browser UI.

                 ┌────────────── auth (OIDC users · API-key agents · scopes) ──────────────┐
  Browser UI ────┤                                                                          │
  REST client ───┼──► pipeline-contract ──► dag-executor ──► datasources (JDBC dialects)    │
  MCP agent ─────┤        │    │                 │      └──► staging (per-execution H2)     │
                 │        │    └── templates (Freemarker SQL, versioned, libraries)         │
                 │        └─────── type-system (canonical types, per-dialect mappers)       │
                 │                                                                          │
                 └── results/events → Redis (cursor, TTL) · metadata → Postgres (Flyway) ───┘

Reading order

New to the project? Read in this order: type-system → pipeline-contract → templates → datasources → staging → dag-executor (the engine), then rest-api → mcp-server → auth (the surfaces), then the rest as needed.

The documents

Contract authorities (read these before anything that cites them)

Doc Status One line
type-system.md v1.1 frozen 11 canonical logical types, wire encodings (BIG* = string), per-dialect JDBC mapping tables, UTC normalization
pipeline-contract.md v1.3 frozen Pipeline/Node JSON schema, caller-node result model, validation rules, the single error-code catalog (§13)
enums.md v1.2 living Every enum's wire value + serialization convention; error-code domain registry
configuration.md v1.2 The only place config keys are defined — YAML path, env derivation rule, defaults, validation
auth.md v2.5 Generic OIDC login, internal JWT, API keys, the scope↔operation matrix (§7.6), Spring Security chain
metadata-db.md v1.1 frozen The only doc that writes DDL — full Postgres schema, Flyway V1 source, operational jobs

Engine

Doc Status One line
templates.md v1.6 frozen Freemarker SQL templates: versioning, library imports ({id, version, alias}), SSTI hardening, parse-only save validation
datasources.md v1.8 Named connections, 8 dialects (incl. LAKE — dp-lake, §8C), Hikari/JDBC property passthrough (§5.6 refusal sets), AES-GCM credential storage
staging.md v1.10 frozen Per-execution in-memory H2: lifecycle, identifier safety, mutex-guarded connection behind withConnection, memory limits
dag-executor.md v1.3 Coroutine executor: topological execution, fail-fast, cancellation (Redis flag), result materialization
calculators.md v1.0 The calculator catalogue: 23 pure kinds a CALCULATOR node can run (calendar/fiscal periods, date arithmetic, rounding, percent change, coalesce/map), the eight always-present Context keys (org_*, current_date, …), $name references vs literals, the topology rule. calculators_list returns the same catalogue live; drift-guarded against the registry.
template-hierarchy-design.md normative The name-is-the-path grammar (2–10 segments since 077, the root holds folders only), virtual folders with no identity, typed templates (sql needs a dialect, html does not) and the explorer's tree contract

Surfaces

Doc Status One line
rest-api.md v1.4 frozen Endpoints, envelopes, SSE execution stream, uniform result-delivery cursor (§7), auth/user-admin endpoints (§16)
mcp-server.md v1.3 frozen Streamable HTTP MCP: 35 tools, resources, prompts, the agent skill's four deliveries (§15) — a thin adapter over REST. There is deliberately no datasource-write tool (§6.2.22): no credential travels through an agent
ui-screens.md v1.1 12 CRUD screens: Thymeleaf + htmx, /partials/** convention, standard states
pipeline-editor.md v1.2 Cytoscape execution/visualization surface: vendored assets, SSE wiring, canvas a11y model
versioning.md v1.4 ratified Draft/release lifecycle (copy-on-write drafts in the version tables, content-hash preconditions, UI-only release), version numbers as cross-env identities, UI-driven promotion with two-sided guards

Operations

Doc Status One line
deployment.md v1.2 Docker image, compose/k8s, multi-instance model, Redis requirements, graceful shutdown, sizing
environments.md v1.0 The two variables an org sets — DATAPIPELINES_ENV (their name) and DATAPIPELINES_POSTURE (development | hardened) — the posture table, the env-var contract, loader recipes, demo as a flag, secrets, first login
observability.md v1.1 draft JSON logs + correlation IDs, metric naming/cardinality rules, redaction (normative), health endpoints
module-structure.md v1.2 Gradle modules, exhaustive dependency table, persistence ownership, version catalog + implementation gates
key-providers.md v1.0 normative Where the AES keys that encrypt datasource credentials come from: the KeyProvider contract, the envelope design, the AWS-KMS recipe, and the provider status table (§7 — only env ships today)

Meta

Doc Purpose
ROADMAP.md Deferred (v1.1/v2), rejected-with-reasoning, operator responsibilities — prevents re-litigation
SPEC-REVIEW-2026-08.md The 2026-08 consistency campaign: findings, ratified decisions D1–D15, per-doc resolutions
ARCH-AUDIT-2026-08.md Service-layer gap + multi-instance readiness audit (2026-08-31): findings M1–M10, S1–S5, drift list — pending review
TEST-GAP-2026-09.md Behaviour-coverage and browser-suite gap audit (2026-09): backfill tiers and the golden-path backlog — findings record, proposed not ratified
semantic-layer-research.md Research brief (pre-spec) on semantic layers for agent-facing data — input to a future semantic-layer.md, not a commitment
../DEVELOPMENT.md Developer setup: local infra, OIDC setup, build/run/test, git workflow
../.agents/skills/datapipelines/ The agent skill — SKILL.md (the operating core) + references/. Its references/tools.md is GENERATED from the tool catalog (./gradlew :modules:mcp-server:skillArtifacts), and every derived copy is drift-tested. Not a spec: it tells an AGENT how to use the product, while these docs define what the product is
../plugins/datapipelines/ The Claude Code plugin — the same skill plus the MCP server entry, installed with /plugin marketplace add msabiransari/datapipelines. Its skills/ directory is a build-time copy of the skill, never a second source

House rules (cross-cutting)

  • Single authority per fact. Config keys → configuration.md. DDL → metadata-db.md. Error codes → pipeline-contract §13. Enum wire values → enums.md. Scopes per operation → auth §7.6. Other docs link; they never restate.
  • Universal save-time validation. Nothing invalid ever reaches the database (pipeline-contract §2.8).
  • Custom headers carry the DP- prefix (rest-api §3.6).
  • Docs are the contract. Frozen docs change additively only; every change lands with a Change Log row and a green docs-audit.sh.

This is the documentation packaged with the running version. The same files live in the repository on GitHub.