datapipelines.co
Menu

Specifications

Documentation

The operations manual and the contracts, packaged into the jar — so what you read here always describes the version that is running. Everything below is also in the AGPL repository.

Operations manual

  • Deployment & Packaging Specification This spec defines how datapipelines.co is packaged, distributed, and deployed in self-hosted environments. The product is open-source and self-hosted…
  • Environments — naming them, and telling the product how careful to be This page is written for the person who deploys datapipelines.co inside an organisation and has never seen it before. It answers one question end to end…
  • Configuration Reference Every configuration key for datapipelines.co, in one place. Environment variables, YAML paths, defaults, and descriptions. A developer or operator should…
  • Auth & Security Specification This spec defines how users authenticate (OIDC via any provider — Google, Microsoft, Okta, Auth0, Keycloak, etc.), how sessions work (internal JWT after…
  • Datasources Specification A Datasource is the environment-specific connection to an external database. Pipelines reference datasources by stable name (e.g., pg-prod); the…
  • Key Providers — implementing a KMS-backed credential key source This document is written for someone — a person or an agent — who has this repository, a customer's cloud account, and no context from the conversation…
  • Observability Specification Observability for datapipelines.co covers logs, metrics, traces, and health endpoints — the signals an operator uses to detect, diagnose, and resolve…
  • MCP Server Specification datapipelines.co is MCP-native. Agentic tools (Claude Desktop, GLM, Copilot, custom LangChain/LlamaIndex agents, etc.) connect to a datapipelines.co…

Contracts

  • Pipeline Contract Specification This spec defines the structure and lifecycle of a Pipeline — the central artifact of datapipelines.co. A Pipeline is a versioned, declarative DAG of…
  • Templates Specification A Template is a versioned, Freemarker-encoded SQL generator. Pipelines reference templates by {id, version} instead of inlining SQL — templates are…
  • Versioning: Draft, Release, Promotion This spec defines the version lifecycle for pipelines and templates: how edits become drafts, how a human releases (locks) a draft into an immutable…
  • REST API + SSE Specification This spec defines the HTTP surface of datapipelines.co: every REST endpoint, request/response shape, error format, the SSE event stream for pipeline…
  • Type System Specification The Type System is the API contract between datapipelines.co and every client — whether that client is an agentic tool (Claude/GLM/Copilot) calling via…
  • Enumerations Reference Specs may inline enum values for readability, but this document is the authoritative reference.
  • Staging (H2) Specification Staging is the in-memory H2 database that holds intermediate result sets during a pipeline execution. Each pipeline execution gets its own isolated H2…
  • DAG Executor Specification The Executor is the runtime engine that takes a validated Pipeline + input parameters and produces the result dataset. It is responsible for:
  • Calculators Design record: Calculators — Configurable Pure Transformations §0.

Reference

  • Metadata Database Schema Specification This spec defines the complete physical schema for the app's own metadata Postgres database. Every table, column, type, constraint, index, and foreign…
  • Module Structure Specification This spec defines how the datapipelines.co codebase is organized into Gradle subprojects (modules), the dependency direction between them, the version…
  • Pipeline Editor UI Specification The Pipeline Editor is the primary human-facing screen of datapipelines.co. It renders a pipeline as an interactive DAG visualization (nodes + edges)…
  • UI Screens Inventory The pipeline editor is fully specified, but the app has many other screens. This spec inventories every page in the application — its URL, purpose, what…
  • ROADMAP Every spec has an "Open Questions / Future Additions" section. Those items are consolidated here by target version, with the source spec tagged. When a…
  • datapipelines.co — Specification Index Spec-first project: these documents ARE the product definition; implementation follows them. Consistency is enforced mechanically — any change here must…