Module Structure Specification

Status: v1.8 (frozen contract — additive-only changes after this point) Owner: datapipelines.co core Depends on: (all other specs — this spec operationalizes them into code structure) Last updated: 2026-10-26

1. Purpose

This spec defines how the datapipelines.co codebase is organized into Gradle subprojects (modules), the dependency direction between them, the version catalog for pinned dependencies, and the build / test conventions that apply to every module.

A multi-module structure (vs. a single monolithic codebase) gives us:

• Clear ownership boundaries — each spec maps to one module.
• Independent testability — modules are unit-tested in isolation.
• Future artifact extraction — typesystem could be published as a client library; mcp-server could be embedded in another product.
• Build incrementality — change in templates doesn't rebuild dag.

2. Design Principles

1. One module per spec. Each major spec becomes a Gradle module. The spec is the contract; the module is the implementation.
2. Layered dependencies, no cycles. Foundational modules (typesystem) depend on nothing internal. Higher-level modules depend on lower. The dependency graph is a DAG, like pipelines.
3. Kotlin-first. Kotlin DSL for Gradle (build.gradle.kts). Kotlin code for implementations. Java interop where libraries require it.
4. Version catalog. All third-party dependency versions pinned in gradle/libs.versions.toml. No version literals in module build files.
5. Convention over configuration. Common build settings (Kotlin version, JVM target, test framework, linting) defined once in the root build.gradle.kts, applied via common-conventions plugin.
6. Test parallelism. Each module's tests run independently. Root test aggregates results. Integration tests in a separate module to keep unit-test runs fast.

3. Module List

datapipelines/
├── settings.gradle.kts
├── build.gradle.kts                     # root build, applies conventions
├── gradle/
│   └── libs.versions.toml               # version catalog
├── buildSrc/
│   └── src/main/kotlin/
│       ├── CommonConventionsPlugin.kt
│       └── KotlinConventionsPlugin.kt
├── modules/
│   ├── typesystem/                      # [Type System spec]
│   ├── graph/                           # the generic Dag<T> primitive, layer 0 (§5.17)
│   ├── persistence/                     # the generic batching writer — ordered group commit, layer 0 (§5.19)
│   ├── calculators/                     # [Calculators catalog] — pure kinds, layer 0 (§5.14)
│   ├── scripting/                       # Script engine seam, layer 0 (§5.15)
│   ├── pipeline-contract/               # [Pipeline Contract spec]
│   ├── templates/                       # [Templates spec]
│   ├── datasources/                     # [Datasources spec]
│   ├── staging/                         # [Staging spec]
│   ├── dag/                             # [DAG Executor spec]
│   ├── parameters/                      # the parameter engine — parameter sets, #194 (§5.18)
│   ├── visualization/                   # visualizations and dashboards — the two documents and their lifecycle, #10 (§5.20)
│   ├── auth/                            # [Auth spec]
│   ├── scheduler/                       # [Scheduler reference] — durable occurrences, #9 (§5.16)
│   ├── application/                     # cross-aggregate use cases (§5.13)
│   ├── mcp-server/                      # [MCP Server spec]
│   ├── web/                             # REST API, SSE, Thymeleaf UI
│   └── app/                             # Spring Boot entry point, assembles everything
├── tests/
│   ├── integration-tests/           # cross-module integration tests
│   └── browser-tests/               # Playwright browser suite — runs in `build`/`check` like every module's `test` (`./gradlew browserTest` runs it alone)
└── docs/                                # these specs

3.1 Module responsibility matrix

Persistence ownership rule (normative)

1. Repositories live in their owning domain module, never in a shared persistence module and never in web. Each owning module takes org.springframework:spring-jdbc (via spring-boot-starter-jdbc) and uses NamedParameterJdbcTemplate per §8.1.
2. The schema is owned by one module: app. The Flyway dependency (flyway-core + flyway-database-postgresql) and every migration script under src/main/resources/db/migration/ live in app only. Domain modules read and write tables; they never create or alter them. DDL authority is Metadata DB §4; the migration file layout is Metadata DB §7.1.
3. Redis is a dependency of exactly two modules: dag (result store, idempotency keys, cancellation flags) and web (post-completion SSE event log, per-user rate-limit counters, and — since 050 — the datasource pool-invalidation pub/sub channel, the one Redis message beside all the Redis state). Both take spring-boot-starter-data-redis (Lettuce client, the starter's default). No other module talks to Redis.
4. The DataSource bean for the metadata DB is a single app-level bean (Spring Boot autoconfiguration from the spring.datasource.* keys defined in configuration.md §3). Modules inject NamedParameterJdbcTemplate, never construct their own pool. This is distinct from the user datasource pools owned by ConnectionPoolManager in the datasources module.

4. Dependency Direction

4.1 Layered dependency graph

This diagram is a rendering of the normative table in §4.2 — it carries no information the table does not. When the two disagree, §4.2 wins and the diagram is the bug. verifyModuleDependencies checks that every module the table names appears below; edge fidelity (the ← lists) is a human read against the table, because ASCII arrows cannot be parsed honestly.

layer 0 — no internal deps
┌──────────────┐ ┌─────────┐ ┌───────────────┐
│  typesystem  │ │  graph  │ │  persistence  │
└──────────────┘ └─────────┘ └───────────────┘

layer 1 — typesystem (auth also persistence)
┌──────────────┐ ┌─────────────┐ ┌──────────────┐ ┌───────────┐ ┌────────┐
│ calculators  │ │  scripting  │ │ datasources  │ │  staging  │ │  auth  │  ← typesystem
└──────────────┘ └─────────────┘ └──────────────┘ └───────────┘ └────────┘  auth ← typesystem, pipeline-contract (the SearchNeedle rule only), persistence

layer 2
┌───────────────────┐
│ pipeline-contract │  ← typesystem, calculators
└───────────────────┘

layer 3
┌──────────────┐ ┌─────────────┐
│  templates   │ │  scheduler  │  templates ← typesystem, pipeline-contract, scripting
└──────────────┘ └─────────────┘  scheduler ← typesystem, pipeline-contract (the name grammar only)

layer 4
┌──────────────┐ ┌──────────────┐
│     dag      │ │  parameters  │  dag ← typesystem, calculators, pipeline-contract, templates,
│  (executor)  │ │              │        datasources, staging, scripting, graph, persistence
└──────────────┘ └──────────────┘  parameters ← typesystem, graph, pipeline-contract, templates, datasources

layer 5
┌─────────────────┐
│  visualization  │  ← typesystem, pipeline-contract, templates, parameters
└─────────────────┘

layer 6
┌──────────────┐
│ application  │  ← typesystem, scripting, pipeline-contract, templates,
│ (use cases)  │    datasources, dag, auth, parameters, visualization
└──────────────┘

layer 7
┌──────────────┐
│  mcp-server  │  ← typesystem, calculators, pipeline-contract, templates,
│              │    datasources, dag, auth, application, parameters, visualization
└──────────────┘

layer 8
┌──────────────┐
│     web      │  ← typesystem, calculators, scripting, pipeline-contract, templates,
│              │    datasources, staging, dag, auth, application, mcp-server, scheduler,
│              │    parameters, persistence, visualization
│              │    (declared explicitly, not transitively)
└──────────────┘

layer 9
┌──────────────┐
│     app      │  ← web only
└──────────────┘

layer 10 — the test suites
┌─────────────────────────┐ ┌─────────────────────┐
│ tests/integration-tests │ │ tests/browser-tests │  ← app (browser-tests: + web)
└─────────────────────────┘ └─────────────────────┘

4.2 The dependency rule (machine-checkable)

There is one layering rule, and it is a table lookup, not a judgment call:

A module's dependencies { implementation(project(...)) } block MUST list a subset of its row below, and every module it uses at compile time MUST be listed explicitly (no reliance on transitive api leakage).

Notes on the shape (explanatory, not additional rules):

• The table is acyclic by construction, so "no cycles" needs no separate rule — Gradle enforces it anyway.
• calculators' row is the shortest one in the table on purpose (072, calculators design §0.4/C12). A calculator kind is a pure function of its inputs; a row that admitted datasources or dag would make that a hope rather than a fact, and the executor's freedom to evaluate a kind anywhere, in any order, rests on it. Adding an entry to that row is the edit a reviewer must refuse.
• scheduler lists pipeline-contract for exactly ONE thing, the published PipelineNameGrammar (a schedule is named like a pipeline — scheduler design revision §6, A2); its SchedulerBoundaryTest fails on any other co.datapipelines.pipeline import, so the edge cannot quietly become pipeline knowledge. It lists no dag, auth or application: the executor adapter, the capacity lease and the system principal live on web's side of its port.
• auth (#504b) lists pipeline-contract for exactly ONE thing, the shared search-needle rule — SearchNeedle and its typed refusal, the same rule every search-owning service applies at its door; its AuthBoundaryTest fails on any other co.datapipelines.pipeline import, so the edge cannot quietly become pipeline knowledge.
• parameters (#194) declares all five of its edges — typesystem (the shared value validator, LogicalType), graph (Dag<T>), pipeline-contract (TemplateRef, the name grammars, the TemplateDryRenderer / DatasourceRegistry / TemplateVersionStatuses / TemplateReleaser ports it validates and releases through, ReadLens, AuthoringGuard), and since lane C templates (the selector runtime renders a pinned template through WorkspaceTemplateEngines) and datasources (it runs through the registry's pools, ReadOnlyStatementLease and ResultRowReader). Since lane D the parameters edges of application, mcp-server and web are DECLARED (the record's §2.4): the surfaces compile against the engine, the reverse-arrow composition in application, the six tools in mcp-server, the routes and the wiring in web.
• persistence (#266) is the second empty row beside graph, for the same reason: it is a primitive every store-owning module can sit on. auth lists it for the audit log's writer and web for the execution-event record's and the replay log's; dag does NOT — its repository offers the batch statement (ExecutionEventRepository.appendAll) and knows nothing of the queue in front of it, which is web's wiring.
• visualization (#10) sits above parameters and below application: it declares typesystem, pipeline-contract (the name grammar, ReadLens, AuthoringGuard, the template ports it validates and releases through) and parameters (the production ParameterSetFacts reads a pinned set release through ParameterSetRepository). templates is allowed and NOT declared — every template fact reaches it through pipeline-contract's ports. The pipeline facts a dashboard needs (released? read-only? the caller output columns) arrive through its PipelineReleaseFacts port, which application implements over ReadOnlyPipelineRule (L1b), so no dag edge exists. Since L1b the visualization edges of application (PipelineReleaseFactsReader, that port's implementation), mcp-server (the eleven visualizations_* / dashboards_* tools) and web (the two controllers and VisualizationConfiguration, the module's only Spring wiring) are DECLARED.
• dag does not list auth: the executor is handed an already-authenticated principal by its caller. mcp-server does list auth (it authenticates its own transport, MCP Server §3.2) and dag (the pipelines_execute / executions_* tools drive the executor directly rather than looping back through HTTP).
• web lists everything it touches explicitly. It could reach most of these transitively through mcp-server; declaring them is what makes the table checkable.
• application is where a use case goes when it needs MORE THAN ONE aggregate. The rule, in one sentence: cross-aggregate use cases live in application; single-aggregate ones live with the aggregate that owns them. PipelineService is therefore in pipeline-contract, and ExecutionLauncher — which needs the pipeline aggregate AND dag's reservation store — is in application. Nothing in application may import a web or mcp type; ArchitectureGuardTest fails the build on one.
• app lists web only. It contains main(), configuration (including the single metadataTransactionManager, §8.5), logback config, and the Flyway migrations (§3.1) — no domain code.
• A module may use api(project(...)) instead of implementation(...) only where its own public API exposes the other module's types (e.g. pipeline-contract exposes ColumnSchema from typesystem). The allowed-set is the same either way.

Enforcement: a Gradle verification task compares each subproject's declared project dependencies against this table and fails the build on any extra entry. Adding an edge means editing this table first — that is the review gate.

4.3 Cross-cutting concerns

Some concerns touch every module:

• Logging — SLF4J + Logback (or structured logging via minlog/logstash-logback-encoder). Each module logs via SLF4J API; the actual logback config lives in app.
• Error handling — every module's exceptions extend a base DatapipelinesException, which lives in typesystem. There is deliberately no common module: the §4.2 table is exhaustive, and a catch-all module is where layering rules go to die.
• Configuration — typed config classes per module, composed into the global app config.
• Metrics — Micrometer API in modules, actual metrics registry configured in app.

5. Module Specs (Detailed)

5.1 typesystem

Dependencies (internal): none.

Dependencies (external): com.fasterxml.jackson.module:jackson-module-kotlin (JSON for the schema envelope). Jackson is the project-wide JSON library — the @JsonValue / @JsonCreator enum mapping in Enums §1 is normative, so there is no second serialization stack.

Public API:

• LogicalType enum
• ColumnSchema data class
• IngressTypeMapper interface + per-dialect implementations (PostgresTypeMapper, OracleTypeMapper, MssqlTypeMapper, MysqlTypeMapper, H2IngressMapper, DuckDbTypeMapper, SqliteTypeMapper)
• H2IngressMapper (H2 JDBC metadata → canonical ColumnSchema) and H2EgressMapper (canonical → H2 DDL type string + java.sql.Types code) — two objects, not inverses; signatures in Staging §5.3
• JsonEncoder (canonical value → wire representation)
• SchemaEnvelope data class

Tests: unit tests for every mapper; round-trip tests for every type.

5.2 pipeline-contract

Dependencies (internal): typesystem.

Dependencies (external):

• com.fasterxml.jackson.module:jackson-module-kotlin — Pipeline JSON ser/deser (Pipeline Contract §17.1 specifies Jackson).
• org.springframework.boot:spring-boot-starter-jdbc (brings org.springframework:spring-jdbc) — for PipelineRepository.

Public API:

• Pipeline data class (top-level entity)
• Node, NodeType, NodeSource data classes
• NodeOutput — flat sealed interface (Tempdb, Caller, Datasource)
• PipelineSettings (with nested TempdbSettings)
• Parameter data class
• TemplateRef data class ({id, version})
• PipelineValidator — runs all §12 validations from the spec
• PipelineSerializer / PipelineDeserializer (Jackson; an omitted output on a DQL node deserializes to NodeOutput.Caller)
• CallerNodeResolver — resolves the caller node per Pipeline Contract §9. Replaces topology-based terminal-node detection, which no longer exists in any form (D1).
• ExecutionContext — runtime mutable map
• PipelineRepository — NamedParameterJdbcTemplate access to pipelines / pipeline_versions (§8.1)

Tests: unit tests for validator (every check + every code path); serialization round-trip tests; PipelineRepository integration tests against a Postgres Testcontainer.

5.3 templates

Dependencies (internal): typesystem, pipeline-contract (for Parameter shape), scripting (script bodies parse and evaluate through its seam, never through Freemarker — transform-nodes design §2.1, D-T9).

Dependencies (external):

• org.freemarker:freemarker (pinned).
• org.springframework.boot:spring-boot-starter-jdbc — for TemplateRepository.

Public API:

• Template, TemplateVersion data classes
• TemplateImport data class ({id, version, alias}) — D12
• TemplateRegistry interface — lookup by id@version, caching
• RegistryTemplateLoader — the Freemarker TemplateLoader; resolves only "{id}@{version}" keys against the registry
• TemplateEngine — wraps Freemarker; owns the render guards (Templates §4.3)
• TemplateValidator
• LibraryResolver — transitive import resolution (depth cap, cycle detection, alias uniqueness)
• TemplateRepository — NamedParameterJdbcTemplate access to templates / template_versions (§8.1)

The module's public API carries no parameter-schema types. The template entity's parameter-schema field was removed entirely (D3) — pipeline parameters is the single declaration point, so there is no ParamsSchema, ParamSpec, or equivalent type here. Matches Templates §12.1.

Tests: unit tests for validator; round-trip tests for sample templates; security tests for forbidden-construct rejection; TemplateRepository integration tests against a Postgres Testcontainer.

5.4 datasources

Dependencies (internal): typesystem (for IngressTypeMapper per dialect).

Dependencies (external):

• com.zaxxer:HikariCP — connection pooling.
• org.postgresql:postgresql — bundled PG driver.
• com.microsoft.sqlserver:mssql-jdbc — bundled MSSQL driver.
• com.h2database:h2 — bundled H2 driver (also for staging).
• org.duckdb:duckdb_jdbc — bundled DuckDB driver.
• org.xerial:sqlite-jdbc — bundled SQLite driver.
• org.springframework.boot:spring-boot-starter-jdbc — for DatasourceRegistry's DatasourceRepository.

BouncyCastle removed (2026-08-07, security review MEDIUM-6). The AES-256-GCM the credential store needs (Datasources §7) is fully served by the JDK's SunJCE (AES/GCM/NoPadding); no spec names a primitive requiring an external provider, and carrying an 8 MB provider with a steady advisory cadence for an unnamed capability fails the dependency rules. If a future implementation genuinely needs one, it comes back through the spec-deviation flow with the primitive named.

• Optional: com.oracle.database.jdbc:ojdbc11 (via -Poracle Gradle property).
• Optional: com.mysql:mysql-connector-j (via -Pmysql Gradle property).

Public API:

• Datasource data class
• Dialect enum — declared in typesystem (2026-08-08: the single authoring authority is Type System §5 per enums.md §5, and TypeMappers.forDialect(dialect: Dialect) lives in typesystem, which depends on nothing internal — so the type cannot be declared here). This module consumes and re-exposes it through its typesystem dependency.
• DatasourceRegistry interface
• DialectAdapter interface + per-dialect implementations
• JdbcDrivers — driver class lookup / availability check
• CredentialEncryptor — AES-256-GCM
• ConnectionPoolManager — HikariCP wrapper for user datasources (distinct from the metadata-DB pool, §3.1 rule 4)
• ConnectionPool.discard and ReadOnlyStatementLease / LeasedStatement (#194 lane C) — a connection that must never return to its pool (HikariCP's evictConnection), and the lease the parameter engine's selector runner reads through: the §7D gate before any lease, the clamped statement timeout, and an abandon() (cancel, then discard) callable from another thread (Datasources §5.3)
• DatasourceRepository — NamedParameterJdbcTemplate access to datasources (§8.1)

5.4.1 Optional driver profiles — implementation sketch

Both optional drivers are runtimeOnly (nothing compiles against them; JdbcDrivers resolves them reflectively by class name — Datasources §10.3). They are gated on Gradle project properties, not Gradle profiles (Gradle has no profiles):

// modules/datasources/build.gradle.kts
dependencies {
    implementation(project(":modules:typesystem"))
    implementation(libs.hikaricp)
    implementation(libs.spring.boot.starter.jdbc)

    runtimeOnly(libs.postgresql)
    runtimeOnly(libs.mssql.jdbc)
    runtimeOnly(libs.h2)
    runtimeOnly(libs.duckdb.jdbc)
    runtimeOnly(libs.sqlite.jdbc)

    // ./gradlew -Poracle build   → OTN-licensed driver bundled (operator accepts the licence)
    if (project.hasProperty("oracle")) {
        runtimeOnly(libs.ojdbc11)
    }
    // ./gradlew -Pmysql build    → GPL+FOSS-exception driver bundled
    if (project.hasProperty("mysql")) {
        runtimeOnly(libs.mysql.connector.j)
    }
}

Two consequences worth stating, because both are easy to get wrong:

• The flag must be passed to every task that builds the artifact. -Poracle on build and not on bootJar produces a JAR without the driver. CI publishes the optional variants as separate jobs (./gradlew -Poracle bootJar), never as a post-hoc patch of the default JAR.
• Property presence, not value, is the switch. -Poracle=false still enables it (hasProperty is true). Documented here so nobody "disables" it that way.

lib/ drop-in (no rebuild). The licence-clean alternative is deploy-time: the operator drops ojdbc11.jar into a lib/ directory beside the application JAR. This is not automatic — the default JarLauncher only reads BOOT-INF/lib/. Extra classpath entries require PropertiesLauncher, selected by the JAR manifest:

// modules/app/build.gradle.kts
tasks.named<BootJar>("bootJar") {
    // PropertiesLauncher is the only launcher that honours loader.path.
    manifest {
        attributes("Main-Class" to "org.springframework.boot.loader.launch.PropertiesLauncher")
    }
}

tasks.named<BootRun>("bootRun") {
    classpath += files("lib")   // dev parity: same drop-in directory, no packaging
}

At runtime the operator points the launcher at the directory with loader.path — a comma-separated list of directories, archives, or directories within archives, resolved relative to loader.home (default: the process working directory). It can be supplied as a system property, the LOADER_PATH environment variable, or a loader.properties file:

java -Dloader.path=lib -jar datapipelines-app.jar
# or, in the container image (absolute: the image sets no WORKDIR, so a relative entry would resolve against /;
# the second entry is the JavaScript isolate's directory, §5.15):
ENV LOADER_PATH=/app/lib,/opt/graaljs

The deployment image therefore ships an empty, read-only /app/lib (root:root, mode 0555, created before the image drops to its app user) and sets LOADER_PATH=/app/lib,/opt/graaljs — the drop-in directory, then the directory holding the image's JavaScript isolate jar (§5.15) — so a drop-in works with no rebuild and no re-configuration (Deployment §6 owns the image contents; the operator mounts the driver jar read-only there, Deployment §3.5). The path is absolute because loader.home defaults to the working directory and the image sets no WORKDIR: LOADER_PATH=lib would put /lib, the OS library directory, on the classpath. The two directories are one list in one ENV, never a -Dloader.path beside it: PropertiesLauncher reads the system property before the environment, so a -Dloader.path naming only the isolate would silently drop the driver drop-in (measured in #8 lane 8a). ImageLoaderPathSpecDriftTest pins the Dockerfile and this section to the same value.

The wiring lives in app (it owns the runnable artifact) even though the drivers are a datasources concern. A driver that is neither bundled nor dropped in fails datasource save with datasource.driver_not_loaded — see Datasources §10 for the licensing rationale and the full driver matrix.

Tests: unit tests for adapter URL validation; integration tests via Testcontainers (real DB containers); a build-level check that -Poracle bootJar contains ojdbc11 and the default bootJar does not.

5.5 staging

Dependencies (internal): typesystem.

Dependencies (external):

• com.h2database:h2
• org.jetbrains.kotlinx:kotlinx-coroutines-core — pool admission is a coroutine Semaphore and every staging operation is suspend (Staging §9.2).

Public API:

• Staging interface
• StagingFactory interface — create(executionId, engine: StagingEngine = H2)
• H2Staging, H2StagingFactory implementations
• StageResult (columns: List<ColumnSchema>), StagingStats data classes

No repository: tempdb lives and dies with one execution and is never persisted (§3.1).

Tests: unit tests for type mapping; integration tests for staging round-trip; streaming tests for memory-bounded behavior.

5.6 dag

Dependencies (internal): pipeline-contract, templates, datasources, staging, typesystem, scripting, graph, persistence (#321 — FailureShape for the two scheduled jobs' failure lines; #326 — ScheduledJob, their failed-tick counter's name and closed tag set; nothing else; §4.2 note); calculators allowed and not declared.

Dependencies (external):

• org.jetbrains.kotlinx:kotlinx-coroutines-core
• org.springframework.boot:spring-boot-starter-data-redis (Lettuce, the starter's default client) — result store, idempotency keys, cancellation flags
• org.springframework.boot:spring-boot-starter-jdbc — for the execution repositories
• io.micrometer:micrometer-core — executor metrics (DAG Executor §15.3)

Public API:

• PipelineExecutor — built on Dag<T>, which lives in graph since #194 (§5.17; same package, co.datapipelines.dag)
• ExecutableNode, NodeSource, NodeType
• NodeResult — the executor's internal in-flight per-node value (§7.1); carries callerResultRef, a Redis key, never a live ResultSet
• NodeStats, NodeStatus — the wire-facing projection of NodeResult
• CancellationRegistry, CancellationHandle — per-node Statement registration and cancel (§8.3.1)
• ExecutionAbortedException, AbortReason (CLIENT_DISCONNECT, CANCELLED, SHUTDOWN) — ExecutionAbortedException extends CancellationException, maps to no error code
• ResultStore — Redis-backed caller-result materialization + paging cursor (D9)
• ExecutionSlots — per-user + global concurrency permits
• ExecutorDispatcher — the module's own bounded IO dispatcher (executor code never touches Dispatchers.IO)
• EventEmitter interface
• ExecutionEvent sealed class
• ExecutionRepository — pipeline_executions; ExecutionEventRepository — execution_events (durable 7-day record; the 1h Redis event log is web's, §5.9)
• StaleExecutionSweeper — the crash sweep's idempotent UPDATE (safe for every replica to run; no leader election by design). Its scheduling lives in web (§5.9): web's SweepSchedulingConfiguration is the project's one @EnableScheduling/@Scheduled surface, introduced 2026-09-01 (036), and — since #316 — declares the scheduled jobs' own scheduler, taskScheduler: ONE daemon thread, dp-scheduled, that every @Scheduled job ticks on and nothing else uses. (Until #316 there was none, and Spring ran every job on the context's unique ScheduledExecutorService — the SSE log streamer's dp-sse-log thread, which serves replays.) Before 036, "no @Scheduled/Quartz/cron anywhere" was a verified property of the codebase — any NEW scheduled job is a lifecycle-surface change and belongs on this record.
• BoundedJdbc — (#326) the one construction of a JDBC template whose every statement carries a queryTimeout (#311's lifecycle writes, the stale sweep's UPDATE, the retention's statements; web hands the auth jobs a template it built here, because persistence holds no JDBC, §5.19)
• ExecutionEventRetention — the execution_events retention job (050/T60): one idempotent DELETE past the retention window, safe for every replica, never touching pipeline_executions. Scheduled by web's RetentionSchedulingConfiguration (@Scheduled, fixed-delay 1h, on the jobs' single-thread scheduler, dp-scheduled — still exactly ONE @EnableScheduling).

Tests: unit tests for executor (mocked dependencies) — the Dag<T> algorithm tests moved to graph with the class; cancellation tests covering all three AbortReason paths incl. the cross-instance Redis flag; integration tests with real H2 + Testcontainers sources + a Redis container.

5.7 auth

Dependencies (internal): typesystem (shared exception base only), pipeline-contract (#504b — the shared SearchNeedle rule, §4.2; boundary-tested to that one use), persistence (#266 — the audit log's batching writer, §5.19).

Dependencies (external):

• de.mkammerer:argon2-jvm
• io.jsonwebtoken:jjwt-api, jjwt-impl, jjwt-jackson
• org.springframework.boot:spring-boot-starter-oauth2-client (Spring Security web/config + OAuth2 client + Jose, per Auth §12.2 — which correctly lists no BouncyCastle; see the §5.4 removal note)
• org.springframework.boot:spring-boot-starter-jdbc — for the user / key / audit repositories

Public API:

• User, ApiKey, AuthenticatedPrincipal, Workspace, WorkspaceMembership, WorkspaceMemberRow, WorkspaceContext data classes; WorkspaceRole, WorkspaceProvisioningMode enums; WorkspacesProperties (configuration §3.17)
• Scope enum
• JwtService, ApiKeyService, UserService, WorkspaceService
• OidcSuccessHandler, JwtAuthenticationFilter, ApiKeyFilter, SecurityConfig
• @RequiredScope annotation + ScopeInterceptor
• AuditLogger — writes audit_log, through its batching writer outside a transaction and directly inside one (Auth §10); PersistenceProperties — datapipelines.persistence.* (Configuration §3.32), read by every writer the application builds
• AuditLogRetention — deletes audit_log rows past datapipelines.audit.retention-days (#310, metadata-db §8.2): one database-clock cutoff per tick, bounded batches; scheduled as the retention sweep's last step by web's RetentionSchedulingConfiguration (no new @Scheduled). AuditProperties binds the key (configuration §3.12; 30–3650, refused at bind)
• UserRepository — users; ApiKeyRepository — api_keys (§8.1)

The per-request is_active / revocation re-check (D13) reads through the same 60s cache as the key-hash lookup; the cache is owned by this module and is in-process per instance, not Redis — it is a read-through cache of Postgres truth, not shared state (Auth §11.4).

Tests: unit tests for each component; integration tests for full auth flow; repository integration tests against a Postgres Testcontainer.

5.8 mcp-server

Dependencies (internal): typesystem, pipeline-contract, templates, datasources, dag, auth.

dag is a real dependency, not an accident of layering: pipelines_execute and the executions_* tools drive PipelineExecutor and ResultStore directly. mcp-server is a thin adapter over the same service layer the REST controllers use — it never loops back through HTTP, and it must not (that would make web a dependency and create a cycle).

Dependencies (external):

• io.modelcontextprotocol.sdk:mcp-core 2.0.0 and io.modelcontextprotocol.sdk:mcp-json-jackson2 2.0.0.

Gate G1 — RESOLVED 2026-08-07 (verified by downloading and inspecting the published jars, not docs). The earlier draft's io.modelcontextprotocol:mcp-core group id was indeed wrong — the real group is io.modelcontextprotocol.sdk. Facts that bind the mcp-server implementation:

• mcp-core-2.0.0 ships plain Jakarta-servlet Streamable HTTP transports (HttpServletStreamableServerTransportProvider, HttpServletStatelessServerTransport) — drop onto Spring MVC directly. The stateless variant matches MCP Server §3.3.
• Use mcp-json-jackson2, never the mcp aggregator — the aggregator pulls mcp-json-jackson3 (Jackson 3.x) onto a Jackson 2.x classpath.
• mcp-spring-webmvc stopped at 0.18.3 (two majors stale) — do not use it. The Ktor-based io.modelcontextprotocol:kotlin-sdk was rejected (second HTTP stack).
• Protocol versions compiled into 2.0.0: 2024-11-05, 2025-03-26, 2025-06-18, 2025-11-25. Which to advertise remains the MCP Server §3.1 gate.

Public API:

• McpServer — Spring Boot autoconfiguration
• McpTool, McpResource, McpPrompt annotation markers
• Tool/resource/prompt implementations (one per top-level entity operation)
• McpAuthFilter — translates API key into MCP session

Tests: unit tests for tool dispatch; integration tests using an in-process MCP client.

5.9 web

Dependencies (internal): typesystem, pipeline-contract, templates, datasources, staging, dag, auth, mcp-server — the aggregation layer, declared explicitly per §4.2.

Dependencies (external):

• org.springframework.boot:spring-boot-starter-web
• org.springframework.boot:spring-boot-starter-thymeleaf
• org.springframework.boot:spring-boot-starter-validation
• org.springframework.boot:spring-boot-starter-data-redis — post-completion SSE event log (1h), per-user rate-limit counters, and the dp:datasource-invalidated pub/sub channel (050 §5.7). The durable 7-day event record is dag's ExecutionEventRepository; these are two different stores for two different retention windows (D9).
• io.micrometer:micrometer-core
• com.fasterxml.jackson.module:jackson-module-kotlin

Vendored front-end libraries: htmx 2.0.10, Alpine 3.14.1, Cytoscape 3.34.0 (+ dagre and node-html-label), dagre 0.8.5 — all static files under static/vendor/, none of them a build dependency. htmx was org.webjars.npm:htmx.org until 096 §B; a webjar puts a /webjars/ glob on the public allowlist over a namespace the dependency graph fills, so it was vendored like the rest (VendoredHtmxAuditTest pins the bytes and the licence).

Styling foundation: @acme/design-tokens (v0.2.0+) — vendored CSS-only design system. NOT an npm dependency; vendored as static files under static/vendor/design-system/. Provides semantic tokens (--surface-*, --text-*, --accent-*), 9 swappable themes (default: saas), and ~80 .ds-* primitive component classes. All UI colors, spacing, typography, shadows, and radii reference design system tokens. No hardcoded hex values anywhere. See Pipeline Editor spec §3.4 for integration details.

Public API:

• RestControllers — REST endpoints (see REST API spec)
• Controllers — Thymeleaf UI controllers
• SSE endpoints (via Spring WebFlux's Flux or SseEmitter)
• Global exception handler (@RestControllerAdvice)
• CORS filter
• Design system CSS vendoring + theme configuration (DATAPIPELINES_UI_THEME env var, default: saas)

Tests: controller unit tests (@WebMvcTest); SSE flow tests; full integration tests in tests/integration-tests.

5.10 app

Dependencies (internal): web (transitively pulls everything).

Dependencies (external):

• org.springframework.boot:spring-boot-starter
• org.springframework.boot:spring-boot-starter-web — serves the root-level /health and /ready probes (rest-api.md §11), which lived here from P0 (added 2026-08-07; if the probe controller moves to web, this dependency reverts with it — the §4.2 internal table is unaffected either way).
• org.springframework.boot:spring-boot-starter-actuator
• org.flywaydb:flyway-core + org.flywaydb:flyway-database-postgresql — schema migration. This module is the only one that depends on Flyway (§3.1 rule 2). The Postgres module is a separate artifact since Flyway 10 and is required for a Postgres target.
• org.postgresql:postgresql — the metadata-DB driver at runtime (also bundled by datasources for user datasources; the version catalog pins it once).

Public API:

• DatapipelinesApplication.kt — @SpringBootApplication main()
• application.yml — top-level config
• logback-spring.xml — logging config
• src/main/resources/db/migration/V*.sql — the Flyway migrations, generated from Metadata DB §4 per §7.1
• bootJar manifest wiring for the lib/ driver drop-in (§5.4.1)

Tests: smoke tests via @SpringBootTest (full context load); a migration test that runs Flyway against a clean Postgres Testcontainer and asserts the resulting schema matches Metadata DB §4.

5.11 tests/integration-tests

Dependencies (internal): app (full app context for end-to-end tests).

Dependencies (external):

• org.springframework.boot:spring-boot-starter-test
• org.testcontainers:postgresql, mysql, mssql, oracle-xe — one container module per supported dialect that needs a real server. DuckDB, SQLite, H2 and LAKE are embedded (no container; LAKE's suite uses a MinIO object-storage container instead). There is no DB2 container — DB2 is not a supported dialect (Type System §5 / Datasources §4.1 list the eight).
• A Redis container (org.testcontainers:testcontainers generic container, or the Redis module) — required for result delivery, idempotency, and cancellation tests (D9/D7).
• io.rest-assured:rest-assured or spring-boot-starter-webflux (for reactive test client)

Purpose: end-to-end tests that spin up real database containers, register datasources, create templates, build pipelines, execute them, verify results. The "cold executable" check — if the integration tests pass, the app works.

5.12 tests/browser-tests

Dependencies (internal): :modules:app (full app context, same allowance as §5.11) and :modules:web (testImplementation, #721: SiteShotsRailOrderTest reads AppNav.ITEMS; app carries web as implementation, so the edge is declared, not leaked).

Dependencies (external):

• com.microsoft.playwright:playwright — pinned in the catalog; the pin covers the driver AND the bundled browser build (a Playwright release pins its browser set).
• org.springframework.boot:spring-boot-starter-test, Testcontainers (PostgreSQL + Redis) — same as §5.11.
• de.mkammerer:argon2-jvm — same declared-exception rationale as §5.11's seeding note: the seeded admin's stored hash must be a real Argon2id of the known password.

Purpose: the mechanical, LLM-free browser suite of the release-critical UI golden paths (TEST-GAP-2026-09.md): login → forced password change → dashboard, datasource CRUD, template/pipeline editors, execute+SSE, history, API keys, workspace switching. Codifies the ad-hoc .playwright-mcp dev loop as a re-runnable gate.

Design rules (ratified 2026-09-03):

• Self-contained boot: @SpringBootTest(RANDOM_PORT) + Testcontainers Postgres + Redis + local auth with a seeded admin — one command, zero manual setup, no dependency on an operator's running stack.
• Runs in build/check (#308): build and check run :tests:browser-tests:test like every module's test (CI's build step lists it; every gate log shows it). The root browserTest lifecycle task is the name for running that suite ALONE. The only browser surface outside build/check is siteShots — it needs a running demo deployment (070 §C), not a browser binary.
• Fails loud, never skips: a deliberate invocation without browser binaries FAILS with the install instructions — a silent skip is not a verdict. (The banner-skip precedent of editorJsTest is for INVOLUNTARY toolchain absence; it does not apply.)
• Chromium only, headless, auto-wait: no timing sleeps; selectors by stable id first (the templates carry them), role/text otherwise. Traces + screenshots saved under tests/browser-tests/build/reports/ on failure.
• Out of scope for v1: multi-browser matrix, visual regression pixel-diffing, accessibility scans.

5.13 application

Added by round 056 (ARCH-AUDIT-2026-08 S1–S5, ruling R6). The number is append-order, not layer-order: web is §5.9 and app is §5.10, and renumbering them would break every inbound anchor in the spec set and the code. Layer position is §4.1's job; this heading is an address.

Dependencies (internal): typesystem, pipeline-contract, templates, datasources, dag, auth. Declared in the build file only as they are actually used at compile time (§4.2) — slice A uses typesystem, pipeline-contract, dag and auth.

Dependencies (external): jackson-module-kotlin, slf4j-api.

Purpose: the use cases that need MORE THAN ONE aggregate, so they belong to no single domain module — and, before this module existed, ended up in web because nowhere else could hold them. That was S4's finding: mcp-server cannot depend on web (the arrow runs the other way), so any use case parked in web is invisible to the MCP surface and gets reimplemented there. Eight of them had been (S2's D1–D8), and one of the copies had a real behavioural difference.

The placement rule, in one sentence:

Cross-aggregate use cases live in application; single-aggregate ones live with the aggregate that owns them.

So PipelineService lives in pipeline-contract (it needs only the pipeline aggregate) and ExecutionLauncher lives here (it needs the pipeline aggregate AND dag's reservation store).

Public API (slice A):

• ExecutionLauncher — parameter binding + the idempotency reservation, shared by POST /pipelines/{id}/execute and the pipelines_execute MCP tool. Sharing it is what gave MCP execute idempotency support, which it had never had (S2's D6).
• ExecutionLaunch, LaunchDecision, IdempotencyMetrics — its request, its answer, and the counter port web adapts onto WebMetrics.

Constraints:

• No web or MCP types, ever. No HttpStatus, no ResponseEntity, no ApiResponse, no MCP wire type. The mapping to a status code stays in ApiErrorCatalog; the mapping to an MCP error stays in the tool. ArchitectureGuardTest fails the build on an offending import, so the rule cannot decay into a convention.
• DatapipelinesException with catalogued codes is the error contract, as everywhere else.
• Every workspace-scoped operation takes workspaceId explicitly — TemplateRepository's "no default anywhere: a missed caller is a compile error" rule applies to services too.
• It ships no Spring configuration of its own: web is the aggregation layer (§5.9) and declares the beans, exactly as it already does for pipeline-contract's repository.

Tests: unit tests over the use cases with the domain modules' own fixtures. The surfaces' own tests shrink to "the controller/tool calls it and maps the result".

Coming in slices B and C: the import services, 055's promotion orchestrator, and the ExecutionService/TemplateService/DatasourceService split of the remaining D3–D8 rows.

5.14 calculators

Dependencies (internal): typesystem — and nothing else, ever (see the §4.2 note).

Dependencies (external): none.

Public API:

• CalculatorKind interface — one pure inputs → output function, with its declared input types, output type and worked example
• CalculatorInput / CalculatorExample — the declaration shape the catalog doc, the validator and calculators_list all read
• CalculatorRegistry — every shipped kind, by kind; additive forever (D4)
• CalculatorEvaluationException — a refusal naming the offending input; the executor maps it to pipeline.node.calculator_failed

Why it is its own module. The kinds are the one part of the system that must be provably free of I/O: the executor evaluates one at an arbitrary DAG position, the validator type-checks its inputs at save time, and a future editor preview would evaluate one with no execution at all. Purity as a build rule (this row) plus a source rule (CalculatorPurityTest) is what makes those three uses safe. It deliberately does not depend on pipeline-contract: JSON literals, $ references, context_key collisions and error codes are contract concerns, and a catalog that knew about them would be a catalog nobody could reuse.

Tests: unit tests per kind (boundary values; the fiscal kinds against both a calendar and a non-calendar fiscal start; add_business_days across a weekend and a holiday; tz_shift across a DST boundary; date_parse with a bad pattern), a registry-invariant test, CalculatorPurityTest, and CalculatorRegistrySpecDriftTest against docs/calculators.md.

5.15 scripting

Dependencies (internal): typesystem — and nothing else, ever (§4.2 row; the root build's allowedInternalDependencies map carries the same closed set).

Dependencies (external):

• com.dashjoin:jsonata (pinned, 0.9.10 at introduction) — the round-one script engine; Apache-2.0. Bus-factor note for the dependency review: a small upstream project (99 stars, 17 open issues, last push 2026-07-08 at dispatch) — recorded on purpose; not a blocker while the seam keeps it replaceable.
• com.fasterxml.jackson.core:jackson-databind (BOM-managed) — the one ObjectMapper behind CanonicalJson, private to the module.
• org.slf4j:slf4j-api (BOM-managed) — the evaluation pool's abandonment log; the registry lives in app.
• org.graalvm.polyglot:polyglot, org.graalvm.sdk:nativebridge, org.graalvm.truffle:truffle-api (pinned, 25.4.4.1.1 at introduction; UPL 1.0) — the JavaScript engine's polyglot API and the isolate's two declared bridge dependencies (#8 lane 8a). They bring org.graalvm.sdk:collections, nativeimage, word and jniutils (UPL 1.0); all seven ride in the boot jar.
• org.graalvm.polyglot:js-isolate-community (same pin; MIT + UPL 1.0) — the GraalJS isolate, test runtime only: the aggregator pulls the four platform binaries into the Gradle cache. No isolate binary enters the jar; the image fetches its platform's into /opt/graaljs, SHA-256-pinned, and puts it on the loader path (§5.4.1). A process without it reports JavaScript unavailable.

Public API:

• ScriptEngine — compile once, evaluate many; JSON-shaped values in and out; JsonataEngine and JavaScriptEngine are the two implementations, wired through one availability-aware map (#8 lane 8b)
• ScriptLanguage, EvaluationLimits, EngineCapabilities — the seam's vocabulary; capabilities state which limits an engine can actually enforce, measured by the breach suite, never intended
• JavaScriptRuntime, JavaScriptRuntimeOptions, Availability, EngineScope — the one factory of every polyglot Engine and Context (SandboxPolicy.UNTRUSTED, spawnIsolate(true), no host, IO, polyglot, native, process or environment access), one Engine per JVM with a fresh Context per evaluation, and the isolate's sizing
• ConsoleSink, ConsoleLine, ConsoleLevel — where a JavaScript body's console lines go
• JavaScriptResourcesMain — the image build's pre-install of the isolate's native resources
• CompiledScript — opaque engine-produced handle
• ScriptEvaluationPool — the bounded pool every production evaluate call runs on; ScriptClock is its injected time source
• TypeGate (+ GateRefusal/GateResult) — the §5.3/R1 output contract of the script design; DECIMAL rounds half-even, BIGDECIMAL is exact
• CanonicalJson — the §5.5 canonical form: keys sorted, shortest doubles, plain decimals
• ScriptingException and its five typed refusals (ScriptSyntaxException, ScriptEvaluationException, ScriptTimeoutException, ScriptResourceLimitException, ScriptPoolExhaustedException) — the codes they carry are the script design's §7 mapping

Why it is its own module. The engine evaluates UNTRUSTED script bodies as pure functions of their JSON input (script design D-T4). That is safe because the module cannot reach anything else: one internal edge, an I/O-refusing source guard (ScriptingPurityTest), and resource bounds that are measured, not believed (JsonataBreachTest and JavaScriptBreachTest write build/reports/jsonata-breach.md and javascript-breach.md, and dag-executor.md's honest-bounds table is pasted from them). The JavaScript engine adds a second wall: its bodies run in a separate GraalJS isolate whose Engines and Contexts only JavaScriptRuntime builds (JavaScriptSandboxGuardTest). It deliberately does not depend on pipeline-contract: modes, contracts, error codes and the Context are callers' concerns (7b/7c), and an engine that knew them would be an engine nobody could sandbox.

Tests: the conformance suite parameterised over every engine (ScriptEngineConformanceTest), the breach suites with their three-way outcome records (JsonataBreachTest, JavaScriptBreachTest) and the JavaScript engine's measurements (JavaScriptMeasurementsTest), the 32-thread determinism proof, the pool contract, the §5.3 type-gate table, the JavaScript engine's own suites (JavaScript*Test, DockerfileGraalJsPinTest), CanonicalJsonTest, and ScriptingPurityTest.

5.16 scheduler

Dependencies (internal): typesystem, pipeline-contract (the name grammar only — §4.2 note).

Dependencies (external):

• com.github.kagkarlsson:db-scheduler-spring-boot-starter (pinned, 16.12.0 at introduction; brings db-scheduler core and db-scheduler-spring-common at the same version) — the durable queue, cluster-wide picking and dead-task detection; Apache-2.0. Its Java-serialization default is replaced by the JSON serializer (scheduler design revision §2.2, A8), and its health indicator is disabled (the record §7.2).
• spring-boot-starter-jdbc, io.micrometer:micrometer-core, Jackson (all BOM-managed).
• org.thymeleaf:thymeleaf-spring6 (BOM-managed, #442b) — the notice templates under scheduler-mail-templates/, rendered by an engine of the module's own exactly as auth renders its notices (no MVC, no view layer; the catalog note says why not bare Thymeleaf).

Public API:

• ScheduleService — the management use cases REST calls (create, edit, pause/resume/unblock, delete, Run now, reads, preview). The only scheduler type a transport may name.
• JobExecutor (+ JobExecutors) — the executor port: validate, prepare, start, inspect, lens; standardized outcomes, no error-code parsing
• CapacityGate / CapacityLease — the acquire-before-claim port (R4); web implements it over dag's ExecutionSlots
• OccurrenceFunction — the ONE occurrence computation (the record §3.2) the dispatcher and the preview share
• SchedulerProperties — datapipelines.scheduler.* (Configuration §3.29)
• the jobs ScheduleDispatcher, ScheduledRunWorker, RunReconciler, ScheduleNotifier (#442b) — declared as db-scheduler tasks by the module's own SchedulerAutoConfiguration; no transport may name them (ArchitectureGuardTest, B5)
• NotificationMailer (#442b) — the mail port: one rendered notice to one recipient, answered with whether the relay could have accepted it; web implements it over auth's one shared SMTP transport (the scheduler cannot see auth)

Why it is its own module. The scheduler must stay pipeline-agnostic (the owner's boundary, record §5): timing, occurrence identity, admission and the run trail are generic, and a report executor is expected to plug into the same port. A module with no dag/application/auth edge makes that a build fact; the fake executor in its own suite proves it at runtime.

Tests: the occurrence function (DST gap/fold against the measured A9 case, month/year boundaries, the min-interval guard); the repositories and the dispatcher/worker/reconciler against a real Postgres with a controlled clock (ManualScheduler, SettableClock) — two instances, one occurrence, one launch; the crash windows; capacity retries; the two spike proofs; SchedulerBoundaryTest.

5.17 graph

Dependencies (internal): none. Dependencies (external): none — Kotlin stdlib only (DAG Executor §2 principle 1: no graph library for ~150 lines of code).

Public API:

• Dag<T> — the immutable, well-formed-by-construction DAG (DAG Executor §3): node ids, dependenciesOf/dependentsOf, topologicalOrder(), independentBatches(), detectCycle()
• DagBuilder<T> — rejects duplicate ids, dangling dependencies and cycles at build()

Why it is its own module. Parameter-engine record P16/P17 (#194): the parameter engine builds its dependency graph with the house Dag<T>, and dag — the executor — carries pipeline-contract, templates, datasources, staging, Redis and JDBC, so nothing below application could depend on it without inheriting that set. The primitive moved byte-identical (git diff -M shows two 100% renames) and in the same package, co.datapipelines.dag, so the executor's two importers did not change — the owner's ruling was that the mature executor is not touched. DagPackageTest pins the package; renaming it is an executor change, not a tidy-up.

Tests: DagTest (moved unchanged: topological order, batches, cycle detection and its path, builder refusals, the deep-chain iterative search); DagPackageTest.

5.18 parameters

Dependencies (internal): typesystem, graph, pipeline-contract, templates, datasources — all declared (the last two since lane C's selector runtime, §4.2 note). Dependencies (external): spring-boot-starter-jdbc (the repository, §8.1), Jackson (BOM-managed), kotlinx-coroutines-core (the evaluator, BOM-managed). No Redis (§3.1 rule 3).

Public API (frozen by #194 lane B — the model, the codes, the probe port and the repository's API; lanes C and D build on them):

• ParameterSetBody / ParameterDefinition / SelectorSource / ConstantOption / Presentation — the definition model (the record's §3), bound by ParameterSetReader (unknown keys and wrong JSON types refused at every level as parameter.validation.body_invalid) and ParameterSetJson's strict mapper. The value-level declaration inside a definition IS typesystem's ParameterDeclaration — never a copy.
• ParameterErrorCodes — the parameter.* family (Pipeline Contract §13.20); PipelineErrorCodes.Parameters mirrors it, pinned equal by reflection.
• SelectorProbe — the port the save-time validator's steps 5–6 call (the record's §4): render a pinned selector template, run it with maxRows = 2. REQUIRED by ParameterSetValidator since lane C, which retired the selector_probe_unavailable stand-in.
• SelectorRunner (lane C) — the selector runtime and the production SelectorProbe (the record's §6.3): renders a pin through the workspace's TemplateEngine, resolves the datasource by the workspace-visible live read (P31), translates :name binds with MULTI list expansion (the empty list as IN (NULL), P29's __<k> slices, every expanded placeholder counted), runs through datasources' ReadOnlyStatementLease (the §7D gate before any lease, the clamped timeout, maxRows) and reads canonical rows with the schema from metadata. Its evaluate-time face is SelectorTasks / SelectorTask / SelectorRequest / SelectorRun / DatasourceResolver — the unit the bulkhead runs and abandons.
• SelectorPool (lane C) — the selector bulkhead (the record's P31, §5.2 step 3), the ScriptEvaluationPool shape copied: size RUNNING statements + waiting more admitted, Saturated at once when full, a suspending cancellable wait for a slot (the evaluate's deadline bounds it), one slot-owning thread per statement (abandoned ones keep their slot), the abandonment — cancel() then ConnectionPool.discard — off the caller's thread, the abandoned LongAdder for the parameters.selectors.abandoned gauge (the web wiring's bean). One per process.
• ParameterEvaluator (lane C) — the runtime (the record's §5): evaluate(workspaceId, set: ParameterSetVersion, selections) / evaluateBlocking → EvaluateResponse (EvaluatedParameter, ParameterState, EvaluatedOption, ParameterError, ValueOrigin, OrgEcho) and EvaluateResponseJson, the one writer the byte cap measures. Step 1 through the shared validator, the set's Dag, one coroutine per parameter awaiting its parents on the bulkhead under withTimeout(evaluate-timeout-seconds), the P26 selection priority, the three whole-request refusals as DatapipelinesExceptions (unknown_parameter, timeout, response_too_large). The route, the tool, the permission and the version resolution are lane D's.
• ParametersProperties / ParametersConfig — datapipelines.parameters.* (Configuration §3.30). The runner applies the statement-level keys from its own config and the evaluator the evaluate-level ones: the wiring builds both, and the one pool, from the same config (the record's §17, C27).
• ParameterSetTemplatePins (#194 lane D) — the parameters side of the templates reverse arrow (the record's §8.4): the working-version, any-version, live-exact-pin and per-version-count scans over the set's pinned templates. ADDITIVE to the frozen repository (a separate class over the same tables). The COMPOSITION with the pipeline scan is application's TemplateUsage; every consumer — templates_used_by, templates_purge_draft, the web delete guards, the template screen's counts — reads the composition.
• ParameterSetValidator — the record's §4 in order (structure, the graph on graph's Dag<T>, the expression AST's static rules, the template pins by namespace, the dry run through SelectorProbe); revalidateSources re-runs steps 4–6 at release and import. Expr / ExpressionParser / ExpressionPrinter / ExpressionEvaluator — the §7 AST (lane C evaluates hidden/disabled with it). ParameterSetGraph — the set's DAG (lane C's evaluation order).
• ParameterSetRepository (jdbc; parameter_sets, parameter_set_versions — V39) and ParameterSetService — the versioning §3.5 verb table (draft create/write with the hash precondition, release with the 142 cascade, purge, discard, restore, switch, import), the working-version read rule and the promoter lens. Every method takes the workspace; every statement filters by it.

Why it is its own module. The owner's intent (the record's P1): an engine for parameters "independent of anything", a decoupled offering. A single-aggregate module — the definition, its validation and its lifecycle — whose only cross-aggregate facts arrive through pipeline-contract's ports; the templates reverse arrow and promotion are application's (the record's §8.4/§8.5).

Tests: the model's strict binding and round trip; the reflection pin between the two code objects; the expression AST (every operator, both cardinalities, the null/empty rules, every cap one past); the validator (every §13.20 validation code reached by a fixture, the probe proven CALLED by a recording fake); the repository and every lifecycle verb against the module's own Postgres container (ParametersTestDb, the shipped migrations through plain JDBC) — workspace scoping both ways, the one-draft index, and both races FORCED with a second connection; ParametersConfigKeysSpecDriftTest. Lane C: SelectorBindsTest (the pinned spring-jdbc's list expansion, the empty list, the slices, the placeholder count); SelectorRunnerIntegrationTest and ParameterEvaluatorIntegrationTest — the REAL runner over the same container as the CUSTOMER database (CustomerDb, a real-pool CustomerRegistry with revocable grants and counted reads) — save end to end, the gate, canonical types, a revoked grant, the row invariants, a statement abandoned at the deadline, the query count per evaluate; SelectorPoolTest over a stub driver that ignores cancel(); ParameterEvaluatorTest, SelectorRowsTest and EvaluatorConcurrencyTest over a scripted, recording selector runtime (every P26 branch, the owner's three scenarios, the caps one off, concurrency on latches).

5.19 persistence

Dependencies (internal): none. Dependencies (external): kotlinx-coroutines-core (the bounded suspending record the event emitter awaits from the executor's coroutines) and slf4j-api (the WARNs a failed batch, a saturated queue and an incomplete drain owe the operator), both BOM-managed; tests only, logback-classic (the BOM's — the suite records the writer's WARN lines, #266b). No Spring, no Micrometer, no JDBC — java.sql.SQLException is read for its SQLState alone (FailureShape).

Public API:

• BatchingWriter<T> — record (blocking: returns once the item is durable or definitively not), recordSuspending (bounded: at most twice record-max-wait-ms, then Indeterminate), submit (non-awaited, refuses when full — no production caller yet, D27's dashboard event), queueDepth/queuedBytes, stop (bounded drain → DrainReport)
• BatchSink<T> — the store's side: write(items) (one unit, throws on failure), writeOne (the direct path), partitionKey, sizeOf, describe (ids for log lines, never payloads), classify (the failure-kind tag), propagates (a failure that belongs to the CALLER — rethrown on its thread by record/recordSuspending; default none, #266b)
• FailureShape — a failure as a log line may carry it: cause (the simple class name) and sqlState (the first SQLState in the chain) — never the message (#266b)
• ScheduledJob — (#326) the four metadata @Scheduled jobs' closed job tag set and the name of their failed-tick counter, as plain strings: each job (dag, auth) registers the counter with its own registry, so this module still carries no Micrometer
• BatchingConfig (the datapipelines.persistence.* values), Outcome, FallbackReason, FailureKinds, BatchingHooks (what the web layer binds to Micrometer), DrainReport

Why it is its own module. #266: the audit log (auth) and the execution-event record and replay log (web) need the SAME group commit, and the only module below both is one that knows neither. A copy in each would be two implementations of the ordering and durability guarantees, and those are exactly the guarantees that must not drift. It holds no store and no Spring type; each module that owns a store supplies its BatchSink and its wiring.

Tests: BatchingWriterTest — one suite over a recording sink, because the primitive is generic: per-key order under 8 writers × 1,000 items, the idle fast path (a sequential producer never lingers), group commit, the batch bounds, linger under load, saturation by count and by bytes (record falls back, submit refuses and counts), the singles retry with the poison row skipped, a hung sink (the bounded wait claims the item back; an in-flight item is never written twice), the bounded suspending record (indeterminate, never unbounded; an abandoned direct write that had not started never runs), the drain (nothing unflushed; a hung sink gives up at shutdown-drain-ms and reports what it lost). BatchingWriterFailureTest (#266b): a failure is logged by class and SQLState and never by its message (a recording appender over a sink whose message carries a marker), and a failure the sink claims is rethrown on the caller's thread, blocking and suspending, from a batch and from the direct write.

5.20 visualization

Dependencies (internal): typesystem, pipeline-contract, parameters — declared; templates allowed and not declared (§4.2 note). Dependencies (external): spring-boot-starter-jdbc (the two repositories, §8.1), Jackson (BOM-managed). No Redis (§3.1 rule 3), no coroutines (nothing here waits on anything).

Public API (#10 L1a — the documents, the codes, the ports and the repositories' API; L1b–L5 build on them):

• VisualizationBody / DashboardBody and their parts — the two documents (dashboards.md §2, the implementation spec §3), bound by VisualizationReader / DashboardReader (an explicit key table per level; unknown keys, wrong JSON types and missing keys refused as *.validation.body_invalid naming the path; the collection bounds checked before their members are walked) and ArtifactJson's strict mapper.
• VisualizationErrorCodes / DashboardErrorCodes — the visualization.* / dashboard.* families (Pipeline Contract §13.22/§13.23); PipelineErrorCodes.Visualization / .Dashboard mirror them, pinned equal by reflection.
• VisualizationValidator / DashboardValidator — the save-time rules (the spec's §3.1/§3.2), exhaustive, one code per rule. The ports they judge through, declared here and implemented by the surfaces: TemplateContractFacts (a pinned template version's type and script contract — TemplateContractFacts.over(TemplateDryRenderer, TemplateVersionStatuses) is the production adapter), RendererConfigValidator (this module's table/kpi schemas and the Plotly shape check; L4 adds the plot-schema depth behind the same port), PipelineReleaseFacts (a pinned pipeline release: status, read-only, parameters, caller output columns — application implements it over ReadOnlyPipelineRule, L1b), ParameterSetFacts (a pinned set version's parameters and their dependents — ParameterSetFacts.over(ParameterSetRepository) is the production adapter).
• VisualizationRepository / DashboardRepository (jdbc; V42) over ONE generic core, ArtifactRepository — the ParameterSetRepository API for each family: every statement takes the workspace, the hash is the database's (HASH_EXPR over the bound body), the one-draft index and the draft race mapped to *.version.conflict.
• VisualizationService / DashboardService — the versioning §3.5 verb table over one generic core (ArtifactLifecycle): create, the draft write, release, purge, discard, restore, switch, import, and the lensed reads (every read takes a ReadLens, no default). A visualization's release cascades its DRAFT transform pins with consent (TemplateReleaser) and passes the ReleaseEvidence gate — whose default REFUSES (visualization.release.tests_missing) until L4 installs the evidence tables' gate. A dashboard's release checks its sources, set and visualizations and cascades DRAFT visualization pins with consent through VisualizationService.release, inside ONE transaction. Since L1b the lifecycle core's repository, transactions and flipDraft, and both services' repository, are module-INTERNAL (the L1a security pass's O3): a surface reads only through the lensed reads — which gained findVersionByName (a pin's address) and DashboardService.pinnedBy (D30's used_by, lensed) for the MCP tools.
• ArtifactTransferService — the export envelopes (rest-api §21.4's shape: templates at a visualization envelope's root, visualizations at a dashboard's) and the import (the lifecycle keys stripped by name, the strict bind, templates → visualizations → the artifact) behind the TemplateBundle port; the routes are L1c's.
• VisualizationProperties / VisualizationConfig — datapipelines.visualization.* (Configuration §3.33).
• The closed vocabularies RendererKind, AssertionKind, TestRunStatus, DashboardObjectType, ActionScope, StateSetting, LayoutPosition (Enumerations §31–§37); ReleaseEvidence (the D56 gate, NOT_INSTALLED by default — refusing) and TemplateBundle (the transfer's template half) are ports L4 and L1c implement.

Why it is its own module. The design record's D29: a separate module whose responsibilities and edges are specified before implementation. Dashboards and visualizations are two aggregates that share one lifecycle shape; the runtime that executes a dashboard (DashboardRuntime, L2) is cross-aggregate and lives in application.

Tests: the readers' key tables pinned against the models' properties by reflection, every reason and bound; the validators — one case per code, the count of codes covered asserted equal to the family's catalogued validation set, the spec's two worked documents accepted unchanged; the two repositories and every lifecycle verb against the module's own Postgres container (VisualizationTestDb, the shipped migrations through plain JDBC), one suite over both families — workspace scoping, the DB-projection hash, the one-draft index and the draft race FORCED; the release cascades rolled back by a stale hash; the transfer strip and order; the spec-drift suites (§13.22/§13.23 both directions, configuration §3.33, enums §31–§37).

6. Version Catalog

gradle/libs.versions.toml — single source of truth for third-party versions.

[versions]
kotlin = "1.9.24"
spring-boot = "3.3.2"
kotlinx-coroutines = "1.8.1"
jackson = "2.17.2"
flyway = "10.17.0"
slf4j = "2.0.13"
logback = "1.5.6"
freemarker = "2.3.33"
hikari = "5.1.0"
postgresql = "42.7.3"
mssql-jdbc = "12.6.1.jre11"
mysql-connector-j = "8.4.0"
h2 = "2.2.224"
duckdb-jdbc = "1.0.0"
sqlite-jdbc = "3.46.1.0"
ojdbc11 = "23.4.0.24.114"
argon2-jvm = "2.11"
jjwt = "0.12.6"
micrometer = "1.13.2"
spring-boot-starter-test = "3.3.2"
testcontainers = "1.20.1"
mcp-sdk = "0.10.0"
htmx = "2.0.0"
junit-jupiter = "5.10.2"
mockk = "1.13.12"
kotest = "5.9.1"

[libraries]
kotlin-stdlib = { module = "org.jetbrains.kotlin:kotlin-stdlib", version.ref = "kotlin" }
kotlinx-coroutines-core = { module = "org.jetbrains.kotlinx:kotlinx-coroutines-core", version.ref = "kotlinx-coroutines" }
jackson-module-kotlin = { module = "com.fasterxml.jackson.module:jackson-module-kotlin", version.ref = "jackson" }
# ... (every dependency declared here, used by ref from build.gradle.kts files)

spring-boot-starter-web = { module = "org.springframework.boot:spring-boot-starter-web", version.ref = "spring-boot" }
spring-boot-starter-jdbc = { module = "org.springframework.boot:spring-boot-starter-jdbc", version.ref = "spring-boot" }
spring-boot-starter-data-redis = { module = "org.springframework.boot:spring-boot-starter-data-redis", version.ref = "spring-boot" }
spring-boot-starter-oauth2-client = { module = "org.springframework.boot:spring-boot-starter-oauth2-client", version.ref = "spring-boot" }

# Flyway — `app` module only (§3.1 rule 2). Since Flyway 10 the Postgres support is a
# separate artifact; flyway-core alone cannot migrate a Postgres target.
flyway-core = { module = "org.flywaydb:flyway-core", version.ref = "flyway" }
flyway-database-postgresql = { module = "org.flywaydb:flyway-database-postgresql", version.ref = "flyway" }
# etc.

Redis client: no explicit entry — spring-boot-starter-data-redis brings Lettuce as its default client, version-managed by the Spring Boot BOM. Adding a second, separately-pinned Lettuce entry would let it drift from the BOM; if Jedis is ever wanted instead, that is an explicit exclusion + dependency, and a spec change here.

Gate G2 — RESOLVED 2026-08-07 (lockfile sub-item closed 2026-08-15, §7.6). Every entry was resolved against repo1.maven.org maven-metadata.xml, pre-releases rejected, and BOM-managed artifacts (verified by parsing spring-boot-dependencies:3.5.16 + its 43 imported BOMs) declared version-less. gradle/libs.versions.toml is now the ratified source of truth for versions; the table above is historical. Anchor decisions: Spring Boot 3.5.16 (4.x exists but §11.1 freezes 3.x for v1), Kotlin 2.4.10 (with an empirically-verified conflict-resolution win over the BOM's kotlin-bom 1.9.25 — see the toml comment). gradle.lockfile (transitive pinning) landed 2026-08-15 per §7.6. The original G2 procedure is retained below for future catalog changes:

1. Resolve the current stable release: curl -s 'https://search.maven.org/solrsearch/select?q=g:"org.flywaydb"+AND+a:"flyway-core"&core=gav&rows=5&wt=json' | jq -r '.response.docs[].v' (repeat per artifact; or run ./gradlew dependencyUpdates once the build exists).
2. Reject anything that is not a released stable version — no ranges, no +, no latest.release, no SNAPSHOT, no RC/M/beta (§6.1, and the project-wide version-pinning rule).
3. Check BOM-managed artifacts are not pinned here at all: anything the spring-boot BOM manages (Jackson, Micrometer, Lettuce, HikariCP, the Spring modules) takes its version from the BOM. A local pin that disagrees with the BOM is a silent runtime-incompatibility source.
4. Verify JDK 21 compatibility for each pinned artifact, then commit gradle.lockfile (§6.1) in the same commit as the catalog.
5. Record the date the catalog was verified in the change log below.

This gate is the reason the versions above are safe to leave stale: they are explicitly labelled unverified, so nobody builds on them believing otherwise.

6.1 Versioning rules

• Pin every dependency to an exact version. No ranges, no +, no SNAPSHOT in production.
• Upgrade deliberately. Dependabot or Renovate can open PRs, but no auto-merge.
• Track security advisories. GitHub Dependabot alerts + Snyk (or equivalent) on the repo.
• Lockfile committed. Gradle generates gradle.lockfile for repeatable builds (§7.6).

7. Build Conventions

7.1 CommonConventionsPlugin

Applied to every module. Configures:

class CommonConventionsPlugin : Plugin<Project> {
    override fun apply(project: Project) {
        project.plugins.apply("org.jetbrains.kotlin.jvm")
        project.plugins.apply("io.spring.dependency-management")

        project.extensions.configure<JavaPluginExtension> {
            sourceCompatibility = JavaVersion.VERSION_21
            targetCompatibility = JavaVersion.VERSION_21
        }

        project.extensions.configure<KotlinJvmProjectExtension> {
            jvmToolchain(21)
            compilerOptions {
                freeCompilerArgs.add("-Xjsr305=strict")
                // NOTE (2026-08-07): -Xcontext-receivers deliberately NOT set — the flag
                // was removed from the Kotlin compiler in 2.2 and is a hard error on 2.4.x.
                // Its successor (-Xcontext-parameters) is unused by v1.
                allWarningsAsErrors = true           // strict
            }
        }

        project.dependencies {
            testImplementation(libs.junit.jupiter)
            testImplementation(libs.mockk)
            // Kotest ASSERTIONS only — deliberately NOT kotest-runner-junit5 (2026-08-08).
            // The runner registers a second JUnit Platform TestEngine in every test JVM;
            // with zero Kotest spec classes it discovers nothing but participates in
            // result reporting, and it has open Gradle 9 incompatibilities that surface
            // as truncated test-result stores (intermittent EOFException in
            // SerializableTestResultStore). §7.4's convention is JUnit 5 as the platform,
            // Kotest as the assertion library. A module that wants Kotest SPECS adds the
            // runner to its own build file and owns that engine's Gradle compatibility.
            testImplementation(libs.kotest.assertions)
        }

        project.tasks.withType<Test>().configureEach {
            useJUnitPlatform()
            testLogging {
                events("passed", "skipped", "failed")
                showExceptions = true
                showCauses = true
            }
        }
    }
}

7.2 JVM target

• JDK 21 (current LTS as of 2025). Use virtual threads (Project Loom) for blocking-IO work in v1.1+.
• Toolchain configured in Gradle; the build fails if the wrong JDK is used (rather than silently producing Java 17 bytecode).

7.3 Linting / formatting

• ktlint via plugin, enforced on build. No code merges with ktlint violations.
• detekt for static analysis. Configured in detekt.yml. Some rules relaxed where they conflict with Kotlin idioms.
• Pre-commit hook runs ./gradlew ktlintCheck detekt (optional but recommended).

7.4 Test conventions

• JUnit 5 as the platform.
• MockK for mocking (idiomatic for Kotlin, vs. Mockito which requires extra setup for final classes).
• Kotest for assertion library (more expressive than JUnit's built-in).
• Testcontainers for integration tests requiring real databases.
• Test file naming: *Test.kt for unit tests, *IntegrationTest.kt for integration tests (different Gradle task).
• Test source root convention: unit tests in each module's src/test/kotlin. Cross-module integration tests live in tests/integration-tests/src/test/kotlin — there is no per-module src/integrationTest source set in v1; the root integrationTest task simply depends on :tests:integration-tests:test. A module needing module-local integration tests (e.g. against a Testcontainer) puts them in its own src/test/kotlin with a *IntegrationTest name; the verifyTestsExecuted guard scans src/test/kotlin only, so tests placed anywhere else are unguarded — don't.

7.5 Build commands

./gradlew build                       # everything: compile, test, lint
./gradlew test                        # unit tests only
./gradlew integrationTest             # integration tests
./gradlew :modules:dag:test           # one module's tests
./gradlew bootJar                     # produce executable app JAR
./gradlew -Poracle build              # build with Oracle driver bundled
./gradlew -Pmysql build               # build with MySQL driver bundled
./gradlew clean build                 # full rebuild

7.6 Dependency locking

Every module — and buildSrc itself — resolves dependencies against a committed gradle.lockfile. Locking covers all configurations (lockAllConfigurations()) in STRICT mode, applied by CommonConventionsPlugin to every module and declared directly in buildSrc/build.gradle.kts (an included build is not reached by the main build's locking). A resolution that drifts from the lock — a version bump, a new or removed dependency, a shifting transitive — fails the build. This is the mechanical twin of §6: the catalog records what we ASK for, the lockfile records what was actually RESOLVED, and STRICT mode additionally fails on any locked configuration with no recorded lock state, so a newly added configuration cannot escape silently.

• Maintenance procedure (the only way locks may change): DEVELOPMENT.md §6.2 — ./gradlew resolveAndLockAll --write-locks, then review and commit the diff with the dependency change that caused it.
• resolveAndLockAll is the documented "lock all configurations in a single build execution" pattern (docs.gradle.org dependency_locking); the built-in dependencies task resolves only one project's configurations, so each project registers its own copy.
• Exactly two artifacts are excluded from lock validation, both declared with their reason in modules/datasources/build.gradle.kts: the §5.4.1 flag-gated drivers ojdbc11 and mysql-connector-j. One committed lockfile cannot validate both the default build and -Poracle / -Pmysql builds.
• The root project locks too (STRICT, same resolveAndLockAll flow, declared directly in the root build.gradle.kts): it resolves the kover aggregation configuration for the cross-module coverage report (§7.7).

7.7 Coverage (Kover)

Every module gets the Kover plugin (org.jetbrains.kotlinx.kover, pinned in the catalog) from CommonConventionsPlugin. The root project applies it too and merges the modules into an aggregated report via kover(project(...)) dependencies — wired reactively through pluginManager.withPlugin("org.jetbrains.kotlinx.kover") (012/F3), so a new module joins the aggregate automatically and a subproject without the Kover plugin cannot break root resolution. tests/integration-tests is deliberately excluded from the ROOT aggregate: pulling its Testcontainers tests into the report's task graph made koverHtmlReport need a Docker daemon, and its integration coverage would break comparability with the unit-only 2026-08-15 baseline the coverage floors derive from. Its own module-level report still exists; only the root aggregate excludes it. ./gradlew koverHtmlReport produces per-module reports plus the aggregate.

check depends on koverVerify. Each module carries a minimum line coverage floor in COVERAGE_FLOORS (CommonConventionsPlugin): the module's measured baseline from the first Kover run (2026-08-15) minus 2 points, rounded down. Floors are a regression tripwire, not a coverage target — raise one only when coverage genuinely improved; never lower one to force a build green. A module absent from COVERAGE_FLOORS fails configuration unless it is in NO_COVERAGE_FLOOR_ALLOWLIST with its reason — currently only tests/integration-tests (no main sources).

Escape hatch: -Pkover.off runs tests without the coverage agent attached (for timing-sensitive diagnosis). The flag also skips the floor rules and the check→koverVerify wiring — with instrumentation off there is no coverage data, so a registered floor would fail on absent data. It is for targeted diagnosis, not for making a red build green: the floors still enforce on every normal build.

7.8 Architecture-as-tests (Konsist)

House layering rules that previously existed only as prose are encoded as Konsist tests (pinned in the catalog, TEST dependency only):

• modules/web — RequiredScopeKonsistTest: every HTTP handler on a @RestController declares @RequiredScope. Deliberately redundant with the reflection-based RequiredScopeCoverageTest: one proves it on the live classpath, the other statically from sources.
• tests/integration-tests — ArchitectureGuardTest, scanning every module's production sources from the cross-module suite:
  • no field injection (@Autowired on properties/fields) — constructor injection only;
  • no DI stereotypes — no class, interface, or object carries @Service, @Component, or @Repository (zero allowlist; every bean is an explicit @Bean method, §8.4);
  • every @Transactional names the metadata transaction manager (056, §8.5) — a bare one fails the build. This replaced the pre-056 outright ban, which was correct while there was no transaction manager at all. The check is a source text scan rather than an annotation walk, so it can name file and line and cannot be satisfied by a constant that reads as bare on the page; it asserts non-vacuity first, because a scan that finds nothing would otherwise pass forever;
  • nothing below the surfaces imports a web or MCP type (056, §5.13): application and every domain module are refused co.datapipelines.web.* / co.datapipelines.mcp.* imports. Gradle's verifyModuleDependencies enforces the declared EDGES; this enforces the source-level rule, names the offending import, and would still fire on a type reached through some future transitive edge;
  • the production scope itself is non-empty, and the layering scan actually covers modules/application (a guard scanning nothing proves nothing).

Konsist lives in existing test source sets only — no dedicated Gradle module.

8. Spring Boot Conventions

8.1 Persistence layer: NamedParameterJdbcTemplate

Decision: NamedParameterJdbcTemplate exclusively. No JPA, no Hibernate, no Exposed.

Reasons:

• Predictable SQL. Every query is explicit SQL — no generated SQL, no N+1 surprises, no lazy-loading traps. What you write is what runs.
• Named parameters (:start_date, :pipeline_id) are far more readable than positional ? for queries with 5+ parameters.
• Full control over type mapping, JSONB handling, array columns. No fighting the ORM.
• Lighter — no Hibernate dependency, no second-level cache, no entity manager.
• Spring-native — NamedParameterJdbcTemplate is part of Spring JDBC, auto-configured by Spring Boot.

Where repositories live: in the module that owns the entity, never in web and never in a shared persistence module — see the §3.1 persistence-ownership rule for the full mapping and the Flyway/Redis boundaries. Each owning module takes spring-boot-starter-jdbc; the DataSource bean itself is app-level.

Pattern per module (pipeline-contract; declared as a @Bean in the consuming configuration — no @Repository stereotype, §8.4):

class PipelineRepository(private val jdbc: NamedParameterJdbcTemplate) {

    fun findById(id: UUID): Pipeline? =
        jdbc.query(
            "SELECT * FROM pipelines WHERE id = :id AND is_deleted = FALSE",
            mapOf("id" to id),
            PipelineRowMapper
        ).singleOrNull()

    fun create(pipeline: Pipeline): Pipeline =
        jdbc.queryForObject(
            """INSERT INTO pipelines (id, name, display_name, description, owner_id)
               VALUES (:id, :name, :displayName, :description, :ownerId)
               RETURNING *""",
            mapOf(
                "id" to pipeline.id,
                "name" to pipeline.name,
                "displayName" to pipeline.displayName,
                "description" to pipeline.description,
                "ownerId" to pipeline.ownerId
            ),
            PipelineRowMapper
        )!!
}

See Metadata DB spec §6 for the full pattern.

8.2 Module auto-configuration

Modules that provide Spring beans expose them via a @AutoConfiguration class in META-INF/spring/org.springframework.boot.autoconfigure.AutoConfiguration.imports. This lets app pull in module configuration without explicit @Import per module.

Example (templates module):

@AutoConfiguration
@ConditionalOnClass(TemplateEngine::class)
@EnableConfigurationProperties(TemplatesProperties::class)
class TemplatesAutoConfiguration {
    @Bean
    @ConditionalOnMissingBean
    fun templateEngine(registry: TemplateRegistry, props: TemplatesProperties): TemplateEngine =
        TemplateEngine(registry, props)
}

8.3 Configuration properties per module

Each module exposes typed configuration:

@ConfigurationProperties(prefix = "datapipelines.templates")
data class TemplatesProperties(
    val cacheSize: Int = 1000,
    val renderTimeoutMs: Long = 10_000
)

Key names, defaults, and env-var bindings are defined once in configuration.md (D8); the snippets here are shape illustrations, not a second definition. Composed into application.yml in app:

datapipelines:
  templates:
    cache-size: 2000
    render-timeout-ms: 5000
  staging:
    h2:
      max-memory-mb: 2048
  executor:
    max-parallel-nodes: 8
  auth:
    jwt:
      ttl-hours: 12

8.4 Beans and DI

• Constructor injection only. No @Autowired on fields (per the user's global rules).
• Zero DI stereotypes. No production class carries @Service, @Component, or @Repository — every bean is declared explicitly as a @Bean method in the module's @Configuration class (auth: AuthConfiguration; engine/domain repositories: EngineConfiguration / DomainConfiguration where their consumers are already wired; templates: TemplatesConfiguration). Component scanning stays on, but the only annotated production classes are configurations (@Configuration / @AutoConfiguration), @ConfigurationProperties, and the web edge (@Controller, @RestController, @ControllerAdvice). Enforced by ArchitectureGuardTest (§7.8).
• Transactions: see §8.5 — @Transactional is allowed since 056 and must always NAME its manager.
• Open classes: Spring's kotlin-spring plugin opens @Configuration classes and @Transactional ones that need CGLIB proxying. No manual open keyword required — and §8.5 explains why that is not a detail.

8.5 Transactions (056, ARCH-AUDIT S3 / ruling R6)

One transaction, one database. There is exactly ONE Spring transaction manager, metadataTransactionManager (declared in app's TransactionConfiguration, over the metadata DataSource), and N Hikari pools for CUSTOMER databases which are not Spring transaction resources and must never become one. No XA, no JTA, no two-phase commit. The full consistency model, including what a partially-failed pipeline leaves behind, is dag-executor §16.

Four rules, each mechanical:

1. The manager is always named: @Transactional("metadataTransactionManager"), never bare. A bare one binds to whichever manager Spring finds — correct by accident with one manager, a trap with two, and silent about which database it means either way. ArchitectureGuardTest (§7.8) fails the build on a bare one.
2. Transactions live on the service layer, never on a repository or a controller. First-choice atomicity is still a single data-modifying CTE (the PipelineRepository stance, unchanged); the annotation is for a unit of work that genuinely spans statements. Two writes that are ALTERNATIVES selected by a caught constraint violation are not such a unit — see PipelineService.discard and PipelineService.update, both deliberately un-annotated with the reasoning on their KDoc.
3. No customer-datasource I/O inside a metadata transaction. ConnectionLease refuses to lease while a transaction is active on the thread, with the catalogued datasource.lease_in_transaction.
4. Rollback is proven, not assumed. @Transactional needs a CGLIB proxy; a Kotlin class is final unless kotlin("plugin.spring") opens it, and an advice that cannot be applied produces no error and no transaction. TransactionRollbackE2eTest asserts two writes plus a throw persist nothing, with a commit control beside it so the assertion cannot pass vacuously.

PromotionReceiveService's explicit TransactionTemplate (055) remains valid: it is the same manager, demarcated in code rather than by annotation.

9. Test Categories

9.1 Unit tests

• Live in src/test/kotlin of each module.
• Run on every ./gradlew build.
• Fast (target: full unit test suite < 60 seconds).
• No Spring context (use plain Kotlin test setup, manual DI).
• Mock external dependencies.

9.2 Integration tests

• Live in tests/integration-tests/src/test/kotlin (cross-module), or in the owning module's src/test/kotlin named *IntegrationTest (module-local, e.g. against a Testcontainer) — see §7.4; no per-module src/integrationTest source set exists in v1.
• Cross-module suite runs on ./gradlew integrationTest (delegates to :tests:integration-tests:test); module-local ones run with the module's normal test task.
• Slower; use real databases (Testcontainers), real H2, real HTTP layer.
• Cover: end-to-end pipeline execution, MCP tool calls, REST endpoints, SSE streams.
• One container per module per test JVM, not one per suite (round 060): each module's suites share a SharedPostgres/SharedRedis singleton started and reset on first touch; app and the E2E module migrate through the first context's Flyway, the domain modules apply the shipped migrations once over JDBC (§3.1 rule 2 keeps the Flyway dependency in app). Sharing is safe by rule, and the rule is load-bearing: each spec cleans the tables it touches (TRUNCATE ... CASCADE + re-seed), seeds carry suite-unique identities, suites with global or exact-set assertions reset via the module's clean helper before seeding, and partial-migration suites take a scratch database on the shared container. Suites that legitimately need isolated deployments keep their own containers (the promotion E2E pair). Verified by shuffled-method-order runs (two seeds, DEVELOPMENT.md §9.3); adding a suite that relies on a fresh container instead of cleaning will fail those runs.

9.3 Smoke tests

• Live in app module.
• Run on every ./gradlew build (full Spring context loads, smoke-tests critical endpoints).
• Run before deployment.

9.4 Performance / load tests

• Live in tests/performance/ (separate directory).
• Run on demand, not in CI fast path.
• Use JMeter or k6 scripts.

10. Documentation Conventions

10.1 Specs

• Live in docs/ (this directory).
• One spec per file.
• Versioned (status: v1 frozen, additive-only).
• Cross-referenced via Markdown links.

10.2 KDoc

• Every public class, function, and property has KDoc.
• KDoc explains what the API does and why, not how (the code is the how).
• Examples for non-trivial APIs.

10.3 README per module

Each modules/{name}/README.md is one page:

• What the module does.
• Its public API.
• Its dependencies.
• How to test it locally.

11. Stability Promise

11.1 Frozen in v1

• The module list (the 10 modules + integration tests).
• The dependency direction — specifically the allowed-dependency table in §4.2, which is the normative form. Adding an edge is a spec change, not a build-file change.
• Persistence ownership (§3.1): repositories in their owning module, Flyway only in app, Redis only in dag and web.
• The version catalog as the single source of dependency versions.
• Kotlin + JDK 21 as the language/runtime baseline.
• Spring Boot 3.x as the application framework.

11.2 Not frozen

• Specific third-party versions (updated deliberately per §6.1).
• Internal class names (only public API is the contract).
• Test framework choice (could evolve — Kotest + MockK is v1 default).
• Build commands (could evolve with Gradle versions).

12. Open Questions / Future Additions

Out of scope for v1:

• Module extraction: publishing typesystem to Maven Central for clients to consume (typed client SDKs).
• Gradle version catalogs merging: when the project grows, split catalogs per concern (database drivers, web libs, etc.).
• Build performance: Gradle configuration-cache, build-cache sharing across CI runs.
• Multi-platform (KMP): extract typesystem to a KMP module so the same types can be consumed by JS / .NET (via tooling) for client SDKs. Probably never needed.
• Polyglot modules: if we add Python SDK or CLI, separate python/ directory at root, not in modules/.

13. Verification Checklist

13.1 Implementation gates (must be closed before the first build lands)

These are the two items this spec deliberately does not resolve on paper. Each has an exact check; neither may be closed by recall.

• [x] G1 — MCP SDK coordinates (§5.8): resolved 2026-08-07 — io.modelcontextprotocol.sdk:mcp-core:2.0.0 + mcp-json-jackson2:2.0.0, Streamable HTTP transports confirmed by jar inspection; see the resolved gate note in §5.8.
• [x] G2 — version catalog vs Maven Central (§6): resolved 2026-08-07 — all entries verified, BOM-managed artifacts version-less, gradle/libs.versions.toml is the ratified source. Lockfile sub-item closed 2026-08-15 (§7.6): committed gradle.lockfile per module + buildSrc, STRICT mode.

13.2 Build checklist

Before considering the module structure "ready":

• [ ] ./gradlew build succeeds from clean state.
• [ ] The §4.2 allowed-dependency verification task passes (no module declares an edge outside its row).
• [ ] ./gradlew -Poracle bootJar produces a JAR containing ojdbc11; the default bootJar does not.
• [ ] A driver mounted read-only into /app/lib/ is picked up at runtime with the image's LOADER_PATH=/app/lib,/opt/graaljs (§5.4.1).
• [ ] Flyway migrations live only in app; no other module declares a Flyway dependency.
• [ ] Only dag and web declare a Redis dependency.
• [ ] All unit tests pass.
• [ ] All integration tests pass.
• [ ] ktlint and detekt clean.
• [ ] ./gradlew bootJar produces an executable JAR.
• [ ] java -jar app/build/libs/datapipelines-app-*.jar starts the app and serves /health.
• [ ] Every module has a README.
• [ ] Every module's public API has KDoc.
• [ ] Version catalog has exact versions for every dependency (no +, no SNAPSHOT).
• [x] Gradle lockfile committed (§7.6 — per-module gradle.lockfile, STRICT mode, 2026-08-15).
• [ ] No internal module depends on app (one-way dependency).
• [ ] tests/integration-tests runs end-to-end pipeline against a real PG container.

Appendix A: Change Log