Scheduler Specification

Status: v1.10 (slices 1–4 of #9 — optional notification delivery implemented; the application credential remains pending) Owner: datapipelines.co core Depends on: REST API §20, Auth, DAG Executor, Metadata DB §4.22–§4.25, Configuration §3.29 Design record: scheduler design revision (ratified 2026-09-25) — the why; this page is the what Last updated: 2026-10-25

1. Purpose

A schedule runs a job at the occurrences of a cron pattern in a timezone. In v1 the one job is a pipeline run: the schedule names a pipeline, follows its current version and passes literal parameter values, plus — since slice 3 — optional per-parameter bindings that resolve on the run's frozen reference time (§2.2). This spec defines:

• when a schedule fires, including across daylight-saving changes (§3);
• what happens to occurrences missed while no instance was dispatching, while a run was still going, or while the schedule was paused (§4);
• what a run is, the states it passes through and why it ends where it ends (§5);
• who a schedule runs as and what that identity may do (§6);
• how many scheduled runs execute at once (§7);
• how instances divide the work and how a deploy ends in-flight runs (§8).

A schedule is not a versioned artifact. It has no draft or release, is never promoted and is edited in place; its revision number exists only so two people editing it cannot overwrite each other. Nothing about a schedule is exposed over MCP.

2. Concepts

2.1 The pipeline payload

{ "pipeline": "finance/daily/revenue", "version": "current" }

Since slice 3 the payload may also carry the optional, additive parameter_bindings object (§2.2) — current and the pipeline name are still the only required keys, and latest is refused by name. current follows the pipeline's current-version pointer at each run (Versioning §3.4) — wherever a person points it, including at a draft under the development posture. latest is refused by name (it is not a selector), and so is a version number. At SAVE, a payload naming a pipeline with no current version to follow is refused 400 schedule.validation.target_not_released (#280) — release it, or switch its current version, then save again (§13.19; it was a payload_invalid / pointer_null refusal before, one code shared with the shape refusals) — while the RUN side's refusal of the same absence stays a not_started / pointer_null run that blocks (§5). Each run freezes what it actually ran — {pipeline_id, version, body_sha256} and, since slice 3, resolved_parameters — before it launches, so a later release, switch or draft edit never changes a run that was already admitted. If the draft is edited between that freeze and the launch, the run is refused not_started / version_changed rather than run on a body nobody reviewed.

A schedule's parameters are literal values for the pipeline's declared parameters, checked by the same binder an interactive run uses — at save, again at unblock, and again when each run is prepared. A value that no longer binds (the pipeline's parameters changed) refuses the run and blocks the schedule; it is never silently dropped.

2.2 Bindings: TODAY and YESTERDAY

A binding computes ONE parameter's value per run, on the run's frozen reference time — the schedule's own logical occurrence time in its timezone, frozen on the run row when it was recorded (never the actual start: a run due 23:55 that starts 00:05 the next day still resolves TODAY to the day it was due for):

The rules an operator can rely on:

• Keywords are an exact allowlist — TODAY and YESTERDAY, uppercase — matched only inside an explicit {"source": "keyword"} binding. A literal STRING "TODAY" in parameters reaches the pipeline as the string; nothing is ever scanned for magic words.
• One source per parameter. The same name in parameters and parameter_bindings is refused (400 schedule.validation.binding_conflict); there is no precedence rule to wonder about.
• A binding must fit the version. A bound name the pipeline's current version does not declare, or declares as anything but DATE, is refused at save (400 schedule.validation.binding_invalid with a reason) and — if it stopped fitting after the save — refuses the run not_started / parameters_invalid and blocks, exactly like a literal that no longer binds.
• The run's time is not a client field. A reference, reference_at or reference_timezone key inside the payload or a binding is refused 400 schedule.validation.payload_invalid; the reference time is minted by the scheduler and frozen on the run.
• A bound run executes as the system identity exactly as a literal one — resolution changes VALUES, never the principal (§6).

Each run's detail shows prepared.resolved_parameters — the literal map the execution actually launched with, bindings already resolved to dates — beside the frozen parameters the schedule holds. A keyword binding is checked structurally at save (the keyword exists, the type fits) and resolved fresh at every run, so a schedule saved on Friday resolves each day's value when it fires.

2.3 Notification settings (#442)

Each schedule stores recipients as a list (entered comma-separated in the form); an empty list means off. The five event checkboxes are start, success, failure, unknown, blocked. Failure, unknown and blocked default on; start and success default off. Addresses are validated and deduplicated server-side, with the configured recipient limit (default 20), one WHATWG valid e-mail address grammar and at most 254 characters. Invalid settings report only a field/index and reason, never an address.

The delivery policy is the owner's 2026-10-03 ruling: start means execution confirmed running, success means successful completion; failure covers failed, every not_started reason (Could not start), cancelled (Stopped — cancelled) and aborted (Stopped — instance shut down), each with its own subject. Unknown is never described as a stop; its later reconciled outcome is an update about the same run under the unknown checkbox. Blocked is a separate schedule-level notice only on the actual transition to blocked. No mail for skipped ticks or pause/resume. Manual runs use the same policy.

Recipient addresses are visible only to roles with schedule.update (author, workspace admin, super admin and their MCP counterparts). Every other schedule reader sees a count and the event set; details never display addresses. Writing settings uses the existing schedule create/update permissions. Audit rows carry counts/events, never addresses. PUT omission preserves stored settings, including omitted members of a present object (the frozen additive REST contract).

Delivery (#442b). Scheduler mail is off unless datapipelines.scheduler.notifications.enabled is true (Configuration §3.29); true needs the shared mail transport and datapipelines.auth.base-url, or startup refuses. The deployment's state is computed, never stored, and every schedule shows it (notifications.delivery, REST §20): on, or off with a reason — disabled (the flag), mail_unconfigured, or no_allowlist (the development posture with an empty recipient-domains allowlist: nothing sends until a domain is listed; under hardened an empty allowlist means no domain restriction). While off, saved settings are kept, nothing is queued and nothing leaves; the form and details say so, and saved settings are never described as delivered.

While on, each transition the schedule's event set selects writes ONE notice — schedule_notification_events, Metadata DB §4.39 — in the transition's own transaction, beside its trail row, plus one delivery per recipient (§4.40), the address snapshotted at that moment. A run reaches each outcome at most once, so a replayed reconciliation or a duplicate callback writes no second notice. A recipient removed later still receives what was already queued, and a schedule deleted after its notice was queued still sends it; a deleted schedule queues nothing new. A recipient outside a non-empty allowlist is written failed / domain_not_allowed at once, and more than backlog-max pending deliveries writes the rest failed / backlog_full — visible on the run, never dropped silently.

Notices render the run's target summary and reference timezone frozen at creation, due/start/end times and generic execution reference when known. Older runs with no frozen target omit “Runs”; executor payloads remain opaque. The schedule's name stays live because a rename retains its identity.

The sender (the schedule-notifier task, every tick-interval-seconds, on dispatching instances only — §8.1) claims due deliveries in batches and sends each as its own message: To is that one address, the subject is <schedule name> — <state words> (Enums §43), the plain-text and HTML bodies name the schedule, the run's outcome and its times, and link to <base-url>/schedules?id=<schedule>&run=<run>. A body never carries rows, SQL, parameter values, error text, a bearer link or any other address. The Message-ID is fixed when the delivery is queued. The sender re-checks the flag and the allowlist at send: a delivery queued while mail was on and sent after it was turned off, or after its domain was narrowed out, ends failed with that reason.

A person holding schedule.update may resend a failed or uncertain delivery (REST §20.13): it is queued again under the same Message-ID, its tries kept — so a retryable failure after a resend is final. Resending an uncertain one warns that the recipient may receive it twice. Each send is a trail row on the run (notification_sent / notification_failed / notification_uncertain, ids and codes, never an address); the resend itself is the person's act and is audited (schedule.notification_resent, Auth §10). The run dialog's Notifications table lists the run's deliveries and offers the resend (UI Screens §4.20); addresses there follow the same rule as the settings.

3. Occurrences and the DST rule

Patterns are five-field Unix cron — minute hour day-of-month month day-of-week. A six-field (seconds) pattern is refused; so is the library's disabled pattern -. The timezone is an IANA region id (America/New_York); a fixed offset (+02:00) is refused, because it cannot follow a region's daylight-saving rules.

The pattern is matched on the local wall clock of the timezone, and each matching local time becomes a UTC instant by these rules:

Measured in America/New_York (the unit suite's cases; the library's own cron evaluation, which the scheduler does not use for timing, is shown for contrast):

GET /schedules/preview (REST API §20.3) shows the next occurrences of any pattern with their local time and offset, from the same function, before anything is saved.

Save-time guards. A pattern whose consecutive occurrences come closer than datapipelines.scheduler.min-interval-seconds (default 300 s) anywhere in the 14 days after the save is refused (schedule.validation.interval_too_short). The 14-day horizon covers every weekday twice; a pattern whose only tight pair straddles a month end beyond it is not caught, and the per-workspace cap (max-schedules-per-workspace, default 100) still bounds it.

4. Missed occurrences, overlap and Run now

A dispatcher tick (§8) records what fell due since the schedule's next_due_at. What it records depends on how late it is and on the schedule's missed-run policy:

Editing a schedule's pattern or timezone also recomputes from now — the old pattern's occurrences are not missed.

Run now (REST API §20.12) records a manual run with the request time as its reference time. It runs like any other run — as the system identity (§6), through the same capacity and admission — and is allowed on a paused schedule (a person asked for it), refused on a blocked one (409 schedule.blocked) and refused while another run is active (409 schedule.run.overlap). It is not a retry: re-running an occurrence whose outcome was unknown is deliberately not offered, because a new execution may repeat effects.

5. Runs: states and reasons

A run is admitted by a worker: capacity first (§7), then the executor prepares it (resolves current, checks the parameters, freezes the snapshot), then the start is claimed — a conditional write that only one worker can win — and only then launched. That order is what makes the states below honest: nothing is ever launched without a committed claim, and a claim is never made without capacity.

5.1 Not started means nothing ran

A scheduled launch fails closed: the execution may begin only after its RUNNING record is written, under an id minted at the claim. So a launch that could not write its record is not_started / record_unwritable — no node ran — rather than a run in an unknown state. Every not_started run still has its trail, which says why. The barrier is now trivially true of the wire too (#306): the RUNNING row is committed before the first frame leaves the emitter, so the every-surface assumption "an id that reached a client resolves" holds for the scheduled path and the interactive ones alike. The insert itself is bounded (datapipelines.executor.lifecycle-write-timeout-seconds, #311): a past-bound insert is the same recorded = false refusal — fail-closed here, a WARN on the interactive paths.

5.2 What blocks

unknown always blocks: a scheduler that fired again after an unknown outcome could run a non-idempotent load twice. Most executor refusals block too, because a person has to fix something — the pipeline has no current version (pointer_null), it is gone (target_not_found), the saved payload or parameters no longer validate, the system identity was refused (authority_refused), the launch path refused before recording (start_refused), or no executor is registered under the run's id (executor_unavailable).

Four do not block, because nothing about the schedule is wrong: capacity (the instance was busy), record_unwritable (the metadata database refused one write), workspace_inactive (a reactivated workspace is handled as an outage) and version_changed (a draft edited between preparation and launch; the next occurrence prepares afresh).

A block names its cause (blocked.reason) and the run that caused it (blocked.run_id); the first cause is kept until a person unblocks. Unblock re-validates the saved payload and parameters, then recomputes next_due_at from now; resume never clears a block.

5.3 A late truth about an unknown run

An unknown run is watched for 24 hours. If its execution later reports a real terminal — a worker that was stalled, not dead, finished after all — the SAME run is updated to that state and its trail gains updated_after_unknown. The schedule stays blocked: an answer arriving late is information for the person, not permission to resume.

6. Who a schedule runs as

A schedule fires as the system identity — the one System account every deployment provisions (Auth §4.5) — never as the person who created it. So a creator who is demoted to viewer, removed from the workspace or deactivated does not stop their schedules, and a schedule never gains or loses power when its creator's role changes.

The system identity holds exactly three permissions, in every workspace: pipeline.read, pipeline.execute and execution.read. It holds no role, no instance permission and no credential: there is no API key, no session and no password for it, and a request that tried to present it is refused. Every launch still ASKS for pipeline.execute — a scheduled run is not exempt from permissions because it is internal.

What people may do (auth.md §7.6): every member reads schedules (a promoter through the promoter lens — the list pages over the schedules the lens admits, read in keyset windows so a create or delete between two windows neither duplicates nor drops a visible one, #257, #277); authors, workspace admins and super admins create, edit, pause, resume, unblock, delete and Run now. Executions a schedule fired are visible to every member with execution.read, not only to their executor — visibility, not ownership: cancelling one still needs execution.cancel_all.

Isolation is per workspace: a schedule, its runs and the pipeline it names all belong to one workspace, and an id from another answers 404.

7. Capacity and budget

At most datapipelines.scheduler.max-concurrent-runs (default 4) scheduled executions run at once per instance. The limit is taken before the start is claimed; a run that finds no slot is retried every 30 seconds within its lateness window, each retry on its trail and counted (datapipelines.scheduler.capacity.retries), and ends not_started / capacity when the window closes. The instance-wide execution ceiling (datapipelines.executor.max-concurrent-executions-per-instance) still applies on top, shared with people's runs.

A scheduled execution has the instance's ordinary execution timeout (datapipelines.executor.execution-timeout-seconds) and asks for the maximum result TTL (datapipelines.result.ttl-max-seconds). A scheduled pipeline's durable output is what it writes to its targets; its caller result, if any, is inspection material that expires within the hour.

8. Instances, deploys and failure

8.1 One dispatcher, any number of instances

Every instance serves the schedule REST routes. An instance with datapipelines.scheduler.enabled: true also dispatches; false is API mode — it runs no scheduler task, so it never sends scheduler mail (§2.3's sender runs only on dispatching instances). Among dispatching instances there is ONE dispatcher: the tick is a single cluster-wide task, and each tick locks the due schedules it handles, so two instances never record the same occurrence twice and never launch one occurrence twice (an occurrence is a unique row, and a start claim is a conditional write). Run tasks are spread over the dispatching instances' worker threads (threads, default 2); a worker holds a thread only while it admits and launches, never for the pipeline's runtime.

If an instance dies, its in-progress tasks are revived elsewhere after six missed heartbeats (heartbeat-interval-seconds, default 30 s). A revived task never launches again: it finds the claim, and either the execution exists (the run is running) or it does not (the run is unknown, and the schedule blocks).

8.2 Deploys and shutdown

For notification activation on a mixed-version fleet and rollback to an image predating V51's trail kinds, follow Deployment §8.3 and §8.4.

On shutdown the instance first stops admitting — a run not yet claimed stays queued and is picked up by another instance or after restart — then waits up to shutdown-wait-seconds (default 5) for launches in progress to reach "started", and only then do the local executions drain. A scheduled execution cut off by the drain ends aborted / shutdown: a conclusive outcome that does not block. Under skip that occurrence does not run again.

The drain can need 20 seconds, and the admission gate's wait up to 15 more. A container runtime whose stop grace is shorter (Docker's default is 10 s) kills the process first; the executions it cut off then end as lost instances, which the scheduler records as unknown, blocking. The shipped deploy/compose.yml gives the application container a stop grace period of 50 s, and the Helm chart's terminationGracePeriodSeconds matches (#251, #277). The arithmetic, at the configured maximum: the admission wait (at most 15 s) + the drain (20 s) + Tomcat's graceful shutdown (10 s) = 45 s, plus Helm's 5 s preStop sleep = 50 s — at 40 s the three scheduler phases alone reached 40 under Helm and left Tomcat nothing (Deployment §8.3.2; ShutdownGraceArithmeticTest pins it).

8.3 Watching it

Metrics (Observability §4.1): datapipelines.scheduler.occurrences{outcome}, datapipelines.scheduler.runs.finished{state}, datapipelines.scheduler.capacity.retries, datapipelines.scheduler.runs.in_flight and the notices' datapipelines.scheduler.notifications.{sent,failed,uncertain}. Log events: Observability §3.4E — scheduler.schedule_blocked is the one that needs a person. There is no scheduler health component: an API-mode instance never starts the scheduler, and a stalled dispatcher is not a reason to restart a pod.

8.4 The page

People manage schedules on the Schedules page (/schedules, in the rail's Operate group — UI Screens §4.20, slice 2). The rail's Schedules item lands on the Schedules home (/overview/schedules, UI Screens §4.20a, #584) — the workspace's schedules counted by condition and by how each one's latest run ended, what fires next and what changed last, all from one read of the same §20.1 list through the reader's lens — and its View all opens this page. It adds nothing to this spec: every read and every write it makes is a call to REST §20 with the person's session, so what the page can do is exactly what §20 and Auth §7.6 allow — every member reads, authors and admins write, a hidden button is never the authority. What it shows is this spec made visible: the next five occurrences as §3's function computes them, a blocked schedule's reason (§5.2) beside Unblock, and a run's history with its trail and — for a reader who may read executions — the execution's own messages merged in time order (the design revision's R10). Its preview of a pattern is §20.3, the same function the dispatcher uses. Elsewhere in the UI the same truth is pre-read before the fact: the pipelines explorer's discard dialog lists the schedules that run the pipeline (the Usage tab's by-target read, UI Screens §4.3d, #273) — a discard succeeds, and each listed schedule then blocks at its next run per §5.2 (pointer_null, target_not_found) until it is repointed or deleted.

9. Not in slice 1

• The Schedules page. Delivered by slice 2 — §8.4.
• Notifications. Settings delivered by slice 4 part a, delivery by part b — §2.3.
• Parameter bindings (values computed per run, e.g. "the day before the occurrence"). Delivered by slice 3 — §2.2.
• An application credential. Slice 1's routes are session-only; organizations that offer scheduling to their own customers call them from a backend with a management credential in slice 5.
• Background one-time runs of a pipeline — slice 6.

Appendix A: Change Log