What ./app.sh --start --demo loads
The demo data, table by table
Three public datasets power every demo on this site: New York City taxi and weather, US trade and exchange rates, and the city's high-volume rideshare trips. This page is the data's own documentation — every table, its row count, where it came from, the questions it is shaped to answer, and the licence it ships under. The tables render from the same published manifests the loader verifies, so what you read here is what your deployment restores.
What ships, and why it is real
Published artifacts, not fixtures
Nothing here is generated test data. Each family is real public data, rebuilt
deterministically from pinned sources and published as a versioned, read-only artifact set on the
datapipelines-co S3 bucket. The application never downloads data on its own: loading a
family is a deployment step, one command —
./app.sh --start --demo nyc,trade,lake
— after which the same version loads the same tables everywhere, byte-identical in content. An example
run on your laptop and a number on this page come from the same rows, because both come from the same
verified artifacts: the loader checks every file's checksum against the family's manifest.json
before it restores anything.
One honest caveat about sampling: the taxi trips are a 1-in-16 hash sample of the TLC's
24-month window (trips_sample_modulus 16 — deterministic, so every deployment samples the
identical trips). That is why the showcase comparisons are built as within-mode ratios, like the
rain-lift question that compares each mode's rainy days against its own dry days, where the sampling
rate cancels — not as raw city-wide counts. The rideshare family carries the same sampling over the same
window, and its full unsampled table is published too.
Family one — nyc, version v8
NYC mobility: one question, three engines
The mobility family is the demo's founding question — how do taxi and rideshare respond to weather? — answered across three engines on purpose: yellow-taxi trips live on Postgres, NOAA daily weather on MySQL, and the TLC zones and rate reference tables on SQLite. A single pipeline joins all three, staging each source's rows in memory for the run. The window is 2023-01 through 2024-12.
The weather is NOAA's GHCN-Daily for five pinned NYC-area stations (among them Central Park,
USW00094728), melted into one row per station, day and element with units already
converted; the reference tables carry the TLC's own zone lookup, rate codes, payment types and a
holiday calendar. Every source URL is pinned in scripts/sample-data/sources.lock.
| Engine | Table | Grain | Rows | Key columns |
|---|---|---|---|---|
| POSTGRES | trips |
one row per sampled yellow-taxi trip | 4,897,311 | trip_id, pickup_ts, pickup_date, pu_location_id, total_amount |
| POSTGRES | trips_daily |
one row per day of the window | 731 | pickup_date, trip_count, total_revenue |
| POSTGRES | trips_monthly |
one row per month × pickup zone | 5,700 | month_start, pu_location_id, trip_count, total_revenue |
| MYSQL | stations |
one row per NOAA GHCN station (the five pinned NYC-area ids) | 5 | station_id, name, latitude, longitude |
| MYSQL | observations |
one row per station × day × element, in standard units | 17,530 | station_id, obs_date, element, value, unit |
| SQLITE | zones |
one row per TLC taxi zone | 265 | location_id, borough, zone, service_zone |
| SQLITE | rate_codes |
one row per rate code | 7 | rate_code_id, description |
| SQLITE | payment_types |
one row per payment type | 7 | payment_type_id, description |
| SQLITE | calendar |
one row per day of the window, with weekend and holiday flags | 731 | cal_date, day_of_week, is_weekend, is_holiday |
Licence verified 2026-09-02 Attribution: the trip records and zone lookup are published by the NYC Taxi & Limousine Commission as part of NYC Open Data, whose FAQ states: “Open Data belongs to all New Yorkers. There are no restrictions on the use of Open Data.” The TLC's own accuracy disclaimer travels with the data: “The trip data was not created by the TLC, and TLC makes no representations as to the accuracy of these data.” The NOAA observations are US Government work, public domain.
When an agent learns something about one of these tables — a unit, a time zone, what a code means — it records the fact with the query that proved it, and every session after it reads the answer beside the columns. That is the learned semantic layer →
Family two — trade, version v4
US trade: Census facts, mirrored and priced
The trade family answers the second kind of question a demo should survive: do the mirrors agree? The US Census Bureau's monthly imports and exports at HS-6 commodity grain live on DuckDB (2.4 million facts across the top 15 partners); the UN Comtrade mirror — what the partner countries report trading with the USA — sits on MySQL; and the Federal Reserve's H.10 daily and G.5 monthly exchange rates for the five partner currencies are on SQLite, so a pipeline can price the same trade flow in the partner's own money.
The window is 2023-01 through 2024-12 (the Comtrade mirror keeps 2022–2024 annual totals). The fifteen Census partners are Canada, Mexico, China, Japan, Germany, South Korea, Taiwan, Vietnam, India, the United Kingdom, Ireland, the Netherlands, France, Italy and Brazil — named by the Census Bureau's four-digit partner codes in the data and crosswalked to their ISO codes for the mirror join; the Comtrade reporters are China, Canada, Mexico, Japan and Germany. The partner list was derived mechanically from the Census enduse endpoints, not picked by hand.
| Engine | Table | Grain | Rows | Key columns |
|---|---|---|---|---|
| duckdb | trade_monthly |
one row per flow × month × partner × HS-6 commodity | 2,373,825 | flow, period, partner_code, hs_code, value_usd |
| duckdb | partners |
one row per Census partner country (the top 15 by 2024 goods trade) | 15 | partner_code, partner_name |
| duckdb | hs_chapters |
one row per HS-2 chapter | 98 | chapter, chapter_name |
| duckdb | trade_flow_monthly |
one row per flow × month, all partners and chapters totalled | 48 | flow, period, total_value_usd, hs6_cell_count |
| duckdb | partner_iso_crosswalk |
one row per Census code the Comtrade reconciliation needs | 5 | census_code, iso_numeric, partner_name |
| mysql | comtrade_annual |
one row per reporter × flow × year at the headline TOTAL level | 27 | reporter_code, flow, period, value_usd |
| mysql | reporters |
one row per Comtrade reporter country | 5 | reporter_code, reporter_name |
| sqlite | fx_daily |
one row per business day × currency, in both quote directions | 2,500 | rate_date, currency, per_usd, usd_per |
| sqlite | fx_monthly |
one row per month × currency, the G.5 monthly average | 120 | month, currency, per_usd, usd_per |
| sqlite | currencies |
one row per currency, with the Board's own series ids | 5 | currency, name, series_daily, series_monthly |
| sqlite | partner_currency |
one row per reconciled Census partner, naming its currency | 5 | census_code, partner, currency |
Licence verified 2026-09-04 Attribution: “This product uses the Census Bureau Data API but is not endorsed or certified by the Census Bureau.” — verbatim and mandatory under the Census API terms of service, and it travels with this data everywhere. The Comtrade slice is deliberately kept below the 100,000-record line that marks Comtrade's fee-free re-dissemination limit. The Federal Reserve's H.10 and G.5 rates are a work of the US Government: no copyright, the credit is courtesy.
Family three — lake, version v1
dp-lake: the rideshare census, read in place
The lake family is the demo's biggest dataset and its different shape: NYC TLC high-volume
for-hire trips — Uber, Lyft, Via and Juno — published as Parquet and Apache Iceberg on S3 and
read in place. Nothing downloads at start; there is no loader. The app's own engine
opens the objects over HTTPS and reads only what a query names, which is why the family's big table is
partitioned by pickup_date: filter on the partition column and the engine prunes to just
the days you asked about — a one-week question reads about seven objects, an unfiltered scan reads
seven gigabytes. The 1-in-16 sample over the same 2023-01 → 2024-12 window as the taxi data is what
lets one question compare taxi and rideshare fairly.
The zone ids are the same TLC zone ids the mobility family's zones table holds, so one
borough lookup serves both trip sources. The licence number to company mapping — the four licensees
Uber, Lyft, Via and Juno named by the TLC's HVFHS data dictionary — ships as its own tiny table.
| Format | Table | Grain | Rows | Objects | Bytes (exact) |
|---|---|---|---|---|---|
| parquet | hvfhv_trips |
one row per high-volume for-hire trip, one directory per day | 471,851,707 | 731 | 6,523,027,848 |
| parquet | hvfhv_trips_sample |
the same trips, hash-sampled 1 in 16, one file per month | 29,492,699 | 24 | 483,114,021 |
| parquet | hvfhv_zone_day |
one row per pickup zone × day × company | 377,148 | 1 | 7,627,553 |
| parquet | hvfhs_companies |
one row per HVFHS licensee code | 4 | 1 | 550 |
| iceberg | hvfhv_trips_iceberg |
the 1-in-16 sample again, as an Iceberg table read by its metadata file | 29,492,699 | 98 | 555,620,647 |
The objects, exactly
The bucket denies listing — you name an object, you get it; you cannot browse. So this page is the
listing, rendered from the manifest's recorded table locations under
s3://datapipelines-co/sample-data/lake/v1/:
-
s3://datapipelines-co/sample-data/lake/v1/hvfhv_trips/pickup_date=*/part-*.parquet -
s3://datapipelines-co/sample-data/lake/v1/hvfhv_trips_sample/part-*.parquet -
s3://datapipelines-co/sample-data/lake/v1/hvfhv_zone_day/part-0.parquet -
s3://datapipelines-co/sample-data/lake/v1/hvfhs_companies/part-0.parquet -
s3://datapipelines-co/sample-data/lake/v1/hvfhv_trips_iceberg s3://datapipelines-co/sample-data/lake/v1/manifest.json
The Iceberg table is opened by its current metadata file —
s3://datapipelines-co/sample-data/lake/v1/hvfhv_trips_iceberg/metadata/00024-040f66d1-2124-4294-8bc5-a136be52b28d.metadata.json —
which the registry row names, because a published prefix has no catalog service.
Licence verified 2026-09-07 Attribution: published by the NYC Taxi & Limousine Commission under the same NYC Open Data programme as the yellow-taxi months — “There are no restrictions on the use of Open Data.” One difference is quoted where it applies: HVFHV carries its own accuracy disclaimer, not the yellow-taxi one — “These records are generated from the FHV Trip Record submissions made by bases, so we cannot guarantee or confirm their accuracy or completeness.”
What you can ask it
Pipelines the demo ships, and the question each answers
Every family seeds worked pipelines into your workspace at first login. The mobility six:
nyc/mobility/revenue_by_borough (which borough earns the most, and who tips best),
nyc/mobility/rainy_vs_dry_ridership (do New Yorkers take more taxis when it rains),
nyc/mobility/borough_od_matrix (where do each borough's trips end up),
nyc/mobility/airport_access_by_borough (who rides to JFK, LaGuardia and Newark, and at
what cost), nyc/mobility/weather_sensitivity_by_borough (which boroughs lose riders when
it rains), and nyc/mobility/mobility_briefing (one row per borough: trips, self-trips,
value).
The showcase: nyc/mobility/taxi_vs_rideshare joins the Postgres taxi trips, the lake's
zone-day rideshare aggregate, the SQLite zones and the MySQL rain series in one four-engine run, and
ranks boroughs by how much ridership shifts in the rain — the reason the sampling note above matters.
The trade three: trade/balance_by_partner (imports minus exports, per partner and month),
trade/reconciliation (US-reported Census facts against the partners' own Comtrade
returns), and trade/fx/imports_in_partner_currency (the same import bill priced in the
partner's currency from the Federal Reserve rates).
Any other question the data can answer, you can ask too — type it, or write the SQL yourself. The mechanics of a cross-engine question are on federated query, and the lake reads on dp-lake.
Licence and attribution
Every source, and the terms it ships under
Each manifest carries its sources as provenance rows — the URL, the transform, the licence string and the date the licence was verified — and the lists below render from those rows, so the page cannot claim a licence the artifact does not. Where a family has no verification stamp, its block says so and makes no licence claim at all.
mobility (v8)
-
nyc_tlc_yellow_trips —
https://d37ci6vzurychx.cloudfront.net/trip-data/yellow_tripdata_{YYYY-MM}.parquet; NYC Open Data / freely usable (nyc.gov terms of use) -
noaa_ghcn_daily —
https://www.ncei.noaa.gov/data/global-historical-climatology-network-daily/access/{STATION}.csv; US Government work, public domain (NOAA/NCEI) -
noaa_ghcn_stations —
https://www.ncei.noaa.gov/pub/data/ghcn/daily/ghcnd-stations.txt; US Government work, public domain (NOAA/NCEI) -
tlc_taxi_zones —
https://d37ci6vzurychx.cloudfront.net/misc/taxi_zone_lookup.csv; NYC Open Data / freely usable (nyc.gov terms of use) -
tlc_reference_codes —
https://www.nyc.gov/site/tlc/about/tlc-trip-record-data.page (Data Dictionary - Yellow Taxi Trip Records, PDF); NYC Open Data / freely usable (nyc.gov terms of use) -
us_federal_holidays —
https://www.opm.gov/policy-data-oversight/pay-leave/federal-holidays/; Facts (statutory schedule); no licence claimed
Combined size of the published files: 118 MB across 4 files.
trade (v4)
-
US Census Bureau International Trade API (imports/hs, exports/hs) —
https://api.census.gov/data/timeseries/intltrade/; US Government work; Census API terms of service This product uses the Census Bureau Data API but is not endorsed or certified by the Census Bureau. -
UN Comtrade preview API (annual TOTAL flows, reporters x USA) —
https://comtradeapi.un.org/public/v1/preview/; UN Comtrade terms of use (preview tier); the slice is deliberately under the 100,000-record fee-free re-dissemination line Source: UN Comtrade (https://comtradeplus.un.org) -
Board of Governors of the Federal Reserve System, H.10 Foreign Exchange Rates (daily) and its G.5 monthly averages, via the Data Download Program —
https://www.federalreserve.gov/releases/h10/; Work of the US Government; no copyright (17 U.S.C. 105). The H.10 release page, its About page, the DDP chooser and the Board's site-wide Website Policies carry no terms-of-use or copyright statement over Board content (checked 2026-09-04). Source: Board of Governors of the Federal Reserve System, H.10
Combined size of the published files: 234 MB across 4 files.
lake (v1)
-
nyc_tlc_hvfhv_trips —
https://d37ci6vzurychx.cloudfront.net/trip-data/fhvhv_tripdata_{YYYY-MM}.parquet; NYC Open Data / no restrictions on use (NYC Open Data FAQ) the publisher's page the NYC Open Data collection -
tlc_hvfhs_licensees —
https://www.nyc.gov/assets/tlc/downloads/pdf/data_dictionary_trip_records_hvfhs.pdf; NYC Open Data / no restrictions on use (NYC Open Data FAQ) the publisher's page
Reproduce it
You do not have to trust the bucket
The artifacts are rebuildable from their pinned sources: each family's
scripts/sample-data*/ directory carries a runbook, a sources.lock pinning
every source URL and its SHA-256 (the two living APIs are pinned at content level instead, so an
upstream revision is the event that fails the build), and the hash-sampled outputs re-derive to the
same row checksums on any machine. On the consuming side,
check-published.sh verifies a published version against its own manifest by hash — the
same check a deployment runs before restoring. The runbooks:
mobility,
trade,
lake.
Questions about the data
About the demo data
Is the demo data real?
Yes — all three families are real public data, not fixtures: NYC Taxi & Limousine Commission trip records and NOAA weather for New York, US Census trade and Federal Reserve exchange rates, and the TLC high-volume for-hire feed on S3. Every family is rebuilt deterministically from pinned sources, so the same version loads the same numbers anywhere. What ships is documented table by table on the demo-data page. (The build and its pins: docs/deployment.md Appendix B.)
Can I use the demo data in my own product?
That is what the licences are for. The NYC-derived families ship under NYC Open Data's no-restrictions statement; NOAA and the Federal Reserve are US Government works with no copyright; the Census slice carries the Bureau's API terms, including its not-endorsed notice; the Comtrade slice stays under the fee-free re-dissemination line. Each family's section on the demo-data page quotes the operative sentence and links the source. (The demo and its gates: docs/deployment.md Appendix B.)
How big is the demo on disk?
The nyc family's files are about 118 MB and the trade family's about 223 MB, restored into the demo containers at start; the lake family is about 7.6 GB of Parquet and Iceberg objects that are never downloaded — the engine reads only the partitions a query names. The exact per-file and per-table figures render from the published manifests on the demo-data page, so they are always the published numbers. (The artifacts the loader fetches: docs/deployment.md Appendix B.)
Can I add my own data next to it?
Yes. The demo families are ordinary datasources — register yours in the app (or in the bootstrap file the demo profiles use) and any pipeline can join them to the demo tables, exactly as the shipped examples join Postgres trips to SQLite zones and MySQL weather. Nothing about the demo is privileged; it is seeded data with example pipelines. (Registration: docs/datasources.md §3.)
Does the demo need internet access?
At start, yes: the nyc and trade artifacts download from the published bucket and the lake datasource points at S3. After that they differ — nyc and trade are local files restored into the demo engines, so running them needs no network, while the lake family is read in place over HTTPS at query time and always needs it. (The consuming side: docs/deployment.md Appendix B.)