Data package format

Generated page

Copied from sample-data/readme.md on every docs build. Edit that file, not this page.

YAC expects data to be in the form of multiple related tables. The information about the tables and their relationships must be recorded in a frictionless data package file.

Frictionless Data Package: https://datapackage.org/

Frictionless data packages support multiple different types of data resources. However, the only kind that YAC currently supports are Tables

Frictionless Table Schema: https://datapackage.org/standard/table-schema/

YAC expects a few additional fields. They are all prepended with udi: for the universal discovery interface project.

At the top level:

Field	Description
udi:name	The name of the full data resource. Can be the same as name.
udi:path	The path of the data resources. It is assumed that all data resources are relative to this path. This can be a relative path, or a full remote URL.
udi:labels	Optional. Friendly labels for categorical values, raw → label ({"Children's Hospital of Philadelphia": "CHOP"}). Applied across every resource, so an institution renamed once covers donors.group_name and samples.group_name alike. Display only — the raw value is what reaches a spec, a query, or the LLM.
udi:example_prompts	Optional. List of example prompts the chat suggests for this package. When absent, the chat asks the agent (/v1/yac/examples). scripts/gen_datapackage.py fills it from an example_prompts.json kept beside the CSVs.

For each table resource:

Field	Description
udi:row_count	The number of rows in the table.
udi:column_count	The number of data columns in the table.
udi:hide_count	Optional. true keeps the table’s row count out of the chat: no count chip in the dashboard header, no count in the data overview. The table still loads, charts and downloads as usual. For a server-side backend, set it in the table’s schemas entry.

For each data field:

Field	Description
title	Optional. Frictionless’s standard field label, used wherever the column is shown to a person: generated chart titles, the tweak pickers, filter chips. Set it when the column name humanizes badly — body_mass_index_value becomes “Body Mass Index” on its own, but the label wanted is “BMI”. Display only; specs and queries always key on name.
udi:cardinality	The number of unique values in the column.
udi:unique	true if the value is unique for each row, false otherwise.
udi:data_type	One of quantitative, ordinal, or nominal.
udi:time_format	Optional. How a time column stores its values, so charts, tables and filters show dates, years and ages instead of raw numbers: epoch_ms / epoch_s (milliseconds / seconds since 1970-01-01 UTC, e.g. HuBMAP’s created_timestamp; shown as dates), year (a calendar year, shown as 1985 rather than 1,985), or days (a duration such as an age, e.g. PCX’s age_at_event_days; shown in years, 12.0 y). Declare such a column udi:data_type: quantitative. Display only — specs, queries and the LLM still see the stored numbers. gen_datapackage.py sets it from an optional time_format column in data_dictionary.csv. Text dates in other layouts (2019-12-02 13:14) are deliberately not parsed: filtering or aggregating them needs numbers in both query engines, so convert such a column to epoch ms when preparing the data.
udi:overlapping_fields	List of fields that are non-null on at least one row together. If all fields overlap, this should be all.

Entity relationships (foreignKeys) — modeling guidance

Cross-entity filtering (brushing one chart filters the others) and LLM join generation resolve relationships from each resource’s standard frictionless schema.foreignKeys. Two relationship shapes are supported:

1. Direct FK — a table references another (samples.donor_id → donors.id).
2. Shared-parent siblings (star schema) — two tables each FK the same parent (Event → Patient ← Surgery). Their FK columns share the parent’s key domain, so filters and joins bridge them automatically — no FK between the siblings themselves is needed.

Current limitation — no multi-hop paths. Relationships are NOT resolved through chains that require traversing an intermediate table’s rows, e.g. two tables linked only through a shared child (donors ← files → studies), or chains with different keys per hop (A.x → B.x, B.y → C.y). When authoring a data package, model it as a star schema where possible: give every child table an FK directly to the parent(s) it should filter against, even if that denormalizes a key column. If a table should cross-filter with another and neither shape above applies, add the linking key column to one of them.

scripts/gen_datapackage.py infers FKs from shared key columns when generating a datapackage.json for a CSV directory; for remote (StarRocks) packages the seed script carries these FKs into the server config — the database itself has no FK metadata, so a package without foreignKeys gets no cross-entity filtering at all. When a key is unique in several tables (pcx’s Patient, Demographics and Treatment Summary are all one row per patient), pass --parent <Table> to choose the hub; otherwise name order picks it, and the agent only joins tables linked directly.

It also fills in table and column descriptions from a data_dictionary.csv in the same directory — columns table,field,description, where table is the CSV filename and a blank field describes the table itself.

Single source of truth

This sample-data/ directory (repo root) is the one canonical copy of the dev/test data. It is synced into each frontend’s static dir on dev/build by scripts/copy-sample-data.mjs (chat → packages/chat/public/data, the site → apps/site/public/app/data); the toolkit’s Storybook mounts it via staticDirs. Those public/data copies are gitignored — edit files here, not there.

Contents

Path	Description
./hubmap/	Full HuBMAP data package (datapackage.json + donors/samples/datasets .tsv), fetched fresh from https://portal.hubmapconsortium.org/metadata/v0/udi/. The only copy of HuBMAP in the repo: chat’s default package, the toolkit stories’ source, and the donors/samples behind the parity goldens.
./penguins.csv	Classic Palmer Penguins test dataset (loose CSV used by toolkit stories, parity goldens, and agent tests).
./penguins/	The same Penguins data as a self-contained package (penguins.csv + datapackage.json) — the committed default for the server-side query quickstart (seed_duckdb.py / seed_starrocks.py) and browser mode via set-chat-data-source.mjs penguins.
./pcx/	De-identified clinical cancer cohort: 970 patients across 7 linked tables (demographics, disease events, medical therapy, radiation, surgery, survival, treatment summary). datapackage.json is generated from the CSVs + data_dictionary.csv + example_prompts.json by scripts/gen_datapackage.py; sessions/ holds predefined session exports. The public Pages site’s default package.
./example_*.csv, ./match_test_*.csv	Small fixtures for tutorial/example specs.

To refresh HuBMAP: re-run the four curls against the portal /udi/ endpoint into ./hubmap/, then set the manifest’s udi:path back to "./data/hubmap/" (consumers resolve it page-relative, i.e. against the served /data mount, not against the manifest’s own location). The portal does not serve the display labels or date formats, so a refresh also drops the hand-authored field titles, the top-level udi:labels block, and the *_timestamp fields’ "type": "integer", "udi:data_type": "quantitative" and "udi:time_format": "epoch_ms" (the portal declares them string / nominal) — re-apply them from git history.

Afterwards, regenerate the parity goldens — they read hubmap/donors.tsv and hubmap/samples.tsv, so a refresh changes their expected rows:

pnpm build:toolkit && node packages/grammar/scripts/gen-parity-goldens.mjs
cd packages/agent && uv run pytest tests/test_query_parity.py

The portal’s schema does drift: assay_type was dropped in favour of dataset_type / soft_assaytype, so check that anything naming a HuBMAP field (stories, specs, packages/agent/data/data_domains/hubmap_data_schema.json) still resolves.