Course data policy
The complete course must be possible without learners providing their own data.
Planned supplied profiles
| Profile | Approximate size | Purpose |
|---|---|---|
| Starter | 5–20 MB | CPU exercises and smoke tests |
| Local | 100–500 MB | normal local training path |
| Extended | 1–5 GB | optional scaling exercises |
The sizes above belong to M1A and are not yet supplied. Current offline fixtures
are the 596-byte Unit-0 corpus, the multilingual BPE generator and the synthetic
data-quality inventory. T05/T06 and E01–E04 use these tiny course examples; no download or
learner-owned dataset is required for their training path. The optional productive
BPE comparison downloads its public tiktoken table on first use.
Required metadata
Every released data package must include a data card, schema, origin or generation method, license and source manifest, stable splits, hashes, version, generator seed, and quality reports.
Gold evaluation separation
The independent gold test set must not be included in model training, synthetic-data prompts, few-shot examples, or hyperparameter selection.
Repository boundary
Only manifests, schemas, generators, and tiny offline fixtures belong in Git. Downloaded datasets, generated corpora, model weights, and training artifacts stay outside version control.
Unit 0 follows this rule with a committed manifest and deterministic generator.
Its 596-byte CC0 corpus and derived prepared checkpoint are materialized under
the ignored artifacts/unit0/ directory and verified by SHA-256.
The data-quality lab audits provenance, duplicates, synthetic
sensitive/toxic markers and split contamination, then measures a misleading
score caused by leakage. Manifests live under datasets/; corresponding data
cards live under docs/data-cards/. Tests regenerate content and verify hashes.