Skip to content
LLM Engineering Course
English
Esc
↑↓navigate↵open⌘Jpreview
On this page

Course data policy

Course home

The complete course must be possible without learners providing their own data.

Planned supplied profiles

Profile Approximate size Purpose
Starter 5–20 MB CPU exercises and smoke tests
Local 100–500 MB normal local training path
Extended 1–5 GB optional scaling exercises

The sizes above belong to M1A and are not yet supplied. Current offline fixtures are the 596-byte Unit-0 corpus, the multilingual BPE generator and the synthetic data-quality inventory. T05/T06 and E01–E04 use these tiny course examples; no download or learner-owned dataset is required for their training path. The optional productive BPE comparison downloads its public tiktoken table on first use.

Required metadata

Every released data package must include a data card, schema, origin or generation method, license and source manifest, stable splits, hashes, version, generator seed, and quality reports.

Gold evaluation separation

The independent gold test set must not be included in model training, synthetic-data prompts, few-shot examples, or hyperparameter selection.

Repository boundary

Only manifests, schemas, generators, and tiny offline fixtures belong in Git. Downloaded datasets, generated corpora, model weights, and training artifacts stay outside version control.

Unit 0 follows this rule with a committed manifest and deterministic generator. Its 596-byte CC0 corpus and derived prepared checkpoint are materialized under the ignored artifacts/unit0/ directory and verified by SHA-256.

The data-quality lab audits provenance, duplicates, synthetic sensitive/toxic markers and split contamination, then measures a misleading score caused by leakage. Manifests live under datasets/; corresponding data cards live under docs/data-cards/. Tests regenerate content and verify hashes.

Was this page helpful?