E02 — Data quality and provenance
← E01 — Reproducible training · E03 — Evaluation → · Glossary
Before you start
Complete T03: distinguish a document, a token window and a split. Review loss if an apparently lower score still feels automatically better. Allow 60–90 minutes; this lab needs only CPU and generates fictional examples in memory.
You will explain where a training example came from, find contamination, repair training data without modifying the benchmark, and measure a false improvement. Provenance is the record of origin and processing. Governance means who may use a source, for what purpose, and under which retention/access rules.
Predict: can perfect accuracy be meaningless?
Suppose training contains eight pairs of arbitrary card IDs and labels. Evaluation contains four different IDs. A program that only memorizes training answers gets 0/4 right. Copy the four evaluation pairs into training: it now gets 4/4 right. The algorithm did not improve; its input included the answers. This is leakage.
python examples/run_data_quality.pyThe report is also saved to artifacts/data-quality/report.json. Expect:
| Field | Before | After repair |
|---|---|---|
| document count | 9 | 4 |
| exact duplicate pairs | 2 | 0 |
| near duplicate pairs | 2 | 0 |
| cross-split exact duplicate pairs | 1 | 0 |
| sensitive marker records | 1 | 0 |
| missing provenance records | 1 | 0 |
In leakage_experiment, clean accuracy is 0, contaminated accuracy 1, and repaired
accuracy 0. These are fractions, so 1 means 100%. The runner evaluates a literal
lookup memorizer on arbitrary labels; it is a deliberately simple counterexample,
not a language-model benchmark. Lower held-out LM loss needs the same protection.
Trace: origin → audit → decision → version
Each QualityDocument holds id, text, split, source and license.
The first two duplicated records have different IDs but identical text: checking
IDs alone cannot find them. Normalizing case/whitespace helps detect superficial
changes. Near matching compares sets of five-character pieces (shingles):
if two sets share 8 pieces and their union contains 10, Jaccard similarity is
8/10 = 0.8. Our teaching threshold is 0.8. Similarity can also flag legitimate
repeated wording, so inspect the evidence before applying this policy to real data.
from llm_course.data_quality import (
generate_quality_documents, audit_documents, repair_documents,
)
raw = generate_quality_documents()
before = audit_documents(raw)
clean, decisions = repair_documents(raw)
assert before["cross_split_duplicates"] == [["leaked", "test"]]
assert audit_documents(clean)["ready"]
for decision in decisions:
print(decision["id"], decision["action"], decision["reason"])Repair retains the held-out copy and removes its training duplicate. It never
silently edits validation/test. A defect inside held-out data raises an error:
an independent review and a new benchmark version are needed. Sensitive markers
such as [EMAIL_001] and [SECRET_001] become [REDACTED]; a [TOXIC] record is
quarantined. These rules illustrate mechanisms. They do not detect all personal
information, secret formats, toxicity or permission violations.
Build: choose and explain a sampling mixture
A sampling weight controls how often a source is drawn. For 20 examples at 75% original / 25% synthetic, draw 15 and 5. Sampling with replacement can repeat rows; repeated draws are not additional unique knowledge.
from llm_course.data_quality import QualityDocument, deterministic_mixture
original = [doc for doc in clean if doc.split == "train"]
synthetic = [QualityDocument("new", "new authored teaching example", "train")]
mix = deterministic_mixture(
{"original": original, "synthetic": synthetic},
{"original": 0.75, "synthetic": 0.25}, size=20, seed=7,
)
assert sum(doc.id == "new" for doc in mix) == 5Try 50/50 with the same size and seed. Before running, predict the new count. Report unique document count separately from sampled example count. Select weights on validation evidence, not on the protected final test.
Break, measure and repair yourself
Create a copy of a validation row with a new ID and split="train". Predict why
that is still leakage. Run the audit, remove the training copy, and show that
held-out text and membership did not change. Then change the last character:
can the near-duplicate rule still catch it? Record the score and any false positives.
Hint
Use dataclasses.replace(doc, id="copied", split="train"). The split must
happen before making windows. Compare content, not only IDs.
Reference reasoning
A new ID does not make the text new. Exact matching detects the copied row; shingles can detect a small edit. Remove the train copy and preserve the original benchmark. The demonstration returns from 100% to 0% accuracy, honestly exposing the memorizer’s lack of generalization.
Source and license inventory
Before a real acquisition, record owner, source URL/version, license/permission, intended purpose, language, access group, personal-data categories, retention and delete/rebuild procedure. Public visibility is not permission to train. Unknown rights stay out of the training snapshot until reviewed. Automated redaction helps minimize exposure; it does not grant authorization or prove anonymization.
The course ships generators and manifests for Unit 0, BPE and this lab. All three
have data cards under docs/data-cards/; each manifest records generator, license,
UTF-8 byte count, SHA-256 and split policy. A hash identifies exact bytes: one
changed space changes the identity. Never update a hash to make an unexplained
mismatch disappear. A new version needs a new baseline and documented migration.
The data policy distinguishes current tiny fixtures from planned
larger supplied data packs. Do not commit generated corpora, real personal data
or weights.
Completion gate
Run pytest tests/test_data_quality.py. Submit the before/after report, one
repaired contamination defect, an explained mixture and a data card for an invented
source. Explain why ready=true means only that these teaching checks passed,
and why preserving test integrity can make a reported score worse.
← E01 — Reproducible training · E03 — Evaluation → · Glossary