Glossary
Examples and derivations live in the linked lessons. Use this page when a term blocks reading.
| Term | Meaning |
|---|---|
| Terminal / Terminal | A window where you type commands and read their output: Terminal on macOS/Linux, PowerShell on Windows (Setup). |
| Virtual environment (venv) / Virtuelle Umgebung (venv) | The .venv folder with its own Python and the course libraries, kept apart from the rest of the computer; activate it in every new terminal (Setup). |
| JSON / JSON | A plain-text format for structured data: named values in {…}, lists in […]. The course’s reports are JSON files. |
| Backend / Backend | The processor that does the arithmetic: cpu, mps (an Apple GPU) or cuda (an NVIDIA GPU) (Unit 0). |
| Corpus / Korpus | The collection of text a model learns from or is measured on. |
| Argmax / Argmax | The position of the largest value in a list; for a row of scores, the most likely next token. |
| Checkpoint / Checkpoint | A saved file with a model’s numbers (during training also the optimizer’s state), so you can reload the model or continue the run. |
| Token / Token | One discrete text unit: a character, byte or learned piece; an ID is its integer label. |
| Vocabulary / Vokabular | All token IDs the model can predict; V is their count. |
| Scalar, vector, matrix, tensor / Skalar, Vektor, Matrix, Tensor | One number; a list of numbers; a table; a number array with any number of axes. |
| Shape, B/T/D/H/V / Shape, B/T/D/H/V | Axis lengths: batch examples, token positions, hidden width, attention heads and vocabulary. |
| Parameter / Parameter | A stored trainable number, such as a weight. Training changes it. |
| Activation / Aktivierung | An intermediate value calculated for a particular input; different from a stored weight. |
| Embedding / Embedding | A lookup table turning a token or position ID into a learned vector. |
| Logit, softmax / Logit, Softmax | An unrestricted score; the operation turning a row of scores into positive probabilities summing to one. |
| Loss, NLL, cross-entropy / Loss, NLL, Kreuzentropie | Prediction penalty. For one true next token, negative log likelihood is -ln(probability assigned to it). |
| Perplexity / Perplexität | exp(mean NLL). A uniform choice among four tokens has NLL ln(4) and perplexity 4; compare only with the same tokenizer/data. |
| Derivative / Ableitung | How fast one number changes when another is nudged: the change in loss divided by a tiny change in a parameter. Its sign says which way lowers the loss (F04). |
| Chain rule / Kettenregel | When a nudge passes through several steps, multiply the rates of the steps (F04). |
| Gradient, backpropagation / Gradient, Backpropagation | Sensitivities of loss to parameters; the chain-rule algorithm computing them backwards through the graph. |
| Optimizer, learning rate / Optimizer, Lernrate | The rule for updating parameters using gradients; the step-size control in that rule. |
| SGD / SGD | Stochastic gradient descent: subtract learning rate × gradient, with the gradient measured on a random batch instead of all data (F06). |
| Central difference / Zentrale Differenz | A derivative estimate without rules: (f(x+h) - f(x-h)) / 2h for a tiny h; it checks hand-derived gradients (F05). |
| MLP / MLP | Multilayer perceptron: layers of neurons, with a bending function such as tanh between them (F05). |
| tanh / tanh | An S-shaped function that squeezes any number into the range −1 to 1; tanh(0)=0 (F05). |
| One-hot / One-Hot | A row of zeros with a single 1 at the position of the true class (F05). |
| Batch, microbatch, accumulation / Batch, Microbatch, Accumulation | Examples processed together; a smaller chunk; collecting several chunks’ gradients before one update. |
| Epoch, step / Epoche, Schritt | One pass through a dataset; one optimizer update. Several microbatches may form one step. |
| Causal mask / Kausale Maske | An allowed-position map preventing a prediction from using future input tokens. |
| Padding, loss mask / Padding, Lossmaske | Filler tokens making sequences equally long; a selection of which targets count toward the loss. |
| Residual, normalization, FFN / Residual, Normalisierung, FFN | Add a block’s input back to its output; rescale feature values; a feed-forward network applied separately at each position. |
| Pretraining / Pretraining | Learning next-token prediction from text. It does not itself teach reliable instruction following. |
| Train / validation / test | Data used for updates / choosing configurations / final evaluation after choices are frozen. |
| Overfitting / Überanpassung | Training gets better while unseen-data quality stagnates or worsens; tiny-batch overfit is intentionally used as a mechanics check. |
| Baseline, ablation / Baseline, Ablation | The unchanged reference experiment; a controlled comparison changing one component. |
| Seed, resume / Seed, Resume | A random-generator starting point; continuing a run from its checkpoint. |
| Leakage, memorization / Leakage, Memorierung | Held-out information enters training/selection; reproducing seen content instead of generalizing. |
| Paired seeds, uncertainty / Gepaarte Seeds, Unsicherheit | Run baseline and change with matching seeds; variation showing how fragile the observed difference may be. |
| FLOP, throughput, memory / FLOP, Durchsatz, Speicher | One floating-point operation; tokens processed per second; bytes needed for parameters, gradients, optimizer and activations. |
| Mixed precision / Gemischte Präzision | Some operations use smaller numeric formats to reduce cost, with a float32 fallback in this course. |
| Temperature / Temperatur | Divisor applied to logits before softmax; below 1 sharpens the distribution, above 1 flattens it, and 0 means greedy. Order never changes (I01). |
| Greedy decoding / Greedy Decoding | Always take the most probable next token; deterministic for a fixed prompt, what do_sample=False and temperature=0 mean (I01). |
| Top-k / top-p (nucleus) sampling / Top-k-/Top-p-Sampling | Sample only among the k most probable tokens / the smallest set whose cumulative probability reaches p, renormalised (I01). |
| EOS token / EOS-Token | End-of-sequence marker; a decoder stops when it is produced, otherwise at max_new_tokens or the context limit (I01). |
| KV cache / KV-Cache | Keys and values of every previous position, stored per block so each new token is projected once and attends over the cache (I01). |
| Streaming / Streaming | Sending each generated token as soon as its step finishes instead of waiting for the whole answer (I01). |
| AdamW / AdamW | Optimizer keeping a smoothed gradient m and squared gradient v per parameter, with decoupled weight decay applied separately from the adaptive step (E01). |
| Warmup / Warmup | The first steps of a schedule use a smaller learning rate; cosine decay then lowers it smoothly to a minimum (E01). |
| Gradient clipping / Gradient Clipping | If the whole gradient vector’s norm exceeds a threshold, rescale it to that norm once per update, before the optimizer step (E01). |
| Held-out data / Held-out-Daten | Data excluded from parameter updates; validation data is held out for choosing settings, a test set only for the final measurement (E03). |
| Contamination / Kontamination | Evaluation content that also appears in training or selection data, so a score no longer measures generalization (E02, C03). |
| Data card / Data Card | A short record of a dataset: source, generator, license, splits, counts, hashes and exclusions (E02). |
| Model card / Model Card | A short record of a trained model: base, data, configuration, measured results, limitations and what it is not approved for (T01, C03). |
| Quantization / Quantisierung | Storing weights in fewer bits (for example 4-bit NF4) to cut memory; QLoRA trains higher-precision adapters over a frozen quantized base (A02). |
| RAG, SFT, LoRA / RAG, SFT, LoRA | Retrieval of external text at request time (R01); supervised fine-tuning on prompt/response pairs with the loss on response tokens only; adaptation through small trainable low-rank matrices beside a frozen base (A01). |
| DPO / DPO | Direct preference optimization: train the policy so the chosen response’s log-probability rises relative to the rejected one, measured against a frozen reference and scaled by beta (A03). |
| Reward model / Reward-Modell | A model trained on chosen/rejected pairs to score responses; it is a proxy for the true task quality and can be exploited (A04). |
| KL divergence / KL-Divergenz | How far one probability distribution has moved from another; used as a penalty to keep a trained policy close to its reference (A04). |
| PPO / PPO | Proximal policy optimization: a policy-gradient update whose probability ratio is clipped so one step cannot move the policy too far (A03, A04). |
| RLHF / RLHF | Reinforcement learning from human feedback: fit a reward model on preferences, then optimize the policy against it under a KL penalty (A04). |