---
title: "Glossary"
sidebar:
  label: "Glossary"
---

<span id="glossary" />


[Course home](/) · [Learning guide](/learning-guide)

Examples and derivations live in the linked lessons. Use this page when a term blocks reading.

| Term | Meaning |
| --- | --- |
| Terminal / Terminal | A window where you type commands and read their output: Terminal on macOS/Linux, PowerShell on Windows ([Setup](/setup)). |
| Virtual environment (venv) / Virtuelle Umgebung (venv) | The `.venv` folder with its own Python and the course libraries, kept apart from the rest of the computer; activate it in every new terminal ([Setup](/setup)). |
| JSON / JSON | A plain-text format for structured data: named values in `{…}`, lists in `[…]`. The course's reports are JSON files. |
| Backend / Backend | The processor that does the arithmetic: `cpu`, `mps` (an Apple GPU) or `cuda` (an NVIDIA GPU) ([Unit 0](/unit-00)). |
| Corpus / Korpus | The collection of text a model learns from or is measured on. |
| Argmax / Argmax | The position of the largest value in a list; for a row of scores, the most likely next token. |
| Checkpoint / Checkpoint | A saved file with a model's numbers (during training also the optimizer's state), so you can reload the model or continue the run. |
| Token / Token | One discrete text unit: a character, byte or learned piece; an ID is its integer label. |
| Vocabulary / Vokabular | All token IDs the model can predict; V is their count. |
| Scalar, vector, matrix, tensor / Skalar, Vektor, Matrix, Tensor | One number; a list of numbers; a table; a number array with any number of axes. |
| Shape, B/T/D/H/V / Shape, B/T/D/H/V | Axis lengths: batch examples, token positions, hidden width, attention heads and vocabulary. |
| Parameter / Parameter | A stored trainable number, such as a weight. Training changes it. |
| Activation / Aktivierung | An intermediate value calculated for a particular input; different from a stored weight. |
| Embedding / Embedding | A lookup table turning a token or position ID into a learned vector. |
| Logit, softmax / Logit, Softmax | An unrestricted score; the operation turning a row of scores into positive probabilities summing to one. |
| Loss, NLL, cross-entropy / Loss, NLL, Kreuzentropie | Prediction penalty. For one true next token, negative log likelihood is -ln(probability assigned to it). |
| Perplexity / Perplexität | exp(mean NLL). A uniform choice among four tokens has NLL ln(4) and perplexity 4; compare only with the same tokenizer/data. |
| Derivative / Ableitung | How fast one number changes when another is nudged: the change in loss divided by a tiny change in a parameter. Its sign says which way lowers the loss ([F04](/foundations-04-neuron)). |
| Chain rule / Kettenregel | When a nudge passes through several steps, multiply the rates of the steps ([F04](/foundations-04-neuron)). |
| Gradient, backpropagation / Gradient, Backpropagation | Sensitivities of loss to parameters; the chain-rule algorithm computing them backwards through the graph. |
| Optimizer, learning rate / Optimizer, Lernrate | The rule for updating parameters using gradients; the step-size control in that rule. |
| SGD / SGD | Stochastic gradient descent: subtract learning rate × gradient, with the gradient measured on a random batch instead of all data ([F06](/foundations-06-pytorch-checkpoint)). |
| Central difference / Zentrale Differenz | A derivative estimate without rules: `(f(x+h) - f(x-h)) / 2h` for a tiny `h`; it checks hand-derived gradients ([F05](/foundations-05-mlp-autograd)). |
| MLP / MLP | Multilayer perceptron: layers of neurons, with a bending function such as tanh between them ([F05](/foundations-05-mlp-autograd)). |
| tanh / tanh | An S-shaped function that squeezes any number into the range −1 to 1; `tanh(0)=0` ([F05](/foundations-05-mlp-autograd)). |
| One-hot / One-Hot | A row of zeros with a single 1 at the position of the true class ([F05](/foundations-05-mlp-autograd)). |
| Batch, microbatch, accumulation / Batch, Microbatch, Accumulation | Examples processed together; a smaller chunk; collecting several chunks’ gradients before one update. |
| Epoch, step / Epoche, Schritt | One pass through a dataset; one optimizer update. Several microbatches may form one step. |
| Causal mask / Kausale Maske | An allowed-position map preventing a prediction from using future input tokens. |
| Padding, loss mask / Padding, Lossmaske | Filler tokens making sequences equally long; a selection of which targets count toward the loss. |
| Residual, normalization, FFN / Residual, Normalisierung, FFN | Add a block’s input back to its output; rescale feature values; a feed-forward network applied separately at each position. |
| Pretraining / Pretraining | Learning next-token prediction from text. It does not itself teach reliable instruction following. |
| Train / validation / test | Data used for updates / choosing configurations / final evaluation after choices are frozen. |
| Overfitting / Überanpassung | Training gets better while unseen-data quality stagnates or worsens; tiny-batch overfit is intentionally used as a mechanics check. |
| Baseline, ablation / Baseline, Ablation | The unchanged reference experiment; a controlled comparison changing one component. |
| Seed, resume / Seed, Resume | A random-generator starting point; continuing a run from its checkpoint. |
| Leakage, memorization / Leakage, Memorierung | Held-out information enters training/selection; reproducing seen content instead of generalizing. |
| Paired seeds, uncertainty / Gepaarte Seeds, Unsicherheit | Run baseline and change with matching seeds; variation showing how fragile the observed difference may be. |
| FLOP, throughput, memory / FLOP, Durchsatz, Speicher | One floating-point operation; tokens processed per second; bytes needed for parameters, gradients, optimizer and activations. |
| Mixed precision / Gemischte Präzision | Some operations use smaller numeric formats to reduce cost, with a float32 fallback in this course. |
| Temperature / Temperatur | Divisor applied to logits before softmax; below 1 sharpens the distribution, above 1 flattens it, and 0 means greedy. Order never changes ([I01](/inference-decoding)). |
| Greedy decoding / Greedy Decoding | Always take the most probable next token; deterministic for a fixed prompt, what `do_sample=False` and `temperature=0` mean ([I01](/inference-decoding)). |
| Top-k / top-p (nucleus) sampling / Top-k-/Top-p-Sampling | Sample only among the k most probable tokens / the smallest set whose cumulative probability reaches p, renormalised ([I01](/inference-decoding)). |
| EOS token / EOS-Token | End-of-sequence marker; a decoder stops when it is produced, otherwise at `max_new_tokens` or the context limit ([I01](/inference-decoding)). |
| KV cache / KV-Cache | Keys and values of every previous position, stored per block so each new token is projected once and attends over the cache ([I01](/inference-decoding)). |
| Streaming / Streaming | Sending each generated token as soon as its step finishes instead of waiting for the whole answer ([I01](/inference-decoding)). |
| AdamW / AdamW | Optimizer keeping a smoothed gradient `m` and squared gradient `v` per parameter, with decoupled weight decay applied separately from the adaptive step ([E01](/pretraining)). |
| Warmup / Warmup | The first steps of a schedule use a smaller learning rate; cosine decay then lowers it smoothly to a minimum ([E01](/pretraining)). |
| Gradient clipping / Gradient Clipping | If the whole gradient vector's norm exceeds a threshold, rescale it to that norm once per update, before the optimizer step ([E01](/pretraining)). |
| Held-out data / Held-out-Daten | Data excluded from parameter updates; validation data is held out for choosing settings, a test set only for the final measurement ([E03](/evaluation)). |
| Contamination / Kontamination | Evaluation content that also appears in training or selection data, so a score no longer measures generalization ([E02](/data-quality), [C03](/company-evaluation)). |
| Data card / Data Card | A short record of a dataset: source, generator, license, splits, counts, hashes and exclusions ([E02](/data-quality)). |
| Model card / Model Card | A short record of a trained model: base, data, configuration, measured results, limitations and what it is not approved for ([T01](/bigram-baseline), [C03](/company-evaluation)). |
| Quantization / Quantisierung | Storing weights in fewer bits (for example 4-bit NF4) to cut memory; QLoRA trains higher-precision adapters over a frozen quantized base ([A02](/adaptation-comparison)). |
| RAG, SFT, LoRA / RAG, SFT, LoRA | Retrieval of external text at request time ([R01](/rag-build)); supervised fine-tuning on prompt/response pairs with the loss on response tokens only; adaptation through small trainable low-rank matrices beside a frozen base ([A01](/sft-lora)). |
| DPO / DPO | Direct preference optimization: train the policy so the chosen response's log-probability rises relative to the rejected one, measured against a frozen reference and scaled by beta ([A03](/dpo)). |
| Reward model / Reward-Modell | A model trained on chosen/rejected pairs to score responses; it is a proxy for the true task quality and can be exploited ([A04](/rlhf-bridge)). |
| KL divergence / KL-Divergenz | How far one probability distribution has moved from another; used as a penalty to keep a trained policy close to its reference ([A04](/rlhf-bridge)). |
| PPO / PPO | Proximal policy optimization: a policy-gradient update whose probability ratio is clipped so one step cannot move the policy too far ([A03](/dpo), [A04](/rlhf-bridge)). |
| RLHF / RLHF | Reinforcement learning from human feedback: fit a reward model on preferences, then optimize the policy against it under a KL penalty ([A04](/rlhf-bridge)). |

[F01](/foundations-01-python-numpy) · [F02](/foundations-02-shapes) · [F03](/foundations-03-probability) · [F04](/foundations-04-neuron) · [F05](/foundations-05-mlp-autograd) · [F06](/foundations-06-pytorch-checkpoint)
