Skip to content
LLM Engineering Course
English
Esc
↑↓navigate↵open⌘Jpreview

Glossary

Course home · Learning guide

Examples and derivations live in the linked lessons. Use this page when a term blocks reading.

Term Meaning
Terminal / Terminal A window where you type commands and read their output: Terminal on macOS/Linux, PowerShell on Windows (Setup).
Virtual environment (venv) / Virtuelle Umgebung (venv) The .venv folder with its own Python and the course libraries, kept apart from the rest of the computer; activate it in every new terminal (Setup).
JSON / JSON A plain-text format for structured data: named values in {…}, lists in […]. The course’s reports are JSON files.
Backend / Backend The processor that does the arithmetic: cpu, mps (an Apple GPU) or cuda (an NVIDIA GPU) (Unit 0).
Corpus / Korpus The collection of text a model learns from or is measured on.
Argmax / Argmax The position of the largest value in a list; for a row of scores, the most likely next token.
Checkpoint / Checkpoint A saved file with a model’s numbers (during training also the optimizer’s state), so you can reload the model or continue the run.
Token / Token One discrete text unit: a character, byte or learned piece; an ID is its integer label.
Vocabulary / Vokabular All token IDs the model can predict; V is their count.
Scalar, vector, matrix, tensor / Skalar, Vektor, Matrix, Tensor One number; a list of numbers; a table; a number array with any number of axes.
Shape, B/T/D/H/V / Shape, B/T/D/H/V Axis lengths: batch examples, token positions, hidden width, attention heads and vocabulary.
Parameter / Parameter A stored trainable number, such as a weight. Training changes it.
Activation / Aktivierung An intermediate value calculated for a particular input; different from a stored weight.
Embedding / Embedding A lookup table turning a token or position ID into a learned vector.
Logit, softmax / Logit, Softmax An unrestricted score; the operation turning a row of scores into positive probabilities summing to one.
Loss, NLL, cross-entropy / Loss, NLL, Kreuzentropie Prediction penalty. For one true next token, negative log likelihood is -ln(probability assigned to it).
Perplexity / Perplexität exp(mean NLL). A uniform choice among four tokens has NLL ln(4) and perplexity 4; compare only with the same tokenizer/data.
Derivative / Ableitung How fast one number changes when another is nudged: the change in loss divided by a tiny change in a parameter. Its sign says which way lowers the loss (F04).
Chain rule / Kettenregel When a nudge passes through several steps, multiply the rates of the steps (F04).
Gradient, backpropagation / Gradient, Backpropagation Sensitivities of loss to parameters; the chain-rule algorithm computing them backwards through the graph.
Optimizer, learning rate / Optimizer, Lernrate The rule for updating parameters using gradients; the step-size control in that rule.
SGD / SGD Stochastic gradient descent: subtract learning rate × gradient, with the gradient measured on a random batch instead of all data (F06).
Central difference / Zentrale Differenz A derivative estimate without rules: (f(x+h) - f(x-h)) / 2h for a tiny h; it checks hand-derived gradients (F05).
MLP / MLP Multilayer perceptron: layers of neurons, with a bending function such as tanh between them (F05).
tanh / tanh An S-shaped function that squeezes any number into the range −1 to 1; tanh(0)=0 (F05).
One-hot / One-Hot A row of zeros with a single 1 at the position of the true class (F05).
Batch, microbatch, accumulation / Batch, Microbatch, Accumulation Examples processed together; a smaller chunk; collecting several chunks’ gradients before one update.
Epoch, step / Epoche, Schritt One pass through a dataset; one optimizer update. Several microbatches may form one step.
Causal mask / Kausale Maske An allowed-position map preventing a prediction from using future input tokens.
Padding, loss mask / Padding, Lossmaske Filler tokens making sequences equally long; a selection of which targets count toward the loss.
Residual, normalization, FFN / Residual, Normalisierung, FFN Add a block’s input back to its output; rescale feature values; a feed-forward network applied separately at each position.
Pretraining / Pretraining Learning next-token prediction from text. It does not itself teach reliable instruction following.
Train / validation / test Data used for updates / choosing configurations / final evaluation after choices are frozen.
Overfitting / Überanpassung Training gets better while unseen-data quality stagnates or worsens; tiny-batch overfit is intentionally used as a mechanics check.
Baseline, ablation / Baseline, Ablation The unchanged reference experiment; a controlled comparison changing one component.
Seed, resume / Seed, Resume A random-generator starting point; continuing a run from its checkpoint.
Leakage, memorization / Leakage, Memorierung Held-out information enters training/selection; reproducing seen content instead of generalizing.
Paired seeds, uncertainty / Gepaarte Seeds, Unsicherheit Run baseline and change with matching seeds; variation showing how fragile the observed difference may be.
FLOP, throughput, memory / FLOP, Durchsatz, Speicher One floating-point operation; tokens processed per second; bytes needed for parameters, gradients, optimizer and activations.
Mixed precision / Gemischte Präzision Some operations use smaller numeric formats to reduce cost, with a float32 fallback in this course.
Temperature / Temperatur Divisor applied to logits before softmax; below 1 sharpens the distribution, above 1 flattens it, and 0 means greedy. Order never changes (I01).
Greedy decoding / Greedy Decoding Always take the most probable next token; deterministic for a fixed prompt, what do_sample=False and temperature=0 mean (I01).
Top-k / top-p (nucleus) sampling / Top-k-/Top-p-Sampling Sample only among the k most probable tokens / the smallest set whose cumulative probability reaches p, renormalised (I01).
EOS token / EOS-Token End-of-sequence marker; a decoder stops when it is produced, otherwise at max_new_tokens or the context limit (I01).
KV cache / KV-Cache Keys and values of every previous position, stored per block so each new token is projected once and attends over the cache (I01).
Streaming / Streaming Sending each generated token as soon as its step finishes instead of waiting for the whole answer (I01).
AdamW / AdamW Optimizer keeping a smoothed gradient m and squared gradient v per parameter, with decoupled weight decay applied separately from the adaptive step (E01).
Warmup / Warmup The first steps of a schedule use a smaller learning rate; cosine decay then lowers it smoothly to a minimum (E01).
Gradient clipping / Gradient Clipping If the whole gradient vector’s norm exceeds a threshold, rescale it to that norm once per update, before the optimizer step (E01).
Held-out data / Held-out-Daten Data excluded from parameter updates; validation data is held out for choosing settings, a test set only for the final measurement (E03).
Contamination / Kontamination Evaluation content that also appears in training or selection data, so a score no longer measures generalization (E02, C03).
Data card / Data Card A short record of a dataset: source, generator, license, splits, counts, hashes and exclusions (E02).
Model card / Model Card A short record of a trained model: base, data, configuration, measured results, limitations and what it is not approved for (T01, C03).
Quantization / Quantisierung Storing weights in fewer bits (for example 4-bit NF4) to cut memory; QLoRA trains higher-precision adapters over a frozen quantized base (A02).
RAG, SFT, LoRA / RAG, SFT, LoRA Retrieval of external text at request time (R01); supervised fine-tuning on prompt/response pairs with the loss on response tokens only; adaptation through small trainable low-rank matrices beside a frozen base (A01).
DPO / DPO Direct preference optimization: train the policy so the chosen response’s log-probability rises relative to the rejected one, measured against a frozen reference and scaled by beta (A03).
Reward model / Reward-Modell A model trained on chosen/rejected pairs to score responses; it is a proxy for the true task quality and can be exploited (A04).
KL divergence / KL-Divergenz How far one probability distribution has moved from another; used as a penalty to keep a trained policy close to its reference (A04).
PPO / PPO Proximal policy optimization: a policy-gradient update whose probability ratio is clipped so one step cannot move the policy too far (A03, A04).
RLHF / RLHF Reinforcement learning from human feedback: fit a reward model on preferences, then optimize the policy against it under a KL penalty (A04).

F01 · F02 · F03 · F04 · F05 · F06

Was this page helpful?