C01 — Turn a company problem into a model decision
← E04 — Profiling & budgets · C02 — Company data → · Glossary
You can train a model now. This unit asks whether training is the right next action. Prerequisites: compare an unchanged baseline with a candidate, distinguish training from evaluation data, and explain why a lower NLL is a limited result. Allow two sessions: one to define the problem, one to interpret measured evidence.
1. Start with a workflow
Nordlicht Workspace is a fictional support-software company. A support agent receives a request about billing, account access or exports. The proposed system returns a small structured routing response; requests for secrets or unauthorized data must be escalated. It does not execute account changes. Invented facts and synthetic identifiers prevent the exercise from requiring real customer data.
Write requirements before selecting a model:
| Requirement | What to measure or inspect |
|---|---|
| Users: support agents | Can they inspect and correct the suggested route? |
| Languages: English and German | Report results separately by language. |
| Must: correct route and valid JSON | Score task and format independently. |
| Must: escalate restricted requests | Measure critical failures separately from average accuracy. |
| Must: local processing | Exclude a remote-only candidate before scoring quality. |
| Must: known model/data provenance | Inspect version, source permissions and license record. |
| Budget: memory and response time | Measure the same input length and device; disclose scope. |
| Can: conversational explanation | Add only after required behavior works. |
The example is an advisory workflow with human review, not an autonomous access control mechanism. Authentication and authorization belong in application code.
2. Match the intervention to the gap
| Observed gap | First intervention | Why |
|---|---|---|
| Frequently changing facts | Retrieval and tools | Fetch the approved version at request time. |
| Domain language is poorly predicted | Test continued pretraining | Learn its distribution, then check general-language regression. |
| Correct format or response workflow is missing | SFT, possibly with LoRA | Provide target responses and supervise their tokens. |
| Several responses work but one is preferred | DPO | Learn a relative preference from chosen/rejected pairs. |
| Baseline already meets requirements | No training | Keep the simpler accepted system. |
The Applied: RAG track builds and measures exactly the first option, retrieval.
SFT specifies the objective and data. LoRA specifies which parameters can change; they are not competing tasks. QLoRA adds a frozen quantized base to low-rank adaptation. Training from scratch would need a reason such as an architecture experiment, not merely access to company documents.
For the routing workflow, compare a transparent rule baseline and a model under the same evaluator. If the rule baseline meets the narrow task, keep it as a serious option. An open model may justify its maintenance cost only when the required capability exceeds that narrow rule coverage.
3. Apply hard constraints before ranking
Suppose an online candidate scores 95% accuracy and a local one 80%. Local processing is mandatory. Averaging privacy and accuracy into a single score can wrongly recommend the online candidate. Instead:
- Eliminate candidates that fail any must requirement.
- Rank the remaining measured candidates by task quality, latency and memory.
- Preserve failed cases and uncertainty, not only the ranking.
Requirements and score_candidates implement these visible steps in
src/llm_course/company_strategy.py. The default quality gate of 0.7 is an
exercise threshold, not a production standard. A measured memory value must
declare whether it means parameter bytes, tensor allocations or process peak.
from llm_course.company_strategy import Requirements, choose_strategy
requirements = Requirements(need="domain_language")
print(choose_strategy(requirements)) # No evidence: collect a baseline.
print(choose_strategy(requirements, domain_nll_gain=0.5,
general_nll_regression=0.1))The second decision only starts a review of continued pretraining. It does not establish that instruction-following or support quality improved.
4. Count the work around training
TCO, total cost of ownership, includes preparation and recurring work. Use hypothetical currency units to practice the arithmetic:
- Setup: 2 hours × 40 units/hour = 80.
- Training: 1 compute-hour × 3 units/hour = 3.
- Six months: each month 1 maintenance-hour × 40 plus 2 compute-hours × 3 = 46.
- Total:
80 + 3 + 6*46 = 359units.
These are invented rates, not vendor prices. Replace them with your own approved estimates, including data review, regression evaluation, deployment and rollback maintenance. Faster training can still be the more expensive system to maintain.
5. Predict → trace → build → break → measure → explain
Predict which approach the requirement selects. Trace one requirement to its measurement, then build a scorecard for the shared company task. Break it by giving the best-quality candidate an incompatible processing location. Measure the remaining candidates on identical cases. Explain at least two rejected alternatives, including what new evidence would change the decision.
Exercise: a company needs fresh export limits and strict JSON. Propose a system with separate responsibilities. Which part needs current facts? Which part could benefit from SFT? Do not put access rights into model weights.
Hint
A single user-facing response can combine retrieved facts, deterministic schema validation and a trained formatter. Select a mechanism for each observed gap.
Reference answer
Retrieve approved current limits, enforce access in the application, and validate the output schema. Try prompting or a deterministic formatter first. Use SFT/LoRA only if the unchanged evaluation shows a remaining response-format gap. Reject continued pretraining for freshness and DPO for adding missing facts.
Checkpoint: another learner can trace every chosen component to a requirement, a measured baseline gap and a reproducible acceptance test. Continue with company data, then protected evaluation.
← E04 — Profiling & budgets · C02 — Company data → · Glossary