AG02 — Native tool use
← AG01 — The agent loop, by hand · AG03 — Prompt injection → · Glossary
Prerequisites: AG01 with its report at artifacts/unit0/report.json,
and ANTHROPIC_API_KEY in your shell — the required route of this unit is hosted, on
claude-sonnet-5. The reference runs below cost about 2–3 cents each on that model;
a full pass of this page including Measure is roughly one dollar. The CPU route is
the local Qwen leg, which uses the 4B you downloaded for AG01. Allow one session of
60–90 minutes.
What changes, and what does not
In AG01 you told the model the rules in prose, parsed one CALL line yourself, and
sent results back as text. A native tool-use API replaces exactly three of those
things: the schema (a JSON Schema per tool instead of a sentence), the parse
(the API returns a structured tool_use block), and the call id (each result is
matched to the call it answers). Everything else is still yours:
| still yours | why the API cannot take it |
|---|---|
| the step budget | the API does not know when a run should give up |
| output truncation | it does not know how large a result may be |
| path containment | it does not know which files are the learner’s |
| believing a result | it cannot tell a true tool result from a lying one |
The first row is not hypothetical. In the reference measurement below, two of the
five text-protocol runs on Claude ended only because the budget ran out: five
search_course calls in a row with rephrased queries, and no answer. The API on
the native leg would not have noticed either.
You will run the same agent on the same question three ways:
| leg | loop | model |
|---|---|---|
| text / Claude | AG01’s loop, hosted | claude-sonnet-5 |
| native / Claude | the new loop | claude-sonnet-5 |
| native / Qwen | the new loop, local | Qwen/Qwen3-4B-Instruct-2507 |
The two loops are src/llm_course/agent/loop.py and native.py. Diff them: that
difference is this unit.
Predict
Read AG01’s transcript again, then this one — the native transcript for the same question. The model asks for both tools in one turn:
assistant: [tool_use read_report {"path": "unit0/report.json"}]
[tool_use search_course {"query": "boost one logit corpus loss worse transition"}]
user: [tool_result …report…] [tool_result …section…]
assistant: It went up, from 1.695 to 1.890 (+0.194) …Write down three predictions: which leg will produce a malformed call first; how many messages each history holds when the run ends; and which history you could hand to a different model unchanged.
Trace
python examples/run_agent.py --protocol native --preset scriptedOpen artifacts/agent/run.json. Compared with AG01’s report:
messages— four entries, not seven. Find the assistant message whosecontentholds twotool_useblocks, each with anid; then the user message whose twotool_resultblocks carry those ids back. The API insists that all results for one turn arrive in one message; the loop asks the backend to build it.steps— two steps with the sameindex: one turn, two calls. AG01’s protocol could not say that.- There is no
CALLline and noRESULTprefix anywhere. Nothing was parsed by you.
Build
Run the three legs. Each writes its own report:
python examples/run_agent.py --protocol text --preset anthropic --model sonnet --report artifacts/agent/text-claude.json
python examples/run_agent.py --protocol native --preset anthropic --model sonnet --report artifacts/agent/native-claude.json
python examples/run_agent.py --protocol native --preset qwen --report artifacts/agent/native-qwen.jsonRead the three answers side by side, then the three messages lists. On the Qwen
leg, find the assistant message: it is plain text containing <tool_call> tags —
Qwen’s template format — and the result went back as a tool message. Open the
rendered prompt in your head: Qwen’s template turns a tool message into a user
turn wrapped in <tool_response>. AG01’s “results go back as a user message” was
not a shortcut; it is what the native format does underneath.
Independent exercise. Write list_reports again, this time as a schema. The
Tool you wrote for AG01 already has typed Parameters; the schema is derived:
from pathlib import Path
from llm_course.agent import Tool, input_schema
def list_reports(artifact_root: Path) -> Tool:
def run(arguments: dict) -> str:
paths = sorted(p.relative_to(artifact_root) for p in artifact_root.rglob("*.json"))
return "\n".join(str(p) for p in paths) or "No reports found."
return Tool("list_reports", "List every JSON report under the artifact directory.", (), run)
print(input_schema(list_reports(Path("artifacts"))))Register it beside the two defaults with run_native_agent (the same shape as AG01’s
runner snippet, with backend=make_backend("anthropic")) and ask “Which reports do
I have, and what did Unit 0 measure?”. Keep the report.
Break
Four deliberate failures. Keep each report.
A history the API refuses. Copy artifacts/agent/native-claude.json, delete one
tool_result block from the user message that holds two, and replay the history
(one API call, about a cent):
python - <<'PY'
import json
from pathlib import Path
from llm_course.agent.presets import make_backend, default_tools
report = json.loads(Path("artifacts/agent/native-claude.json").read_text())
messages = report["messages"][:3]
assert len(messages[2]["content"]) == 2, "no batched turn; pick the user message with two results"
messages[2]["content"] = messages[2]["content"][:1] # drop the second result
backend = make_backend("anthropic")
try:
backend.call("You are an assistant with tools.", tuple(messages), default_tools(Path("content"), Path("artifacts")))
except Exception as error:
print(type(error).__name__, error)
PYThe API rejects the conversation: a tool_use without its tool_result is an
error, not a guess. AG01’s loop would have accepted the same gap silently.
A wrong type in the schema. In your agent_tools.py, declare k as "string"
on a copy of search_course’s parameters and ask a question that needs k. Read the
call the model makes and what search_course returns for "k": "3".
The budget is still yours.
python examples/run_agent.py --protocol native --preset anthropic --model sonnet --max-steps 1 --report artifacts/agent/native-budget.jsonstopped is budget. The API never noticed. In 2 of 5 measured text-protocol runs
on Claude the budget was the only thing that ended the run — five rephrased searches
in a row and no answer.
Qwen’s own format. Rerun the Qwen leg into a report of its own, so the Build report stays intact:
python examples/run_agent.py --protocol native --preset qwen --report artifacts/agent/native-qwen-format.jsonRead every <tool_call> block in that report’s messages; if a parse_error
appears in its steps, read the block that caused it. Reference measurement: 0 of
5 — Qwen’s template format parsed every time.
Measure
python examples/run_agent.py --protocol text --preset anthropic --model sonnet --repeat 5 --report artifacts/agent/measure-ag02/text-claude.json
python examples/run_agent.py --protocol native --preset anthropic --model sonnet --repeat 5 --report artifacts/agent/measure-ag02/native-claude.json
python examples/run_agent.py --protocol native --preset qwen --repeat 5 --report artifacts/agent/measure-ag02/native-qwen.jsonReference measurement, Apple Silicon (MPS) for Qwen and the API for Claude, 2026-09-21
(claude-sonnet-5 with adaptive thinking, effort low). The two AG01 rows are copied
from that page for comparison. In the “two calls in one turn” column, – means the
protocol cannot express two calls in one turn, and 0 means it could and the model
never did:
| leg | protocol_ok (of 5) | answered (of 5) | read_report (of 5) | search_course (of 5) | exact heading (of 5) | two calls in one turn (of 5) | median steps | median latency (s, total) | median cost (USD) |
|---|---|---|---|---|---|---|---|---|---|
| text / Claude sonnet-5 | 5 | 3 | 5 | 5 | 1 | – | 5 | 8.3 | 0.0257 |
| native / Claude sonnet-5 | 5 | 5 | 5 | 5 | 3 | 5 | 4 | 9.6 | 0.0240 |
| native / Qwen 4B | 5 | 5 | 5 | 5 | 5 | 0 | 2 | 13.6 | 0.0000 |
| text / Qwen 4B (AG01, 2026-09-21) | 5 | 5 | 5 | 5 | 5 | – | 2 | 13.5 | 0.0000 |
| text / Qwen 1.7B (AG01, 2026-09-21) | 5 | 5 | 5 | 0 | 0 | – | 1 | 4.6 | 0.0000 |
Your first prediction has a plain answer: no leg produced a malformed call.
protocol_ok is 5 of 5 on all three, and the Qwen <tool_call> JSON parsed every
time. The interesting failure was elsewhere. On the text protocol Claude answered
only 3 of 5: runs 1 and 3 hit the 6-step budget issuing five consecutive
search_course calls with rephrased queries — “expected corpus loss increase after
boost”, “corpus loss expected to rise after controlled change”, “why corpus loss got
worse is this expected” — because no Unit 0 section literally says “expected”
(grep -i expected content/unit-00.md is empty), and the model never decided it had
enough evidence. On the native protocol the same model answered 5 of
5, and in 5 of 5 runs it opened with one assistant turn calling both tools
(messages[1].content is thinking, tool_use, tool_use). Same model, same
question, different plumbing, different behaviour: the protocol changed what the
model did, not just the bookkeeping.
Cost and tokens: the text protocol used fewer output tokens per run (236 versus 527
on average — a CALL line is a few tokens, a native turn carries a thinking block
and structured tool_use blocks) but more turns, each resending the growing
history, so the medians came out close: 0.0240 for
native. The five-run measurement of both hosted legs together cost $0.20. Median
latency was 8.3 s (text / Claude), 9.6 s (native / Claude) and 13.6 s (native /
Qwen on MPS).
Two things the table shows are model traits, not protocol traits. Qwen 4B never
batched two calls in one turn (0 of 5) although the protocol allows it and the
scripted transcript does it; Claude did it in every run. And the question asks for
the exact heading, which search_course returned verbatim every time, yet Claude
quoted 30–42 min · Build one controlled change exactly in only 1 of 5 text runs and
3 of 5 native runs — the others paraphrased it as “Build one controlled change” and
dropped the time badge. Qwen 4B quoted it exactly in 5 of 5, on both protocols. Its
native leg also produced identical token counts in all five runs (3391 in, 192 out):
the local backend is deterministic here.
Explain
Complete in your lab notes:
- The API now guarantees … ; it still cannot know …
- All results for one turn travel in one message because …
- Qwen’s
toolrole differs from a user message in that … — and does not differ in that … - Per run, the text protocol used … output tokens than the schema but cost about the same, because …
- If a tool result lied on the native leg, the loop would …
Completion gate
You pass AG02 with: the three reference reports; the four Break reports with one
sentence each; the five-run table for all three legs with cost; your schema-based
list_reports and the report of a question it answered. Your own words count; the
words on this page do not.
What’s next
AG03 measures what each defence against prompt injection actually buys, on this same agent and on R02’s RAG pipeline. AG04 will connect the agent to the course’s MCP server and then have you build your own (coming soon). G01 will turn the “still yours” column into a layer of its own.
← AG01 — The agent loop, by hand · AG03 — Prompt injection → · Glossary