Week 1
Build this week's exercise, then run the gate.
make check WEEK=1
| Session | Session 2: Direction, 45 min |
| Build | The extract endpoint and the agent's first two tools. You design the tools: descriptions, constraints, what comes back, refusals in words. |
| Files | app/api/extract.py, app/tools/desk.py |
| Measure | make evidence WEEK=1 on a real model, into evidence/week-1.json |
| Record | decisions/week-1.md from the evidence, ai-use/week-1.yml, reflections/week-1.md |
| Guides | The model standard · Glossary |
submitted 1 of 16 checks pass
Red is the normal state of a week you have not built yet. Each name is a test in the file beside it.
Written last: make evidence WEEK=1, then decisions/week-1.md and ai-use/week-1.yml.
The exercise is in your repository, where you can run it:
code weeks/1/README.md make next
A service that takes documents, parses and stores them, with the desk's customers and tickets already loaded. No model anywhere yet.
A model behind an interface: it survives the provider misbehaving, never pays twice, records what it cost and swaps provider from a setting. Output against a schema, and the agent's first two tools, get_customer and get_ticket.
Week 2 gives the agent its main tool, search. That is only worth measuring if the model and the tools under it cannot fall over or quietly change.
A model is an unreliable, expensive, non-deterministic third-party dependency. Treat it like one, and give it nothing you have not described exactly.
Read this once (about 20 minutes). Then write the sentence above in your own words in
reflections/week-1.md, Q1, before you touch any code.
Everything in this week's exercise exists because of one of these.
1. It fails. Rate limits (HTTP 429), outages (5xx), network drops. Some failures pass if you try again in a second. Some (a 400: you sent something wrong) never will, and retrying them burns money and hides your bug. A good client knows the difference.
sequenceDiagram
participant S as Service (retry.py)
participant P as Provider
S->>P: attempt 1
P-->>S: 429 rate limited (retryable)
Note over S: wait 0.5 s
S->>P: attempt 2
P-->>S: 429 (retryable)
Note over S: wait 1.0 s
S->>P: attempt 3
P-->>S: 200 OK
Note over S,P: three calls billed, one answer
S->>P: a different request
P-->>S: 400 bad request (NOT retryable)
Note over S: stop at once: retrying our own mistake
costs money and hides the bug
2. It stalls. A call that usually takes 800 ms sometimes takes 40 s, or never returns. Without a timeout, one slow call holds a connection, then a worker, then your whole service.
3. It costs. Every call bills tokens in and tokens out. Ten thousand documents a day at half a cent each is 50 dollars a day, 18,000 a year, for one endpoint. Sending the same document twice is paying twice for the same answer. Sending a 60-page document when the first page would do is paying for 59 pages of nothing.
flowchart LR
A["POST /documents/{id}/extract"] --> K{"extractions row for
doc_id + provider + model + prompt_version?"}
K -- hit --> C["return it
cached: true, $0"]
K -- miss --> T[cap to MAX_INPUT_CHARS]
T --> M["model call
timeout, retry"]
M --> V{fits DocumentExtract?}
V -- no --> R["one repair call
with the validation error"]
R --> V2{fits now?}
V2 -- no --> E["502 schema_error
never a 500"]
V -- yes --> W["store row:
tokens, latency, cost"]
V2 -- yes --> W
W --> O["return it
cached: false"]
4. It varies. Ask the same question twice and get two different answers. That is fine for prose and fatal for data. Free text from a model is not data; it becomes data only after it has passed a schema. When it does not fit, you ask once more with the error attached, then you fail loudly.
5. It changes under you. Providers deprecate models, change defaults, alter pricing. Free tiers
appear and vanish. If the model's name is in your code, a provider change is a code change, a
review, a deploy. If it is in configuration, it is two lines in .env.
flowchart LR
subgraph service [your service: nothing here names a provider]
E[extract.py] --> C["ModelClient protocol
app/llm/client.py"]
C --> G["registry.py
reads MODEL_PROVIDER"]
end
G -- ".env says gemini" --> P1[(Gemini)]
G -- ".env says groq" --> P2[(Groq)]
G -- "tests" --> P3[("fake_a / fake_b")]
style service fill:#eef3f8,stroke:#1f5f8b
This happened to this repo. Its first default provider was GitHub Models, chosen because it needed
no signup. GitHub retired the service on 30 July 2026. The fix was one entry in registry.py and
two lines in .env.example; not one line of the service changed. That is the whole argument.
From this week the model does not only answer. It asks for things: "call get_customer with
C-1004". That request is function calling, and it is nothing more mysterious than structured
output with consequences. The model is shown each tool's name, a description and a JSON schema of
its arguments, and replies with a tool name and arguments in that shape.
Two things follow, and the week is built on both.
The description is a prompt. It is the only thing the model knows about the tool. "Gets data" leaves the model guessing which of two tools a request needs, and it guesses wrong about half the time. "Look up one of this desk's customers by id and get back which version and plan they are on" does not. A tool's description has to say when to call it and what comes back, in the words a request will use.
Arguments are validated before anything runs. A model will one day send "1001", "c1001",
an empty string or "C-1001; DROP TABLE". The schema's constraints reject them before the tool
runs, and the model gets back a sentence it can act on, not a stack trace. The same goes for a
refusal: a customer on another desk is a boundary, answered in words, not an exception.
flowchart LR
M["model: {tool, args}"] --> V{"args fit
the schema?"}
V -- no --> E["error: invalid arguments ...
(the tool never runs)"]
V -- yes --> A{"writes or
spends?"}
A -- "yes, no approval" --> N["needs approval
(week 4)"]
A -- no --> T["the tool runs"] --> R["text the model can read:
a record, or refused: why"]
You measure both on a real model: how often it calls the right tool with valid arguments, with a
first-draft description and with yours (scripts/tool_drill.py), and how often plain "reply in
JSON" breaks the schema compared with schema-enforced output (scripts/schema_drill.py).
| Betrayal | Where the defence lives | What you do this week |
|---|---|---|
| Fails | app/llm/retry.py |
Retry 429 and 5xx with backoff; never retry 400 |
| Stalls | app/llm/retry.py, MODEL_TIMEOUT_S |
Every attempt gets its own timeout |
| Costs | app/llm/cost.py, the extractions table |
Record tokens and cost per call; never pay twice |
| Varies | app/llm/structured.py, DocumentExtract |
Validate against a schema; repair once; typed error |
| Changes | app/llm/registry.py, .env |
Provider and model are configuration |
| Calls the wrong tool | app/tools/, the argument models |
Describe each tool exactly; validate before running; refuse in words |
git diff main shows no provider-specific code after you switched providers.Now run the three scripts in explore/ (see README.md, "See it").
Checked September 2026. Read the first two; the rest when the exercise makes you curious.
retry.py.Answers are kept in this browser only, under hub.quiz in local storage. They are never sent anywhere: not to the repository, not to the progress export, not to your mentor, and they have no effect on your route. Clearing your browser data clears them. The explanations are the point, not the score.