Observability, security and privacy

make check WEEK=5

code weeks/5/README.md
make next

Once the system can act, content it reads can attack it. Remove the capability first, and see everything: you cannot fix what you cannot trace.

Read this once (about 20 minutes). Write the sentence in your own words in reflections/week-5.md, Q1, before you touch code.

Observability: trace every step, attribute every cent

A request to an AI feature is several steps: retrieve, call the model (maybe twice), run a tool, call again. When it is slow, or wrong, or expensive, the question is which step. So:

• Trace every step. A request is a tree of spans; each has a duration, tokens, cost and, when it failed, a kind. The request id goes back to the caller so a complaint can be looked up.
• Attribute latency and cost to the step that caused it. "This request cost 0.4 cents" is a fact. "The repair call cost 0.3 of it" is something you can act on.
• An error taxonomy, not one bucket. provider_timeout and schema_error need different fixes. A dashboard that says "12 failures" says nothing; one that says "9 timeouts, 3 schema" says which provider to call and which prompt to look at.
• A dashboard a non-engineer can read. Cost per query is now a standard interview question, and the person asking it will not read a span tree.

One /ask request as its trace. Cost sits on the leaf that spent it and rolls up to the root.

flowchart TD
  R["POST /ask  ·  root
412 ms · $0.00094 · error_kind: none"]
  R --> S["retrieval.search
18 ms · $0"]
  R --> M["model.call
380 ms · 706 in / 115 out · $0.00094"]
  R --> T["tool.get_ticket (agents only)
gen_ai.tool.name · …"]
  style M fill:#fff4e5,stroke:#9a5b00

Security: the model reads instructions in the data

Prompt injection is not a prompt problem; it is a data problem. Any text the model reads (an uploaded file, a retrieved passage, a web page, a tool result) can contain instructions, and the model cannot reliably tell the difference between your instruction and one in the data. It arrives through retrieval as easily as through a user, and it looks like a normal document.

The disciplines, none sufficient alone:

• Treat data as data. Fence untrusted text, label it, and tell the model that instructions never come from inside the fence. Cheap, and it removes most casual injections.
• Filter input, and know what filtering cannot catch. Known patterns can be stripped. New phrasings, other languages and encoded text walk straight past a pattern list.
• Treat model output as untrusted input to whatever comes next. A figure the model reports that the source document never contains did not come from the document. Check.
• Least agency. The minimum permission the task actually needs. A tool that can spend money or delete data is a tool an attacker can spend or delete with.
• Kill switches and blast radius. One setting that stops every model call, and a budget that caps how far a runaway loop, or an attacker, can get before someone notices.

flowchart LR
  U["uploaded or retrieved text
may carry instructions"] --> L{"check_limits
kill switch? budget?"}
  L -- blocked --> B["503 / 429
no model call"]
  L -- ok --> DI["detect_injection
strip known patterns, report them"]
  DI --> WR["wrap_untrusted
fence as data + GUARD_PREAMBLE"]
  WR --> M[model]
  M --> SO["scan_output
figures not in the source?"]
  SO --> N["whatever reads the output next
treat it as untrusted input"]
  style M fill:#fff4e5,stroke:#9a5b00

Once the system can act

Since week 4 an injection does not have to change an answer to do harm: it can ask for an action. A ticket says the customer is pre-approved for a credit; a page of the manual says to escalate; a tool result says to look up another customer. Approval in code stops the action itself, but an agent that keeps reaching for it is one careless approval away from paying out. So, in order:

1. Remove the capability. The agent handling a lookup is never offered issue_credit. The offer is decided from the request a person on the desk made, never from the content the agent reads, and the door refuses any tool that was not offered. Nothing to call, nothing to trick.
2. Then filter, as a second layer. Third-party text reaches the model fenced as data, known patterns stripped. Say plainly what that cannot catch.
3. Then bound what is left. For every action tool that is still offered, write down its blast radius: the worst it could do if every defence failed, and what caps it.

Personal data

A ticket carries a name, an email address and a phone number, and none of them helps a model answer the question. Before every model call they are replaced with placeholders, the same person always the same placeholder within a run, and only an output a person approves (a draft) gets the real values back. The proof is not the code: it is a check of everything the model was actually shown, finding nothing.

Traces anyone can read

Name the spans the way OpenTelemetry does (gen_ai.request.model, gen_ai.usage.input_tokens, gen_ai.tool.name), and a trace from this service reads the same in whatever tool an employer uses. Then turn it into the numbers a business asks for: what a ticket costs, and how that moves over time.

What this looks like in the service

Idea	Where it lives	What you do this week
Spans and cost attribution	app/trace.py, GET /traces	Read it; use it to answer questions about your own service
Error taxonomy	ErrorKind in app/trace.py	Map every failure to a kind
Dashboard	scripts/trace_report.py	Run it; explain it to someone who does not code
Least agency	app/agents/routing.py, the door in app/tools/base.py	Decide what each request is offered
Guard	app/guard.py	Get attacked; build the defences, the second layer
Personal data	app/privacy.py, app/llm/masking.py, scripts/privacy_check.py	Mask before every model call; prove nothing leaked
OpenTelemetry	GET /traces/{id}/otel	Read a trace the way an employer's tool would
Attack set	eval/attacks.jsonl, scripts/attack.py	Attack a peer; add what worked to the set
Limits	KILL_SWITCH, DAILY_BUDGET_USD	Flip them; watch the service refuse

What you should be able to say by Friday

• The cost of one /ask request, step by step, from a trace.
• Which error kind your service produced most this week, and what you changed because of it.
• The attack that got through your peer's service, and the one that got through yours.
• What your input filter cannot catch, and what you rely on instead for that case.
• Which capability you removed rather than filtered, and the blast radius of what is left.
• What a ticket costs, from your trace report, and how you would bring it down.

Read more

Checked September 2026.

• Simon Willison: prompt injection — the series that defined the problem and keeps tracking it. Read the first two posts; then the newest one.
• OWASP Top 10 for LLM Applications — Prompt Injection, Insecure Output Handling, Excessive Agency: this week's three, with real incidents.
• OpenTelemetry: semantic conventions for generative AI — the industry's names for the span attributes trace.py records (tokens, model, cost). If you later move to a real tracer, this is the vocabulary.
• Google SRE book: Monitoring distributed systems — the four golden signals; trace_report.py shows three of them.