Context and retrieval, as the agent's main tool

The same model gives a different answer when you give it different context. Choosing the context is the job, and retrieval is how you choose it.

Read this once (about 20 minutes). Write the sentence in your own words in reflections/week-2.md, Q1, before you touch code.

Two ideas, one week

Context engineering is deciding what the model sees on each call. Everything you put in the window competes for attention with everything else. More context is not more knowledge; past a point it is more noise, more latency and more money for the same answer, and sometimes a worse one. The named failure modes:

The disciplines that follow: select (only what this call needs), compress (summarise what persists), isolate (keep tasks in separate contexts), cache (reuse a stable prefix so you do not pay to re-read it), and be explicit about memory: what persists between turns and sessions, and what must not.

flowchart LR
  Q[question] --> S["select
retrieve only what this call needs"]
  S --> W["context window
an attention budget, not a bucket"]
  W --> M[model]
  subgraph failures [what goes wrong when the window is treated as storage]
    L[lost in the middle] ~~~ P[poisoning] ~~~ D[distraction] ~~~ X[clash]
  end
  W -.- failures

Retrieval is how you select. Cut the corpus into chunks, embed them, find the nearest to the question, hand those to the model. Every choice in that sentence is a measurable decision:

• Chunking. Fixed-size splits cut tables and sentences in half. Sentence and paragraph splits keep meaning whole but vary in size. Heading-aware splits keep a section's title with its body. Which is right depends on the documents and the questions, so you measure.
• Embeddings and the store. Chosen with a stated reason, not a default. This repo starts with a lexical embedder (hash) so you can see the mechanics, and a real one (fastembed) so you can see the difference.
• Hybrid search. Keyword search (BM25) and meaning search fail on different questions. Meaning search finds "how long does it wait for the source" when the page says "connection timeout"; it misses E1104 and KESTREL_CONNECT_TIMEOUT, which carry little meaning and must match exactly, and a support desk is full of them. Keyword search is the opposite. Reciprocal rank fusion merges the two lists by rank, because their scores are on different scales. You build both halves this week and measure what fusing them buys.
• Evaluating retrieval separately from generation. If the right chunk was never retrieved, the best model in the world cannot answer. So measure retrieval on its own, with its own metrics: precision at k, recall at k, mean reciprocal rank, and whether the context was relevant at all.

The same clause, cut four ways. Only one strategy keeps "3. Fees." with its fee.

flowchart TB
  subgraph fixed [fixed, 500 chars]
    f1["…sixty (60) days written no"] --- f2["tice. 3. Fees. Client will pay Provider 6,500 GBP…"]
  end
  subgraph sentence [sentence, ~400 chars]
    s1["…written notice. 3. Fees."] --- s2["Client will pay Provider 6,500 GBP per month…"]
  end
  subgraph paragraph [paragraph, ~800 chars]
    p1["3. Fees. Client will pay Provider 6,500 GBP per month, invoiced monthly in advance, payable within 14 days of invoice."]
  end
  subgraph heading [heading-aware, per section]
    h1["## Fees
Client will pay Provider 6,500 GBP…"]
  end

Where the measurement sits: before the model, so a wrong answer can be traced to the step that produced it.

flowchart LR
  Q[question] --> E1[embed]
  D[("chunks + vectors
one index per strategy")] --> C
  E1 --> C[cosine over every chunk]
  C --> K[top k hits]
  K --> MEAS{{"measure here
P@k · R@k · MRR · hit rate"}}
  K --> M[model, Week 3]

The same model, different context

The cleanest way to see this week's sentence is to hold everything still except the context. scripts/context_experiment.py asks the same twenty questions of the same model with the same prompt, five times each: with no passages, with the wrong version's page, with the right page, with the right page and forty irrelevant ones, and with the right page buried in the middle of eighty. On the right page the answers are mostly right. On the wrong version's page they are confidently wrong, and they cite it. With too much, they cost ten times as much, and on smaller models they start to slip. The model did not change. The answers did. Your decision on how much context to send is read off that table.

Retrieval as a tool

From this week retrieval is not only an endpoint: it is search_docs, the agent's second tool, called with the version get_customer returned. The tool's schema makes the version required, so the model cannot leave it out and fall back into the version trap. That is the pattern every later week repeats: the boundary lives in the schema and the code, not in the prompt.

What this looks like in the service

What you should be able to say by Friday

• Which of your six configurations you chose, with which k, and the numbers that decided it.
• What hybrid search fixed that meaning search alone missed, with an example question.
• How much the answers moved in the context experiment when only the context changed.
• Why recall@k and precision@k can move in opposite directions when you change chunk size.
• What lost-in-the-middle is, and whether you saw it at the context sizes you tried.
• Why measuring retrieval separately from the answer is the only way to know where a wrong answer came from.

Read more

Checked September 2026.

• Lost in the Middle: How Language Models Use Long Contexts (Liu et al., 2023) — the paper that named the failure. Read the figures; then re-run explore/w2_02_lost_in_the_middle.py with --filler 200.
• Anthropic: Effective context engineering for AI agents — select, compress, isolate, as a discipline rather than a trick.
• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (Lewis et al., 2020) — where the term RAG comes from; the architecture is what /search plus Week 3's /ask reproduce in miniature.
• Okapi BM25 (Wikipedia) — the formula in hybrid.py, with what each of k1 and b does.
• Pinecone: Chunking strategies — a practical survey of the choices in chunkers.py and a few this repo leaves to you.
• Reciprocal Rank Fusion (Cormack et al., 2009) — two pages; the fusion method you build this week.
• BAAI/bge-small-en-v1.5 — the embedding model behind EMBED_PROVIDER=fastembed: what it was trained on, and what its similarity scores mean.