🛩️ Module 4B · Project — RAG over a corpus

① Problem: a single rule, sliced in half

What goes wrong: chunk every file the same blind way — a cut every ~1000 characters — and a regulation like §91.151 gets split across two chunks. Half the rule lands in chunk 7, the other half in chunk 8.

❌ Before — one blind cut for everything

flowchart LR
  D["📄 §91.151
(one whole rule)"] --> CUT["✂️ blind
1000-char cut"]
  CUT --> A["chunk 7
…first half"]
  CUT --> B["chunk 8
second half…"]
  A --> X["❌ retrieval gets a fragment
no chunk IS §91.151 → can't cite it"]
  B --> X
  classDef s fill:#0d1f33,stroke:#1f6feb88,color:#e6edf3;
  classDef danger fill:#2d1215,stroke:#f8514988,color:#e6edf3;
  class D,CUT,A,B s;
  class X danger;
          

The flaw: the clause is broken, so the model only ever sees a piece of it — and because no chunk equals §91.151, the answer has nothing to cite.

The fix: look at the text first. FAA files carry <!-- §… --> tags, so cut on those § boundaries — one whole rule per chunk, keeping its section + part. Untagged text keeps the fixed windows. Routing the text to the right chunker by its content is content-routed chunking.

✅ After — route by content

flowchart LR
  T["📄 a document"] --> Q{"has § tags?"}
  Q -->|"yes · FAA"| S["chunk_by_section
1 chunk = 1 rule
✅ cite §91.151 (Part 91)"]
  Q -->|no| C["chunk_text
~1000 chars + overlap"]
  classDef b fill:#11271a,stroke:#3fb95088,color:#e6edf3;
  classDef d fill:#241a2e,stroke:#a371f788,color:#e6edf3;
  class T,S,C b;
  class Q d;
          

② Problem: which model built this index?

What goes wrong: the index is just a pile of vectors on disk. Nothing on it records which embedding model produced them. Weeks later you query with a different model — say minilm instead of the bge you built with.

❌ Before — vectors with no label

flowchart LR
  BLD["🏗️ build with bge"] --> PKL[("index.pkl
just vectors,
no label")]
  PKL --> QRY["🔎 query with minilm
(different model)"]
  QRY --> X["💥 incompatible vectors
meaningless matches — and no error"]
  classDef s fill:#0d1f33,stroke:#1f6feb88,color:#e6edf3;
  classDef k fill:#2a2410,stroke:#d2932288,color:#e6edf3;
  classDef danger fill:#2d1215,stroke:#f8514988,color:#e6edf3;
  class BLD,QRY s;
  class PKL k;
  class X danger;
          

The flaw: vectors from two models live in different spaces (often different sizes). The "nearest" chunks come back as nonsense — and nothing crashes, so it's confident garbage you might never notice.

The fix: stamp the model name into a sidecar, index.meta.json (e.g. {"embed_model": "bge"}). The query reads it first and loads the same model. An index that carries its own build settings is a self-describing index — the single source of truth.

✅ After — the index carries its model

flowchart LR
  BLD["🏗️ build with bge"] --> PKL[("index.pkl")]
  BLD --> MJ[("index.meta.json
model = bge")]
  MJ -. "read first" .-> QRY["🔎 query with bge
✅ same model"]
  PKL --> QRY
  classDef b fill:#11271a,stroke:#3fb95088,color:#e6edf3;
  classDef s fill:#0d1f33,stroke:#1f6feb88,color:#e6edf3;
  classDef k fill:#2a2410,stroke:#d2932288,color:#e6edf3;
  class BLD b;
  class QRY s;
  class PKL,MJ k;
          

③ Problem: the model just makes things up

What goes wrong: hand the question straight to Claude and it answers from its own training memory — fluent, confident, and sometimes wrong. No sources attached, no way for the reader to check.

❌ Before — answer from memory

flowchart LR
  Q["❓ question"] --> M["🤖 answers from
its own memory"]
  M --> X["❌ fluent but unsourced,
sometimes invented (hallucination)"]
  classDef s fill:#0d1f33,stroke:#1f6feb88,color:#e6edf3;
  classDef danger fill:#2d1215,stroke:#f8514988,color:#e6edf3;
  class Q,M s;
  class X danger;
          

The flaw: nothing forces the answer to come from your documents, so there's no grounding and nothing to cite — exactly where hallucinations slip in.

The fix: wrap the call in a SYSTEM_PROMPT with hard rules — use only the CONTEXT, cite every claim [n], and if the context doesn't cover it, say so. Invalid [n] get dropped in code afterward. Tying the answer to the retrieved sources this way is a grounded prompt.

✅ After — answer only from context

flowchart TB
  CX["📋 CONTEXT [1]…[5]
+ your QUESTION"] --> M["🤖 Claude"]
  RU["📏 RULES
only use context ·
cite [n] · no context → say so"] --> M
  M --> A["✅ grounded answer
every claim tagged [n]"]
  A --> D["🧹 code drops
any invalid [n]"]
  classDef s fill:#0d1f33,stroke:#1f6feb88,color:#e6edf3;
  classDef r fill:#241a2e,stroke:#a371f788,color:#e6edf3;
  class CX,M,A,D s;
  class RU r;
          

④ Problem: one §-section blows the token budget

What goes wrong: §-sections vary wildly — median ~1,000 chars, but some run to ~200,000. Stuff the raw top-5 into the prompt and a single giant section pushes past 15k input tokens: slow and costly. The obvious fix — just keep the first few thousand characters — is worse: the answer often lives in a late subsection like (a)(1)(iv), which front-truncation throws away.

❌ Before — stuff the raw sections (or truncate the front)

flowchart LR
  H["top-5 §-sections
(one is ~200k chars)"] --> CX["📋 stuff all 5
into CONTEXT"]
  CX --> X["💥 15k+ input tokens — slow & costly
(or front-truncate → drop clause (a)(1)(iv))"]
  classDef s fill:#0d1f33,stroke:#1f6feb88,color:#e6edf3;
  classDef danger fill:#2d1215,stroke:#f8514988,color:#e6edf3;
  class H,CX s;
  class X danger;
          

The flaw: position is the wrong thing to cut on. The relevant clause can sit anywhere in a long section, so keeping the front both wastes tokens and risks dropping the answer.

The fix: cut on relevance, not position. select_context splits each hit into ~500-char windows, ranks them by similarity to the question, and keeps the best ones up to a ~5,000-char budget — about 1,250 tokens. Same budget, better recall, no re-index (it runs at request time on the already-loaded model).

✅ After — sub-chunk rerank (select_context)

flowchart LR
  H["top-5 §-sections"] --> W["✂️ split into
~500-char windows"]
  W --> R["rank windows by
similarity to the QUESTION"]
  R --> K["keep best-first
up to ~5,000 chars"]
  K --> OK["✅ ≈1,250 tokens,
keeps the right clause
even if it's (a)(1)(iv)"]
  classDef s fill:#0d1f33,stroke:#1f6feb88,color:#e6edf3;
  classDef b fill:#11271a,stroke:#3fb95088,color:#e6edf3;
  class H,W,R,K s;
  class OK b;
          

⑤ Problem: a follow-up question loses its topic

What goes wrong: someone asks "What are the fuel-reserve requirements?" then follows with "what about at night?". Embed that follow-up on its own and there are no topic words to match — retrieval returns random sections and the app wrongly says "out of scope".

❌ Before — embed the bare follow-up alone

flowchart LR
  P["Q1: fuel reserves,
day vs night?"] -.-> Q2["Q2: “what about
at night?”"]
  Q2 --> E["🔍 embed “what about
at night?” by itself"]
  E --> X["❌ no topic words →
retrieves unrelated §§ →
false “out of scope”"]
  classDef s fill:#0d1f33,stroke:#1f6feb88,color:#e6edf3;
  classDef danger fill:#2d1215,stroke:#f8514988,color:#e6edf3;
  class P,Q2,E s;
  class X danger;
          

The flaw — and the trap: the follow-up needs the previous topic. But blindly prepending the prior question to every query is also wrong — a full standalone question gets polluted (a "Class B/C airspace" prefix once buried a fuel-reserve question and retrieved airspace §§ instead).

The fix: only prepend when the turn looks like a bare follow-up (≤5 words, or opens with a cue like "what about"). Standalone questions search on their own. Generation still gets the full history either way. This is query contextualization.

✅ After — contextualize only bare follow-ups

flowchart LR
  Q2["Q2: “what about at night?”"] --> D{"bare follow-up?
≤5 words / cue"}
  D -->|yes| J["prepend Q1's topic →
“fuel reserves … at night”"]
  D -->|"no · standalone"| S["search as-is
(don't pollute it)"]
  J --> OK["✅ inherits the topic →
finds §91.151"]
  classDef s fill:#0d1f33,stroke:#1f6feb88,color:#e6edf3;
  classDef b fill:#11271a,stroke:#3fb95088,color:#e6edf3;
  classDef d fill:#241a2e,stroke:#a371f788,color:#e6edf3;
  class Q2,S s;
  class J,OK b;
  class D d;
          

⑥ Problem: the user stares at a frozen screen

What goes wrong: retrieval + generation take several seconds. A plain spinner shows nothing, then the whole answer appears at once. It feels broken, and you can't see how the answer was found — which is exactly what a reviewer wants to inspect.

❌ Before — spinner, then a wall of text

flowchart LR
  Q["🧑 question"] --> WsP["⏳ spinner…
5–10s of nothing"]
  WsP --> A["answer dumps
all at once"]
  A --> X["❌ feels frozen ·
no view into retrieval"]
  classDef s fill:#0d1f33,stroke:#1f6feb88,color:#e6edf3;
  classDef danger fill:#2d1215,stroke:#f8514988,color:#e6edf3;
  class Q,WsP,A s;
  class X danger;
          

The flaw: no feedback during the wait, and the retrieval reasoning is invisible.

The fix: stream the answer token-by-token behind a live cursor, and drive a process panel that prints the real steps (embed → top-5 sections → trim N→M chars) with actual numbers — which then persists in a collapsed panel under the answer, so it's re-viewable later.

✅ After — stream + live retrieval panel

flowchart LR
  Q["🧑 question"] --> P["🔍 live panel:
embed → top-5 → trim
(real numbers)"]
  P --> S["✍️ answer streams
token-by-token ▌"]
  S --> OK["✅ fast & transparent;
trace persists under the answer"]
  classDef s fill:#0d1f33,stroke:#1f6feb88,color:#e6edf3;
  classDef b fill:#11271a,stroke:#3fb95088,color:#e6edf3;
  class Q,P,S s;
  class OK b;
          

⓪ 🧪 Before any tuning — build the ruler (and a war story about killing a run)

You can't pick a "best" config until you can measure configs, so nothing was tuned until an evaluation harness existed: a verified holdout question set (each with its expected §-sections), a retrieval scorer computing recall coverage MRR (kept independent of the chunking axis), and a final split held back to guard against overfitting.

The four knobs — chunking × embedding × method × K — were swept overnight: 2 chunkers × 4 embeddings × 3 methods × 3 K = 792 runs, logged append-only (one line per config×question) with content-keyed resume. Then the character-chunk half froze at 495 / 792 — the process had ballooned to 16 GB and was thrashing swap.

Kill it. Safe to do, because the section run — the real winner — was already complete and append-only had preserved all 495 lines. Lesson: build the ruler first, and design experiments to be resumable and safe to interrupt — a free bonus config isn't worth cooking the machine all night.

① 🤖 Which embedding model? → bge (measured, not famous)

An offline grid search compared four embedding models on this corpus — bge · gte · minilm · e5. Top rows of the leaderboard (over 11 questions):

Highlighted row = shipped config. "input-tok proxy" is a per-config total across the 11 questions.

Three models tied on coverage at 0.864 — so coverage alone can't name a winner.

The tie broke on two finer signals: rank quality — bge's MRR 0.636 beat gte's 0.602 and minilm's 0.530 — and small-K efficiency — bge hits 0.818 at K=5 while minilm needs K=8 for the same score. A better embedding literally buys back tokens.

Ship bge at vector · K5. The "strong" model e5 came last — losing even to gte smelled like a usage bug (missing query:/passage: prefixes), so it was benched, not trusted. Benchmark fame doesn't reproduce on your data; only measurement does. And because vectors from different models aren't comparable, the winner gets engraved into the index (see ⑥).

② ✍️ How to stop hallucination? → grounded prompt + code enforcement

Hand a question straight to claude-sonnet-4-6 and it answers from training memory — fluent, uncited, sometimes wrong.

A SYSTEM_PROMPT with three hard rules:

• Answer only from the retrieved CONTEXT.
• Attach a [n] citation to every claim.
• If the context doesn't support it, say "I don't know" — never invent.

The prompt itself was a measured choice, not a vibe — three candidates were generated and scored by an Opus rubric judge:

Prompt guides, code enforces: shipped SYSTEM_PROMPT = GROUNDING_RULES + BALANCED_STYLE (overall 4.93). After generation the app also parses the [n] markers, drops out-of-range ones, and dedupes — so even a stray citation can't slip in. *C_warm was truncated at max_tokens.

③ 💸 How many chunks per question? → K = 5 (the cost lever)

K sets how many retrieved chunks go into the prompt — and it's the one knob that really moves the bill.

On real API calls the output is a flat ~100 tokens, and 91–96% of cost is input (K × passage size). Input tokens by K: K3 = 1,762 · K5 = 2,656 · K8 = 5,136.

flowchart LR
  K8["K = 8
coverage 0.864"] --> C8["5,136
input tokens"]
  K5["K = 5
coverage 0.818"] --> C5["2,656
input tokens"]
  C8 -->|"−0.046 coverage = tiny"| V["✅ ship K = 5
≈ half the tokens"]
  C5 --> V
  classDef s fill:#0d1f33,stroke:#1f6feb88,color:#e6edf3;
  classDef b fill:#11271a,stroke:#3fb95088,color:#e6edf3;
  class K8,K5,C8,C5 s;
  class V b;
          

Coverage barely dips (0.864 → 0.818) from K8→K5 but tokens roughly halve — a great trade for the rubric's 15-point Cost Management score, which literally asks for "as few tokens as possible".

④ 🧭 Single-shot or agentic? → ship single-shot, cap the agent

The experiment: give the model a search_cfr tool and let it search as many times as it wants. On "make a table of all airspace classes" the agent searched 13 times, noticed Part 71 only says where airspace is (not the operating rules in Part 91), and searched Part 91 on its own — real self-correction, complete table.

Every turn re-sends the whole history, so cost grows like 1 + 2 + … + N ≈ N²:

The rubric scores results only and rewards low cost, and 4 of 5 practice questions live in one Part — which static answers perfectly and cheaply. So single-shot ships. The agent isn't thrown away: it's capped at MAX_SEARCHES = 3 (keeping even a broad question near ~15k) and kept on the bench as a ready fallback.

⑤ 🎯 Two traps where the naive fix is worse — cut on relevance, not position

Both live-app problems have an obvious fix that quietly makes things worse; the craft is spotting that:

• Giant §-sections blow the token budget. Naive fix — keep the first few thousand chars — throws away late clauses like (a)(1)(iv). Real fix: select_context re-ranks ~500-char windows by similarity to the question and keeps the best up to ~1,250 tokens.
• Bare follow-ups ("what about at night?") lose their topic. Naive fix — always prepend the previous question — pollutes standalone queries (a "Class B/C" prefix once dragged a fuel question into airspace §§). Real fix: query contextualization prepends only for short follow-ups (≤5 words / cue words).

Full before/after diagrams for both are in 4B.4; the point to carry away is the pattern — cut and contextualize on meaning, never on position or reflex.

⑥ ⚙️ What keeps build-time and query-time in sync? → one source of truth

• CHAMPION_* — four lines (chunker=section · embed=bge · method=vector · K=5) that both indexer.py and the app read, so the index and the query can never silently disagree.
• Self-describing index: index.meta.json records the build model; a startup dimension probe fails loud (SystemExit) on a mismatch instead of returning confident garbage.
• Two frontends, one retrieval: the Flask/React backend and the Streamlit app share retrieve() and format_context(), so [n] points to the same chunk everywhere.

The reasoning behind each of these is expanded in 4B.6 · Why it's built this way.

⑦ 🎓 The honesty check — score the champion once on held-back data

Before shipping, the champion config (bge / vector / K5 + B_balanced) was run exactly once on the final split it had never seen: 3.5 / 4 — 2 correct, 1 correct refusal, and 1 grounded-partial where retrieval missed a § but the model did not hallucinate.

Did not retune on the final split. Hold out a set you touch exactly once, so the reported score isn't inflated by tuning to it — that's the difference between a real number and a flattering one.

• rag-starter/indexer.py — chunk → embed → save; owns CHAMPION_* and embed()
• harness/retrieval.py — retrieve() (vector / bm25 / hybrid), format_context(), select_context()
• rag-starter/backend/app.py — Flask /api/chat: retrieve → prompt → Claude → citations
• rag-starter/streamlit_app.py — same retrieval, Streamlit UI, streaming, live panel, inline source cards
• harness/agent_rag.py — the agentic-loop experiment (search_cfr tool), kept for comparison
• rag-starter/phase0_smoke.py — one-shot smoke test through the real /api/chat path
• index.pkl / index.meta.json — the self-describing index the two phases meet through