FAA RAG Contest · Build Log
Building a RAG Chatbot
for FAA Flight Rules
How the answer pipeline works, the one experiment I tried,
and why I shipped the simple design.
2,184
indexed law chunks (14 CFR)
bge · 1024d
embedding model
The idea
What is RAG? An open-book exam.
The model did not memorize the law. It answers from pages we hand it.
Open-book exam: the student (the model) knows nothing about aviation law.
A librarian reads the question, pulls the 5 most relevant pages,
and puts them on the desk. The student writes the answer using only those pages —
and cites them. No outside guessing. This stops hallucination.
RAG = Retrieval Augmented Generation.
First retrieve the right law text, then generate an answer grounded in it.
Chunk = one small piece of the law, cut so it is easy to search.
Our whole library is 2,184 chunks.
Why it fits this contest: answers must be cited and verifiable —
exactly what "answer only from the given pages" gives us.
The design I shipped
Static RAG — one search, one answer
A straight line. No loop. This is the deployed chatbot.
The heart of it: the model gets one message with the found pages,
and no search tool. So it cannot ask for more. One round trip → cost stays flat and
predictable (~2.3k tokens).
- Search is free. It is just math (comparing number vectors), done locally — $0 API. Only the single generate call costs money.
- Search and generate are separate. I can tune retrieval all I want without raising the API bill.
Under the hood
Inside one retrieval (real log)
Question: "make a table covering all airspace classes"
🔎 Embed
question → 1024 numbers,
compare vs 2,184 chunks
→
📑 Top 5
§71.1 · 71.33 · 71.31
· 71.71 · 71.61
→
✂️ Trim
10,420 → 6,235 chars
(~40% smaller)
→
✍️ Answer
grounded + cited
~1,558
tokens sent (14 passages)
⚠️ Look at the Top 5 — all from Part 71.
Part 71 only says where airspace is drawn, not the operating rules (those live in
Part 91). One search landed on the wrong Part, so the table came out with holes
(Class B / C left blank). A single search can't go back for more.
The experiments · offline
First I tuned search — not the agent
A deterministic grid search over the retrieval half. Same input → same result.
No agent, no model loop — just measuring.
Embedding
bge · gte
minilm · e5
×
Chunking
§ boundary
vs characters
×
Method
vector · bm25
· hybrid
×
🏆 Winner: § chunking + bge + hybrid + K8 →
coverage 0.864 (the right section is in the top 8, 86% of the time).
Value pick: bge + vector + K5 = 0.818 with far fewer tokens.
- Ensemble wins: hybrid 0.864 > meaning-only 0.773 > keyword-only 0.636.
- A good embedding saves tokens: bge hits 0.818 at K5; minilm needs K8 to match.
- § boundaries beat character chunks: the law is written in sections, so answers are too.
The deliberations
Three hard calls I had to make
The why behind each choice — the part I care about most.
① Kill the overnight run?
The character-chunk index froze at 495/792 — the process had swollen to
16 GB and was thrashing swap. Decision: kill it.
Why: the section run (my real winner) was already done and safe — append-only
kept all 495 lines. Character chunking was a free bonus, not worth cooking the Mac all night.
② Which embedding is #1?
Three models tied on coverage (0.864). And the "strong" model e5 came
last. Decision: bge (best MRR, 0.636).
Why: when coverage ties, rank quality + small-K efficiency decide. e5 losing even to
gte smelled like a usage bug (missing query:/passage: prefix), so I didn't throw it away.
③ Drop K from 8 to 5?
I measured real calls: output is a flat ~100 tokens, and 91–96% of cost is
input (K × passages). K8 = 5,136 vs K5 = 2,656. Decision: K5.
Why: K is the only real cost lever. Coverage dips 0.864→0.818 (tiny) but tokens
halve — a great trade for the 15-point Cost score.
One theme: every call was solved by measuring, not guessing — and by
weighing quality ↔ cost.
The experiment · online
Then I tried an agentic loop — great answers, huge cost
Give the model a search tool and let it search as many times as it wants.
✅ The win
On the airspace table, the agent searched 13 times. It noticed Part 71 was
only designations and searched Part 91 by itself — real self-correction.
It filled the complete B/C/D/E/G table.
❌ The disaster
Tokens exploded. Each turn re-sends the whole history, so 13 searches pile up.
The input grew to ~80k. That is the built-in cost of the loop.
81,320
agentic loop (tokens)
Why it grows so fast: the model has no memory, so every turn we
resend all earlier search results. Search N times → cost adds up like 1+2+3+…+N.
That is roughly N² (quadratic) — a few searches are fine, 13 is a bomb.
The decision
I rolled back to static RAG
Better answers weren't worth blowing the cost budget.
| Question type | Static RAG | Agentic | Quality |
|---|
| Specific (contest style) | 2,281 | 4,282 · 1 search | both correct |
| Broad ("all classes") | 2,949 · gaps | 81,320 · 13 searches | agentic wins |
- The rubric rewards cost. Cost Management is 15 points and literally says
"as few tokens as possible". 81k works against that.
- Contest questions are specific. 4 of 5 practice questions live in a single Part —
static answers them perfectly and cheaply.
- Agentic is kept, but capped. A hard limit of
MAX_SEARCHES = 3 keeps even
a broad question near ~15k. It waits on the bench as a ready backup.
Rule check: the contest never requires an agentic loop. It scores
results only (quality, citations, cost). Static RAG is fully compliant — and safest on cost.
Summary
Final setup & the decisions behind it
The build
- Embedding: bge (1024d) — best recall, wins with a small K
- Retrieval: § boundary chunking + hybrid/vector, K=5
- Generate: single-shot, one API call
- Citations: real § numbers, e.g.
§91.151 (Part 91)
Key decisions
- K 8→5: half the tokens, tiny recall loss
- Static over agentic: flat cost beats 27× spikes
- Cap the agent: 3 searches max, kept as backup
- Log every attempt: tokens/cost to
chat_attempts.jsonl
One theme across every choice: measure, don't guess — and always
weigh quality ↔ cost. That is how each decision here was made.