LongBook Verifier

Five retrieval strategies

Implemented and benchmarked head-to-head: naive first-context and last-context baselines, flat chunk RAG over the whole document, a chapter-summary chain (rank extractive chapter summaries, then retrieve chunks inside the best chapters), and hierarchical book RAG (score chapters → retrieve chunks within → expand to neighboring chunks).

Grounding measured separately from correctness

A gold-question benchmark (JSONL: gold answer, evidence terms, expected source location per question) scores evidence-term coverage and answer-term coverage independently — because a system can produce a correct-sounding answer without ever finding the evidence. Outputs of nine consumer and enhanced-access AI systems (ChatGPT, Claude, Gemini, Copilot, Grok variants) were scored on the same gold sets under separated, clearly-labeled protocols.

Retrieval — Experiment B (240k words)

Structure-aware retrieval beat the naive baseline 3.27× on context recall. Retrieval quality degraded measurably as the corpus grew from 64k to 240k words — retrieval strategy must be re-benchmarked per corpus, not assumed.

Experiment C — follow-up ablation. Because hierarchical book RAG underperformed even flat chunking, I ran a stage-level ablation to find out why. Removing neighbor expansion alone raised hierarchical recall from 0.3365 to 0.4771 (matching the chapter-summary chain), and forcing the correct chapter (oracle) raised it to 0.7844 with perfect hit@1. The loss came from two failure modes on this corpus — wrong first-stage chapter selection (27 of 80 questions) and neighbor-expansion dilution (32 of 80) — which supports, but refines, the error-compounding hypothesis. Multi-stage retrieval pipelines need stage-level ablation before their failures can be interpreted. Published separately: DOI 10.5281/zenodo.20692451.

Model outputs — the divergence

The two systems with perfect answer-term coverage had the worst evidence grounding. Confident, correct-sounding answers — weakest factual support. This failure mode is invisible unless grounding is measured separately from answer quality.

Free-tier protocol — Experiment A

64,000-word book · 40 gold questions · consumer free-tier access.

"Zero-evidence" counts answers containing none of the expected evidence from the book — ChatGPT and Grok produced perfectly-phrased answers (1.0000 answer coverage) while having the weakest grounding and the only zero-evidence answers in the set.

Enhanced-access protocol — Experiment B

240,767-word five-book corpus · 80 gold questions · full-access agentic runs.

Reasoning depth mattered more than brand: the same model family moved from 0.7281 to 0.8698 evidence coverage when thinking effort increased.

Every question in the benchmark is a gold question — written in advance with three things attached: the gold answer (what a correct answer must contain), a list of evidence terms (the specific facts and phrases that exist in the source text and prove the answer), and the expected source location (which chapter the evidence lives in).

• Answer-term coverage asks: does the system's answer contain the key terms of the gold answer? This measures whether the answer sounds correct and complete.
• Evidence-term coverage asks: does the answer contain the actual evidence from the book? This measures whether the answer is grounded in the source — not reconstructed from the model's general knowledge or invented.
• Zero-evidence questions count answers with none of the expected evidence at all — the most dangerous case: a fluent, confident answer with no factual support.
• Retrieval metrics (context precision / recall) are scored against the expected source location: did the retrieval step actually fetch the passages that contain the evidence?

The two coverages are reported separately because they diverge — that divergence is the central finding of this work, and it is exactly what an evaluation that only grades "answer quality" can never see.

• Grounding must be a first-class metric. A RAG system evaluated only on answer quality will ship hallucinations that read well.
• Document structure is leverage. Chapter/section-aware retrieval outperformed flat chunking on long documents — most real corpora (contracts, filings, manuals) have exploitable structure.
• Evaluation must be reproducible and local. Gold sets, versioned runs, and deterministic fallbacks make results auditable — and confidential data never leaves the environment.
• Cost and latency are part of the result. Every method is reported with latency and token estimates, because the best-scoring method is not automatically the deployable one.