A retrieval benchmark over the Epstein Files
Epstein Bench
Steve Bannon emailed Jeffrey Epstein eight words: “we r on the cusp of real power.”
He sent it at 10:22 on a Monday morning in February 2018. It sits somewhere in the roughly 3.5 million pages of records released by Congress and the Justice Department, between flight logs, scanned bank statements, and endless OCR sludge. We distilled the release into an 84,000-document haystack and asked the best AI systems in the world to find that email. The best one succeeds a little more than half the time.
This is the scoreboard.
The current high score, set by the Claude Sonnet 5 agent at about $0.05 a question: 54% answered correctly with a valid citation. It gets the other 46% wrong.
Most retrieval benchmarks quiz AI on clean text. The Epstein Files are the opposite. This benchmark distills that release into an 84,000-document haystack of garbled scans, near-duplicate emails, and legalese, plus 1,038 questions a system can only answer by finding the right document and citing it.
From the files
Real questions from the benchmark. Each answer is a sentence hiding somewhere in the corpus, quoted exactly as the documents record it.
Hover or tap the black bars to unredact.
The hardest questions ask a system to reconstruct the entire documented timeline of one person’s contacts with Epstein, scattered across dozens of files. Our best baseline scores near zero. the dossier tasks, 7 of 1,038
The scoreboard
full split, dataset v1.0, 1,038 tasks. Scored as
cited answer correctness: the judged answer has to match the reference and
cite a gold document. Higher is better.
| # | system | correct & cited | 95% CI | $ / task |
|---|---|---|---|---|
| 1 | Claude Sonnet 5 agentreference | 53.8% | 50.4 to 57.3 | $0.053 |
| 2 | Hybrid retrieval (BM25 + dense)reference | 49.0% | 45.4 to 53.3 | n/a |
| 3 | Claude Opus 4.8 agentreference | 48.8% | 45.9 to 52.0 | $0.114 |
| 4 | BM25 keyword searchreference | 48.3% | 44.5 to 52.2 | n/a |
| 5 | Dense embeddingsreference | 43.4% | 39.8 to 46.8 | n/a |
| — | Closed bookcontrol | 32.8% | 31.8 to 33.3 | n/a |
| — | Parametric probecontrol | 30.4% | 28.0 to 32.4 | n/a |
The two controls answer from the question alone and never see the documents. Nearly all of their credit comes from correctly refusing the trap questions; on every question that requires the documents, they score zero. They sit on the board as a floor, unranked.
Full metrics: per-type scores, citation precision, retrieval recall, tokens
| system | overall | micro | uncited | single_hop | aggregation | timeline | dossier | unanswerable | false_premise | cit. prec | recall@5 | recall@20 | tok/task |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| agentic-sonnet-5 (reference) | 0.538 | 0.532 | 0.607 | 0.535 | 0.286 | 0.407 | 0.000 | 1.000 | 1.000 | 0.485 | 0.250 | 0.321 | 14,787 |
| hybrid (reference) | 0.490 | 0.488 | 0.541 | 0.486 | 0.313 | 0.296 | 0.032 | 0.812 | 1.000 | 0.483 | 0.272 | 0.594 | n/a |
| agentic-opus-4-8 (reference) | 0.488 | 0.460 | 0.598 | 0.463 | 0.170 | 0.296 | 0.000 | 1.000 | 1.000 | 0.368 | 0.272 | 0.357 | 19,586 |
| bm25 (reference) | 0.483 | 0.475 | 0.518 | 0.476 | 0.247 | 0.333 | 0.000 | 0.844 | 1.000 | 0.462 | 0.271 | 0.589 | n/a |
| dense (reference) | 0.434 | 0.393 | 0.481 | 0.383 | 0.217 | 0.185 | 0.000 | 0.844 | 0.974 | 0.406 | 0.219 | 0.517 | n/a |
| closed_book (reference) | 0.328 | 0.066 | 0.346 | 0.000 | 0.000 | 0.000 | 0.000 | 0.969 | 1.000 | 0.000 | 0.000 | 0.000 | n/a |
| parametric (reference) | 0.304 | 0.062 | 0.346 | 0.000 | 0.000 | 0.000 | 0.000 | 0.875 | 0.947 | 0.000 | 0.000 | 0.000 | n/a |
overall is the macro-average across task types; micro is the task-weighted average. The two diverge for the controls, which score only on the rejection types. uncited is correctness with the citation gate removed. false_premise rewards rejecting a fabricated meeting. Raw numbers in leaderboard.json.
How scoring works
An answer counts only if two things hold at once. A pinned LLM judge has to rule it equivalent to the reference, and at least one cited document has to belong to the pooled gold set. A right answer with no supporting citation scores zero. Confident prose gets no partial credit.
We also run the whole benchmark with the documents taken away. Given nothing but the question, a closed-book model answers under 2 percent of the single-hop questions correctly, and the model that wrote the questions manages about 5 percent from its training weights alone. That number is the contamination gauge. As future models train on this public release it will climb, and we will be tracking it.
The set also includes 38 questions built on a meeting that never happened. Every system we tested refuses them, so the trap itself is saturated. The difference shows up one level down: the agents name the specific fabricated claim, and nothing else can.
Every task survived a four-stage verification gauntlet before release. Full details in the methodology and dataset card.
Add your system
The rows above are reference baselines we built so a new system has something to measure against. Every score is recomputed from a submission’s raw predictions before it lands on the board.
Run your system on the full split, open a pull request with your predictions, and get on the board.