Epstein Bench Dataset Card

Current release: v1.0 (2026-07-07). This card documents the methodology; the Release statistics section is updated whenever a version ships.

Source corpus

• aurora2424/Epstein-Files (Hugging Face): the full public Epstein Files release, ~4.1M rows (340GB including raw media bytes). Only the text columns are consumed, via parquet column projection; rows without text_content (images, audio, video) are skipped. Emails, depositions, flight logs, scanned letters. The media columns (image, audio, video, online_url) open the door to multimodal task families later; v1 is text-only.
• Documents are quality-screened into clean / degraded / garbage. Tasks are generated only from clean text; degraded documents remain in the retrieval corpus as natural distractors; garbage is excluded entirely.

Task types

Corpus selection is entity-complete: a wide scan indexes entity mentions across the source dataset, an LLM notability check picks target people (public figures only; entities appearing in more than max_entity_docs documents are excluded as impractically pervasive), and the corpus is all documents mentioning any target plus a seeded random backbone. Single-hop facts are salience-filtered (newsworthiness ≥3/5: notable people, money flows, meetings/travel, legal exposure, never speculation; facts must be document-stated).

Generation is fact-first: atomic facts are extracted from clean documents and questions are written against the fact, in investigator phrasing. Aggregation questions are bounded, scoped to an entity whose candidate documents are enumerable via an alias index, because unbounded "list all X" gold sets cannot be verified at corpus scale.

Verification

Every shipped task passed all of:

1. Standalone: interpretable without the source document (concrete entities, no deixis, no boilerplate targets).
2. Answerability: an independent prompt, shown the gold documents, recovers the reference answer (semantic match + token-F1 floor; ≥80% item recovery for aggregation).
3. Necessity: closed-book and random-distractor attempts fail; for multi-document types, no single gold document suffices.
4. Adjudication: a stronger model passes/fails with a category.

Unanswerable tasks run stages 1 and 4, plus a generation-time absence check (top BM25 hits confirmed non-answering). All rejections are logged with the failing stage (build/rejected.jsonl).

False-premise tasks are anchored on entity-complete targets (so "no document supports this" is bounded) and fabricate an interaction between the target and a prominent outside figure, rotated across a diverse pool. They skip stages 2–3 and run a generation-time absence check plus a two-stage adjudication: a neutral support check that drops any premise the on-topic documents actually support (catching premises that merely perturb a detail of a real meeting), and a quality check for plausibility and that the wording does not reveal the premise is false.

Retrieval ground truth (pooled)

Gold document sets come from TREC-style pooling: the union of top-20 results from three diverse retrievers (BM25, dense embeddings, hybrid RRF) plus the source documents, each judged supports/partial/irrelevant. Gold = all 'supports' documents. A sample is re-judged by the strong model; tasks with supports↔irrelevant flips are dropped as unstable.

Limitation: pooled relevance sets are not exhaustive. A document outside the pool that happens to state the answer will be scored as non-gold. Pool composition is versioned with each release.

Models (pinned per release)

The scoring judge is a strong model because correctness judging approaches human agreement at that tier; generation and gauntlet filtering tolerate the cheaper model. Changing the scoring judge or its prompt is a new benchmark version; scores across versions are not comparable.

Release statistics

v1.0 (2026-07-07)

• Corpus: entity-complete selection over the full source dataset, a wide scan of all 634 parquet shards (~1.38M text-bearing docs) indexed entity mentions; 40 notable target people were chosen (LLM notability check) and the retrieval corpus is every document mentioning a target plus a 30,000-doc random backbone, for a corpus of 83,810 documents and 159,564 chunks.
• Tasks: 1,038 (full): 823 single_hop / 111 aggregation / 27 timeline / 7 dossier / 32 unanswerable / 38 false_premise. dev: 50. Multi-document types (timeline, dossier) survive verification at low rates, so they remain a small share.
• Gauntlet: 1,098 of 4,034 candidates passed (27%) in the original run. Rejections: 1,528 standalone, 1,080 answerability, 786 adjudication, 229 necessity. The 38 false_premise tasks were added from 39 candidates that passed the two-stage adjudication (1 dropped as non-standalone; earlier candidates dropped when the corpus supported the premise or the premise merely perturbed a real meeting).
• Pooling: 1,018 of 1,098 kept; 74 dropped as unstable under strong-model re-judging, 6 as source-not-supportive. One non-person dossier target was retracted, leaving 1,000 tasks, plus 38 false_premise (no pooling; empty gold set), for 1,038 released.
• Baselines: reference-system scores are not duplicated here; the leaderboard is canonical and the methodology analyzes the findings.
• Grounding check: an automated pass verifies that each gold answer appears in a gold document (verbatim or ≥60% token overlap): single_hop 99.0% (815/823), timeline 96.3% (26/27), with the residual traced to date-format matching artifacts. No answerable task lacks a gold document and no gold reference dangles. All 7 dossiers and ~20 sampled task/answer/ document triples were additionally reviewed by hand; independent third-party review remains a roadmap item.

Known limitations

• Questions are synthetic (LLM-written), human-spot-checked rather than fully human-authored.
• One cheap model (gpt-4o-mini) both drafts tasks and runs the answerability/necessity gauntlet stages, so the shipped tasks skew toward questions that model family finds answerable, and a consistent OCR misreading by that model can survive verification. The strong model enters only at adjudication and pool stability.
• Pooled (non-exhaustive) relevance judgments; see above. Because the headline metric is citation-gated against the pool, this limits correctness, not just retrieval recall: a right answer citing a supporting document outside the pool scores zero.
• Pool and gauntlet judges read a fixed-length excerpt of each document (2,500 chars in the v1.0 build; 3,000 in the current pipeline), while systems see full documents; a supporting passage past the cutoff can keep a relevant document out of the gold set.
• The absence check for unanswerable/false_premise verifies against the top BM25 hits for the question (strong-model adjudication re-reads the same documents), not the full pooled or entity-complete document set; a premise supported only by a document those probes miss could slip through.
• The v1.0 split is not regenerable end-to-end from the pipeline alone: one non-person dossier target was retracted by hand after pooling and the 38 false_premise tasks were appended, and the generator models are not bit-deterministic without the released LLM cache. The shipped files are pinned by dataset/v1.0/manifest.json.
• OCR noise in degraded documents is uncorrected by design: it is part of the haystack, never part of the answer key.
• The alias index driving aggregation/timeline bounding is heuristic; entities with unusual name forms may be under-covered.
• false_premise saturates on the headline (all systems refuse), so it does not drive the sort key; its informative signal is the premise-identification diagnostic, which separates the agentic systems (1.0) from the one-shot and no-context baselines (0.0).
• This is a self-run benchmark with a public answer key (tasks.jsonl ships beside questions.jsonl): CI proves committed scores follow from the predictions, but cannot prove the predictions never read the key.

Ethics

All source documents are public records. Tasks are generated exclusively from already-public text; no new personal information is synthesized or inferred. Retracted or erroneous tasks are removed in point releases (v1.x), never silently edited.