DocFieldBench
An open benchmark for structured extraction from documents.
Leaderboard
| 95% interval | |||
|---|---|---|---|
| Gemini 3.8 Flash | 94.6 | 92.9–96.1 | 7.2 s |
| DeepSeek v4.1 Flash | 93.2 | 91.3–94.9 | 3.0 s |
| Mistral Large 4 | 91.4 | 89.4–93.2 | 27.4 s |
| Refractr (live model) | 89.4 | 87.2–91.4 | 0.7 s |
| GPT-5.4 mini | 86.2 | 83.9–88.5 | 2.2 s |
| Ministral 8B | 85.5 | 83.0–87.7 | 3.3 s |
| Llama 3.1 8B | 73.9 | 70.5–77.2 | 0.5 s |
| NuExtract 2.0 (2B) | 58.6 | 54.7–62.4 | 1.4 s |
| GLiNER2 (base) | 24.2 | 20.8–27.6 | 0.5 s |
Refractr's time is measured on its own server (one RTX 3080 Ti). The hosted models' times include their API round trip.
Provider based in the European Union: Mistral AI in France, Refractr in the Netherlands. Under the GDPR, personal data in documents may leave the European Economic Area only with extra safeguards, such as an adequacy decision or standard contractual clauses. With a provider that is established and hosts in the EU, that question does not arise, which is why some organisations that handle EU personal data prefer or require one. NuExtract and GLiNER2 are open models run self-hosted here, so data stays wherever they are run.
Score against median time per document
What it measures
Every field of a document earns a point when it comes back right, including a field correctly left empty. A missing value earns nothing. A wrong or invented value given without warning costs a point.
The reason is how document pipelines fail. A wrong value that looks fine is stored without anyone noticing. A blank gets caught. The score punishes the first more than the second.
Scoring is deterministic. No LLM judges the answers.
Design choices
- Three tracks that weigh the same. 260 documents: 100 business documents (invoices, receipts, bank statements, delivery notes, payslips), 100 records (court opinions, Federal Register rules, SEC filings) and 60 US government forms. Each track counts for a third of the score, so the smaller forms track is not drowned out.
- Systems get the original file. Reading it, through a text layer, OCR or vision, is part of the test, because that is where a real pipeline starts.
- Scans and photos are reported separately. 86 documents also come as a scan or a phone photo. Those do not enter the headline score, so every document counts once.
- Some fields are absent on purpose. About one field in seven is not in its document. The right answer there is an empty field, which is how invented values are measured.
- Languages. Mostly English, plus German, Spanish, Dutch and French in the business track, where documents from several countries are the normal case.
- Answer key. Taken from the documents' own structured data or from human labels. No value in the key was generated by a model. Before release, 309 fields where a system disagreed with the key were reviewed blind by an LLM reviewer, and every change is logged in the repository (
ADJUDICATION_LOG.md). - One protocol for every system. Every system gets the same published prompt, one request per document, and the first reply is scored. Differences then come from the system, not from prompt tuning or retries.
- Recent documents. 31 documents were published in November 2026, after current models' training cutoffs, so they cannot have been seen in training.
Run it yourself
The documents, the answer key and the scoring code are in the repository. From a clone of it:
# from a clone of this repository: the documents and gold are in data/v1.0.0
pip install -e .
docfieldbench data verify data/v1.0.0 # every file is checked against the manifest
docfieldbench run data/v1.0.0/tasks --system my_system.json --out preds.jsonl
docfieldbench score data/v1.0.0/tasks preds.jsonl --summary summary.json
docfieldbench baseline data/v1.0.0/tasks --out all_null.jsonl # the all-null baseline, for comparison
- Submit a result. See
CONTRIBUTING.md. Results come in as GitHub issues with the predictions file attached, so anyone can rescore them. - Report an error in the answer key. Open an issue in the repository's issues.
How to cite
@misc{docfieldbench,
title = {DocFieldBench: an open benchmark for structured extraction from documents},
author = {{DocFieldBench contributors}},
year = {2026},
version = {1.0.0},
howpublished = {\url{https://github.com/Refractr-io/DocFieldBench}}
}
Version history
- v1.0.0, 18 November 2026. First public release: 260 documents in three tracks, plus 86 scan and photo variants.
DocFieldBench is built and maintained by Refractr, a document extraction API.