DocFieldBench

An open benchmark for structured extraction from documents.

v1.0.0 · 18 November 2026 · 260 documents · Apache-2.0 code · github.com/Refractr-io/DocFieldBench

Leaderboard

95% interval
Gemini 3.8 Flash 94.6 92.9–96.1 7.2 s
DeepSeek v4.1 Flash 93.2 91.3–94.9 3.0 s
Mistral Large 4 Provider based in the European Union 91.4 89.4–93.2 27.4 s
Refractr (live model) Provider based in the European Union 89.4 87.2–91.4 0.7 s
GPT-5.4 mini 86.2 83.9–88.5 2.2 s
Ministral 8B Provider based in the European Union 85.5 83.0–87.7 3.3 s
Llama 3.1 8B 73.9 70.5–77.2 0.5 s
NuExtract 2.0 (2B) 58.6 54.7–62.4 1.4 s
GLiNER2 (base) 24.2 20.8–27.6 0.5 s

Refractr's time is measured on its own server (one RTX 3080 Ti). The hosted models' times include their API round trip.

Provider based in the European Union Provider based in the European Union: Mistral AI in France, Refractr in the Netherlands. Under the GDPR, personal data in documents may leave the European Economic Area only with extra safeguards, such as an adequacy decision or standard contractual clauses. With a provider that is established and hosts in the EU, that question does not arise, which is why some organisations that handle EU personal data prefer or require one. NuExtract and GLiNER2 are open models run self-hosted here, so data stays wherever they are run.

Score against median time per document

Score against median time per document. Gemini 3.8 Flash: score 94.6, 7.2 s. DeepSeek v4.1 Flash: score 93.2, 3.0 s. Mistral Large 4: score 91.4, 27.4 s. Refractr (live model): score 89.4, 0.7 s. GPT-5.4 mini: score 86.2, 2.2 s. Ministral 8B: score 85.5, 3.3 s. Llama 3.1 8B: score 73.9, 0.5 s. NuExtract 2.0 (2B): score 58.6, 1.4 s. GLiNER2 (base): score 24.2, 0.5 s.

What it measures

Every field of a document earns a point when it comes back right, including a field correctly left empty. A missing value earns nothing. A wrong or invented value given without warning costs a point.

The reason is how document pipelines fail. A wrong value that looks fine is stored without anyone noticing. A blank gets caught. The score punishes the first more than the second.

Scoring is deterministic. No LLM judges the answers.

Design choices

Run it yourself

The documents, the answer key and the scoring code are in the repository. From a clone of it:

# from a clone of this repository: the documents and gold are in data/v1.0.0
pip install -e .
docfieldbench data verify data/v1.0.0                             # every file is checked against the manifest
docfieldbench run data/v1.0.0/tasks --system my_system.json --out preds.jsonl
docfieldbench score data/v1.0.0/tasks preds.jsonl --summary summary.json
docfieldbench baseline data/v1.0.0/tasks --out all_null.jsonl   # the all-null baseline, for comparison

How to cite

@misc{docfieldbench,
  title        = {DocFieldBench: an open benchmark for structured extraction from documents},
  author       = {{DocFieldBench contributors}},
  year         = {2026},
  version      = {1.0.0},
  howpublished = {\url{https://github.com/Refractr-io/DocFieldBench}}
}

Version history

DocFieldBench is built and maintained by Refractr, a document extraction API.