Scope

V1: this is the full scope of the data as of June 2026.

550
questions
30×
runs each
5
full models
82,500
graded cells
ModelAPI identifierProvider
Claude Opus 4.7claude-opus-4-7Anthropic
Claude Sonnet 4.6claude-sonnet-4-6Anthropic
GPT-5.5gpt-5.5-2026-04-23OpenAI
Grok 4.3grok-4.3xAI
Pixtral Largepixtral-large-2411Mistral
Gemini 3.1 Pro — 25 runs remaining, pending Tier 2 API permissioninggemini-3.1-pro-previewGoogle

Methodology

Each page is rendered as a full-resolution PNG and sent to a vision-language model with a fixed extraction prompt. The returned answer is then graded against a golden answer.

app interfaces websites emails PDFs etc Image vision-language model ANSWER: $200,000.00 text out

The Parameters

  • max_tokens: 2048
  • no temperature
  • no seed pinned
  • GPT-5.5: reasoning_effort none
  • base64-encoded PNG, full resolution
  • no preprocessing
  • no per-model tuning

The Prompt

Answer the question using only what is visible on the page. Be brief.

Return your answer in exactly this format:
ANSWER: <value>
EVIDENCE: <short verbatim phrase from the page>

If the answer is not on the page:
ANSWER: not found
EVIDENCE: none

Grading

The goal of each question is to single out a plain, intuitive piece of information on the image. Each returned answer was categorized as one of the following:

Exact match

Deterministic: answer and golden answer are normalized* and a direct equality check.

*lowercase, strip $ and commas, collapse whitespace

Human evaluated
Materially correct

Exact match fails, human review decides materiality. The approved answer is then stamped deterministically.

Materially wrong

Exact match fails, human review decides materiality. The identified error is then stamped deterministically.

Grading Rubric

The rubric is built around the most intuitive interpretation of each answer and we acknowledge that reasonable people can read the same answer differently:

Please submit your feedback on any Q&A you find issue with

Materiality

The boundary between materially correct and materially wrong generally looks like this:

Materially correct looks like
  • unit or prefix already in the question
  • glyph variant (en-dash, subscript, µ)
  • answer compressed
  • numeral vs spelled-out
  • article dropped or swapped
  • equivalent phrasing
  • title / honorific handling
  • answer expanded
  • magnitude in a parenthetical
  • list separator / conjunction
Materially wrong looks like
  • magnitude dropped or mangled
  • wrong value
  • wrong count
  • wrong category (e.g. NAYS vs YEAS)
  • unwarranted abstention
  • wrong label (on a diagram)
  • wrong spelling / mis-OCR
  • extraction failure (broken output)
  • catastrophic miss

Reliability Over Thirty Runs

Thirty runs provide a basis for evaluating accuracy, consistency, and output variability.

30/30 exact match
30/30 error
30/30 acceptable with variety
1 ≤ n < 30 answers error

Audit the data

Feel free to download the data. In it you will find: every question, golden answer, model answer, run index, verdict, and failure tag.

Download the full dataset (JSON)

Source document images are referenced by content hash.

Submit feedback