Mistral OCR vs space-ocr, round two: the photos nobody benchmarks
The controlled space-ocr vs Mistral OCR benchmark, extended to 15 cases, 14 documents, 769 fields: field accuracy, run-to-run stability, a vision LLM collapsing on rotated documents, and a receipt that prints ¥711 on four lines.
Earlier this month we published a controlled benchmark against Mistral Document AI. It used hard documents. But production sends hard photos: paper that is rotated, crumpled, stacked, stamped, and shot one-handed at night.
So we added seven cases of exactly that, and re-ran everything: 15 cases over 14 documents, 769 fields, 3 runs per arm, same schema and scorer for every arm. This time the space-ocr arm runs through the public API — the same POST /ocr/fields any account calls.
| 769 fields · 3 runs | space-ocr | Mistral OCR | Mistral vision LLM |
|---|---|---|---|
| Field accuracy (exact match) | 89.2% | 69.3% | 66.8% |
| Score spread across 3 identical runs | 6 fields | 106 fields | 53 fields |
Two reading notes. The ground truth freezes one notation, so absolute scores are a lower bound for every arm — read the gaps, not the absolutes. And the spread row matters more than it looks: run the same input twice and one engine gives you a reviewable diff, the other gives you noise.
Where the gap opens
On the cleanest document the engines tie. Every point of the gap comes from structure and shooting conditions:
| Score by condition | space-ocr | Mistral OCR | Mistral vision LLM |
|---|---|---|---|
| Clean dense printed table | 1.000 | 0.986 | 0.960 |
| Receipt with two-line items | 0.718 | 0.179 | 0.077 |
| Dot-matrix slip, photographed sideways | 0.872 | 0.778 | 0.217 |
| Pile of carbon-copy slips, sideways + stamp | 0.833 | 0.611 | 0.130 |
| Night shot on a clipboard | 0.831 | 0.488 | 0.441 |
| Crumpled note in low light | 0.912 | 0.898 | 0.830 |
The sharpest line in that table: the vision LLM collapses on rotation — 0.217 and 0.130 on the two sideways documents, while the arms with a real OCR front end kept reading (0.611–0.872). An OCR pass detects and normalizes orientation before reading; a raw vision pass gets the rotated pixels as-is. Two documents is not a law of nature, so we say it narrowly: in this corpus, that is what happened, three runs out of three.
One receipt, four ¥711s
The last new case makes an abstract argument concrete. On this Seven-Eleven receipt the string 711 appears in eight places — the total, the iD payment, the charged amount and a tax line are all ¥711, and 711 also hides in the phone number, the 07:11 timestamp, the dates and the slip number.
Ask for total, id_payment and amount, and a correct engine returns the same three characters three times. Values cannot distinguish themselves; only a coordinate can say which line each one came from. In all three runs, all three fields came back with boxes on their own correct lines — identical coordinates every run. A values-only response has no way to even express which line it read. Right or wrong, it looks the same.
The corpus, in one gallery
Every new photo from this run. Pick one from the list — every green box is a coordinate the engine actually returned, and business details are pixelated by us.
What still gets through
The verification layer this API ships checks sourcing: that box and text agree, character by character, between two independent readers. On these hard photos, most of the errors that survived were the class we have always documented as out of scope — both readers agreeing on the same misread of a glare-bleached or blurred glyph. A cross-check cannot catch a consensus error. That is why review flags route human attention rather than replace it, and why schema and business-rule validation still belong on top.
Caption for every number here: 15 cases over 14 documents · 769 fields · 3 runs each · August 2026 · same images, schemas and grading script for every arm · space-ocr measured through the public API. Corpus leans toward dense Japanese business documents. New photos carry real business details, so they are published only in the pixelated form above. Field extraction only — no claims about markdown conversion, speed or price.
The original write-up has the full method and raw outputs for the published set; measuring the verification layer itself is a separate post. To see what your own worst photo does, the landing page demo reads one without an account.