Same documents, same schema, different answers: space-ocr vs Mistral

space-ocr is our document reading API. You send a photo and a list of the values you want out of it, and you get those values back as data, each one carrying the exact spot on the page it was read from. This post is a comparison of it against Mistral Document AI, run on the documents that are actually hard.

Every OCR vendor's demo reads a clean printed invoice perfectly, ours included, and that tells you nothing about the decision you are making. Three questions do. What happens on awkward documents. Whether the answer changes when you run the same page twice. And how you find the wrong values without re-checking every one of them by hand. We measured all three, and everything below is in here, including the parts that don't flatter us.

What we ran

Eight scoring cases, 463 values in total, over seven photographs: three Japanese delivery slips, a supermarket receipt that prints an item's name on one line and its quantity and price on the next, a 24-row order sent as a chat message, a commission statement, and a 22-row wholesale price sheet photographed off a monitor. They are the image-bearing cases of our own regression suite, the set we use to catch ourselves breaking things.

Seven photographs but eight cases, because two of the cases are the same chat-order photo with the same answer key, kept separate in our suite to exercise different code paths. Both engines simply saw that image twice. It is worth knowing that one photo therefore accounts for 144 of the 463 values.

Three arms, all given identical images, an identical list of fields with identical descriptions, and the same instruction to copy what is printed:

Three runs per arm per document, 72 runs in total, all of them archived. Scoring is one script for all three arms: strip whitespace, compare to a hand-written answer key, exact match or nothing. August 2026.

The headline, and the caveat that goes with it

space-ocr read 91.1% of the 463 values to an exact match, averaged over three runs. Mistral's OCR annotations landed at 70.0%, its vision LLM at 73.5%.

Some of every engine's misses are cosmetic: a currency symbol kept, a unit suffix left on a number. So we also stripped out every mismatch that differed only in delimiters or spacing and counted content errors alone. Roughly 23 per run for space-ocr, against 119 for Mistral OCR and 93 for the vision LLM. The gap survives the cleanup.

One more thing before the numbers. The answer key was transcribed against our own pipeline's conventions for where one value ends and the next begins, so every live engine, ours included, loses points to boundary disagreements. Read the gap between the arms. The absolute scores are floors, not grades.

Every document, every score

Mean values correct out of the total, averaged over the three runs.

The top row is the one to look at first. On the cleanest document in the set, a dense printed table with 141 values, Mistral OCR scored exactly what we did. Reading characters is not where these engines differ, and any comparison that hides that is selling you something.

The bottom of the table is where they do differ, and the pattern is consistent: the harder the layout, the wider the gap. Note also the two chat-order rows. Same photo, same answer key, same schema, sent twice. We came out 3 values apart on the two passes. Mistral OCR came out 18 apart.

Run the same page three times

Totals out of 463, one column per run.

A 100-value spread is not noise around a mean, it is two different products. One Mistral OCR run returned 3 of 72 values on a document it had scored 58 of 72 on an hour earlier, because the annotation layer scrambled every column of a 43-row table. Nothing in the response distinguishes that run from a good one.

What the misses actually look like

Here is the two-line receipt, run 1, the document with the widest gap. The receipt prints the product name on one line and the quantity and unit price on the next, and the totals block sits directly underneath in the same column.

Read the last row before you read the rest. On the total, both engines returned the same thing and both are scored wrong, because the answer key has no currency symbol. That is the kind of miss that inflates every engine's error count, ours included, and it is why the content-only number above matters more than the raw one.

The rest of the column is a different failure. Mistral is not misreading characters there. It is stapling the shelf code to the product name and the price to the quantity line, because it never worked out that two printed lines are one record.

The 24-row chat order fails in a way that is worth seeing in full, because it is the failure mode that costs the most downstream:

Every row has slid up by one. Individually each value is a real string from the page, spelled correctly. As a table it is wrong from the second row down, and nothing about the response looks unusual.

Our own misses on that page were smaller but they were there: ヒチョウ came back as ヒチョウ背, 冷凍 as 冷凍 2L. We merged two adjacent cells instead of shifting the table. On delivery slip B we read a unit price of 1510 as 510, dropping a digit that the neighbouring column had glued on.

The part no score table can show

Ask both engines for the same 141 values on the price sheet and you get two answers back that look equally usable, until you ask where each value came from.

space-ocr returns a box per value, anchored to the pixels it was read from. In the Mistral responses we archived for this benchmark (mistral-ocr-latest, August 2026), the same price sheet came back as a single rectangle covering the whole table. Those responses carried no per-value coordinates, and a paragraph block was the finest granularity we saw in them. What the API returns today is worth checking against Mistral's current documentation before you design around either behaviour.

Those responses carried no verification signal either. Every space-ocr value arrives in data.cells[path]: a box and a quad on a 0 to 1000 grid, a verified verdict, a review object whose reasons array names in machine-readable codes what did not check out, and an evidence object holding the character cross-check itself (text_match), the match_ratio behind it and, when the recogniser can report one, ocr_confidence. The values that want a human are collected in one place, data.review.flagged. Declaring a label on a field anchors which occurrence to take when the same value repeats down a page. Read verified: true for what it is: the checks that ran found nothing, not a promise that the value is the one you wanted.

That is what changes the work. With boxes and a review list, values that are quietly wrong get surfaced, and a person starts from the handful that are flagged instead of re-reading all 141. It is not a net that catches everything, since a cross-check has nothing to raise when both readings agree on the same answer, which is exactly what the total row above looks like. But a flagged wrong value costs a glance, and an unflagged one costs whatever you built on top of it.

The whole set, both engines

All seven photographs, left and right the same image, with the boxes drawn straight from the archived responses of the benchmark's first run. Green is one space-ocr value, orange is one the engine flagged for review in that run, and blue is one Mistral OCR block. Third-party names, addresses, phone numbers, account numbers and staff names are pixelated before publishing, which is also why some boxes sit over grey rectangles.

What this does not measure

Field extraction, on eight cases, in August 2026. Not speed, not price, not Mistral's markdown conversion, which is a different product doing a different job and was not part of any run here. Not general document reading either: eight cases from one company's regression suite is a small, deliberately nasty sample, weighted towards Japanese business forms and photographs taken in bad light.

The raw material is all kept: the runner script, the 72 scored runs, every raw response from both vendors, and the script that drew the images above.

The most useful benchmark is still your own. Take five documents that give you trouble, write the answer key by hand first, send the same field list to both APIs, and run each of them three times. The steps below are what we did. A successful scan costs $0.05, failed scans are never charged, and the first 100 pages every month are free, so the running cost of checking this yourself is roughly nothing.