What a self-verifying OCR pipeline is actually worth, measured
A controlled ablation of an OCR verification pipeline on 333 hand-labelled cells: turning every check off drops value accuracy from 93.7% to 91.3% and loses coordinates on 10 cells. 22 cells fixed, 0 broken — plus a measurement method you can run on your own pipeline.
"Self-verifying OCR" is a claim, and a claim you cannot switch off is not a measurement. So we switched ours off.
This article is the result: a controlled ablation of the verification and repair stages in the space-ocr engine — each stage switched off in isolation while every input stays frozen — scored against two truth sources that never touch each other. It walks through the numbers, the scoring method — no snapshots, no circularity — and exactly what each signal guarantees. If you are building extraction on top of an LLM, the method matters more than our numbers — the last section is about how to run it on your own pipeline.
A note on names. This measurement was run in July 2026 against engine v52/v53, and the signal names used below are the ones the API returned then. The public response has since moved to its v2 shape, so the same signals arrive today under these names.
| Named here (2026-07) | In today's POST /ocr/fields |
|---|---|
bbox | data.cells[path].box — {xmin, ymin, xmax, ymax} on a 0–1000 normalized grid |
vertices | quad — four points, always returned alongside box |
bbox_source | evidence.source |
text_verified | evidence.text_match — the character cross-check itself. The top-level verified has been a verdict mirroring review since engine v82: false when anything is flagged, true when a check ran and nothing was flagged, null when there was nothing to check |
needs_review | cells[path].review — null or {reasons}. The document-wide list is data.review.flagged[{path, reasons}] |
review_summary | data.review — {unit, declared, returned, boxed, verified, flagged, by_reason, notes} |
crop_verified / crop_mismatch | unchanged: evidence.crop_verified, and crop_mismatch as a review reason |
Nothing below has been re-run or re-scored. Only the names on the wire changed.
The circularity problem
Most OCR regression suites score against a snapshot of a previous known-good run. That works for catching regressions and is worthless as evidence: if yesterday's output is the definition of correct, today's output agrees with it by construction. Our own suite reports bbox 318/318 = 1.000, and that number means nothing outside the suite — we flagged it internally as a circular metric and do not publish it.
Measuring whether verification works needs truth from somewhere the pipeline cannot reach. We used two sources:
- Values —
ground_truth.json, transcribed by hand from the source image. Made by a person looking at a photo, so the engine has no influence over it. - Coordinates — the page's own vision OCR. Read the vision words whose centre falls inside a returned box, concatenate them, and ask whether the returned value is spelled there. The snapshot is never consulted.
# The page's own OCR is the judge — no snapshot, no circularity.
def spells(box, vision_words, value):
inside = [
w["text"] for w in vision_words
if box["xmin"] <= (w["bbox"]["xmin"] + w["bbox"]["xmax"]) / 2 <= box["xmax"]
and box["ymin"] <= (w["bbox"]["ymin"] + w["bbox"]["ymax"]) / 2 <= box["ymax"]
]
return norm(value) in norm("".join(inside))
# A returned coordinate is wrong when the words under it do not spell
# the value that coordinate was attached to.
bad = [c for c in boxed_cells if not spells(c["bbox"], words, c["value"])]The second one is the interesting half. A returned coordinate is not "the box the matcher produced last month"; it is a claim that this value was read from this rectangle. The page can check that claim, because the page is covered in OCR text with known positions. If the words under the box don't spell the value, the coordinate is wrong — no snapshot required.
The corpus: 11 regression cases, 333 hand-labelled cells, 317 of which come back with a coordinate. Replayed from frozen vision and model output, so the measurement is deterministic — running it ten times gives the same numbers. It is our own fixture set, deliberately weighted toward hard documents, and it is not a public benchmark. Every number below carries that caveat.
Turning the pipeline off
The engine's post-model stages are individually switchable, so each one can be removed while the frozen model output stays byte-identical. Three matter here:
- Word-token anchoring — the model cites which OCR words on the page it read a value from; the engine checks that those words actually spell the value before trusting them for the location.
- Coordinate hint — a rough "the value is around here" location from a second model call, used to pick the right occurrence when the same value appears on the page more than once.
- Delimiter restore — when the model normalises what the page prints (
2025年09月05日returned as2025-09-05), the verbatim vision text is put back.
| Configuration | Value accuracy (vs. hand-labelled truth) | Cells with a box | Box actually spells the value |
|---|---|---|---|
| Full pipeline | 312/333 = 93.7% | 317/333 | 292/317 = 92.1% |
| Everything off | 304/333 = 91.3% | 307/333 | 278/307 = 90.6% |
Aggregate numbers hide direction — "+2.4 points" could equally mean twenty cells fixed and ten quietly broken. So the comparison is per cell:
| Stage removed | Values saved | Values broken | Coordinates saved | Coordinates broken |
|---|---|---|---|---|
| Word-token anchoring | 0 | 0 | 10 | 0 |
| Coordinate hint | 0 | 0 | 2 | 0 |
| Delimiter restore | 8 | 0 | 2 | 0 |
| All three | 8 | 0 | 14 | 0 |
22 cells fixed, 0 cells broken. Not one cell in the corpus got worse when a verification stage was switched on. That is the result we actually care about, and it is stronger than the headline percentages: a repair stage that fixes eight cells while quietly corrupting three is a bad trade even when the average improves.
Where the eight repaired values came from. All eight are the delimiter class: three delivery_date, two subtotal, two total, one line-item unit_price. The model had normalised what the page prints — 2025年09月05日 became 2025-09-05, ¥1,451 became 1451. The digits were never in doubt; the rendering was. Restoring the verbatim vision text is only safe because the check is narrow: identical digit sequence, both sides' non-digit characters on a whitelist, interior currency marks rejected, and comma grouping has to be plausible. One candidate was refused by that last guard during the original rollout, which is the behaviour you want from a repair stage.
Do the flags point at real problems?
Every boxed cell comes back with text_verified (does the vision text under the box agree with the model's value?) and needs_review (should a human look at this?). The obvious question is whether those flags land on the cells that are actually wrong.
Define a problem cell as one where the value disagrees with the hand transcription or the coordinate does not spell its own value. That's 39 of 333 cells — an 11.7% base rate.
| Signal | What the measurement shows |
|---|---|
text_verified: false | 75% precision (6/8) — 6.4× the 11.7% base rate. When the two engines disagree about what the box says, three times out of four something really is wrong there |
needs_review | catches 40% of coordinate errors (10/25) — a recall number a snapshot-scored suite cannot measure at all |
text_verified: false is the sharp instrument: few cells carry it, and one that does deserves a look. needs_review casts wider by design — a routing signal built to over-include rather than stay silent, which is exactly what you want feeding a review queue.
Stating the guarantee precisely
Reading these numbers right requires stating the guarantee precisely. Cross-checking guarantees that a box and its text agree — that the value you received was really read from the place its coordinate points at. Whether the right field was read — which column a unit comes from, whether an honorific belongs in a name — is a semantic question, and the right tool for it is schema validation and business rules on top of the structured output. The two layers are complementary: coordinates make every value auditable, schemas make the record consistent. Production pipelines run both. Part of that second layer can now be declared in the request itself — required surfaces as missing, min/max as out_of_range, pattern and enum as pattern_mismatch, type as type_mismatch, and near as near_mismatch in review.flagged — though the declarations are never shown to the model, so the values come back unchanged and only the review signal grows.
Live numbers, for scale
The corpus is small and deliberately hard — the routing pays off on real traffic. Over 2,165 production requests, 9,685 fields came back and 97.2% carried coordinates. The flags did their job as a router: roughly four fields in five arrived with no review signal at all, ready to auto-accept, while needs_review concentrated human attention on the remaining fifth. Latency sat at a p50 of 7.2 s and a p90 of 10.5 s over 262 requests on one representative day — observed distributions from our own logs, not a service level agreement.
Running this on your own pipeline
The transferable part is not our percentages, it is the shape of the measurement:
- Never let yesterday's output define correct. If your regression suite compares against a stored run, it can tell you what changed and nothing about whether you are right.
- Find truth the pipeline cannot influence. Hand transcription for values. For coordinates, your OCR layer already gives you positioned text for the whole page — that is a free, independent judge for every box you emit, and almost nobody uses it that way.
- Score per cell, not per corpus. Averages hide trade-offs. "Fixed 22, broke 0" is a much stronger statement than "+2.4 points", and it is the one that tells you whether to ship.
- Replay frozen inputs. Freeze the vision output and the model response, then toggle only the stage under test. A live A/B of an LLM stage measures LLM variance more than it measures your change.
- Pair it with schema validation. Character-level cross-checking guarantees provenance — that each value came from where its coordinate says. Semantic constraints (totals that add up, dates in range, required fields present) guarantee the record. The pipelines that hold up in production run both.
If you want to see the signals this article is about, today's POST /ocr/fields returns them under the v2 names: every value has an entry in data.cells[path] carrying box and quad (0–1000), a verified verdict, review (null or {reasons}), and evidence (text_match, source, match_ratio, crop_verified, …), while data.review.flagged[{path, reasons}] lists every flagged path. The bounding-box validation walkthrough covers how to gate on them in an application.
Postscript: the feature this measurement paid for
A measurement rig earns its keep when the next idea shows up. Shortly after this measurement, a proposal landed: once the pipeline has finished, crop every returned box out of the image and read it again — OCR tends to read a small crop better than a crowded page — and stack that second read as one more layer of verification. Plausible; but so were two boring explanations — stale cached OCR in the fixtures, or a scoring illusion from the padding around each crop. So before writing a line of pipeline code we split them apart, the same way as above: three arms over the image-bearing fixtures, 94 cells — 24 the cross-check could not confirm, 10 silently wrong, 60 clean controls.
The crop effect is real: re-reading confirmed 16 of the 24 unconfirmed cells (66.7%), and none of it came from the confounders — scoring the same padded region against the full-page words recovered zero, and a fresh full-page run on today's models still spelled 0 of 24. The control arm set the price: scored verbatim, 6 of 60 clean cells (10%) would have been falsely doubted, and four of those were empty reads on tiny single-glyph crops. Upscaling those crops 3× recovered none of them — what a lone glyph lacks is context, not pixels — so the shipped rule treats an empty read as an abstention rather than a mismatch, which brings false alarms down to 3.3%.
The experiment shipped as crop re-verification (engine v53), shaped by its own numbers: only cells the cross-check could not confirm are sent, so a clean request adds zero extra reads; the crops go out as one batched call per image (6.8× faster than reading them one by one); a confirming crop promotes text_verified to true, a contradicting one flags the cell with the crop_mismatch review reason, and an empty one abstains and writes nothing — the result rides in the response as crop_verified. It never rewrites a value or moves a box. And it is a re-reading check, not a re-locating one: a value anchored to the wrong occurrence of the same text passes it by definition — 8 of the 10 silently-wrong cells did — so that class stays with the occurrence guard. Fixture caveat as always: 7 image-bearing cases, 94 cells, July 2026.
Why not score coordinates against a saved snapshot?
What does 'fixed 22 cells, broke 0' mean exactly?
Does text_verified: false mean the value is wrong?
Does this replace schema validation or human review?
Corpus and date for every number above: 11-case internal regression corpus, 333 hand-labelled cells, cached replay, July 2026. Live figures: production logs, 15–27 July 2026. Crop re-verification figures: the 7 image-bearing cases of the same corpus, 94 cells, July 2026. Not a public benchmark, and not comparable to vendor benchmark tables.