What a self-verifying OCR pipeline is actually worth, measured
A controlled ablation of an OCR verification pipeline on 333 hand-labelled cells: turning every check off drops value accuracy from 93.7% to 91.3% and loses coordinates on 10 cells. 22 cells fixed, 0 broken — plus a measurement method you can run on your own pipeline.
"Self-verifying OCR" is a claim, and a claim you cannot switch off is not a measurement. So we switched ours off.
This article is the result: a controlled ablation of the verification and repair stages in the space-ocr engine — each stage switched off in isolation while every input stays frozen — scored against two truth sources that never touch each other. It walks through the numbers, the scoring method — no snapshots, no circularity — and exactly what each signal guarantees. If you are building extraction on top of an LLM, the method matters more than our numbers — the last section is about how to run it on your own pipeline.
The circularity problem
Most OCR regression suites score against a snapshot of a previous known-good run. That works for catching regressions and is worthless as evidence: if yesterday's output is the definition of correct, today's output agrees with it by construction. Our own suite reports bbox 318/318 = 1.000, and that number means nothing outside the suite — we flagged it internally as a circular metric and do not publish it.
Measuring whether verification works needs truth from somewhere the pipeline cannot reach. We used two sources:
- Values —
ground_truth.json, transcribed by hand from the source image. Made by a person looking at a photo, so the engine has no influence over it. - Coordinates — the page's own vision OCR. Read the vision words whose centre falls inside a returned box, concatenate them, and ask whether the returned value is spelled there. The snapshot is never consulted.
# The page's own OCR is the judge — no snapshot, no circularity.
def spells(box, vision_words, value):
inside = [
w["text"] for w in vision_words
if box["xmin"] <= (w["bbox"]["xmin"] + w["bbox"]["xmax"]) / 2 <= box["xmax"]
and box["ymin"] <= (w["bbox"]["ymin"] + w["bbox"]["ymax"]) / 2 <= box["ymax"]
]
return norm(value) in norm("".join(inside))
# A returned coordinate is wrong when the words under it do not spell
# the value that coordinate was attached to.
bad = [c for c in boxed_cells if not spells(c["bbox"], words, c["value"])]The second one is the interesting half. A returned coordinate is not "the box the matcher produced last month"; it is a claim that this value was read from this rectangle. The page can check that claim, because the page is covered in OCR text with known positions. If the words under the box don't spell the value, the coordinate is wrong — no snapshot required.
The corpus: 11 regression cases, 333 hand-labelled cells, 317 of which come back with a coordinate. Replayed from frozen vision and model output, so the measurement is deterministic — running it ten times gives the same numbers. It is our own fixture set, deliberately weighted toward hard documents, and it is not a public benchmark. Every number below carries that caveat.
Turning the pipeline off
The engine's post-model stages are individually switchable, so each one can be removed while the frozen model output stays byte-identical. Three matter here:
- Word-token anchoring — the model cites which OCR words on the page it read a value from; the engine checks that those words actually spell the value before trusting them for the location.
- Coordinate hint — a rough "the value is around here" location from a second model call, used to pick the right occurrence when the same value appears on the page more than once.
- Delimiter restore — when the model normalises what the page prints (
2025年09月05日returned as2025-09-05), the verbatim vision text is put back.
| Configuration | Value accuracy (vs. hand-labelled truth) | Cells with a box | Box actually spells the value |
|---|---|---|---|
| Full pipeline | 312/333 = 93.7% | 317/333 | 292/317 = 92.1% |
| Everything off | 304/333 = 91.3% | 307/333 | 278/307 = 90.6% |
Aggregate numbers hide direction — "+2.4 points" could equally mean twenty cells fixed and ten quietly broken. So the comparison is per cell:
| Stage removed | Values saved | Values broken | Coordinates saved | Coordinates broken |
|---|---|---|---|---|
| Word-token anchoring | 0 | 0 | 10 | 0 |
| Coordinate hint | 0 | 0 | 2 | 0 |
| Delimiter restore | 8 | 0 | 2 | 0 |
| All three | 8 | 0 | 14 | 0 |
22 cells fixed, 0 cells broken. Not one cell in the corpus got worse when a verification stage was switched on. That is the result we actually care about, and it is stronger than the headline percentages: a repair stage that fixes eight cells while quietly corrupting three is a bad trade even when the average improves.
Where the eight repaired values came from. All eight are the delimiter class: three delivery_date, two subtotal, two total, one line-item unit_price. The model had normalised what the page prints — 2025年09月05日 became 2025-09-05, ¥1,451 became 1451. The digits were never in doubt; the rendering was. Restoring the verbatim vision text is only safe because the check is narrow: identical digit sequence, both sides' non-digit characters on a whitelist, interior currency marks rejected, and comma grouping has to be plausible. One candidate was refused by that last guard during the original rollout, which is the behaviour you want from a repair stage.
Do the flags point at real problems?
Every boxed cell comes back with text_verified (does the vision text under the box agree with the model's value?) and needs_review (should a human look at this?). The obvious question is whether those flags land on the cells that are actually wrong.
Define a problem cell as one where the value disagrees with the hand transcription or the coordinate does not spell its own value. That's 39 of 333 cells — an 11.7% base rate.
| Signal | What the measurement shows |
|---|---|
text_verified: false | 75% precision (6/8) — 6.4× the 11.7% base rate. When the two engines disagree about what the box says, three times out of four something really is wrong there |
needs_review | catches 40% of coordinate errors (10/25) — a recall number a snapshot-scored suite cannot measure at all |
text_verified: false is the sharp instrument: few cells carry it, and one that does deserves a look. needs_review casts wider by design — a routing signal built to over-include rather than stay silent, which is exactly what you want feeding a review queue.
Stating the guarantee precisely
Reading these numbers right requires stating the guarantee precisely. Cross-checking guarantees that a box and its text agree — that the value you received was really read from the place its coordinate points at. Whether the right field was read — which column a unit comes from, whether an honorific belongs in a name — is a semantic question, and the right tool for it is schema validation and business rules on top of the structured output. The two layers are complementary: coordinates make every value auditable, schemas make the record consistent. Production pipelines run both.
Live numbers, for scale
The corpus is small and deliberately hard — the routing pays off on real traffic. Over 2,165 production requests, 9,685 fields came back and 97.2% carried coordinates. The flags did their job as a router: roughly four fields in five arrived with no review signal at all, ready to auto-accept, while needs_review concentrated human attention on the remaining fifth. Latency sat at a p50 of 7.2 s and a p90 of 10.5 s over 262 requests on one representative day — observed distributions from our own logs, not a service level agreement.
Running this on your own pipeline
The transferable part is not our percentages, it is the shape of the measurement:
- Never let yesterday's output define correct. If your regression suite compares against a stored run, it can tell you what changed and nothing about whether you are right.
- Find truth the pipeline cannot influence. Hand transcription for values. For coordinates, your OCR layer already gives you positioned text for the whole page — that is a free, independent judge for every box you emit, and almost nobody uses it that way.
- Score per cell, not per corpus. Averages hide trade-offs. "Fixed 22, broke 0" is a much stronger statement than "+2.4 points", and it is the one that tells you whether to ship.
- Replay frozen inputs. Freeze the vision output and the model response, then toggle only the stage under test. A live A/B of an LLM stage measures LLM variance more than it measures your change.
- Pair it with schema validation. Character-level cross-checking guarantees provenance — that each value came from where its coordinate says. Semantic constraints (totals that add up, dates in range, required fields present) guarantee the record. The pipelines that hold up in production run both.
If you want to see the signals this article is about, POST /ocr/fields returns bbox, vertices, bbox_source, text_verified and needs_review on every value, plus a review_summary naming the flagged paths. The bounding-box validation walkthrough covers how to gate on them in an application.
Why not score coordinates against a saved snapshot?
What does 'fixed 22 cells, broke 0' mean exactly?
Does text_verified: false mean the value is wrong?
Does this replace schema validation or human review?
Corpus and date for every number above: 11-case internal regression corpus, 333 hand-labelled cells, cached replay, July 2026. Live figures: production logs, 15–27 July 2026. Not a public benchmark, and not comparable to vendor benchmark tables.