Reading documents with AI: what accuracy to expect

Document understanding is the most reliably valuable AI application in most businesses, and the most commonly mis-specified. The problem is usually that the conversation starts with an accuracy percentage, which is not a useful unit.

Why one accuracy number is misleading

Accuracy varies enormously by field and by document type in the same process:

• Structured fields on consistent layouts — an invoice number in the same position on a supplier’s standard template — perform very well.
• Numbers that can be cross-checked — totals that must equal the sum of lines — can be validated by rules regardless of what the model read.
• Dates are read well and interpreted badly, because format ambiguity is a genuine problem rather than a model failure.
• Line items on variable layouts are harder, particularly where a table runs across pages.
• Free text — descriptions, notes, handwritten annotations — is the weakest, and often the least important.

A vendor’s headline figure is an average across a benchmark set that does not resemble your documents.

The measure that matters

Straight-through rate at an acceptable error rate. What proportion of documents can be processed with no human involvement, while errors stay inside a tolerance you have defined?

That framing forces the useful questions. What is the tolerance? What does a wrong answer cost? Which fields must be right and which are conveniences?

Set the threshold from the consequence

Confidence thresholds decide what goes through and what a person sees. The threshold should come from the cost of being wrong, not from a target.

Where an error is caught downstream and costs a correction, a lower threshold and higher throughput makes sense. Where it results in a payment, a commitment, or a safety-relevant instruction, set it high and accept more exceptions. Different fields in the same document can reasonably have different thresholds.

Validate with rules, not just with the model

A great deal of accuracy comes from checks that have nothing to do with AI. Does the total match the sum of the lines? Is the supplier one we have? Is the date within a plausible range? Does the purchase order exist and have remaining value?

These catch errors regardless of how the field was read, and they are cheap. Design them alongside the extraction, not afterwards.

Design the exception path first

Whatever the threshold, a proportion of documents will need a person. That path has to be fast: the document on screen, the extracted values beside it, the uncertain fields marked, and a single action to correct and proceed.

Where the exception path is clumsy, the whole process is slower than it was before, even when the model is performing well. This is the most common reason a technically successful implementation disappoints.

Measure your baseline before you compare

Manual data entry has an error rate. It is rarely measured and it is never zero. Before setting a target, sample your current process and find out what it actually is.

Most organizations discover the comparison is more favourable than they assumed — and, more usefully, that they now have a number to improve against rather than an expectation of perfection.