How to Test Document AI Accuracy Claims Before You Buy

By the Wynstra team · · 9 min read

Every document AI vendor will show you a number: 99% accuracy, 95% straight-through processing, field-level extraction you can trust. The trouble is that these numbers rarely agree with each other, and they almost never describe what will happen on your invoices, your claims forms or your contracts. Before you commit budget, it helps to understand why the published figures conflict, which two metrics actually matter, and how to run a short test that tells you the truth about your own documents.

Why the published numbers don't agree

Part of the confusion is structural. Even a single, well-regarded research firm can publish different cost and performance figures across different report editions. Ardent Partners' State of ePayables 2025 puts the average cost per invoice at $12.42, its AP Metrics that Matter 2025 report at $9.40, and the 2026 edition of that series at $9.90. The underlying survey populations, question wording and sponsor shift each time, which is exactly why two research firms can publish cost-per-invoice figures you can trace to a current report, and still measure different things.

Straight-through processing (STP) figures show the same pattern. Ardent Partners' Accounts Payable Metrics that Matter in 2026 benchmarks AP performance based on research with nearly 200 finance and AP leaders worldwide, and finds Best-in-Class AP teams process a single invoice for $2.67, while all others pay $12.50, nearly 4.7 times as much. Yet a separate pull of the same broad body of research puts the industry average touchless rate at 32.6% and best-in-class at 49.2%, figures that sit well below the 85-95% STP numbers some software vendors advertise. None of these figures is necessarily wrong; they simply measure different populations, time periods and definitions of "touchless." The practical lesson is to ask which report, which edition and which population a number comes from before you repeat it in a board paper.

Two metrics worth insisting on

Cut through the noise by asking vendors for two specific numbers, defined precisely.

Field-level accuracy is whether a specific field, such as an invoice total or a claim number, is completely correct. A system can have 99% character-level accuracy and still extract an invoice total incorrectly, because one wrong digit in a long string barely moves the character score but breaks the field. Field-level accuracy, not character accuracy, is the number that determines whether a document can move forward untouched.

Straight-through processing rate is the share of documents that complete the whole workflow, extraction, validation, matching and posting, with no human involved. Vendors often quote headline STP figures that assume a mature deployment. In reality, most AP automation vendors will tell you that their platforms achieve 95%+ STP rates, but realistic STP rates typically start at 40-50% at implementation and improve to 70-85% at six months as rules and matching logic are tuned. If a vendor's number doesn't come with a ramp timeline, ask for one. And ask the obvious follow-up: what rate does the platform actually reach on invoices that look like yours, not on the vendor's cleanest reference account.

What independent benchmarks actually show

Independent, disclosed benchmarks are more useful than vendor case studies, but even they show how much accuracy depends on document complexity. A 2026 academic benchmark testing frontier language models on schema-guided extraction from real enterprise documents found that across 210 extraction attempts, six models achieved only a 51% valid JSON rate, with an aggregate pass rate of 4.6% across all field evaluations. Even when the output was valid, field-level accuracy ranged from 65% to 80%, with an aggregate of 72.9% across all valid extractions. That is a long way from the 99%-plus numbers quoted in marketing pages, and it reflects what happens on documents with nested fields, nested tables and unusual layouts rather than clean, single-page invoices.

Other published benchmarks are more favourable, but they disclose their method, which is the real point. An independent 2026 comparison of six document AI systems across 3,365 documents reported that the leading system reached a mean extraction accuracy of 89.1%, measured by comparing each extracted field against a human-verified reference value, with a field counted as correct from a similarity of 0.8 upwards. A separate benchmark aimed specifically at long, repeated tables, such as line items on large invoices, found mean accuracy of 99.2% across 45 financial, clinical and legal PDFs when testing whether systems preserve complete, schema-faithful arrays as output grows from a dozen rows to thousands. The gap between these results and the schema-extraction study above shows that accuracy is highly task-specific: the same underlying models can look brilliant on one benchmark and mediocre on another, depending on document structure.

One more thing worth knowing: vendor confidence scores are usually conservative by design. An audit-focused study of document intelligence in financial statement review noted that the divergence between model-reported confidence scores and observed performance reflects the conservative nature of Document AI confidence estimates, which are designed to understate certainty rather than overstate extraction accuracy. A low confidence score is not necessarily a wrong answer, but a high one is not a guarantee either.

A practical pre-purchase test

Run this before you sign anything, ideally over two to four weeks rather than a single demo session.

  1. Build your own test set. Pull 100-300 real documents spanning your worst formats, not just the clean scans a vendor will ask for. Include faxes, mobile photos, non-native-language invoices and multi-page contracts if those exist in your volume.
  2. Create ground truth first. Have someone manually key the correct answer for every field before running the vendor's tool, so you are not grading against the vendor's own output.
  3. Score field by field. Report accuracy per field (total, date, tax amount, claim code, clause type), not as a single blended average, since a 98% average can hide a critical field that is wrong one time in five.
  4. Measure STP at the workflow level, including matching and validation rules, not just extraction in isolation. Set your own success thresholds against a discovery baseline, covering touchless rate, extraction accuracy and time to completion, keep humans in the loop, and document everything for regulators.
  5. Test the exception path. Deliberately feed in a batch of messy or incomplete documents and check how the system flags, routes and explains failures, since exception handling is usually what separates a 65% STP rate from an 88% one.
  6. Ask for the vendor's benchmark methodology. Sample size, document types, and whether an independent party reviewed the results. If a vendor can't describe its method the way a published benchmark does, treat the headline number as marketing, not evidence.

Claims and contracts need their own bar

Healthcare claims and contracts raise the stakes further. Many payer contracts set a hard accuracy floor: current AI medical coding systems report around 99.2% coding accuracy, which exceeds the 95% accuracy threshold required for most payer contracts. That threshold matters because claims extraction errors cascade into downstream denials, and claims intake automation has been reported to cut intake time from 15-45 minutes to 2-3 minutes per claim while data accuracy climbs from 75-85% to 95-99%, but only once the pipeline is tuned on the payer's specific document formats.

Contracts are harder to benchmark at all, because many of the fields that matter most, such as a liability cap buried in a schedule or a termination clause with cross-references, don't have a single unambiguous "correct" extraction the way an invoice total does. Treat contract extraction as a drafting aid that still needs a qualified human reviewer on anything with financial or legal consequence, rather than as a straight-through process.

Conclusion

Accuracy and straight-through processing numbers for document AI vary enormously depending on which report, which vendor and which document type you're looking at. The only number that matters is the one you generate on your own documents, measured field by field, over a realistic test window with your messiest formats included. If you want help designing and running that kind of test before committing budget to a document AI platform for invoices, claims or contracts, Wynstra can help you set it up.

Sources

  • Document Ai
  • Invoice Processing
  • Claims Automation
  • Vendor Evaluation
  • Contract Ai

Have a project in mind?

Tell us what you need. We'll reply within one business day with ideas, a rough timeline and an estimate. No obligation.