The gap between an impressive AI demo and a system that saves money is wider than most vendors admit. Four questions that separate the two before you commit budget.
Every AI demo works. That is what makes them demos. The interesting question is what happens on the four hundredth document, the one with a coffee stain and a handwritten correction in the margin.
Question one: what is the baseline?
Before anything is built, someone should measure how the process performs today. How many documents per hour? What is the current error rate? How much staff time does it consume?
Without that number, any result can be presented as a success. With it, the conversation becomes concrete: the model achieves 86 percent straight-through processing against a manual baseline of 100 percent accuracy at 40 documents per hour, and here is what that trade is worth.
Question two: what happens when the model is unsure?
This is the question that separates production systems from demos. A model that is right 90 percent of the time is useless if you cannot tell which 90 percent.
Confidence scoring and a human review path are not optional extras. In our experience the review interface takes as much design effort as the model integration, because the reviewer needs the source document, the extraction and the correction path in one view.
A working pattern
- High confidence: process automatically, log for sampling.
- Medium confidence: queue for review with the extraction pre-filled.
- Low confidence: route to manual handling with the document attached.
The economics come from moving the boundary, not from eliminating review entirely.
Question three: would a rules engine be better?
A surprising number of problems presented as AI problems are deterministic. If your invoice formats come from nine known suppliers, nine templates will outperform a model — more accurately, more cheaply and more predictably.
We have talked several clients out of AI projects for exactly this reason. It is a short conversation and it saves everyone money.
Question four: who maintains it?
Models drift. Input formats change, suppliers switch systems, seasonal patterns shift. A model deployed and forgotten degrades quietly until someone notices the error rate a quarter later.
Monitoring, drift detection and a retraining path need to be part of the original scope. If the proposal does not mention them, the proposal is for a demo.
Where it does pay
Applied AI earns its keep on high-volume, repetitive, expensive processes where partial automation is still valuable. Document extraction, demand forecasting, ticket triage and quality inspection all qualify. What they share is a clear baseline, a tolerance for partial automation, and a measurable cost per unit of work.
Start there. The demo can wait.