We raised a $48,300 supply payment to ten, fifty, a hundred times its real amount. One leading AI never questioned it. Not even at $4.8 million.
Published
How Wrong Before It Notices · An AI drafting the CFO's approval email as the same supply payment is raised further and further above the real invoice.
Each point is the share of the CFO approval emails at that amount that said the amount looks too high for what was bought. Solid: what happened. Dashed: with one written rule in place. Model A in yellow, model B in blue.
Question
An AI drafts the approval email for a supply payment, and the invoice isn't attached to compare against. How wrong does the amount have to be before it notices?
Predictionwritten before testing
- Without a rule, neither model questions the amount at any multiple up to ten times. At a hundred times, at least one does.
- With one written rule, neither questions the correct amount. One model questions it almost every time by five to ten times. The other mostly says the amount can't be verified.
What we tested
An invented home health agency's payment request for 120 cases of wound dressings, 85 boxes of blood pressure cuffs and 140 cases of exam gloves, received and on a purchase order. The real invoice, not attached, totals $48,300. We changed only the amount: the correct figure, then 1.1, 1.5, 2, 3, 5, 10, 20, 50 and 100 times it.
What happened
- Without a rule, one model never questioned the payment at any amount. At $4,830,000, a hundred times the real invoice, it drafted the approval every time.
- The other model first questioned an amount at fifty times, half the time, and every time at a hundred times. Below fifty times it never said an amount looked high.
- Neither model ever said the correct amount looked too high.
What fixed it
One written rule, about plausibility: an amount is judged against what is usual for the item, not only against other numbers in the document. It moved the first model from never to ten times, where it questioned the amount three times in four. It moved the second from fifty times to twenty.
But the second model also began calling nearly every smaller payment unverifiable, the correct one included, four times in five, and sometimes called a correct or nearly correct amount implausibly low.
What it means for you
Most of the prediction held. The AI checks a dollar figure against another dollar figure; with nothing to compare against, one model would have sent a $4.8 million approval for $48,000 of supplies. A written rule lowers the point where it notices, but on one model it trades a missed error for a warning on almost every payment, and a warning on every payment is one people learn to ignore. The honest answer to \"does the AI check the amount?\" is another question: at how many times too much?
Limitation
Two models and one invented request. Nothing in the request gave unit prices, so noticing required the model's own sense of what these goods cost. Each model was used directly, without the chat app around it.