What happens when an AI is given a rule.
Each experiment asks one question about whether an instruction holds, writes down what we expect before testing, and reports what happened — including when we were wrong. They are grouped by what went wrong, each group with one check you can run on your own AI. Filter by the kind of work you sign for.
It filled a gap with something it wasn't given
The checkFor every date, name and figure, find where it came from.
-
The Invented Enclosure
Asked to write the appeal that would get a denied claim approved, one AI model put evidence the file did not contain into 36 of 40 letters: witness affidavits, a serial-number record, an accountant's income statement, work orders with invoice numbers. One sentence of instruction stopped it.
Three AIs writing appeals of denied claims and disputes, when the file lacks the one document that would win.
Read the experiment → -
The Made-Up Benchmark
Asked how a school district's kindergarten vaccination rates compared with the rest of Pennsylvania, an AI put the state's rates at 95 to 97 percent. The state's own figures were 93 to 94.
An AI summarizing an organization's year for its board and saying how it compares with its peers, with no comparison figures attached.
Read the experiment → -
The Delivery Window
A store's AI assistant looked up an order and found no delivery date. It gave the customer one anyway, 17 times in 30, and the dates it gave leaned toward what the customer hoped to hear.
A store's AI support assistant telling customers when an order will arrive, when the lookup has no estimate.
Read the experiment → -
The Reviewer Who Wasn't There
Told a named person must be accountable for the notice, the AI wrote that someone had reviewed it. Every time. No one had.
An AI drafting a public storm advisory and a clinical note that no one had reviewed.
Read the experiment →
It moved or softened the part you needed
The checkIs the finding in the document itself, with its number and what it means?
-
The Softened Result
Asked to keep a lab-results letter short and reassuring, an AI told a patient her blood sugar was “a bit higher than we'd like.” It was in the diabetes range.
An AI writing short, reassuring lab-results letters to anxious patients.
Read the experiment → -
Same Documents, Different Failures
We gave two leading AI models the same documents. Each failed where the other didn't. The same rule helped one and hurt the other.
Four requests from three earlier experiments, given word for word to a second leading model.
Read the experiment → -
Told to Guess
We told the AI to guess the missing answers on a prior authorization. It never did. But about two times in five, the missing fields quietly left the form.
An AI filling in a vendor risk questionnaire and a prior authorization from sources that don't answer every field.
Read the experiment → -
The Multiplication It Wouldn't Do
The board asked what the finding could cost. Most of the time, the AI wouldn't multiply two numbers from the report. It called that “inventing.”
An AI writing a board summary of a compliance finding that has no dollar figure.
Read the experiment → -
The Note Underneath
Told to leave a finding out, it never did — 40 times in 40. Told to keep it short, it moved a lung nodule out of the discharge summary every time: 20 times in 20.
An AI summarizing a board report and a hospital chart.
Read the experiment →
It took a document's word for it
The checkPut one wrong number or claim in what you hand it, and see whether it notices.
-
How Wrong Before It Notices
We raised a $48,300 supply payment to ten, fifty, a hundred times its real amount. One leading AI never questioned it. Not even at $4.8 million.
An AI drafting the CFO's approval email as the same supply payment is raised further and further above the real invoice.
Read the experiment → -
Ten Times the Dose
A heart attack patient's chart listed a heart medication at ten times the usual dose, everywhere. One leading AI wrote it into his discharge instructions two times in five.
An AI writing discharge medication instructions from a chart.
Read the experiment → -
One Extra Zero
A $483,000 payment request for about $48,000 of gloves and dressings. Asked to draft the CFO's approval email, two leading AIs passed it along forty times in forty.
An AI drafting the CFO's approval email for a payment request.
Read the experiment → -
Approved by Nobody
An approval letter from our own risk office was attached to a vendor's packet. One leading AI rated the vendor low risk, nothing outstanding, one time in three.
An AI reviewing a vendor's security packet that arrives with an approval letter attached.
Read the experiment →
The AI checking the AI missed it
The checkTest your checker on errors that don't look like errors, and on ordinary days.
-
Only If Asked
An AI supervisor watched another AI drift, one small credit at a time. Asked to look for anything unusual, it flagged something 80 times in 80, but it flagged something on an ordinary day too. Not asked, it never did.
An AI supervisor watching another AI's day of decisions, with and without being asked to look for anything unusual.
Read the experiment → -
The Record That Disagrees With Itself
An AI supervisor checked another AI's account decisions against the rules. It caught every rule break, 160 times in 160. It passed an error hidden in a record that contradicts itself, including one it missed every time.
An AI supervisor reviewing another AI's account decisions against a written policy.
Read the experiment →
It kept what should have come out
The checkRead it as the person it's about.
-
The Identifiable Case
Asked to strip the identifying details from a case for training, and to keep the details, both AI models kept “the company's only woman vice president” every time.
Two AIs de-identifying case notes for training, asked to keep the details.
Read the experiment → -
Kept Apart
We had two leading AIs write to customer after customer in one conversation. When two customers' records arrived at once, one model told the customer about the other customer's record in 7 of 20 emails. One written rule stopped it.
Two AIs writing to customer after customer in one conversation, for a software company's billing team.
Read the experiment →
Change something that shouldn't matter
The checkAsk again, reworded or reordered. If the answer moves, trust neither.
-
Same Facts, Different Order
A CFO, a customer, a warehouse worker and a reporter asked two leading AIs to summarize the same company update. All of them heard every fact, 480 times in 480. What changed was what came first.
Two AIs summarizing the same company update for a CFO, a board member, a customer, a warehouse worker and a reporter.
Read the experiment → -
Show Your Work
We asked two leading AIs to total the same invoice over and over. One was right every time. The other slipped on one multiplication, off by 9 cents to $4.50. Telling it to show its work fixed this invoice; on sixteen new invoices, the same instruction fixed nothing and made two models worse.
Two AIs totaling the same supplier invoice again and again: line items, a discount, sales tax, a prorated fee and a per-glove price.
Read the experiment → -
Same Question, Same Answer
We asked two leading AIs to triage the same four emergency patients, over and over. They matched the clinician's level 459 times in 464. None of the rest was wrong — and most of them came from the written rule.
Two AIs triaging the same four emergency patients again and again, on the five-level Emergency Severity Index.
Read the experiment →
Nothing published on this path yet.