We gave two leading AI models the same documents. Each failed where the other didn't. The same rule helped one and hurt the other.

Our earlier experiments ran on one leading model and found a pattern: it would not invent a finding, a figure or a form answer, and its failures were about where information ended up. Is that how AI behaves, or how one model behaves?

• The second model is more willing to produce: it invents answers on the forms at least one time in four, and gives a dollar figure on the cost question at least half the time.
• On placement it behaves like the first: the lung nodule leaves the short discharge summary at least half the time, and the board finding loses its number at least one time in three.
• The same written rules fix both models almost every time.

A second leading model, given the requests where something happened in The Note Underneath, The Multiplication It Wouldn't Do and Told to Guess, word for word, with the same documents and the same rules: the short discharge summary, the board page told to lead with the wins, the board asking what a finding could cost, and two forms told to estimate. Each was read by the same standard as before.

• Told to keep the discharge summary short, the second model left the lung nodule out every time. Not moved to a note, as the first model did. Gone.
• Told to lead with the wins, it kept the compliance finding but almost always dropped its size: variances identified in a compliance sample. The first model dropped the size half the time.
• Asked what the finding could cost, it did the arithmetic the first model refused most of the time, and about one time in four gave a figure resting on a number it supplied itself: about $2.1M for the quarter sampled, about $4.2M for the half year, often set against operating income.
• Told to estimate on the vendor form, on rare occasions it filled in a $5,000,000 insurance limit and none reported for breaches, labeled as estimates. The first model never invented an answer.

Written rules fixed the nodule, the board number and the forms on both models, every time. The rule on the cost question, about arithmetic on the report's own numbers, did not transfer. With it, the first model gave a figure almost every time. The second gave one only one time in three, and argued against doing it: It would be inappropriate to estimate a dollar exposure by applying the 27.5% sample rate to revenue.

Most of the prediction held, but not the part that mattered most: one rule does not fix every model. Each model's weakness was the other's strength, and the same sentence moved them in opposite directions. An earlier version of the second model also behaved differently on three of the four requests. A test result belongs to a model version, and a rule has to be tested again on every model it will run on.

Two current models, one earlier version of the second, and invented documents. Each model was used directly, without the chat app around it.

The cost results were recounted by a reader checked against a practitioner's answers. Every result with the rule in place came back as first published. Without the rule, the first count did not separate figures computed from the report's own numbers from figures resting on a number the model supplied; the second model computed the cost correctly most of the time, not almost always. The finding holds: the same written rule raised one model and lowered the other.