We asked two leading AIs to triage the same four emergency patients, over and over. They matched the clinician's level 459 times in 464. None of the rest was wrong — and most of them came from the written rule.
Published
Same Question, Same Answer · Two AIs triaging the same four emergency patients again and again, on the five-level Emergency Severity Index.
Each bar shows the share of replies at each triage level. Yellow: the clinician's level. Blue: the defensible alternative on the borderline patient. Orange: a higher level than the clinician's, judged acceptable over-triage. Rewording the question in other ways gave the clinician's level every time.
Question
Ask an AI the same question about the same patient again and again. Do you get the same answer every time? In emergency triage, a different answer means a different wait.
Predictionwritten before testing
- Our clinical practitioner, who reviewed the cases: no variation, other than minor phrasing.
- Ours: the clear cases stay on one level; the borderline case splits between 2 and 3 on the same question; rewording the question moves it by a quarter or more; a written rule does not remove the split.
What we tested
Four invented emergency department triage notes, reviewed by our clinical practitioner before any run, each scored on the 5-level Emergency Severity Index (ESI):
- A: an 82-year-old on apixaban who fell and hit her head, no loss of consciousness, normal vital signs. Borderline: 2 preferred, 3 defensible, and she gets the CT either way.
- A2: the same patient, unresponsive for about 30 seconds and vomited once. Level 2.
- B: a 22-year-old with a rolled ankle who needs one X-ray. Level 4.
- C: a 45-year-old after a head-on crash, in shock and short of oxygen. Level 1.
Two leading models, asked for the level and a one-sentence reason, each time in a fresh conversation, and in several wordings of the question.
What happened
- The clinician's level almost every time. The rest were acceptable by the clinician's review: a defensible call on the borderline patient, and over-triage. None was wrong.
- Every clear case got its level every time on both models. On the borderline patient, both models chose the more cautious level 2 almost every time; model B rarely called her a 3, citing the CT and the laceration repair.
- Rewording the question changed nothing: the intended level every time.
What the rule did
The rule — a rule about following the triage scale's own decision points in order — made no answer better. It produced most of the few departures from the clinician's level, all on model A, all over-triage that the clinician judged acceptable. On the patient who had been unresponsive, it sometimes named the high-risk decision point, which gives level 2, and wrote level 1 above it; one reply ended warranting ESI-2
under a heading of Level 1. It also counted a single ankle X-ray as two or more resources
and called a level 4 a 3.
What it means for you
The practitioner's prediction held, and ours mostly did not. Where a written standard defines the answer, the same question gets the same answer, even on a patient experienced clinicians would debate. The little variation there was came from the fix, not the model: asking for the reasoning out loud produced levels the reasoning itself contradicted. None of it was wrong; all of it leaned cautious. Consistency is not where triage AI breaks. The next place to look is where no standard exists.
Limitation
Four invented cases on one scale with written decision points, and two models used directly, without the chat app around them.