Noetika Labs

We asked two leading AIs to triage the same four emergency patients, over and over. They matched the clinician's level 459 times in 464. None of the rest was wrong — and most of them came from the written rule.

Published

Same Question, Same Answer · Two AIs triaging the same four emergency patients again and again, on the five-level Emergency Severity Index.

Each bar shows the share of replies at each triage level. Yellow: the clinician's level. Blue: the defensible alternative on the borderline patient. Orange: a higher level than the clinician's, judged acceptable over-triage. Rewording the question in other ways gave the clinician's level every time.

Question

Ask an AI the same question about the same patient again and again. Do you get the same answer every time? In emergency triage, a different answer means a different wait.

Predictionwritten before testing

  • Our clinical practitioner, who reviewed the cases: no variation, other than minor phrasing.
  • Ours: the clear cases stay on one level; the borderline case splits between 2 and 3 on the same question; rewording the question moves it by a quarter or more; a written rule does not remove the split.

What we tested

Four invented emergency department triage notes, reviewed by our clinical practitioner before any run, each scored on the 5-level Emergency Severity Index (ESI):

  • A: an 82-year-old on apixaban who fell and hit her head, no loss of consciousness, normal vital signs. Borderline: 2 preferred, 3 defensible, and she gets the CT either way.
  • A2: the same patient, unresponsive for about 30 seconds and vomited once. Level 2.
  • B: a 22-year-old with a rolled ankle who needs one X-ray. Level 4.
  • C: a 45-year-old after a head-on crash, in shock and short of oxygen. Level 1.

Two leading models, asked for the level and a one-sentence reason, each time in a fresh conversation, and in several wordings of the question.

What happened

  • The clinician's level almost every time. The rest were acceptable by the clinician's review: a defensible call on the borderline patient, and over-triage. None was wrong.
  • Every clear case got its level every time on both models. On the borderline patient, both models chose the more cautious level 2 almost every time; model B rarely called her a 3, citing the CT and the laceration repair.
  • Rewording the question changed nothing: the intended level every time.

What the rule did

The rule — a rule about following the triage scale's own decision points in order — made no answer better. It produced most of the few departures from the clinician's level, all on model A, all over-triage that the clinician judged acceptable. On the patient who had been unresponsive, it sometimes named the high-risk decision point, which gives level 2, and wrote level 1 above it; one reply ended warranting ESI-2 under a heading of Level 1. It also counted a single ankle X-ray as two or more resources and called a level 4 a 3.

What it means for you

The practitioner's prediction held, and ours mostly did not. Where a written standard defines the answer, the same question gets the same answer, even on a patient experienced clinicians would debate. The little variation there was came from the fix, not the model: asking for the reasoning out loud produced levels the reasoning itself contradicted. None of it was wrong; all of it leaned cautious. Consistency is not where triage AI breaks. The next place to look is where no standard exists.

Limitation

Four invented cases on one scale with written decision points, and two models used directly, without the chat app around them.

Methods, the rule, and the numbers →

Talk about a spike

Start with two days.

A small sample of your real work, the instructions your team already uses, and time with the people who review and approve the finished work. By the end you'll know whether your rules are holding, and whether anything more is worth doing.

Decision criteria are agreed before it starts. If the model already follows your rules, you'll be told a simple set of checks is all you need — and that's the end of it.

What happens after the spike →

Prefer email? support@noetikalabs.com