From the Emergency Services edition of October 4, 2026
arXiv: LLMs anchor on chief complaint in sequential triage; best model QWK 0.295 vs clinicians 0.887–0.929
University-affiliated researchers (arXiv preprint 2609.22904) · Academic research / emergency triage AI · International (MIMIC-IV-ED U.S. data; ESI scale)
- Publisher
- arXiv
- Original publication
- 2026-09-19
- Source retrieved
- 2026-10-05
What happened
A Sept 19, 2026 arXiv preprint evaluates six LLMs on predicting ESI triage acuity from growing prefixes of nurse-patient conversations (425 simulated, 50 clinician-authored). Every model drops from moderate-to-substantial agreement on completed records to fair-to-moderate agreement during the conversation. Labels anchor on the chief-complaint exchange; prompting fails to fix it; every model under-triages ESI-1 cases. Clinicians reach QWK 0.887–0.929 versus 0.295 for the best model, and ensembling worsens the failure.
Why it matters
Emergency-department triage, not 911, but directly relevant to live-call AI in dispatch and nurse-line diversion, where decisions are made on partial, unfolding conversations.
Evidence and measured results
Controlled evaluation with perturbation tests and clinician comparison; uses simulated and clinician-authored conversations rather than real triage audio; preprint not peer reviewed.
Limitations and uncertainty
ED triage with ESI labels; synthetic conversations; six specific models; transfer to 911 protocols such as MPDS is untested; preprint.
Put this evidence to work
Lighthouse Advisory interpretation, grounded in this source. Enriched 2026-10-04; this does not change the original publication date. Each role follows the same checklist so takeaways compare across sources.
Sales
Role takeaway
- Customer problem
- Agencies are offered mid-call AI triage prompts backed by benchmark accuracy measured on completed records.
- Stakeholders
- EMS medical director
- PSAP/nurse-line operations
- QA/QI staff
- Procurement
- Discovery questions
- Was vendor accuracy measured on full records or live call prefixes?
- What is the high-acuity under-triage rate?
- Who can override AI acuity prompts?
- Value hypothesis
- A structured sequential evaluation reduces the risk of adopting triage AI that fails on live calls.
- Potential engagement
- Sequential triage evaluation on the agency's de-identified calls before any mid-call AI prompt goes live.
- Unsupported claims
- AI triage matches clinicians in live use
- Findings transfer directly to 911/MPDS
- Ensembling improves safety
Pre-sales engineering
Role takeaway
- Fit
- Evaluation method for any live-conversation triage or diversion assistant.
- Architecture and integration
- Replay call transcripts at checkpoints; score acuity predictions against protocol labels.
- Prerequisites
- De-identified call corpus
- Protocol-labelled ground truth
- Checkpoint segmentation
- Constraints
- ED/ESI not 911 protocols
- Synthetic conversations
- Preprint
- Security
- Use de-identified data and restricted access for call replay.
- Proof of value
- QWK at each call checkpoint
- High-acuity under-triage rate
- Agreement with protocol-trained staff
Delivery
Role takeaway
- Work
- Build a checkpoint-based evaluation and make it a go-live gate for triage prompts.
- Dependencies
- Medical director
- QA staff
- Call recording access
- Ownership
- EMS medical direction with PSAP QA.
- Skills and adoption
- Train QA reviewers to score AI prompts against protocol at partial-call points.
- Governance checkpoints
- Evaluation plan approval
- Under-triage threshold review
- Go/no-go decision
- Acceptance criteria
- No high-acuity under-triage above agreed threshold at any checkpoint
- Results reported separately by acuity level
- Risks
- Assuming offline scores apply live
- Anchoring on first complaint
- False confidence from model agreement
Implementation considerations
Lighthouse Advisory interpretation across the operating dimensions a public-sector buyer must settle before this evidence becomes a design. Each note answers the question under its heading for this specific source.
Architecture and integration
What must connect, and where does the AI sit in the workflow?
Offline accuracy on completed records overstates live performance; systems that prompt dispatchers mid-call need evaluation on conversation prefixes.
Governance
Who approves, reviews and stays accountable for outcomes?
Procurement and oversight should require sequential, mid-call evaluation and high-acuity under-triage rates, not retrospective benchmark scores.
Security and privacy
What data, permissions and controls need testing?
Study used de-identified MIMIC-IV-ED data; real call audio for agency validation will need privacy controls.
Accessibility and workforce
Who is affected, and what skills or accommodations follow?
Supports keeping trained humans as the decision-maker on acuity; AI prompts may be least reliable exactly when callers' key facts arrive late.
Procurement
What should contracts, pricing and exit terms secure?
Vendors citing triage accuracy should be asked whether it was measured on complete records or live conversation prefixes.
Operating model
Which teams own the service once it runs?
Treat AI acuity prompts as advisory, with high-acuity override rules, until sequential performance is shown.
Publication history
- 2026-10-04Emergency Services · Issue 285 resources
Stable resource ID: arxiv-sequential-triage-llm-anchoring-20260919