{"resourceId":"arxiv-sequential-triage-llm-anchoring-20260919","versions":[{"version":"external-04decfdfe6d6ed8074a816397cd4f3673d9923fc5598d461a5a804483cdbf70c","resource":{"id":"arxiv-sequential-triage-llm-anchoring-20260919","title":"arXiv: LLMs anchor on chief complaint in sequential triage; best model QWK 0.295 vs clinicians 0.887–0.929","organization":"University-affiliated researchers (arXiv preprint 2609.22904)","sector":"Academic research / emergency triage AI","geography":"International (MIMIC-IV-ED U.S. data; ESI scale)","publishedAt":"2026-09-19","publicationDate":"2026-09-19","eventDate":null,"sourceName":"arXiv","sourceLabel":"Preprint evaluating six LLMs on sequential ED triage conversations","sourceUrl":"https://arxiv.org/html/2609.22904v1","evidenceClass":"academic-research","outcomeClass":"cautionary","topics":["knowledge-work","governance-procurement","operating-model"],"finding":"A Sept 19, 2026 arXiv preprint evaluates six LLMs on predicting ESI triage acuity from growing prefixes of nurse-patient conversations (425 simulated, 50 clinician-authored). Every model drops from moderate-to-substantial agreement on completed records to fair-to-moderate agreement during the conversation. Labels anchor on the chief-complaint exchange; prompting fails to fix it; every model under-triages ESI-1 cases. Clinicians reach QWK 0.887–0.929 versus 0.295 for the best model, and ensembling worsens the failure.","sledRelevance":"Emergency-department triage, not 911, but directly relevant to live-call AI in dispatch and nurse-line diversion, where decisions are made on partial, unfolding conversations.","evidence":"Controlled evaluation with perturbation tests and clinician comparison; uses simulated and clinician-authored conversations rather than real triage audio; preprint not peer reviewed.","architectureImplications":"Offline accuracy on completed records overstates live performance; systems that prompt dispatchers mid-call need evaluation on conversation prefixes.","governanceImplications":"Procurement and oversight should require sequential, mid-call evaluation and high-acuity under-triage rates, not retrospective benchmark scores.","securityPrivacyImplications":"Study used de-identified MIMIC-IV-ED data; real call audio for agency validation will need privacy controls.","caveats":"ED triage with ESI labels; synthetic conversations; six specific models; transfer to 911 protocols such as MPDS is untested; preprint.","streamIds":["emergency-services"],"roles":{"sales":"Use this paper to reset expectations: models that look physician-level on completed records fell to a best QWK of 0.295 when judging conversations as they unfolded, and all under-triaged the most urgent cases. For agencies weighing mid-call AI prompts or nurse-line diversion, the honest offer is an evaluation on their own call prefixes before relying on any acuity suggestion. This is ED evidence; say so.","engineering":"The key technical finding is anchoring: labels lock onto the chief complaint and later evidence does not move them, and ensembles agree with each other more than with ground truth. Any dispatch or triage assistant should be tested at multiple checkpoints during a call, with ESI-1-equivalent under-triage reported separately. Prompting fixes did not help in this study, so design for human override rather than tuning alone.","delivery":"Add a sequential-evaluation gate to any project that surfaces AI acuity or diversion prompts during live calls: replay de-identified calls at several points, compare against protocol-trained staff, and block go-live if high-acuity under-triage appears. Document that ED results may not carry over to 911 protocols, and plan a local validation rather than citing the paper as proof either way."},"roleDetails":{"sales":{"customerProblem":"Agencies are offered mid-call AI triage prompts backed by benchmark accuracy measured on completed records.","stakeholders":["EMS medical director","PSAP/nurse-line operations","QA/QI staff","Procurement"],"discoveryQuestions":["Was vendor accuracy measured on full records or live call prefixes?","What is the high-acuity under-triage rate?","Who can override AI acuity prompts?"],"valueHypothesis":"A structured sequential evaluation reduces the risk of adopting triage AI that fails on live calls.","potentialEngagement":"Sequential triage evaluation on the agency's de-identified calls before any mid-call AI prompt goes live.","unsupportedClaims":["AI triage matches clinicians in live use","Findings transfer directly to 911/MPDS","Ensembling improves safety"]},"engineering":{"fit":"Evaluation method for any live-conversation triage or diversion assistant.","architectureIntegration":"Replay call transcripts at checkpoints; score acuity predictions against protocol labels.","prerequisites":["De-identified call corpus","Protocol-labelled ground truth","Checkpoint segmentation"],"constraints":["ED/ESI not 911 protocols","Synthetic conversations","Preprint"],"security":"Use de-identified data and restricted access for call replay.","proofOfValue":["QWK at each call checkpoint","High-acuity under-triage rate","Agreement with protocol-trained staff"]},"delivery":{"work":"Build a checkpoint-based evaluation and make it a go-live gate for triage prompts.","dependencies":["Medical director","QA staff","Call recording access"],"ownership":"EMS medical direction with PSAP QA.","skillsAdoption":"Train QA reviewers to score AI prompts against protocol at partial-call points.","governanceCheckpoints":["Evaluation plan approval","Under-triage threshold review","Go/no-go decision"],"acceptanceCriteria":["No high-acuity under-triage above agreed threshold at any checkpoint","Results reported separately by acuity level"],"risks":["Assuming offline scores apply live","Anchoring on first complaint","False confidence from model agreement"]}},"retrievedAt":"2026-10-04T22:00:50-05:00","enrichedAt":"2026-10-04T22:00:50-05:00","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Supports keeping trained humans as the decision-maker on acuity; AI prompts may be least reliable exactly when callers' key facts arrive late.","procurementImplications":"Vendors citing triage accuracy should be asked whether it was measured on complete records or live conversation prefixes.","operatingModelImplications":"Treat AI acuity prompts as advisory, with high-acuity override rules, until sequential performance is shown.","sourceVerification":{"responseId":"curl-arxiv-triage-2026-10-04T22:00:50-05:00","openedUrl":"https://arxiv.org/html/2609.22904v1","referenceExcerpt":"Three expert clinicians on the same conversations reach a QWK of 0.887–0.929, while the best model reaches 0.295.","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}