{"resourceId":"arxiv-counterfactual-bias-audit-llm-ed-triage-20261001","versions":[{"version":"external-dfe24f1d55b1491cace449b8eb06eea291bddbd4403b80b147fec8a34b7823e1","resource":{"id":"arxiv-counterfactual-bias-audit-llm-ed-triage-20261001","title":"arXiv audit: open-source LLMs change pediatric ED triage levels when only context changes, including EMS arrival and overnight surge","organization":"Virginia Tech, University of Washington/Seattle Children's Hospital and Children's National Hospital (arXiv preprint 2610.01963)","sector":"Academic research / emergency triage AI fairness","geography":"United States","publishedAt":"2026-10-01","publicationDate":"2026-10-01","eventDate":null,"sourceName":"arXiv","sourceLabel":"Preprint auditing counterfactual bias in ten open-source LLMs for pediatric Emergency Severity Index prediction","sourceUrl":"https://arxiv.org/html/2610.01963v1","evidenceClass":"academic-research","outcomeClass":"cautionary","topics":["knowledge-work","governance-procurement","data-security"],"finding":"An Oct 1, 2026 preprint audits ten open-source LLMs predicting pediatric ED Emergency Severity Index levels on 352 institutional encounters and 245 handbook vignettes, changing one contextual detail while holding the clinical picture fixed. Sensitivity varied widely and did not reliably fall with model size or medical pretraining. The authors' fine-tuned Qwen2.5-7B had a 5.27% any-shift rate vs 16.02% for the base model. Even for it, ambulance arrival with EMS pre-notification shifted 10.51% of predictions, and overnight/weekend surge 13.35%, with 10.51% undertriage.","sledRelevance":"Limited, stated relevance: this is ED triage, not prehospital care, but EMS agencies and dispatch centers are being offered locally hosted LLMs for triage and handoff. The finding that arrival mode and system strain alone move acuity predictions is a transferable test for any EMS-facing triage model.","evidence":"Counterfactual audit with repeated-run stability checks, external vignette replication and positive-control injections; preprint, not peer reviewed; best model was fine-tuned by the authors.","architectureImplications":"Targets locally deployable open-weight models (7B to 120B), relevant to on-premises or private hosting for agencies with patient-data limits.","governanceImplications":"Supports requiring counterfactual testing, with undertriage reported separately, before any triage model is used for decision support.","securityPrivacyImplications":"Open-source models can run locally, keeping patient data in house, but local hosting does not remove bias risk.","caveats":"Evaluation set filtered for complete documentation; nurse-assigned labels are not perfect ground truth; templated injected phrases; single-variable perturbations only; pre-arrival activation cases and most ESI-1 encounters excluded.","streamIds":["emergency-services"],"roles":{"sales":"Use this preprint to set expectations on triage AI: across ten open-source models, changing only context, such as arriving by ambulance or arriving during an overnight surge, moved predicted acuity, and bigger or medical models were not reliably better. For EMS and dispatch buyers, the bounded offer is a fairness and stability audit of any proposed triage model. It is an ED study, so do not present it as prehospital evidence.","engineering":"The method is portable: take your own cases, change one non-clinical detail at a time, and count shifts and undertriage. Engineers should run this against any triage or prioritization model, including locally hosted open-weight ones, and include EMS-specific context such as transport mode and pre-notification. Add positive controls so a model that ignores all context does not look falsely fair.","delivery":"Make a counterfactual audit a go-live gate for triage decision support: build the test set with clinicians, report undertriage separately, and repeat after every model update. Keep clinicians as the final decision makers, and monitor shift rates during surge periods, since overnight surge produced the largest shift in the best model."},"roleDetails":{"sales":{"customerProblem":"EMS and emergency care leaders are offered LLM triage support without evidence that it treats identical patients consistently.","stakeholders":["EMS medical director","ED leadership","Health equity officer","Clinical informatics"],"discoveryQuestions":["Has the triage model been tested for context-driven shifts?","How is undertriage reported?","Is the model hosted locally?"],"valueHypothesis":"A counterfactual audit may reveal unsafe or unfair shifts before a triage model reaches patients.","potentialEngagement":"Counterfactual fairness and stability audit of a proposed triage or prioritization model.","unsupportedClaims":["Results apply to prehospital triage","Fine-tuned model is safe for deployment","Larger models are fairer"]},"engineering":{"fit":"Evaluation method for triage and prioritization models.","architectureIntegration":"Paired counterfactual inputs, repeated runs and positive controls against locally hosted models.","prerequisites":["Labeled local cases","Clinician-defined context variables","Model hosting"],"constraints":["Preprint","ED pediatric scope","Templated phrases"],"security":"Local hosting keeps patient data in house; audit data must be de-identified.","proofOfValue":["Any-shift and undertriage rates by variable","Stability across repeated runs","Positive-control responsiveness"]},"delivery":{"work":"Build local counterfactual test set, audit before go-live, repeat after updates.","dependencies":["Clinicians","Informatics","Model vendor"],"ownership":"Clinical informatics with EMS or ED medical direction.","skillsAdoption":"Clinicians learn to read shift and undertriage reports.","governanceCheckpoints":["Pre-deployment audit","Audit after each model update","Surge-period monitoring"],"acceptanceCriteria":["Undertriage shift rate reported per variable","Audit repeated after updates"],"risks":["Undertriage during surge","Equity harms","False confidence from aggregate accuracy"]}},"retrievedAt":"2026-10-06T21:59:34-05:00","enrichedAt":"2026-10-06T21:59:34-05:00","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Shifts tied to language, insurance and housing raise equity concerns for patients EMS serves; clinicians must keep final triage authority.","procurementImplications":"Buyers of triage AI should ask vendors for counterfactual audit results by context variable, not only aggregate accuracy.","operatingModelImplications":"Model as decision support behind clinician triage, audited before and after deployment.","sourceVerification":{"responseId":"curl-arxiv-triage-2026-10-06T21:59:34-05:00","openedUrl":"https://arxiv.org/html/2610.01963v1","referenceExcerpt":"Arrival by ambulance with EMS pre-notification produced a 10.51% shift rate in Fine-tuned Qwen2.5-7B, while private-vehicle arrival produced 8.81%.","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}