From the Emergency Services edition of October 6, 2026
arXiv audit: open-source LLMs change pediatric ED triage levels when only context changes, including EMS arrival and overnight surge
Virginia Tech, University of Washington/Seattle Children's Hospital and Children's National Hospital (arXiv preprint 2610.01963) · Academic research / emergency triage AI fairness · United States
- Publisher
- arXiv
- Original publication
- 2026-10-01
- Source retrieved
- 2026-10-07
What happened
An Oct 1, 2026 preprint audits ten open-source LLMs predicting pediatric ED Emergency Severity Index levels on 352 institutional encounters and 245 handbook vignettes, changing one contextual detail while holding the clinical picture fixed. Sensitivity varied widely and did not reliably fall with model size or medical pretraining. The authors' fine-tuned Qwen2.5-7B had a 5.27% any-shift rate vs 16.02% for the base model. Even for it, ambulance arrival with EMS pre-notification shifted 10.51% of predictions, and overnight/weekend surge 13.35%, with 10.51% undertriage.
Why it matters
Limited, stated relevance: this is ED triage, not prehospital care, but EMS agencies and dispatch centers are being offered locally hosted LLMs for triage and handoff. The finding that arrival mode and system strain alone move acuity predictions is a transferable test for any EMS-facing triage model.
Evidence and measured results
Counterfactual audit with repeated-run stability checks, external vignette replication and positive-control injections; preprint, not peer reviewed; best model was fine-tuned by the authors.
Limitations and uncertainty
Evaluation set filtered for complete documentation; nurse-assigned labels are not perfect ground truth; templated injected phrases; single-variable perturbations only; pre-arrival activation cases and most ESI-1 encounters excluded.
Put this evidence to work
Lighthouse Advisory interpretation, grounded in this source. Enriched 2026-10-06; this does not change the original publication date. Each role follows the same checklist so takeaways compare across sources.
Sales
Role takeaway
- Customer problem
- EMS and emergency care leaders are offered LLM triage support without evidence that it treats identical patients consistently.
- Stakeholders
- EMS medical director
- ED leadership
- Health equity officer
- Clinical informatics
- Discovery questions
- Has the triage model been tested for context-driven shifts?
- How is undertriage reported?
- Is the model hosted locally?
- Value hypothesis
- A counterfactual audit may reveal unsafe or unfair shifts before a triage model reaches patients.
- Potential engagement
- Counterfactual fairness and stability audit of a proposed triage or prioritization model.
- Unsupported claims
- Results apply to prehospital triage
- Fine-tuned model is safe for deployment
- Larger models are fairer
Pre-sales engineering
Role takeaway
- Fit
- Evaluation method for triage and prioritization models.
- Architecture and integration
- Paired counterfactual inputs, repeated runs and positive controls against locally hosted models.
- Prerequisites
- Labeled local cases
- Clinician-defined context variables
- Model hosting
- Constraints
- Preprint
- ED pediatric scope
- Templated phrases
- Security
- Local hosting keeps patient data in house; audit data must be de-identified.
- Proof of value
- Any-shift and undertriage rates by variable
- Stability across repeated runs
- Positive-control responsiveness
Delivery
Role takeaway
- Work
- Build local counterfactual test set, audit before go-live, repeat after updates.
- Dependencies
- Clinicians
- Informatics
- Model vendor
- Ownership
- Clinical informatics with EMS or ED medical direction.
- Skills and adoption
- Clinicians learn to read shift and undertriage reports.
- Governance checkpoints
- Pre-deployment audit
- Audit after each model update
- Surge-period monitoring
- Acceptance criteria
- Undertriage shift rate reported per variable
- Audit repeated after updates
- Risks
- Undertriage during surge
- Equity harms
- False confidence from aggregate accuracy
Implementation considerations
Lighthouse Advisory interpretation across the operating dimensions a public-sector buyer must settle before this evidence becomes a design. Each note answers the question under its heading for this specific source.
Architecture and integration
What must connect, and where does the AI sit in the workflow?
Targets locally deployable open-weight models (7B to 120B), relevant to on-premises or private hosting for agencies with patient-data limits.
Governance
Who approves, reviews and stays accountable for outcomes?
Supports requiring counterfactual testing, with undertriage reported separately, before any triage model is used for decision support.
Security and privacy
What data, permissions and controls need testing?
Open-source models can run locally, keeping patient data in house, but local hosting does not remove bias risk.
Accessibility and workforce
Who is affected, and what skills or accommodations follow?
Shifts tied to language, insurance and housing raise equity concerns for patients EMS serves; clinicians must keep final triage authority.
Procurement
What should contracts, pricing and exit terms secure?
Buyers of triage AI should ask vendors for counterfactual audit results by context variable, not only aggregate accuracy.
Operating model
Which teams own the service once it runs?
Model as decision support behind clinician triage, audited before and after deployment.
Publication history
- 2026-10-06Emergency Services · Issue 315 resources
Stable resource ID: arxiv-counterfactual-bias-audit-llm-ed-triage-20261001