From the Emergency Services edition of September 26, 2026
JHIR: ED-triage LLMs beat classic ML yet show rampant intersectional sex/race acuity bias
Journal of Healthcare Informatics Research research team (Lee et al.) · Academic clinical informatics / emergency department triage · United States-centric MIMIC-IV-ED plus ESI handbook and KTAS evaluation settings (transferability limits for field EMS)
- Publisher
- Journal of Healthcare Informatics Research
- Original publication
- 2026-06-22
- Source retrieved
- 2026-09-27
What happened
Lee et al. (Journal of Healthcare Informatics Research, published online 2026-06-22) evaluate LLMs for emergency department triage on distribution shift/missing data and intersectional sex/race bias via counterfactuals across ESI-Handbook, KTAS, and MIMIC-IV-ED. LLMs outperform traditional ML baselines and provide rationales; retrieving similar prior cases (KATE) outperforms chain-of-thought for triage accuracy. Bias findings are stark: under GPT-4o-mini vanilla prompting, Native Hawaiian women receive average acuity 2.38 versus 3.05 for White men (lower = higher priority)—nearly a full triage level—and the authors state biases are rampant across all models, with at least one significant demographic subcategory after multiple-testing correction. CoT prompting reduces some bias. Authors conclude LLMs should support—not replace—nurses, with equity and calibration still open.
Why it matters
High-value Pass B scrutiny for any EMS/ED or nurse-line triage LLM pitched to SLED buyers. Why now: not previously archived; supplies quantified intersectional acuity gaps that contradict 'neutral assistant' marketing as agencies expand clinical AI after recent Seattle/Corti oversight coverage.
Evidence and measured results
Opened Springer PDF via curl/pdftotext. Abstract claims, Table 1 method comparison narrative, Table 3 intersectional bias example (2.38 vs 3.05), rampant-bias statement, and conclusion on support-not-replace transcribed from the PDF.
Limitations and uncertainty
Hospital ED triage audit—not a live 911/PSAP study; retrospective/counterfactual design; KTAS/ESI handbook mix limits direct U.S. field-EMS transfer.
Put this evidence to work
Lighthouse Advisory interpretation, grounded in this source. Enriched 2026-09-26; this does not change the original publication date. Each role follows the same checklist so takeaways compare across sources.
Sales
Role takeaway
- Customer problem
- EDs exploring LLM triage need evidence on both accuracy and demographic fairness before go-live.
- Stakeholders
- ED medical directors
- Triage nursing leads
- Hospital compliance/equity offices
- EMS medical directors (handoff)
- Discovery questions
- Have you run counterfactual sex/race acuity audits?
- Will demographics be in the prompt?
- Is the LLM advisory or allowed to set ESI?
- How will overrides and disparity metrics be logged?
- Value hypothesis
- LLM triage support may improve robustness over classic ML only if fairness audits and human final acuity are mandatory.
- Potential engagement
- Local counterfactual fairness and calibration evaluation—not a production autonomous triage rollout.
- Unsupported claims
- Neutral/unbiased acuity across demographics
- Safe replacement of nurse triage
- Chain-of-thought removes all bias
Pre-sales engineering
Role takeaway
- Fit
- Advisory LLM acuity/rationale for ED triage with governed example retrieval—not autonomous routing.
- Architecture and integration
- Triage note/vitals → LLM (optional KATE retrieval) → rationale + suggested acuity → clinician confirmation + bias monitors.
- Prerequisites
- Protocol-aligned labels
- Fairness test harness
- Override audit store
- Missing-data policy
- Constraints
- ED not PSAP setting
- Retrospective/counterfactual evidence
- Model-specific bias axes
- Security
- Protect ED notes and demographic fields; credentialed data access analogous to MIMIC controls.
- Proof of value
- Accuracy/QWK vs baseline
- Intersectional acuity gap tests
- Undertriage rate on critical cases
Delivery
Role takeaway
- Work
- Run fairness audit, configure advisory UI, train staff to challenge biased suggestions.
- Dependencies
- ED nursing education
- Privacy/IRB
- EHR integration
- Equity office
- Ownership
- ED medical director owns clinical policy; analytics owns monitoring.
- Skills and adoption
- Teach that fluent rationales can still encode demographic preference.
- Governance checkpoints
- Pre-go-live fairness gate
- Monthly disparity review
- Prompt/demography policy review
- Acceptance criteria
- Human final acuity
- Published local bias report
- Override logging complete
- Risks
- Scaled inequity
- Automation bias
- Prompt drift reintroducing demographics
Implementation considerations
Lighthouse Advisory interpretation across the operating dimensions a public-sector buyer must settle before this evidence becomes a design. Each note answers the question under its heading for this specific source.
Architecture and integration
What must connect, and where does the AI sit in the workflow?
If used at all, deploy as advisory scoring with demographic-ablation tests, retrieval-of-similar-cases carefully governed, and mandatory clinician final acuity—never autonomous routing.
Governance
Who approves, reviews and stays accountable for outcomes?
Require counterfactual fairness audits and multiple-testing–aware bias reports in procurement; treat CoT as a bias mitigator to evaluate, not a cure-all.
Security and privacy
What data, permissions and controls need testing?
MIMIC-style ED data and prompts containing demographics are sensitive; log access and restrict demographic fields in production prompts where policy requires.
Accessibility and workforce
Who is affected, and what skills or accommodations follow?
Biased acuity suggestions can worsen disparities for already underserved groups; training must teach staff to challenge model priority shifts.
Procurement
What should contracts, pricing and exit terms secure?
Write fairness audit and override-logging requirements into triage-AI RFPs; do not accept accuracy-only demos.
Operating model
Which teams own the service once it runs?
Keep nurse/physician triage authoritative; LLM outputs advisory with equity monitoring dashboards.
Publication history
- 2026-09-26Emergency Services · Issue 215 resources
Stable resource ID: jhir-llm-ed-triage-bias-20260622