{"resourceId":"jhir-llm-ed-triage-bias-20260622","versions":[{"version":"external-42a6691ed728cb8fb64026df7b292dff5dabcafff32b3fcf0f348c9ac6e8bbdc","resource":{"id":"jhir-llm-ed-triage-bias-20260622","title":"JHIR: ED-triage LLMs beat classic ML yet show rampant intersectional sex/race acuity bias","organization":"Journal of Healthcare Informatics Research research team (Lee et al.)","sector":"Academic clinical informatics / emergency department triage","geography":"United States-centric MIMIC-IV-ED plus ESI handbook and KTAS evaluation settings (transferability limits for field EMS)","publishedAt":"2026-06-22","publicationDate":"2026-06-22","eventDate":null,"sourceName":"Journal of Healthcare Informatics Research","sourceLabel":"Peer-reviewed LLM ED triage robustness and intersectional bias audit (DOI 10.1007/s41666-026-00241-z)","sourceUrl":"https://link.springer.com/content/pdf/10.1007/s41666-026-00241-z.pdf","evidenceClass":"academic-research","outcomeClass":"cautionary","topics":["developers-agents","governance-procurement","accessibility-workforce","data-security","operating-model"],"finding":"Lee et al. (Journal of Healthcare Informatics Research, published online 2026-06-22) evaluate LLMs for emergency department triage on distribution shift/missing data and intersectional sex/race bias via counterfactuals across ESI-Handbook, KTAS, and MIMIC-IV-ED. LLMs outperform traditional ML baselines and provide rationales; retrieving similar prior cases (KATE) outperforms chain-of-thought for triage accuracy. Bias findings are stark: under GPT-4o-mini vanilla prompting, Native Hawaiian women receive average acuity 2.38 versus 3.05 for White men (lower = higher priority)—nearly a full triage level—and the authors state biases are rampant across all models, with at least one significant demographic subcategory after multiple-testing correction. CoT prompting reduces some bias. Authors conclude LLMs should support—not replace—nurses, with equity and calibration still open.","sledRelevance":"High-value Pass B scrutiny for any EMS/ED or nurse-line triage LLM pitched to SLED buyers. Why now: not previously archived; supplies quantified intersectional acuity gaps that contradict 'neutral assistant' marketing as agencies expand clinical AI after recent Seattle/Corti oversight coverage.","evidence":"Opened Springer PDF via curl/pdftotext. Abstract claims, Table 1 method comparison narrative, Table 3 intersectional bias example (2.38 vs 3.05), rampant-bias statement, and conclusion on support-not-replace transcribed from the PDF.","architectureImplications":"If used at all, deploy as advisory scoring with demographic-ablation tests, retrieval-of-similar-cases carefully governed, and mandatory clinician final acuity—never autonomous routing.","governanceImplications":"Require counterfactual fairness audits and multiple-testing–aware bias reports in procurement; treat CoT as a bias mitigator to evaluate, not a cure-all.","securityPrivacyImplications":"MIMIC-style ED data and prompts containing demographics are sensitive; log access and restrict demographic fields in production prompts where policy requires.","caveats":"Hospital ED triage audit—not a live 911/PSAP study; retrospective/counterfactual design; KTAS/ESI handbook mix limits direct U.S. field-EMS transfer.","streamIds":["emergency-services"],"roles":{"sales":"Lead with caution: this June 2026 JHIR audit shows ED-triage LLMs can outperform classic ML yet encode intersectional acuity bias—e.g., nearly a full ESI-level gap between Native Hawaiian women and White men under GPT-4o-mini—so do not sell 'neutral' triage AI. Discovery should ask for local fairness audits, demographic prompt policies, and whether buyers want retrieval-augmented examples. Bounded engagement: counterfactual bias evaluation on de-identified local triage data. Unsupported claims include unbiased acuity, safe autonomous triage, or that chain-of-thought eliminates disparity.","engineering":"Fit is LLM-based ED acuity suggestion with rationale—best when retrieving similar prior cases—not autonomous ESI assignment. Architecture needs clinician confirmation, demographic ablation tests, and audit logs. Prerequisites: protocol-aligned labels, bias test harness, and clear handling of missing vitals. Constraints: hospital ED setting; counterfactual not prospective RCT; model-specific bias patterns. Proof of value pairs discrimination metrics with intersectional fairness deltas, not accuracy alone.","delivery":"Pilot advisory displays only, with equity dashboards and override logging before any protocol change. Dependencies include ED nursing leadership, privacy/IRB review, and model operations staff. Ownership stays with the EM medical director for clinical use and analytics for monitoring. Acceptance criteria: documented fairness audit, human final acuity on every case, and calibration checks addressing hesitancy on highest acuity. Risks include scaled inequity across high visit volumes and automation bias when fluent rationales sound confident."},"roleDetails":{"sales":{"customerProblem":"EDs exploring LLM triage need evidence on both accuracy and demographic fairness before go-live.","stakeholders":["ED medical directors","Triage nursing leads","Hospital compliance/equity offices","EMS medical directors (handoff)"],"discoveryQuestions":["Have you run counterfactual sex/race acuity audits?","Will demographics be in the prompt?","Is the LLM advisory or allowed to set ESI?","How will overrides and disparity metrics be logged?"],"valueHypothesis":"LLM triage support may improve robustness over classic ML only if fairness audits and human final acuity are mandatory.","potentialEngagement":"Local counterfactual fairness and calibration evaluation—not a production autonomous triage rollout.","unsupportedClaims":["Neutral/unbiased acuity across demographics","Safe replacement of nurse triage","Chain-of-thought removes all bias"]},"engineering":{"fit":"Advisory LLM acuity/rationale for ED triage with governed example retrieval—not autonomous routing.","architectureIntegration":"Triage note/vitals → LLM (optional KATE retrieval) → rationale + suggested acuity → clinician confirmation + bias monitors.","prerequisites":["Protocol-aligned labels","Fairness test harness","Override audit store","Missing-data policy"],"constraints":["ED not PSAP setting","Retrospective/counterfactual evidence","Model-specific bias axes"],"security":"Protect ED notes and demographic fields; credentialed data access analogous to MIMIC controls.","proofOfValue":["Accuracy/QWK vs baseline","Intersectional acuity gap tests","Undertriage rate on critical cases"]},"delivery":{"work":"Run fairness audit, configure advisory UI, train staff to challenge biased suggestions.","dependencies":["ED nursing education","Privacy/IRB","EHR integration","Equity office"],"ownership":"ED medical director owns clinical policy; analytics owns monitoring.","skillsAdoption":"Teach that fluent rationales can still encode demographic preference.","governanceCheckpoints":["Pre-go-live fairness gate","Monthly disparity review","Prompt/demography policy review"],"acceptanceCriteria":["Human final acuity","Published local bias report","Override logging complete"],"risks":["Scaled inequity","Automation bias","Prompt drift reintroducing demographics"]}},"retrievedAt":"2026-09-26T22:03:10-05:00","enrichedAt":"2026-09-26T22:03:10-05:00","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Biased acuity suggestions can worsen disparities for already underserved groups; training must teach staff to challenge model priority shifts.","procurementImplications":"Write fairness audit and override-logging requirements into triage-AI RFPs; do not accept accuracy-only demos.","operatingModelImplications":"Keep nurse/physician triage authoritative; LLM outputs advisory with equity monitoring dashboards.","sourceVerification":{"openedUrl":"https://link.springer.com/content/pdf/10.1007/s41666-026-00241-z.pdf","referenceExcerpt":"biases are rampant across all models: in every case, at least one demographic","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}