{"resourceId":"arxiv-equitriage-llm-ed-triage-gender-bias-20260505","versions":[{"version":"external-553260e728591e9af68dfb464ac74aace13dfa72e36399ae86591d8b43b005d1","resource":{"id":"arxiv-equitriage-llm-ed-triage-gender-bias-20260505","title":"EQUITRIAGE (arXiv): five-model fairness audit finds LLM ED triage gender sensitivity above 5% flip threshold","organization":"University of Nevada, Las Vegas / DeepNeuro AI (Young & Matthews)","sector":"Emergency department triage / clinical LLM fairness","geography":"United States (MIMIC-IV-ED, Beth Israel Deaconess Medical Center cohort; vignette audit)","publishedAt":"2026-05-05","publicationDate":"2026-05-05","eventDate":"2026-05-05","sourceName":"arXiv HTML","sourceLabel":"arXiv preprint EQUITRIAGE fairness audit of gender bias in LLM-based ESI triage","sourceUrl":"https://arxiv.org/html/2605.03998","evidenceClass":"academic-research","outcomeClass":"cautionary","topics":["governance-procurement","data-security","knowledge-work","accessibility-workforce"],"finding":"Young & Matthews (arXiv:2605.03998, submitted 5 May 2026) present EQUITRIAGE, auditing five LLMs (Gemini-3-Flash, Nemotron-3-Super, DeepSeek-V3.1, Mistral-Small-3.2, GPT-4.1-Nano) across 374,275 evaluations on MIMIC-IV-ED-derived vignettes with gender-swapped counterfactuals. All five models produced counterfactual flip rates above a pre-registered 5% threshold, ranging from 9.9% (GPT-4.1-Nano) to 43.8% (Nemotron-3-Super). Two models showed directional female undertriage (DeepSeek-V3.1 F/M 2.15; Gemini-3-Flash 1.34); chain-of-thought degraded human-concordance κw for all five and increased flip rates for four. Demographic blinding helped some models but not uniformly; authors conclude per-model counterfactual auditing should precede clinical deployment. Why now: newly relevant as emergency services scale AI for triage/QA assist (e.g., recent PSAP EMD QA and EMS AI prompting deployments), despite this being a text-vignette ED ESI audit—not live 911.","sledRelevance":"Cautionary evidence for any emergency-services AI that assigns or prompts acuity/priority from language—transferable as a fairness-testing requirement even though the study is ED ESI vignettes, not PSAP protocols.","evidence":"Large-scale counterfactual audit with pre-registered hypotheses/thresholds on deidentified MIMIC-IV-ED structured vignettes; explicitly not real nurse triage with gestalt cues; single-center source data.","architectureImplications":"If LLMs assist acuity assignment, require counterfactual/demographic stress tests per model and prompt strategy; do not assume CoT or fairness prompts are safe defaults.","governanceImplications":"Group calibration can look acceptable while counterfactual gender flips remain high (Chouldechova-style dissociation)—audits must include pair-level invariance, not only aggregate parity.","securityPrivacyImplications":"Study used deidentified structured fields plus synthetic names via cloud inference under PhysioNet DUA; production triage AI would face stricter PHI and re-identification controls.","caveats":"Text vignettes omit nurse gestalt; human ESI labels themselves may be biased; May 2026 preprint; ED not PSAP; blind strategy also strips age; not a deployed system trial.","streamIds":["emergency-services"],"roles":{"sales":"Bring EQUITRIAGE into EMS/ED/PSAP AI conversations as a caution: all five audited LLMs exceeded a 5% gender counterfactual flip threshold on ESI vignettes, with some female undertriage ratios above 2:1. Discovery should ask what fairness tests vendors ran, whether chain-of-thought is enabled, and how overrides are logged. Bounded engagement: design a counterfactual audit harness for the agency’s candidate triage/prompting model before go-live. Unsupported claims include direct applicability to live 911 protocols or proof that any specific commercial PSAP AI is biased.","engineering":"Before any LLM acuity assist, run gender (and intersectional) counterfactual pairs on local protocol or ESI-like labels; test baseline versus CoT versus blind/debiased prompts; monitor flip rate and directional F/M, not only accuracy. Prerequisites: representative vignette set, frozen prompt/model versions, and human-in-the-loop confirmation for high-acuity outputs. Constraints: vignette studies are not bedside triage, and debiasing effectiveness is model-specific across model vendors and prompt stacks.","delivery":"Add fairness-gate acceptance criteria to AI triage pilots, including a maximum flip-rate threshold, subgroup dashboards, and rollback if CoT or model updates worsen measured bias metrics. Ownership: clinical/quality lead for acuity policy; vendor for model cards and prompt change control. Risks include deploying the highest-accuracy model that is most directionally biased, and mistaking ED vignette results for PSAP operational clearance."},"roleDetails":{"sales":{"customerProblem":"Agencies piloting LLM triage/priority assist lack evidence that models are stable under demographic counterfactuals.","stakeholders":["ED/EMS medical directors","Quality/equity officers","Privacy counsel","AI vendors"],"discoveryQuestions":["What counterfactual tests exist?","Is CoT used in production prompts?","Who reviews high-acuity AI suggestions?","How are model updates controlled?"],"valueHypothesis":"Requiring EQUITRIAGE-style audits before go-live can catch gender sensitivity that accuracy metrics miss.","potentialEngagement":"Build a counterfactual ESI/protocol audit pack and score vendor models against a pre-registered flip threshold.","unsupportedClaims":["This proves a named 911 product is biased","Blinding always fixes bias","Vignette results equal bedside triage"]},"engineering":{"fit":"Offline fairness audit harness for LLM acuity assignment using counterfactual pairs.","architectureIntegration":"Candidate model+prompt → vignette batch → flip/F/M/κw reports → go/no-go gate before clinical UI.","prerequisites":["Vignette corpus","Frozen model/prompt IDs","HITL for ESI1/2","Change-control on prompts"],"constraints":["ED vignettes ≠ PSAP","Single-center MIMIC source","Age stripped in full blind"],"security":"Keep PHI local; if using cloud eval, send only deidentified synthetic vignettes under BAA/DUA.","proofOfValue":["Flip rate < pre-registered threshold","Directional F/M near 1.0","κw not collapsed under prompt strategy"]},"delivery":{"work":"Implement fairness gate in AI triage pilot plan with dashboards and rollback triggers.","dependencies":["Clinical quality","Vendor model access","Equity review board","Eval dataset steward"],"ownership":"Medical/quality lead owns acuity policy; vendor owns prompt/model change logs.","skillsAdoption":"Train reviewers to interpret flip/F/M vs accuracy and to challenge CoT defaults.","governanceCheckpoints":["Pre-prod fairness review","Post-update re-audit","Quarterly subgroup report"],"acceptanceCriteria":["Documented counterfactual audit","HITL on high acuity","Rollback threshold defined"],"risks":["Accuracy-only procurement","Prompt drift","Over-transfer from ED vignettes to 911"]}},"retrievedAt":"2026-10-01T22:07:40-05:00","enrichedAt":"2026-10-01T22:07:40-05:00","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Bias concentrated in some models for older female patients and chest-pain categories—equity monitoring must be part of clinical AI staffing, not an afterthought.","procurementImplications":"Require per-model counterfactual fairness evidence and prompt-strategy testing in RFPs for triage/acuity AI; reject CoT-as-default without local validation.","operatingModelImplications":"LLM acuity outputs should remain decision support with mandatory clinician confirmation for high-acuity classes and continuous subgroup monitoring.","sourceVerification":{"responseId":"curl-arxiv-equitriage-20261001T220740","openedUrl":"https://arxiv.org/html/2605.03998","referenceExcerpt":"All five models produced counterfactual flip rates above a pre-registered 5% threshold","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}