From the Emergency Services edition of October 1, 2026
EQUITRIAGE (arXiv): five-model fairness audit finds LLM ED triage gender sensitivity above 5% flip threshold
University of Nevada, Las Vegas / DeepNeuro AI (Young & Matthews) · Emergency department triage / clinical LLM fairness · United States (MIMIC-IV-ED, Beth Israel Deaconess Medical Center cohort; vignette audit)
- Publisher
- arXiv HTML
- Original publication
- 2026-05-05
- Source retrieved
- 2026-10-02
- Event date
- 2026-05-05
What happened
Young & Matthews (arXiv:2605.03998, submitted 5 May 2026) present EQUITRIAGE, auditing five LLMs (Gemini-3-Flash, Nemotron-3-Super, DeepSeek-V3.1, Mistral-Small-3.2, GPT-4.1-Nano) across 374,275 evaluations on MIMIC-IV-ED-derived vignettes with gender-swapped counterfactuals. All five models produced counterfactual flip rates above a pre-registered 5% threshold, ranging from 9.9% (GPT-4.1-Nano) to 43.8% (Nemotron-3-Super). Two models showed directional female undertriage (DeepSeek-V3.1 F/M 2.15; Gemini-3-Flash 1.34); chain-of-thought degraded human-concordance κw for all five and increased flip rates for four. Demographic blinding helped some models but not uniformly; authors conclude per-model counterfactual auditing should precede clinical deployment. Why now: newly relevant as emergency services scale AI for triage/QA assist (e.g., recent PSAP EMD QA and EMS AI prompting deployments), despite this being a text-vignette ED ESI audit—not live 911.
Why it matters
Cautionary evidence for any emergency-services AI that assigns or prompts acuity/priority from language—transferable as a fairness-testing requirement even though the study is ED ESI vignettes, not PSAP protocols.
Evidence and measured results
Large-scale counterfactual audit with pre-registered hypotheses/thresholds on deidentified MIMIC-IV-ED structured vignettes; explicitly not real nurse triage with gestalt cues; single-center source data.
Limitations and uncertainty
Text vignettes omit nurse gestalt; human ESI labels themselves may be biased; May 2026 preprint; ED not PSAP; blind strategy also strips age; not a deployed system trial.
Put this evidence to work
Lighthouse Advisory interpretation, grounded in this source. Enriched 2026-10-01; this does not change the original publication date. Each role follows the same checklist so takeaways compare across sources.
Sales
Role takeaway
- Customer problem
- Agencies piloting LLM triage/priority assist lack evidence that models are stable under demographic counterfactuals.
- Stakeholders
- ED/EMS medical directors
- Quality/equity officers
- Privacy counsel
- AI vendors
- Discovery questions
- What counterfactual tests exist?
- Is CoT used in production prompts?
- Who reviews high-acuity AI suggestions?
- How are model updates controlled?
- Value hypothesis
- Requiring EQUITRIAGE-style audits before go-live can catch gender sensitivity that accuracy metrics miss.
- Potential engagement
- Build a counterfactual ESI/protocol audit pack and score vendor models against a pre-registered flip threshold.
- Unsupported claims
- This proves a named 911 product is biased
- Blinding always fixes bias
- Vignette results equal bedside triage
Pre-sales engineering
Role takeaway
- Fit
- Offline fairness audit harness for LLM acuity assignment using counterfactual pairs.
- Architecture and integration
- Candidate model+prompt → vignette batch → flip/F/M/κw reports → go/no-go gate before clinical UI.
- Prerequisites
- Vignette corpus
- Frozen model/prompt IDs
- HITL for ESI1/2
- Change-control on prompts
- Constraints
- ED vignettes ≠ PSAP
- Single-center MIMIC source
- Age stripped in full blind
- Security
- Keep PHI local; if using cloud eval, send only deidentified synthetic vignettes under BAA/DUA.
- Proof of value
- Flip rate < pre-registered threshold
- Directional F/M near 1.0
- κw not collapsed under prompt strategy
Delivery
Role takeaway
- Work
- Implement fairness gate in AI triage pilot plan with dashboards and rollback triggers.
- Dependencies
- Clinical quality
- Vendor model access
- Equity review board
- Eval dataset steward
- Ownership
- Medical/quality lead owns acuity policy; vendor owns prompt/model change logs.
- Skills and adoption
- Train reviewers to interpret flip/F/M vs accuracy and to challenge CoT defaults.
- Governance checkpoints
- Pre-prod fairness review
- Post-update re-audit
- Quarterly subgroup report
- Acceptance criteria
- Documented counterfactual audit
- HITL on high acuity
- Rollback threshold defined
- Risks
- Accuracy-only procurement
- Prompt drift
- Over-transfer from ED vignettes to 911
Implementation considerations
Lighthouse Advisory interpretation across the operating dimensions a public-sector buyer must settle before this evidence becomes a design. Each note answers the question under its heading for this specific source.
Architecture and integration
What must connect, and where does the AI sit in the workflow?
If LLMs assist acuity assignment, require counterfactual/demographic stress tests per model and prompt strategy; do not assume CoT or fairness prompts are safe defaults.
Governance
Who approves, reviews and stays accountable for outcomes?
Group calibration can look acceptable while counterfactual gender flips remain high (Chouldechova-style dissociation)—audits must include pair-level invariance, not only aggregate parity.
Security and privacy
What data, permissions and controls need testing?
Study used deidentified structured fields plus synthetic names via cloud inference under PhysioNet DUA; production triage AI would face stricter PHI and re-identification controls.
Accessibility and workforce
Who is affected, and what skills or accommodations follow?
Bias concentrated in some models for older female patients and chest-pain categories—equity monitoring must be part of clinical AI staffing, not an afterthought.
Procurement
What should contracts, pricing and exit terms secure?
Require per-model counterfactual fairness evidence and prompt-strategy testing in RFPs for triage/acuity AI; reject CoT-as-default without local validation.
Operating model
Which teams own the service once it runs?
LLM acuity outputs should remain decision support with mandatory clinician confirmation for high-acuity classes and continuous subgroup monitoring.
Publication history
- 2026-10-01Emergency Services · Issue 265 resources
Stable resource ID: arxiv-equitriage-llm-ed-triage-gender-bias-20260505