{"resourceId":"bmc-ems-triage-ml-vs-manual-20260424","versions":[{"version":"external-828409e150075cfaa38ca4d17f6ac6c1e3f128aaa5f7e39a7d4fc627c2bafea1","resource":{"id":"bmc-ems-triage-ml-vs-manual-20260424","title":"BMC EM: EMS red-zone ML matches physician triage (AUROC ~0.84) but not significantly better","organization":"Tertiary academic emergency department research team (Selvi et al.)","sector":"Emergency medicine / EMS academic research","geography":"Single-center high-volume tertiary ED (international; transferability limits for U.S. EMS)","publishedAt":"2026-04-24","publicationDate":"2026-04-24","eventDate":null,"sourceName":"BMC Emergency Medicine","sourceLabel":"Peer-reviewed comparative study of manual triage systems vs ML for EMS-transported ED patients","sourceUrl":"https://link.springer.com/article/10.1186/s12873-026-01595-8","evidenceClass":"academic-research","outcomeClass":"mixed","topics":["operating-model","developers-agents","accessibility-workforce","data-security"],"finding":"Open-access BMC Emergency Medicine study (published 2026-04-24): single-center analysis of 899 adult EMS patients triaged to the ED red zone compared ESI/MTS/TMHTS with RF, GBM, and CatBoost for ICU admission or in-hospital death (9.1% adverse outcomes). On the held-out test set (n=270), physician ESI AUROC 0.814 vs RF 0.840 / GBM 0.841 / CatBoost 0.842; all bootstrap DeLong pairwise p>0.05 amid only 21–25 adverse events. Authors conclude ML performance is robust and comparable—supporting complementary decision support—and call for multicenter validation. Limitations include single-center scope, consent/missing-data exclusions, and no external validation.","sledRelevance":"Measured EMS→ED triage evidence that ML can match but not statistically beat manual systems on rare adverse outcomes—useful counterweight to dispatch-AI marketing. Why now: after recent MADLAD/npj EMS ML coverage, this adds a high-acuity comparative negative on superiority claims.","evidence":"Opened Springer/BMC HTML via curl. Abstract methods/results, Table 4 AUROC/CI/p-values, conclusions, and limitations transcribed from the article text.","architectureImplications":"Score-as-complement beside ESI/MTS with explanation UI; retain clinician final acuity.","governanceImplications":"Demand powered superiority/non-inferiority designs and external validation before replacing triage protocols.","securityPrivacyImplications":"Clinical variables and outcomes require hospital/EMS data agreements and audit of score access.","caveats":"International single center; red-zone only; underpowered for DeLong differences; not a 911-dispatch RCT.","streamIds":["emergency-services"],"roles":{"sales":"Present this BMC EM study as measured but non-superior ML for high-acuity EMS→ED triage: CatBoost/RF/GBM AUROCs ~0.84 vs physician ESI 0.814 with all DeLong p>0.05 on a small adverse-event test set. Discovery should ask for local outcome labels (ICU/death), whether red-zone-only results generalize, and appetite for complementary—not replacement—decision support. Bounded engagement: retrospective external validation on local ePCR/ED data. Unsupported claims include statistically proven superiority over ESI/MTS or safe autonomous triage.","engineering":"Fit is tabular ML (RF/GBM/CatBoost) on routinely available clinical variables for adverse-outcome risk among EMS-transported red-zone adults—not NLP dispatch or CAD automation. Integrate as ED/EMS decision-support scores with SHAP explanations. Prerequisites: complete triage and outcome data, external validation plan, and power for rare events. Constraints: single-center Turkey tertiary ED; 21–25 adverse events in test split; no external validation. Proof of value requires significant improvement or clear calibration gains in prospective multicenter tests.","delivery":"Delivery should keep physician triage authoritative while piloting score displays beside ESI or MTS. Wire ePCR and ED outcome extracts, stand up override logging, and schedule drift reviews before any protocol change. Dependencies include EMS and ED data stewards, ethics review, and triage educators who can explain non-significant AUROC edges. Ownership stays with EM medical leadership for clinical use and analytics for model operations. Acceptance criteria: preregistered comparison plan, multicenter validation commitment, and documented human final acuity. Risks include overinterpreting underpowered DeLong tests and automation complacency on rare adverse events."},"roleDetails":{"sales":{"customerProblem":"ED needs better risk stratification for EMS arrivals but lacks evidence ML beats trained triage nurses/physicians.","stakeholders":["ED medical directors","EMS medical directors","Triage nursing leads","Hospital analytics"],"discoveryQuestions":["What adverse-outcome rate and sample size can you power?","Will ML be complementary to ESI/MTS or proposed as replacement?","Can you externally validate beyond one center?","How will overrides be logged?"],"valueHypothesis":"ML may match manual triage discrimination and slightly improve calibration as complementary support if prospectively validated.","potentialEngagement":"Local retrospective validation and sample-size planning—not a production triage AI rollout.","unsupportedClaims":["Statistically significant superiority over physician ESI","Generalization beyond red-zone EMS adults at one center","Safe replacement of manual triage"]},"engineering":{"fit":"Complementary adverse-outcome risk model for high-acuity EMS arrivals using structured clinical features.","architectureIntegration":"Batch/real-time scoring service with SHAP explanations beside ESI/MTS; no autonomous routing.","prerequisites":["Linked EMS-ED outcomes","Feature dictionary","Prospective monitoring","External validation cohort"],"constraints":["Single-center","Few adverse events","Red-zone adults only","Internal validation only"],"security":"Protect clinical features under hospital privacy rules; limit model training exports.","proofOfValue":["DeLong-tested AUROC vs ESI","Brier calibration","Prospective decision-impact study"]},"delivery":{"work":"Reproduce features locally, validate, and design HITL display with override audit.","dependencies":["EHR/ePCR extracts","IRB/ethics","Triage educators"],"ownership":"ED medical director owns clinical use; analytics owns model ops.","skillsAdoption":"Train triage staff that NS AUROC edges are not mandates to change acuity.","governanceCheckpoints":["External validation gate","Power analysis","Go-live HITL review"],"acceptanceCriteria":["Documented non-inferiority/superiority plan","Override logging","Multicenter commitment"],"risks":["Overfit to rare events","Consent/selection bias","Automation complacency"]}},"retrievedAt":"2026-09-25T22:04:35-05:00","enrichedAt":"2026-09-25T22:04:35-05:00","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"May reduce cognitive load if used as a second look, but training must prevent automation bias given non-significant gains.","procurementImplications":"Buy validation services and data plumbing before triage-AI licenses; avoid sole-source claims of beating ESI.","operatingModelImplications":"Keep manual triage authoritative; ML outputs advisory pending multicenter prospective trials.","sourceVerification":{"openedUrl":"https://link.springer.com/article/10.1186/s12873-026-01595-8","referenceExcerpt":"None of the pairwise comparisons between ML models and their corresponding physician triage reference reached statistical significance","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}