{"resourceId":"healthcare-mdpi-ai-ems-call-taking-dispatch-scoping-review-20260924","versions":[{"version":"external-7efc54f054a84ba30b20dd1ce4dabff5f3d0c355f9da65520e64084100c91ee7","resource":{"id":"healthcare-mdpi-ai-ems-call-taking-dispatch-scoping-review-20260924","title":"Scoping review of AI in EMS call-taking and dispatch: no independent external validation in 11 model-development studies, and two trials without consistent benefit","organization":"Yonsei University and Yonsei University Health System (published in Healthcare, MDPI)","sector":"EMS call-taking and dispatch research","geography":"International (18 studies from 13 countries and one multinational setting); authors in Republic of Korea","publishedAt":"2026-09-24","publicationDate":"2026-09-24","eventDate":null,"sourceName":"Healthcare (Basel) via PubMed Central","sourceLabel":"Peer-reviewed scoping review","sourceUrl":"https://pmc.ncbi.nlm.nih.gov/articles/PMC13665075/","evidenceClass":"academic-research","outcomeClass":"cautionary","topics":["governance-procurement","operating-model"],"finding":"A PRISMA-ScR scoping review searched six databases for studies from January 2021 to August 2026 and included 18 studies from 13 countries and one multinational setting. Eleven studies (61.1%) addressed patient recognition and classification and five (27.8%) dispatch support. Eleven were retrospective model development with internal validation, and only two were randomized trials. None of the 11 model-development studies did independent external validation or calibration assessment, and the two trials (one at high risk of bias, one with some concerns) showed no consistent improvement across dispatch outcomes. The authors conclude evidence is insufficient to support widespread implementation and call for prospective real-world studies, local validation and monitoring for drift.","sledRelevance":"Independent, peer-reviewed context for how much weight to give vendor and pilot claims about AI triage and dispatch support. Newly relevant now because the archive already holds several live-deployment and simulation items (Seattle, Seoul, Wellington) and this review supplies the evidence-quality frame. It is global, not U.S.-specific, so transferability is limited.","evidence":"Scoping review in Healthcare (Basel) 2026;14(19):3176, received 14 Jul 2026, accepted 21 Sep 2026, published 24 Sep 2026. Descriptive synthesis, no meta-analysis. Findings as stated in the opened article's abstract and discussion.","architectureImplications":"Interpretation: predictive performance is not clinical or operational utility; designs should allow local recalibration and drift monitoring, because the authors note performance can degrade as terminology, protocols and case mix change.","governanceImplications":"The authors call for prospective and real-world implementation studies before integration, local validation and, where needed, recalibration for language, protocols, prevalence and resources. Interpretation: make external validation and ongoing monitoring a procurement and go-live gate.","securityPrivacyImplications":"Not the focus of the review. Interpretation: call audio and text used for model development raise data-governance requirements that this review does not assess.","caveats":"Scoping review, descriptive and not a meta-analysis; no pooled effect sizes. Studies are heterogeneous and mostly retrospective, from outside the U.S. in many cases. Search window ended August 2026. The review is not an evaluation of any specific product.","streamIds":["emergency-services"],"roles":{"sales":"Use this review to set honest expectations. Across 18 studies, most AI for call-taking and dispatch was tested retrospectively with internal validation, none of the 11 model-development studies reported independent external validation, and the two randomized trials showed no consistent benefit. Customers asking for AI triage should expect conversations about local validation, drift monitoring and prospective pilots. Avoid promising clinical or response-time gains; position offerings around governed, measured pilots with clear stop criteria.","engineering":"The key engineering message is that offline accuracy does not guarantee operational value. Build proofs of value that validate on the agency's own data, check calibration, test subgroups such as language and accent, and monitor performance over time as protocols change. Plan for human-in-the-loop override logging and for retraining or recalibration. Treat any vendor accuracy figure without external or prospective validation as unverified for the customer's setting.","delivery":"Delivery plans should include a validation phase before go-live, a shadow or parallel run, and post-launch drift monitoring with named owners. Governance checkpoints should require evidence of local validation, clear dispatcher authority and documented override handling. Acceptance criteria should be set with the medical director and the dispatch center, and should measure operational outcomes rather than model scores alone, since the review shows the two can diverge."},"roleDetails":{"sales":{"customerProblem":"Agencies are pitched AI triage and dispatch tools whose reported accuracy rarely comes from independent or prospective testing.","stakeholders":["911 center director","EMS medical director","Fire/EMS chief","Quality assurance manager","Procurement and risk officers"],"discoveryQuestions":["What validation evidence would you require before using AI prompts on live calls?","Who would review performance across languages and accents?","What outcome, not model score, would define success?"],"valueHypothesis":"A validation-and-monitoring framework may lower adoption risk for AI call-taking tools; the review supports the need but not any specific benefit.","potentialEngagement":"Bounded evidence review and pilot design: local validation plan, shadow-mode run and monitoring dashboard specification.","unsupportedClaims":["That AI improves dispatch outcomes","That any specific product has been externally validated","That results from these 18 studies apply to a given U.S. center"]},"engineering":{"fit":"Directly relevant to designing proofs of value for call-taking and dispatch AI.","architectureIntegration":"Shadow-mode inference alongside CAD and call-taking with logging of model output, dispatcher action and outcome for later calibration checks.","prerequisites":["Historical call and outcome data for local validation","Data-sharing and privacy approvals","Baseline performance metrics"],"constraints":["Evidence is mostly retrospective and non-U.S.","Performance may drift as protocols and terminology change","Subgroup performance is often unreported"],"security":"Not assessed by the review; call audio and clinical data need agency-level privacy review.","proofOfValue":["External-style validation on a held-out local dataset","Calibration check and subgroup analysis","Shadow-mode comparison with dispatcher decisions before any live prompt"]},"delivery":{"work":"Sequence the project as local validation, shadow-mode run, limited live pilot, then monitoring and recalibration.","dependencies":["Local data access","Medical director protocol alignment","Analytics staff for monitoring"],"ownership":"Dispatch center director with the EMS medical director; QA lead owns monitoring.","skillsAdoption":"Dispatcher training on advisory prompts and override documentation; analyst training on drift monitoring.","governanceCheckpoints":["Go/no-go after local validation","Review after shadow-mode run","Periodic drift and equity review"],"acceptanceCriteria":["Proposed: documented local validation with calibration and subgroup results","Proposed: shadow-mode agreement and miss analysis reviewed by medical director","Proposed: monthly drift report with named owner"],"risks":["Model degradation over time","Over-reliance on prompts","Unrepresentative training data","Metrics focused on model scores rather than outcomes"]}},"retrievedAt":"2026-11-11T04:02:08Z","enrichedAt":"2026-11-11T04:02:08Z","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"The authors note language, documentation practices and resources differ across systems, so local validation matters; dispatcher workload effects are not quantified in the review.","procurementImplications":"Interpretation: require evidence of external or prospective validation, local recalibration support and drift monitoring in solicitations.","operatingModelImplications":"The review concludes AI should be integrated only after prospective and real-world implementation studies establish safety and utility; ongoing monitoring is an operating responsibility.","peerReview":"peer-reviewed","sourceVerification":{"openedUrl":"https://pmc.ncbi.nlm.nih.gov/articles/PMC13665075/","referenceExcerpt":"None of the 11 model-development studies conducted independent external validation or calibration assessment","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}