{"resourceId":"mdpi-healthcare-scoping-review-ai-ems-dispatch-no-external-validation-20260924","versions":[{"version":"external-4efd7b0692660050c4b899cda06843161d7c778c203e79d9d9690992d6409abb","resource":{"id":"mdpi-healthcare-scoping-review-ai-ems-dispatch-no-external-validation-20260924","title":"Scoping review: AI for EMS call-taking and dispatch lacks external validation, and two randomized trials show no consistent benefit","organization":"Yonsei University College of Medicine","sector":"Emergency medical dispatch research","geography":"International (18 studies, 13 countries)","publishedAt":"2026-09-24","publicationDate":"2026-09-24","eventDate":null,"sourceName":"MDPI Healthcare","sourceLabel":"Healthcare 14(19):3176 scoping review of AI in EMS call-taking and dispatch","sourceUrl":"https://www.mdpi.com/2227-9032/14/19/3176","evidenceClass":"academic-research","outcomeClass":"cautionary","topics":["knowledge-work","governance-procurement","data-security","operating-model"],"finding":"A PRISMA-ScR scoping review in MDPI Healthcare (vol. 14, issue 19, article 3176; Crossref publication date Sept 24, 2026) by Yonsei University authors searched six databases for January 2021 to August 2026 and included 18 studies from 13 countries on AI in EMS call-taking and dispatch; 11 focused on patient recognition and classification, five on dispatch support. Eleven were retrospective model-development studies with internal validation only, and none of these conducted independent external validation or calibration assessment; all 11 were rated high risk of bias under PROBAST+AI. The two randomized trials showed no consistent improvement: in Denmark, ML cardiac-arrest alerts had higher sensitivity (85.0% vs 77.5%) but PPV of 17.8% vs 55.8% and did not improve dispatcher recognition (93.1% vs 90.5%, p=0.15); Sweden's MADLAD showed modestly better low-acuity prioritization (68.3% vs 62.5%) with high risk of bias. The authors call for prospective multicenter studies, local recalibration, drift monitoring, dispatcher override authority and auditable logs. Why now: most recent synthesis of dispatch AI evidence, directly relevant to tonight's Korean guide recommending real-time 119 call-audio AI.","sledRelevance":"911 and EMS dispatch leaders weighing AI call-audio or triage tools can use it as a checklist of what evidence vendors should show: external validation, calibration and operational outcomes.","evidence":"Peer-reviewed PRISMA-ScR scoping review with risk-of-bias assessment.","architectureImplications":"Systems combine ASR, NLP and structured CAD data; must integrate with CAD without adding latency and with fallback for low-confidence or incomplete audio.","governanceImplications":"Dispatchers should keep override authority; AI outputs, overrides and final decisions should be logged and auditable; models need drift monitoring and recalibration.","securityPrivacyImplications":"Emergency-call audio contains health, location and bystander data requiring lawful-use, de-identification, retention and third-party access rules.","caveats":"No registered protocol; English search terms only; conference papers and preprints excluded, so fast-moving technical work is missed; risk-of-bias assessment by one reviewer and added during peer review; funded by a Korean fire/EMS technology program; MADLAD trial already archived separately; published about four weeks before this edition; opened via WebFetch because box curl returned 403.","streamIds":["emergency-services"],"roles":{"sales":"For sales, this review sets the evidence bar buyers should apply to any dispatch AI pitch: has the model been validated outside the data it was built on, and did outcomes improve in practice? Use it to position an independent evaluation offer. Do not cite model accuracy figures as proof of benefit; the randomized trials show accuracy did not translate into consistent operational gains.","engineering":"For engineering, plan for what the studies lacked: local validation on your own call audio and languages, calibration checks, and drift monitoring after go-live. Measure alert burden too; the Danish trial's PPV of 17.8% meant many false alerts. Integrate with CAD so recommendations add no latency and log every override for later analysis. Test separately for accents, languages and distressed callers, since speech models degrade there.","delivery":"For delivery, run any dispatch AI first in a silent or parallel mode with local validation, then a controlled rollout with dispatcher override and auditable logs. Define clinically meaningful metrics such as recognition time and under-triage, train dispatchers on automation bias, and schedule periodic recalibration reviews as protocols and case mix change. Publish the evaluation results to staff so dispatchers see why the tool is trusted or not."},"roleDetails":{"sales":{"customerProblem":"Dispatch centers face vendor claims for AI triage and recognition tools without evidence that they work locally.","stakeholders":["911 or EMS dispatch director","EMS medical director","Quality assurance manager","Procurement officer"],"discoveryQuestions":["Has the vendor validated on data from other centers?","What false-alert rate can your dispatchers absorb?","Which outcomes would prove value to you?"],"valueHypothesis":"An evidence-based evaluation plan could avoid buying tools whose accuracy does not translate into better dispatch outcomes.","potentialEngagement":"Independent evaluation design and silent-mode validation for a proposed dispatch AI tool.","unsupportedClaims":["Improved survival from dispatch AI","Generalizable model accuracy","Reduced dispatcher workload"]},"engineering":{"fit":"Evidence base for evaluating call-audio, transcript and structured-data AI in EMS dispatch.","architectureIntegration":"ASR and NLP on live call audio feeding CAD-integrated alerts, with logging of outputs and overrides.","prerequisites":["Local labeled call data","CAD integration","Outcome linkage to EMS records"],"constraints":["Accent, dialect and multilingual speech degrade models","Different protocols across centers","Latency limits in time-critical calls"],"security":"Govern secondary use of call audio, de-identify for development, and audit vendor access.","proofOfValue":["External validation on local calls","Calibration plot and PPV at chosen threshold","Silent-mode comparison with dispatcher decisions"]},"delivery":{"work":"Stage dispatch AI from silent validation to controlled rollout with monitoring.","dependencies":["Vendor","CAD team","Medical direction","Data governance"],"ownership":"Dispatch center director with EMS medical director.","skillsAdoption":"Dispatchers trained on interpreting alerts and resisting automation bias.","governanceCheckpoints":["Local validation sign-off","Override and output logging live before rollout","Scheduled drift and recalibration review"],"acceptanceCriteria":["Local sensitivity and PPV meet thresholds agreed before go-live","Every override logged and reviewed monthly","No increase in under-triage over six months"],"risks":["Alert fatigue from low PPV","Performance drift after protocol changes","Language-related misses"]}},"retrievedAt":"2026-10-20T22:05:00-05:00","enrichedAt":"2026-10-20T22:05:00-05:00","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Models may underperform for non-native speakers and distressed callers; dispatchers need training on automation bias.","procurementImplications":"Contracts should require external validation evidence, calibration data, logging and monitoring obligations.","operatingModelImplications":"AI as decision support with dispatcher final authority and fallback procedures.","sourceVerification":{"responseId":"webfetch-mdpi-healthcare-3176-2026-10-20","openedUrl":"https://www.mdpi.com/2227-9032/14/19/3176","referenceExcerpt":"None of the 11 model-development studies conducted independent external validation or calibration assessment","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}