{"resourceId":"ceem-llm-multiagent-em-scoping-20260408","versions":[{"version":"external-42a6691ed728cb8fb64026df7b292dff5dabcafff32b3fcf0f348c9ac6e8bbdc","resource":{"id":"ceem-llm-multiagent-em-scoping-20260408","title":"CEEM scoping review: EM LLM evidence still mostly non-agentic tasks; prospective safety eval rare","organization":"Clinical and Experimental Emergency Medicine (Kim, Jo, Lim, Choi)","sector":"Academic emergency medicine / AI digital health review","geography":"International literature (Korea-based journal; studies spanning global EM/ED settings)","publishedAt":"2026-04-08","publicationDate":"2026-04-08","eventDate":null,"sourceName":"Clinical and Experimental Emergency Medicine","sourceLabel":"Peer-reviewed scoping review of non-agentic LLMs, agents, and multi-agent systems in emergency medicine","sourceUrl":"https://www.ceemjournal.org/journal/view.php?number=673","evidenceClass":"academic-research","outcomeClass":"mixed","topics":["developers-agents","operating-model","governance-procurement","accessibility-workforce"],"finding":"Kim et al. (Clin Exp Emerg Med, published online 2026-04-08) scoped Web of Science/Scopus/PubMed/CINAHL literature from 2021-03-08 to 2026-03-07 on non-agentic LLMs, LLM-based agents, and multi-agent systems in emergency medicine/ED, analyzing 35 studies (26 application, 6 framework, 3 benchmark). Non-agentic LLMs dominated (n=25) versus agents (n=7) and multi-agent systems (n=3). Applications clustered on triage, diagnostic/treatment support, and documentation; inputs were mostly text; evaluation relied mainly on expert comparison, retrospective review, vignettes, and task metrics. The authors state that workflow-level, prospective, and safety- and trustworthiness-oriented evaluations were limited, concluding the field remains focused on task-level non-agentic support rather than dynamic ED multi-agent validation.","sledRelevance":"Independent map of evidence maturity for EMS/ED buyers hearing multi-agent dispatch hype. Why now: not previously archived; pairs with today’s JHIR bias audit to show agentic EM claims outpace prospective safety evidence, after yesterday’s DORA/DispatchMAS themes.","evidence":"Opened CEEM HTML via curl (PDF also retrieved). Abstract counts (35; 26/6/3; n=25/7/3), task concentration, evaluation-method limitations sentence, and conclusion on non-agentic focus taken from the article page.","architectureImplications":"Prefer narrow, evaluable task assistants with logging before multi-agent orchestration; require multimodal and workflow-level test harnesses if agents are proposed.","governanceImplications":"Make prospective safety/trustworthiness evaluation a procurement gate for any agentic EM/ED system.","securityPrivacyImplications":"Agent tool-calling across EHR/CAD expands privacy and accountability surface; demand traceability the review finds lacking.","caveats":"Scoping review without pooled effect sizes; hospital EM-heavy corpus; journal pre-proof PDF notes possible further edits; limited PSAP-specific trials.","streamIds":["emergency-services"],"roles":{"sales":"Use this April 2026 CEEM scoping review to reset expectations: of 35 EM LLM studies, 25 are non-agentic task tools and only 3 are multi-agent, with prospective safety/trustworthiness evaluations still limited. Discovery should ask what prospective workflow trial the vendor will fund and which tasks (triage vs documentation) are in scope. Bounded engagement: evidence-gap workshop and pilot protocol design. Unsupported claims include mature multi-agent ED/EMS autonomy or that vignette accuracy equals operational safety.","engineering":"Fit is a literature-derived maturity map—not a product—showing text-centric, expert or retrospective evaluation still dominates emergency LLM research. If building agents, instrument workflow-level traces, multimodal inputs, and safety tests the review finds rare. Prerequisites include a clear task taxonomy, human override paths, and a prospective evaluation plan. Constraints: thin multi-agent evidence (n=3) and ED-centric studies. Proof of value is prospective safety and trust metrics, not another vignette leaderboard score.","delivery":"Sequence documentation or triage copilots with prospective evaluation before any multi-agent orchestration. Dependencies include EM research partners, IRB or ethics review, and operations teams who can instrument real workflows. Ownership: clinical lead for patient-facing use; AI governance for agent scope decisions. Acceptance criteria: prospective protocol registered, safety metrics defined, and a non-agentic baseline compared on workflow outcomes. Risks include compounding opacity and responsibility gaps as agent counts grow without accountability design."},"roleDetails":{"sales":{"customerProblem":"Buyers hear multi-agent EMS/ED promises but lack a map of what evidence actually exists.","stakeholders":["EM/ED medical directors","EMS chiefs","Hospital AI governance","Research/QA leads"],"discoveryQuestions":["Is the proposed system non-agentic, agentic, or multi-agent—and where is prospective evidence?","Which workflow metrics—not vignette accuracy—will be measured?","How is safety/trustworthiness evaluated pre-go-live?","What human override and responsibility model exists?"],"valueHypothesis":"Task-level LLM support may be justifiable where prospective evaluation is funded; multi-agent autonomy is premature on current evidence.","potentialEngagement":"Evidence-gap assessment and prospective pilot protocol—not a multi-agent production rollout.","unsupportedClaims":["Mature multi-agent EM autonomy","Vignette accuracy equals operational safety","Literature already validates workflow-level agents"]},"engineering":{"fit":"Guidance to constrain EM LLM architectures to evaluable tasks until prospective multi-agent evidence exists.","architectureIntegration":"Start with logged single-task assistants; add tools/agents only with traceable workflow evaluation harnesses.","prerequisites":["Task taxonomy","Override paths","Prospective eval plan","Trace/audit store"],"constraints":["Only n=3 multi-agent studies in review","Text-heavy inputs","Sparse safety eval"],"security":"Agent tool access to clinical/CAD data needs strict scopes and immutable traces for accountability.","proofOfValue":["Prospective workflow endpoints","Safety incident rates","Trust/override metrics"]},"delivery":{"work":"Pilot one non-agentic task with prospective measures before any multi-agent expansion.","dependencies":["Clinical champions","IRB/ethics","Ops instrumentation","Vendor eval cooperation"],"ownership":"Clinical lead owns patient impact; governance owns agent-scope decisions.","skillsAdoption":"Train staff on limits of vignette-validated tools in live workflows.","governanceCheckpoints":["Evidence-class review","Prospective protocol approval","Multi-agent expansion gate"],"acceptanceCriteria":["Registered prospective plan","Defined safety metrics","Human accountability documented"],"risks":["Opacity as agents multiply","Responsibility gaps","Premature autonomy"]}},"retrievedAt":"2026-09-26T22:03:11-05:00","enrichedAt":"2026-09-26T22:03:11-05:00","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Without prospective workflow studies, frontline staff bear unmeasured cognitive and trust burdens when agents are introduced.","procurementImplications":"Discount multi-agent marketing that cites only vignette accuracy; require prospective safety protocols in contracts.","operatingModelImplications":"Keep humans accountable for triage/dispatch decisions until workflow-level prospective evidence exists.","sourceVerification":{"openedUrl":"https://www.ceemjournal.org/journal/view.php?number=673","referenceExcerpt":"In contrast, workflow-level, prospective, and safety- and trustworthiness-oriented evaluations were limited.","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}