From the Emergency Services edition of September 26, 2026
CEEM scoping review: EM LLM evidence still mostly non-agentic tasks; prospective safety eval rare
Clinical and Experimental Emergency Medicine (Kim, Jo, Lim, Choi) · Academic emergency medicine / AI digital health review · International literature (Korea-based journal; studies spanning global EM/ED settings)
- Publisher
- Clinical and Experimental Emergency Medicine
- Original publication
- 2026-04-08
- Source retrieved
- 2026-09-27
What happened
Kim et al. (Clin Exp Emerg Med, published online 2026-04-08) scoped Web of Science/Scopus/PubMed/CINAHL literature from 2021-03-08 to 2026-03-07 on non-agentic LLMs, LLM-based agents, and multi-agent systems in emergency medicine/ED, analyzing 35 studies (26 application, 6 framework, 3 benchmark). Non-agentic LLMs dominated (n=25) versus agents (n=7) and multi-agent systems (n=3). Applications clustered on triage, diagnostic/treatment support, and documentation; inputs were mostly text; evaluation relied mainly on expert comparison, retrospective review, vignettes, and task metrics. The authors state that workflow-level, prospective, and safety- and trustworthiness-oriented evaluations were limited, concluding the field remains focused on task-level non-agentic support rather than dynamic ED multi-agent validation.
Why it matters
Independent map of evidence maturity for EMS/ED buyers hearing multi-agent dispatch hype. Why now: not previously archived; pairs with today’s JHIR bias audit to show agentic EM claims outpace prospective safety evidence, after yesterday’s DORA/DispatchMAS themes.
Evidence and measured results
Opened CEEM HTML via curl (PDF also retrieved). Abstract counts (35; 26/6/3; n=25/7/3), task concentration, evaluation-method limitations sentence, and conclusion on non-agentic focus taken from the article page.
Limitations and uncertainty
Scoping review without pooled effect sizes; hospital EM-heavy corpus; journal pre-proof PDF notes possible further edits; limited PSAP-specific trials.
Put this evidence to work
Lighthouse Advisory interpretation, grounded in this source. Enriched 2026-09-26; this does not change the original publication date. Each role follows the same checklist so takeaways compare across sources.
Sales
Role takeaway
- Customer problem
- Buyers hear multi-agent EMS/ED promises but lack a map of what evidence actually exists.
- Stakeholders
- EM/ED medical directors
- EMS chiefs
- Hospital AI governance
- Research/QA leads
- Discovery questions
- Is the proposed system non-agentic, agentic, or multi-agent—and where is prospective evidence?
- Which workflow metrics—not vignette accuracy—will be measured?
- How is safety/trustworthiness evaluated pre-go-live?
- What human override and responsibility model exists?
- Value hypothesis
- Task-level LLM support may be justifiable where prospective evaluation is funded; multi-agent autonomy is premature on current evidence.
- Potential engagement
- Evidence-gap assessment and prospective pilot protocol—not a multi-agent production rollout.
- Unsupported claims
- Mature multi-agent EM autonomy
- Vignette accuracy equals operational safety
- Literature already validates workflow-level agents
Pre-sales engineering
Role takeaway
- Fit
- Guidance to constrain EM LLM architectures to evaluable tasks until prospective multi-agent evidence exists.
- Architecture and integration
- Start with logged single-task assistants; add tools/agents only with traceable workflow evaluation harnesses.
- Prerequisites
- Task taxonomy
- Override paths
- Prospective eval plan
- Trace/audit store
- Constraints
- Only n=3 multi-agent studies in review
- Text-heavy inputs
- Sparse safety eval
- Security
- Agent tool access to clinical/CAD data needs strict scopes and immutable traces for accountability.
- Proof of value
- Prospective workflow endpoints
- Safety incident rates
- Trust/override metrics
Delivery
Role takeaway
- Work
- Pilot one non-agentic task with prospective measures before any multi-agent expansion.
- Dependencies
- Clinical champions
- IRB/ethics
- Ops instrumentation
- Vendor eval cooperation
- Ownership
- Clinical lead owns patient impact; governance owns agent-scope decisions.
- Skills and adoption
- Train staff on limits of vignette-validated tools in live workflows.
- Governance checkpoints
- Evidence-class review
- Prospective protocol approval
- Multi-agent expansion gate
- Acceptance criteria
- Registered prospective plan
- Defined safety metrics
- Human accountability documented
- Risks
- Opacity as agents multiply
- Responsibility gaps
- Premature autonomy
Implementation considerations
Lighthouse Advisory interpretation across the operating dimensions a public-sector buyer must settle before this evidence becomes a design. Each note answers the question under its heading for this specific source.
Architecture and integration
What must connect, and where does the AI sit in the workflow?
Prefer narrow, evaluable task assistants with logging before multi-agent orchestration; require multimodal and workflow-level test harnesses if agents are proposed.
Governance
Who approves, reviews and stays accountable for outcomes?
Make prospective safety/trustworthiness evaluation a procurement gate for any agentic EM/ED system.
Security and privacy
What data, permissions and controls need testing?
Agent tool-calling across EHR/CAD expands privacy and accountability surface; demand traceability the review finds lacking.
Accessibility and workforce
Who is affected, and what skills or accommodations follow?
Without prospective workflow studies, frontline staff bear unmeasured cognitive and trust burdens when agents are introduced.
Procurement
What should contracts, pricing and exit terms secure?
Discount multi-agent marketing that cites only vignette accuracy; require prospective safety protocols in contracts.
Operating model
Which teams own the service once it runs?
Keep humans accountable for triage/dispatch decisions until workflow-level prospective evidence exists.
Publication history
- 2026-09-26Emergency Services · Issue 215 resources
Stable resource ID: ceem-llm-multiagent-em-scoping-20260408