From the Emergency Services edition of September 25, 2026
DORA benchmark: frontier LLM disaster agents top out near 54% vs gold, with 56% gaps on long pipelines
University of Tokyo / RIKEN AIP / Waseda / Stanford (Wang et al.) · Academic AI research / disaster operations agents · Global disaster events corpus (45 sites / 10 disaster types); transferability caution for U.S. EOC ops
- Publisher
- arXiv cs.AI preprint 2605.11633
- Original publication
- 2026-05-12
- Source retrieved
- 2026-09-26
What happened
DORA (arXiv 2605.11633, submitted 2026-05-12) evaluates LLM agents on 515 expert-authored tasks across 45 real disasters with 3,500 gold tool-call steps and a 108-tool MCP geospatial library. Strongest model (Gemini-3.0-Flash) reaches 53.74% average final-answer accuracy versus 80.48% gold-trajectory execution—about 26 points below. Gold tool-order hints improve accuracy only 1.08–4.40%; best scaffold gain +3.24%. Agent-to-gold gap widens from ~7% on short pipelines to 56% on ≥11-step heterogeneous pipelines. Dominant failure modes: disaster-pipeline composition (56.3%), damage-semantic grounding (20.3%), sensor-modality mismatch (14.8%).
Why it matters
Hard scrutiny for emergency-management buyers hearing agentic disaster-ops pitches. Why now: complements today’s RAND adoption landscape and FEMA inventory—shows that even when tools exist, long-horizon operational agents remain unreliable.
Evidence and measured results
Opened arXiv abs page via curl; cross-checked abstract metrics with HTML full text (WebFetch) for table values and failure-mode percentages.
Limitations and uncertainty
Preprint; simulated tool environment not a live EOC; model names/versions will age; not EMS clinical triage.
Put this evidence to work
Lighthouse Advisory interpretation, grounded in this source. Enriched 2026-09-25; this does not change the original publication date. Each role follows the same checklist so takeaways compare across sources.
Sales
Role takeaway
- Customer problem
- Vendors pitch LLM disaster agents, but buyers lack rigorous end-to-end operational benchmarks.
- Stakeholders
- EOC/GIS leads
- Emergency managers
- AI/research partners
- Mutual-aid coordinators
- Discovery questions
- Was the agent tested on multi-step tool pipelines with expert gold trajectories?
- How does accuracy change as trajectory length grows?
- What happens under SAR/optical/vector fusion?
- Who verifies every operational recommendation?
- Value hypothesis
- Agentic disaster AI may assist perception tasks but needs major gains before trusted planning/report synthesis.
- Potential engagement
- Benchmarking workshop mapping vendor claims to DORA-style dimensions and failure modes.
- Unsupported claims
- Frontier LLMs can autonomously run disaster operations
- Scaffolding (ReAct/PE) closes the gold gap
- Tool-order hints alone fix argument grounding
Pre-sales engineering
Role takeaway
- Fit
- Evaluation benchmark for disaster-response LLM agents using MCP tool libraries—not a production EOC system.
- Architecture and integration
- ReAct-style agent loop over 108 typed MCP tools with trajectory logging and typed final answers.
- Prerequisites
- Geospatial data manifests
- Expert gold trajectories
- Dual-level scoring harness
- Constraints
- Domain grounding failures
- Argument grounding bottleneck
- Fragility on long heterogeneous pipelines
- Security
- Sandbox tools and data; do not connect unvalidated agents to live CAD/alerting.
- Proof of value
- Final-answer accuracy vs gold
- Tool-exact-match / parameter accuracy
- Length-binned gap analysis
- Modality-stratified scores
Delivery
Role takeaway
- Work
- Reproduce subset evals, map failure modes to local data, and gate any pilot on human-verified outputs.
- Dependencies
- GIS/RS experts
- Compute for multi-model eval
- EOC doctrine for answer schemas
- Ownership
- EM operations owns decisions; research lead owns benchmark runs.
- Skills and adoption
- Train staff to distrust long-horizon agent plans until argument grounding is proven.
- Governance checkpoints
- No live alerting without HITL
- Publish failure-mode report
- Scaffolding ablation review
- Acceptance criteria
- Documented accuracy by dimension
- Override logging
- Explicit non-autonomy statement
- Risks
- Error cascades on long pipelines
- Sensor modality mistakes
- Overtrust from fluent reports
Implementation considerations
Lighthouse Advisory interpretation across the operating dimensions a public-sector buyer must settle before this evidence becomes a design. Each note answers the question under its heading for this specific source.
Architecture and integration
What must connect, and where does the AI sit in the workflow?
Separate tool selection from argument grounding; log full trajectories; keep humans on planning/report synthesis outputs.
Governance
Who approves, reviews and stays accountable for outcomes?
Require benchmarked, dimension-wise scores before any EOC agent pilot; prohibit autonomous alerting.
Security and privacy
What data, permissions and controls need testing?
Sandbox geospatial tools; avoid wiring unvalidated agents to live population or facility layers.
Accessibility and workforce
Who is affected, and what skills or accommodations follow?
Fluent but wrong agent reports can mislead fatigued EOC staff; training must emphasize verification.
Procurement
What should contracts, pricing and exit terms secure?
Write eval gates (DORA-like tasks) into RFPs for disaster agents rather than demo scripts alone.
Operating model
Which teams own the service once it runs?
Keep agents advisory for perception; retain human GIS/EM control for routing, allocation, and public reports.
Publication history
- 2026-09-25Emergency Services · Issue 205 resources
Stable resource ID: arxiv-dora-disaster-llm-agents-260511633