{"resourceId":"arxiv-dora-disaster-llm-agents-260511633","versions":[{"version":"external-828409e150075cfaa38ca4d17f6ac6c1e3f128aaa5f7e39a7d4fc627c2bafea1","resource":{"id":"arxiv-dora-disaster-llm-agents-260511633","title":"DORA benchmark: frontier LLM disaster agents top out near 54% vs gold, with 56% gaps on long pipelines","organization":"University of Tokyo / RIKEN AIP / Waseda / Stanford (Wang et al.)","sector":"Academic AI research / disaster operations agents","geography":"Global disaster events corpus (45 sites / 10 disaster types); transferability caution for U.S. EOC ops","publishedAt":"2026-05-12","publicationDate":"2026-05-12","eventDate":null,"sourceName":"arXiv cs.AI preprint 2605.11633","sourceLabel":"Academic agentic benchmark for end-to-end disaster operational response (DORA)","sourceUrl":"https://arxiv.org/abs/2605.11633","evidenceClass":"academic-research","outcomeClass":"cautionary","topics":["developers-agents","infrastructure","operating-model","knowledge-work"],"finding":"DORA (arXiv 2605.11633, submitted 2026-05-12) evaluates LLM agents on 515 expert-authored tasks across 45 real disasters with 3,500 gold tool-call steps and a 108-tool MCP geospatial library. Strongest model (Gemini-3.0-Flash) reaches 53.74% average final-answer accuracy versus 80.48% gold-trajectory execution—about 26 points below. Gold tool-order hints improve accuracy only 1.08–4.40%; best scaffold gain +3.24%. Agent-to-gold gap widens from ~7% on short pipelines to 56% on ≥11-step heterogeneous pipelines. Dominant failure modes: disaster-pipeline composition (56.3%), damage-semantic grounding (20.3%), sensor-modality mismatch (14.8%).","sledRelevance":"Hard scrutiny for emergency-management buyers hearing agentic disaster-ops pitches. Why now: complements today’s RAND adoption landscape and FEMA inventory—shows that even when tools exist, long-horizon operational agents remain unreliable.","evidence":"Opened arXiv abs page via curl; cross-checked abstract metrics with HTML full text (WebFetch) for table values and failure-mode percentages.","architectureImplications":"Separate tool selection from argument grounding; log full trajectories; keep humans on planning/report synthesis outputs.","governanceImplications":"Require benchmarked, dimension-wise scores before any EOC agent pilot; prohibit autonomous alerting.","securityPrivacyImplications":"Sandbox geospatial tools; avoid wiring unvalidated agents to live population or facility layers.","caveats":"Preprint; simulated tool environment not a live EOC; model names/versions will age; not EMS clinical triage.","streamIds":["emergency-services"],"roles":{"sales":"Position DORA as independent proof that frontier LLM agents are not ready for end-to-end disaster operations: best models ~54% average accuracy vs ~80% gold trajectories, with gaps widening to 56% on long pipelines. Discovery should ask whether any proposed 'disaster agent' was evaluated on multi-step geospatial tool use with expert gold paths. Bounded engagement: tabletop using DORA-like tasks before any operational agent pilot. Unsupported claims include autonomous evacuation planning or situational reporting without human GIS/EM review.","engineering":"Fit is an evaluation harness—108 MCP tools, 515 tasks, dual trajectory/answer metrics—not a deployable EOC agent. If building agents, instrument tool-selection and argument grounding separately; gold tool-order hints only lift accuracy 1–4%. Prerequisites: curated geospatial data, expert trajectories, and typed answer schemas. Constraints: damage-semantic grounding, sensor-modality mismatch, and pipeline composition dominate failures. Proof of value must beat strong baselines on long heterogeneous pipelines, not chat demos.","delivery":"Delivery for any agent pilot should mirror DORA’s dual metrics: trajectory fidelity and final-answer correctness, plus efficiency against gold length. Map failure modes to local imagery and vector layers before EOC exposure. Dependencies include RS/GIS specialists, compute for multi-model eval, and EOC doctrine owners for answer schemas. Ownership: EM operations retains decision authority; research/IT owns agent evaluation harnesses. Acceptance criteria: published dimension-wise scores, human override on every operational recommendation, and an explicit non-autonomy statement. Risks include compositional error cascades and modality fusion mistakes under cloud, night, or SAR conditions."},"roleDetails":{"sales":{"customerProblem":"Vendors pitch LLM disaster agents, but buyers lack rigorous end-to-end operational benchmarks.","stakeholders":["EOC/GIS leads","Emergency managers","AI/research partners","Mutual-aid coordinators"],"discoveryQuestions":["Was the agent tested on multi-step tool pipelines with expert gold trajectories?","How does accuracy change as trajectory length grows?","What happens under SAR/optical/vector fusion?","Who verifies every operational recommendation?"],"valueHypothesis":"Agentic disaster AI may assist perception tasks but needs major gains before trusted planning/report synthesis.","potentialEngagement":"Benchmarking workshop mapping vendor claims to DORA-style dimensions and failure modes.","unsupportedClaims":["Frontier LLMs can autonomously run disaster operations","Scaffolding (ReAct/PE) closes the gold gap","Tool-order hints alone fix argument grounding"]},"engineering":{"fit":"Evaluation benchmark for disaster-response LLM agents using MCP tool libraries—not a production EOC system.","architectureIntegration":"ReAct-style agent loop over 108 typed MCP tools with trajectory logging and typed final answers.","prerequisites":["Geospatial data manifests","Expert gold trajectories","Dual-level scoring harness"],"constraints":["Domain grounding failures","Argument grounding bottleneck","Fragility on long heterogeneous pipelines"],"security":"Sandbox tools and data; do not connect unvalidated agents to live CAD/alerting.","proofOfValue":["Final-answer accuracy vs gold","Tool-exact-match / parameter accuracy","Length-binned gap analysis","Modality-stratified scores"]},"delivery":{"work":"Reproduce subset evals, map failure modes to local data, and gate any pilot on human-verified outputs.","dependencies":["GIS/RS experts","Compute for multi-model eval","EOC doctrine for answer schemas"],"ownership":"EM operations owns decisions; research lead owns benchmark runs.","skillsAdoption":"Train staff to distrust long-horizon agent plans until argument grounding is proven.","governanceCheckpoints":["No live alerting without HITL","Publish failure-mode report","Scaffolding ablation review"],"acceptanceCriteria":["Documented accuracy by dimension","Override logging","Explicit non-autonomy statement"],"risks":["Error cascades on long pipelines","Sensor modality mistakes","Overtrust from fluent reports"]}},"retrievedAt":"2026-09-25T22:04:34-05:00","enrichedAt":"2026-09-25T22:04:34-05:00","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Fluent but wrong agent reports can mislead fatigued EOC staff; training must emphasize verification.","procurementImplications":"Write eval gates (DORA-like tasks) into RFPs for disaster agents rather than demo scripts alone.","operatingModelImplications":"Keep agents advisory for perception; retain human GIS/EM control for routing, allocation, and public reports.","sourceVerification":{"openedUrl":"https://arxiv.org/abs/2605.11633","referenceExcerpt":"the agent-to-gold gap widening from 7% to 56% on long pipelines","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}