{"resourceId":"arxiv-disasterbench-llm-planning-20260527","versions":[{"version":"external-d37cc357cd81f1553c7b3e1c6d59379cee1cf0519dc1898699ea3d1319655121","resource":{"id":"arxiv-disasterbench-llm-planning-20260527","title":"arXiv DisasterBench: LLM disaster-response planners fail first on tool mismatch and parameter binding","organization":"Texas A&M University et al.","sector":"Disaster-response AI planning research / emergency management tooling","geography":"International research (U.S. author institutions); transferability limited pending local tool schemas","publishedAt":"2026-05-27","publicationDate":"2026-05-27","eventDate":null,"sourceName":"arXiv (cs.CL)","sourceLabel":"arXiv HTML for DisasterBench LLM planning under typed tool interface constraints (2605.27957v1)","sourceUrl":"https://arxiv.org/html/2605.27957","evidenceClass":"academic-research","outcomeClass":"cautionary","topics":["developers-agents","infrastructure","operating-model","knowledge-work"],"finding":"Chen et al. (arXiv:2605.27957v1, cs.CL, 27 May 2026) introduce DisasterBench for LLM planning under typed tool-interface constraints in disaster-response multi-agent workflows. From expert-designed workflows they derive 233 planning tasks over a pool of 26 tools spanning perception to multi-stage branching (e.g., satellite analysis, nowcasting, flood modeling, damage assessment). Models receive tool specs/schemas without underlying workflow structure. First-Point-of-Failure (FPoF) localizes earliest root causes. Findings: planning-method gains depend strongly on model capacity; tool mismatch and parameter-binding errors dominate first failures (semantic grounding vs execution consistency); verbose intermediate reasoning can clash with structured-output requirements. Frontier example: Gemini 3.1 Pro Preview ToT reaches 73.39 exact-match in reported table cells, with lower scores for weaker models/methods. Why now: newly relevant as agencies consider agentic EM orchestration amid expanding PSAP/EMS AI tooling.","sledRelevance":"Provides measurable failure modes for agentic disaster-management AI—directly informative for emergency-management stream boundaries (not policing).","evidence":"Academic benchmark paper with task counts, tool pool size, FPoF taxonomy, and model/method score tables; not an operational deployment study.","architectureImplications":"Requires typed tool registries, admissible workflow edges, and structured-output planners; unconstrained chat agents are a poor fit for multi-stage EM toolchains.","governanceImplications":"Supports require-before-pilot evaluation of executable plans and human approval gates when FPoF classes are tool-mismatch heavy.","securityPrivacyImplications":"Multi-tool agents that can invoke sensing/APIs need scoped credentials and invocation audit trails (paper focuses on correctness, not auth).","caveats":"Synthetic/expert tasks; May 2026 preprint; scores are plan exact-match not lives saved; why-now interpretive relative to concurrent PSAP/EMS AI expansion.","streamIds":["emergency-services"],"roles":{"sales":"Position DisasterBench as a caution for agentic emergency-management copilots: across 233 tasks and 26 tools, first failures concentrate in tool mismatch and parameter binding—not witty plans. Discovery should ask which EM tools have typed schemas and how first-point-of-failure will be tested before duty-officer use. Bounded engagement: run a DisasterBench-style schema-validation drill on a jurisdiction’s tool stack. Unsupported claims include live wildfire/flood outcome proof from this benchmark alone.","engineering":"Benchmark requires LLMs to emit executable multi-step plans over typed disaster tools without seeing gold workflows; FPoF separates primary errors from cascades. Prerequisites for analogous ops: tool I/O schemas, admissible edge rules, and offline evaluation harness. Constraints: synthetic tasks; English schemas. Proof of value: exact-match plan validity and FPoF mix under your tools—not chatbot demos.","delivery":"Before agentic EM assist goes near watch centers, inventory tools with typed interfaces, ban unconstrained tool selection, and require schema validation plus human approve on first failure classes. Ownership: EM tech lead for tool registry; vendors for adapters. Risks: verbose CoT breaking structured outputs; cascading wrong tool chains in time-critical incidents."},"roleDetails":{"sales":{"customerProblem":"Agencies want multi-step AI for disaster workflows but lack evidence models can bind the right tools safely.","stakeholders":["Emergency managers","GIS/remote-sensing analysts","EM IT","Vendors of hazard tools"],"discoveryQuestions":["Which tools have typed schemas?","How are inadmissible edges blocked?","What is our FPoF test set?","Who approves agent plans?"],"valueHypothesis":"Agentic EM help is credible only after schema-validated planning beats unconstrained LLM baselines on local tools.","potentialEngagement":"Build a 20–40 task local tool-planning eval with FPoF tagging before any pilot.","unsupportedClaims":["Benchmark equals field readiness","Any frontier model is safe for duty use"]},"engineering":{"fit":"Evaluation harness for LLM multi-agent disaster-response planning under typed tool constraints.","architectureIntegration":"Planner LLM ↔ tool registry with I/O schemas ↔ workflow edge constraints ↔ FPoF diagnostics.","prerequisites":["Tool schema catalog","Admissible dependency graph","Gold plan set or expert review","Eval compute"],"constraints":["Not live incidents","Curated 26-tool pool","Language/schema limits"],"security":"Tool agents that call external sensing/APIs need auth scoping and audit logs for every invocation.","proofOfValue":["Exact-match plan rate","FPoF taxonomy mix","Latency under structured-output enforcement"]},"delivery":{"work":"Register tools, write schemas, create tasks, run DP/CoT/ReAct comparisons, gate pilots on FPoF thresholds.","dependencies":["Tool owners","EM SMEs","Eval engineers","Model API budgets"],"ownership":"EM authority owns accept/reject of agent plans; engineering owns schema registry.","skillsAdoption":"Analysts learn to read FPoF reports and challenge tool-mismatch failures.","governanceCheckpoints":["Schema completeness review","Pilot gate on FPoF classes","Change control on tool registry"],"acceptanceCriteria":["No unconstrained tool calls in pilot","FPoF dashboard live","Human approval on multi-stage plans"],"risks":["Instruction clash from verbose reasoning","Schema drift","False confidence from leaderboard scores"]}},"retrievedAt":"2026-10-03T22:01:45-05:00","enrichedAt":"2026-10-03T22:01:45-05:00","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Highlights analyst/EM skill needs to interpret FPoF diagnostics rather than trusting fluent plans.","procurementImplications":"Prefer vendors who expose typed tool schemas and evaluation harnesses over opaque multi-agent demos.","operatingModelImplications":"Agentic assist should remain proposal-only until schema validation and human approval are operating procedures.","sourceVerification":{"responseId":"curl-arxiv-html-disasterbench-20261003T220145","openedUrl":"https://arxiv.org/html/2605.27957","referenceExcerpt":"tool mismatch and parameter-binding errors dominate first","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}