{"resourceId":"arxiv-llms-emergency-text911-mt-eval-gaps-202607","versions":[{"version":"external-b25d50b1de6e2a0d6cda923c6959286ceed84c2cf1da20fbafa2918b91800be7","resource":{"id":"arxiv-llms-emergency-text911-mt-eval-gaps-202607","title":"ACL Findings/arXiv: LLMs in emergency text-2-911 MT deploy without evaluation data or city AI policy","organization":"The Ohio State University / Community Refugee & Immigration Services (ACL Findings 2026)","sector":"Peer-reviewed NLP / emergency communications language-access research","geography":"United States (Ohio PSAP case study; transferable caution for SLTT 911)","publishedAt":"July 2026","publicationDate":"2026-07-01","eventDate":"2025-09-30","sourceName":"arXiv HTML (ACL Findings 2026)","sourceLabel":"Court, Downing, and Elsner ACL Findings 2026 paper on LLM MT in a text-2-911 emergency deployment","sourceUrl":"https://arxiv.org/html/2607.00019","evidenceClass":"academic-research","outcomeClass":"cautionary","topics":["governance-procurement","accessibility-workforce","data-security","operating-model"],"finding":"ACL Findings 2026 paper (arXiv:2607.00019; July 2026 anthology; submitted 29 May 2026) presents a case study of an LLM-based machine-translation application in a U.S. text-2-911 system advertising 55 languages. Authors visited the 911 center (end of September 2025): PSAP software uses Microsoft Azure language detection/translation; staff said Microsoft does not provide underlying model/training data; implementers had not been provided evaluation data or QA services by the software provider. Ohio has a state AI administrative policy, but no equivalent city departmental policy. Live probing found Arabic dialect/MSA and Nepali Latin-vs-Devanagari mismatches; callers are not shown MT disclosure/language detection. Recommendations include model cards, informed consent, continuous monitoring, and human-interpreter backup.","sledRelevance":"Independent academic scrutiny of real PSAP text-2-911 MT—evaluation voids, dialect failures, and governance gaps—balancing today’s QA procurement enthusiasm. Why now: not archived; newly relevant as PSAPs buy assistive AI (e.g., Lake County EMD QA) without parallel language-access evaluation infrastructure.","evidence":"Opened arXiv HTML https://arxiv.org/html/2607.00019 via curl. 55-language text-2-911 case; Azure MT; no evaluation data quote; state-vs-city policy gap; Arabic/Nepali test failures; July 2026 ACL Findings metadata cross-checked on anthology page.","architectureImplications":"MT for text-2-911 needs language-pair scorecards, human-interpreter fallback, dual-visibility of source/translation, and logging—not opaque Azure API wrappers alone.","governanceImplications":"Require pre-deployment evaluation packages, city AI policy alignment, and community-partner testing (dialects/scripts) before marketing 55-language emergency MT.","securityPrivacyImplications":"Emergency texts may include location and victim details sent to third-party MT APIs—demand subprocessors, retention, and non-training clauses.","caveats":"Single anonymized PSAP case study; authors note limited interaction volume for statistical MT evaluation; not a vendor product benchmark table.","streamIds":["emergency-services"],"roles":{"sales":"Use this ACL Findings case study to caution buyers that advertising 55-language text-2-911 MT can outrun evaluation: the studied PSAP lacked vendor evaluation data, city AI policy, and caller-facing MT disclosure, and live tests stumbled on Arabic dialects and Nepali script practices. Discovery should ask for language-pair emergency-domain scores, interpreter fallback SLAs, and community test plans. Bounded engagement: language-access evaluation design with refugee/LEP partners before go-live. Unsupported claims include equal quality across 55 languages or MT replacing Title VI interpreter obligations.","engineering":"Fit is high-stakes MT for text-2-911 with human dispatcher mediation. Architecture needs source+translation dual display, language-ID confidence, escalation to human interpreters, and API logging/retention controls for Azure-class subprocessors. Prerequisites: language inventory vs community needs, dialect test sets, and city AI policy. Constraints: proprietary models without training-data access; single case study. Proof of value is measured translation adequacy on emergency phrases across priority languages—not feature checklists.","delivery":"Delivery work includes community-partner testing, dispatcher training on MT limits, and policy for disclosure/consent tradeoffs. Dependencies: PSAP ops, city attorney/equity office, MT vendor, and LEP advocates. Ownership: 911 director for deployment; equity/language-access lead for evaluation. Acceptance: published language-pair metrics, documented interpreter backup, and caller/dispatcher transparency design. Risks include silent mistranslation harm and false confidence from “AI-powered” marketing."},"roleDetails":{"sales":{"customerProblem":"PSAPs face Title VI language-access pressure and are sold multi-language text-2-911 MT without evaluation packages.","stakeholders":["911 directors","City equity/LEP offices","Community refugee advocates","Procurement/legal"],"discoveryQuestions":["Which languages are actually spoken locally?","Any emergency-domain MT scores?","Interpreter fallback SLA?","Caller MT disclosure?"],"valueHypothesis":"MT can add text access only if languages/scripts match real users and human backup is funded—marketing lists are insufficient.","potentialEngagement":"Community-partner language-access evaluation before expanding text-2-911 MT.","unsupportedClaims":["Equal quality across 55 languages","MT replaces interpreters","No need for city AI policy"]},"engineering":{"fit":"API-based MT for text-2-911 with human dispatcher review and interpreter escalation.","architectureIntegration":"SMS → language ID/MT API → dual text to dispatcher → optional interpreter bridge → logged transcript.","prerequisites":["Priority language list","Dialect/script test sets","Subprocessor DPA","Escalation workflow"],"constraints":["Opaque commercial MT models","No vendor eval data in case","Single-site study"],"security":"Minimize retention of emergency SMS at MT subprocessors; ban training use; log access.","proofOfValue":["Human-rated adequacy on emergency phrases","Escalation rate/time","Dispatcher error-catch rate"]},"delivery":{"work":"Run community tests, train dispatchers, and write disclosure/escalation SOPs.","dependencies":["PSAP ops","Vendor","LEP community partners","City counsel"],"ownership":"911 director owns go-live; language-access lead owns ongoing evaluation.","skillsAdoption":"Teach dispatchers to verify critical slots (address, threat) and escalate early.","governanceCheckpoints":["Pre-deploy eval package","Quarterly language metrics","Complaint/grievance review"],"acceptanceCriteria":["Published language-pair scores","Interpreter backup funded","Transparency design approved"],"risks":["Dialect/script failures","Undisclosed AI to callers","Accountability gaps after harm"]}},"retrievedAt":"2026-09-30T22:07:36-05:00","enrichedAt":"2026-09-30T22:07:36-05:00","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Poor dialect/script support can worsen language access for LEP callers; invest in interpreters and community testing, not MT marketing alone.","procurementImplications":"Demand language-pair emergency-domain metrics, model cards, and QA services in RFP—reject “55 languages” claims without evidence.","operatingModelImplications":"Prefer voice+human interpreter when safe; treat text MT as disclosed assistive path with easy escalation to humans.","sourceVerification":{"responseId":"curl-arxiv-html-llms-emergency-20260930T220736","openedUrl":"https://arxiv.org/html/2607.00019","referenceExcerpt":"staff managing implementation of the tool had also not been provided any evaluation data","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}