From the Emergency Services edition of September 30, 2026
ACL Findings/arXiv: LLMs in emergency text-2-911 MT deploy without evaluation data or city AI policy
The Ohio State University / Community Refugee & Immigration Services (ACL Findings 2026) · Peer-reviewed NLP / emergency communications language-access research · United States (Ohio PSAP case study; transferable caution for SLTT 911)
- Publisher
- arXiv HTML (ACL Findings 2026)
- Original publication
- July 2026
- Source retrieved
- 2026-10-01
- Event date
- 2025-09-30
What happened
ACL Findings 2026 paper (arXiv:2607.00019; July 2026 anthology; submitted 29 May 2026) presents a case study of an LLM-based machine-translation application in a U.S. text-2-911 system advertising 55 languages. Authors visited the 911 center (end of September 2025): PSAP software uses Microsoft Azure language detection/translation; staff said Microsoft does not provide underlying model/training data; implementers had not been provided evaluation data or QA services by the software provider. Ohio has a state AI administrative policy, but no equivalent city departmental policy. Live probing found Arabic dialect/MSA and Nepali Latin-vs-Devanagari mismatches; callers are not shown MT disclosure/language detection. Recommendations include model cards, informed consent, continuous monitoring, and human-interpreter backup.
Why it matters
Independent academic scrutiny of real PSAP text-2-911 MT—evaluation voids, dialect failures, and governance gaps—balancing today’s QA procurement enthusiasm. Why now: not archived; newly relevant as PSAPs buy assistive AI (e.g., Lake County EMD QA) without parallel language-access evaluation infrastructure.
Evidence and measured results
Opened arXiv HTML https://arxiv.org/html/2607.00019 via curl. 55-language text-2-911 case; Azure MT; no evaluation data quote; state-vs-city policy gap; Arabic/Nepali test failures; July 2026 ACL Findings metadata cross-checked on anthology page.
Limitations and uncertainty
Single anonymized PSAP case study; authors note limited interaction volume for statistical MT evaluation; not a vendor product benchmark table.
Put this evidence to work
Lighthouse Advisory interpretation, grounded in this source. Enriched 2026-09-30; this does not change the original publication date. Each role follows the same checklist so takeaways compare across sources.
Sales
Role takeaway
- Customer problem
- PSAPs face Title VI language-access pressure and are sold multi-language text-2-911 MT without evaluation packages.
- Stakeholders
- 911 directors
- City equity/LEP offices
- Community refugee advocates
- Procurement/legal
- Discovery questions
- Which languages are actually spoken locally?
- Any emergency-domain MT scores?
- Interpreter fallback SLA?
- Caller MT disclosure?
- Value hypothesis
- MT can add text access only if languages/scripts match real users and human backup is funded—marketing lists are insufficient.
- Potential engagement
- Community-partner language-access evaluation before expanding text-2-911 MT.
- Unsupported claims
- Equal quality across 55 languages
- MT replaces interpreters
- No need for city AI policy
Pre-sales engineering
Role takeaway
- Fit
- API-based MT for text-2-911 with human dispatcher review and interpreter escalation.
- Architecture and integration
- SMS → language ID/MT API → dual text to dispatcher → optional interpreter bridge → logged transcript.
- Prerequisites
- Priority language list
- Dialect/script test sets
- Subprocessor DPA
- Escalation workflow
- Constraints
- Opaque commercial MT models
- No vendor eval data in case
- Single-site study
- Security
- Minimize retention of emergency SMS at MT subprocessors; ban training use; log access.
- Proof of value
- Human-rated adequacy on emergency phrases
- Escalation rate/time
- Dispatcher error-catch rate
Delivery
Role takeaway
- Work
- Run community tests, train dispatchers, and write disclosure/escalation SOPs.
- Dependencies
- PSAP ops
- Vendor
- LEP community partners
- City counsel
- Ownership
- 911 director owns go-live; language-access lead owns ongoing evaluation.
- Skills and adoption
- Teach dispatchers to verify critical slots (address, threat) and escalate early.
- Governance checkpoints
- Pre-deploy eval package
- Quarterly language metrics
- Complaint/grievance review
- Acceptance criteria
- Published language-pair scores
- Interpreter backup funded
- Transparency design approved
- Risks
- Dialect/script failures
- Undisclosed AI to callers
- Accountability gaps after harm
Implementation considerations
Lighthouse Advisory interpretation across the operating dimensions a public-sector buyer must settle before this evidence becomes a design. Each note answers the question under its heading for this specific source.
Architecture and integration
What must connect, and where does the AI sit in the workflow?
MT for text-2-911 needs language-pair scorecards, human-interpreter fallback, dual-visibility of source/translation, and logging—not opaque Azure API wrappers alone.
Governance
Who approves, reviews and stays accountable for outcomes?
Require pre-deployment evaluation packages, city AI policy alignment, and community-partner testing (dialects/scripts) before marketing 55-language emergency MT.
Security and privacy
What data, permissions and controls need testing?
Emergency texts may include location and victim details sent to third-party MT APIs—demand subprocessors, retention, and non-training clauses.
Accessibility and workforce
Who is affected, and what skills or accommodations follow?
Poor dialect/script support can worsen language access for LEP callers; invest in interpreters and community testing, not MT marketing alone.
Procurement
What should contracts, pricing and exit terms secure?
Demand language-pair emergency-domain metrics, model cards, and QA services in RFP—reject “55 languages” claims without evidence.
Operating model
Which teams own the service once it runs?
Prefer voice+human interpreter when safe; treat text MT as disclosed assistive path with easy escalation to humans.
Publication history
- 2026-09-30Emergency Services · Issue 245 resources
Stable resource ID: arxiv-llms-emergency-text911-mt-eval-gaps-202607