{"resourceId":"arxiv-nashville-genai-911-training-20260709","versions":[{"version":"external-d37cc357cd81f1553c7b3e1c6d59379cee1cf0519dc1898699ea3d1319655121","resource":{"id":"arxiv-nashville-genai-911-training-20260709","title":"arXiv: GenAI 9-1-1 call-taking trainer at Metro Nashville—formal methods beat pure LLM assessment (94.3–95.9% vs 87.8%)","organization":"Vanderbilt University / Metro Nashville Department of Emergency Communications","sector":"PSAP training / emergency communications workforce","geography":"United States (Nashville, Tennessee)","publishedAt":"2026-07-09","publicationDate":"2026-07-09","eventDate":null,"sourceName":"arXiv (cs.CY)","sourceLabel":"arXiv HTML for Chen & Ma GenAI 9-1-1 calltaking training deployment study (2602.13241v3)","sourceUrl":"https://arxiv.org/html/2602.13241","evidenceClass":"academic-research","outcomeClass":"effective","topics":["accessibility-workforce","knowledge-work","developers-agents","operating-model","governance-procurement"],"finding":"Chen & Ma (arXiv:2602.13241v3, cs.CY, 9 Jul 2026) report a six-month GenAI-powered 9-1-1 call-taking training deployment with Metro Nashville Department of Emergency Communications (MNDEC). Context: U.S. call-takers handle >240M calls/year; many centers face >25% staffing shortages; new-hire training can require up to 720 hours of one-on-one instruction. The system generates scenarios across 57 incident types and assesses trainees against 1,651 protocol requirements translated to formal specifications, with foundation models role-playing callers via telephony. Deployment scaled from an 18-workstation pilot to 190 operational users and 1,120 training sessions; logs include 98,429 user interaction events. Evaluation on 1,244 real-world calls: formal-methods hybrid assessment achieved 94.3–95.9% agreement with expert assessments vs 87.8% for the strongest pure LLM baseline (ICL+CoT+RAG). Authors emphasize HITL co-design, verifiable assessment traces, and four lessons on delivery, rigor, resilience, and human factors.","sledRelevance":"Actionable U.S. PSAP workforce/AI training evidence with measured agreement metrics—supports emergency-services dispatch capacity without claiming live AI call-taking.","evidence":"Peer-style arXiv research paper with deployment logs and agreement evaluation against expert assessments; not a randomized clinical outcomes trial.","architectureImplications":"Hybrid GenAI scenario generation plus formal-specification assessment over telephony training interfaces; emphasizes verifiable traces over opaque LLM scores.","governanceImplications":"Government–university co-design with ethical/HITL sections; competency decisions remain with human trainers/QA.","securityPrivacyImplications":"Training audio and interaction logs are sensitive even when not live 911; access and retention controls are implied requirements.","caveats":"Training only; single jurisdiction; agreement ≠ field clinical outcomes; arXiv HTML preprint lineage.","streamIds":["emergency-services"],"roles":{"sales":"Frame Vanderbilt/MNDEC’s GenAI 911 training deployment—190 users, 1,120 sessions, formal-methods assessment at 94.3–95.9% expert agreement—as a workforce-capacity play for PSAPs facing 25%+ shortages and 720-hour training loads, not as live call-taking AI. Discovery should ask which protocol library can be formalized and who owns QA. Bounded engagement: pilot scenario generation + assessor hybrid for a regional training academy. Unsupported claims include live 911 replacement, clinical outcome gains, or universal protocol portability.","engineering":"System combines GenAI caller role-play over telephony with assessment against 1,651 protocol requirements encoded as formal specs, outperforming pure LLM judges. Prerequisites: protocol text rights, expert validators, telephony training pods, and logging. Constraints: training environment only; single-jurisdiction co-design. Proof of value: expert-agreement deltas, training hours reclaimed, and trainee field-readiness measures—not scenario novelty demos.","delivery":"Co-design formal specifications with trainers/QA, instrument workstations, and keep human review of assessment traces. Ownership: PSAP training/QA for accept; research/IT partners for model ops. Risks: over-trusting LLM-only scoring, scenario bias, and diverting trainers without measuring field performance after hire."},"roleDetails":{"sales":{"customerProblem":"PSAP training does not scale under staffing shortages and long one-on-one instruction hours.","stakeholders":["PSAP training/QA","New call-takers","IT/telephony","Labor/leadership"],"discoveryQuestions":["How many training hours per hire?","Which protocols can be formalized?","Who validates AI assessments?","What telephony sandbox exists?"],"valueHypothesis":"GenAI scenario+formal assessment can expand training throughput if expert agreement stays high and HITL co-design continues.","potentialEngagement":"90-day academy pilot: formalize a protocol subset and measure assessor agreement.","unsupportedClaims":["Live 911 AI replacement","Guaranteed shorter time-to-solo","Cross-state protocol portability"]},"engineering":{"fit":"GenAI training simulator and formal-methods performance assessor for call-takers.","architectureIntegration":"Telephony trainee interface ↔ foundation-model caller ↔ formal assessment engine with expert-traceable rules.","prerequisites":["Protocol corpus","Expert validators","Instrumented training seats","Logging/analytics"],"constraints":["Not live public 911","MNDEC-specific co-design","Model/telephony dependencies"],"security":"Training audio and call content require protected storage and access controls even in sandbox.","proofOfValue":["Expert agreement rate","Training hours per competency","Post-hire QA scores"]},"delivery":{"work":"Translate protocols to formal specs, deploy pods, train trainers on AI debriefs, audit assessment traces.","dependencies":["Domain experts","IT telephony","Model vendor/API","QA leadership"],"ownership":"PSAP training/QA owns competency decisions; technical partners own system reliability.","skillsAdoption":"Trainers learn to interpret formal assessment traces and challenge model errors.","governanceCheckpoints":["Spec validation sign-off","Bias/scenario review","Go-live for each protocol family"],"acceptanceCriteria":["≥ agreed expert-agreement threshold","Every score traceable to rules","Human override path"],"risks":["LLM-only scoring drift","Scenario demographic bias","Trainer deskilling"]}},"retrievedAt":"2026-10-03T22:01:35-05:00","enrichedAt":"2026-10-03T22:01:35-05:00","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Targets the training bottleneck (720 hours; >25% shortages) that limits PSAP accessibility of service under load.","procurementImplications":"Favors systems that expose formal assessment specs and expert-validation workflows over black-box scoring SaaS.","operatingModelImplications":"Keeps humans as competency authorities while AI scales scenario practice and first-pass assessment.","sourceVerification":{"responseId":"curl-arxiv-html-nashville-genai-20261003T220135","openedUrl":"https://arxiv.org/html/2602.13241","referenceExcerpt":"The formal methods approach achieved 94.3-95.9 percent agreement with expert assessments","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}