{"resourceId":"prepared-st-johns-911-aqa-20260831","versions":[{"version":"external-42ea534e13e3023b33bec309df7943594ea2e079f3cfca84510a83c71b075862","resource":{"id":"prepared-st-johns-911-aqa-20260831","title":"Prepared/Axon case study: St. John's County 911 AI QA from ~2% sampling to 100% coverage","organization":"Prepared (Axon 911 Intelligence) featuring St. John's County 911, Florida","sector":"Vendor case study / emergency communications QA","geography":"United States (St. John's County / St. Augustine, Florida)","publishedAt":"2026-08-31","publicationDate":"2026-08-31","eventDate":"2026-08-31","sourceName":"Prepared911.com","sourceLabel":"Axon 911 Intelligence by Prepared case study on St. John's County automated QA","sourceUrl":"https://www.prepared911.com/case-studies/st-johns-county-911-went-from-reviewing-2-of-calls-to-every-single-one","evidenceClass":"vendor-claim","outcomeClass":"mixed","topics":["accessibility-workforce","operating-model","governance-procurement","knowledge-work"],"finding":"Prepared's August 31, 2026 case study reports St. John's County 911 (St. Augustine, FL) moved from manually reviewing about 2% of ~101,000 annual 911 calls (20–30 minutes per review) to Automated QA + Training from Axon 911 Intelligence by Prepared. Quoted floor manager Debra Smith: the center went from QAing very few calls to doing 100%. In the past 90 days the center QA'd 4,928 calls including 100% of failing calls (score <90%); manual review time is down 90%, with a weekend example of ~120 failing calls in ~3 hours versus ~30 hours manually. Auto and manual scores are said to align within 3%. Managers keep AQA up during shifts to catch issues in progress; coaching happens within days. Humans still review failures and own coaching.","sledRelevance":"Direct PSAP workforce-AI signal with quantified QA coverage and review-time claims for SLTT emergency communications. Why now: not in emergency-services archive; published 2026-08-31 and newly relevant as 911 AI QA spreads beside non-emergency answering coverage after 2026-09-27's wildfire/triage edition.","evidence":"Opened Prepared case-study HTML via curl. Date August 31, 2026; 2%→100% QA framing; 101,000 911 calls; 4,928 calls/90 days; 90% manual-time reduction; within-3% score alignment; HITL review of <90% scores taken from page.","architectureImplications":"Treat automated QA as a scoring/transcription layer alongside recording systems with supervisor queues for fails—not as live call-taking or CAD replacement.","governanceImplications":"Define fail thresholds, dispute processes, and how AI scores feed employee evaluations; separate vendor case-study metrics from agency-audited KPIs.","securityPrivacyImplications":"Full-call AI scoring implies broad access to 911 audio/transcripts; require data-processing agreements, retention limits, and access controls for vendor platforms.","caveats":"Vendor marketing case study with single-site self-reported metrics; no independent audit of 90% time savings or 3% score agreement; no outcome data on call quality or patient/public safety.","streamIds":["emergency-services"],"roles":{"sales":"Frame St. John's reported shift from ~2% manual 911 QA sampling to automated 100% coverage with 90% less manual review time as a workforce capacity story—not autonomous call-taking. Discovery should map current QA sample rates, SOP scorecards, union/eval rules, and who owns coaching. Bounded engagement: 90-day AQA pilot measuring human–auto score agreement and coaching latency. Unsupported claims include guaranteed 90% time savings everywhere, improved clinical outcomes, or AI replacing supervisors.","engineering":"Fit is post-call (and in-shift) automated scoring against agency SOPs with supervisor fail queues. Architecture should integrate recording/NG911 audio, transcription, score persistence, and audit of auto vs manual scores. Prerequisites include defined scorecards, fail threshold (e.g., <90%), and dispute workflow. Constraints: vendor-reported metrics; model drift vs local dialects/protocols. Proof of value is measured agreement rates and review-hours saved under local SOPs, not case-study headlines alone.","delivery":"Delivery work includes SOP digitization, supervisor training on fail queues, and change management for floor culture/disputes. Dependencies span ECC leadership, IT/recording vendors, and HR/eval policy. Ownership stays with the QA/floor manager for coaching decisions. Acceptance criteria: 100% auto-score coverage with logged human review of fails and published local agreement metrics. Risks include evaluation misuse, alert overload on fails, and opaque vendor scoring."},"roleDetails":{"sales":{"customerProblem":"PSAPs cannot manually QA more than a tiny sample of 911 calls without burning out staff.","stakeholders":["ECC directors","QA/floor managers","Telecommunicators","HR/labor"],"discoveryQuestions":["What % of 911 calls are QA'd today?","Who owns coaching?","How are scores used in evals?","What SOP scorecard exists?"],"valueHypothesis":"Automated QA may expand coverage and cut manual review hours if humans still coach fails—not if scores become unreviewed punishment.","potentialEngagement":"90-day AQA pilot with agreement and coaching-latency KPIs.","unsupportedClaims":["Guaranteed 90% time savings","Better patient outcomes proven","AI replaces QA supervisors"]},"engineering":{"fit":"Automated 911 call scoring and fail surfacing against agency QA SOPs.","architectureIntegration":"Recorder/NG911 audio → AI score/transcript → fail queue → supervisor review/coaching log.","prerequisites":["Recording access","Digitized SOP scorecard","Fail threshold","Identity/access controls"],"constraints":["Vendor self-reported metrics","Local protocol mismatch","Dialect/noise robustness"],"security":"Limit vendor access to 911 audio/PII; set retention and subprocessors in the DPA.","proofOfValue":["Auto–manual score agreement","Manual QA hours","Time-to-coaching after fail"]},"delivery":{"work":"Configure scorecards, train supervisors on fail queues, and stand up dispute/coaching SOPs.","dependencies":["ECC ops","Recording/IT","HR evaluation policy"],"ownership":"Floor/QA manager owns coaching; ECC director owns policy on score use.","skillsAdoption":"Train coaches to treat AI scores as prompts for listening, not verdicts.","governanceCheckpoints":["Monthly agreement audit","Dispute log review","Eval-policy check"],"acceptanceCriteria":["Human review of all fails","Documented dispute path","Local KPI dashboard"],"risks":["Eval misuse","Fail-queue overload","Opaque model changes"]}},"retrievedAt":"2026-09-28T22:06:14-05:00","enrichedAt":"2026-09-28T22:06:14-05:00","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Positions AI as burnout relief for collateral-duty QA staff while keeping supervisors as coaches; train telecommunicators on dispute/feedback workflows.","procurementImplications":"Evaluate AQA against local SOPs, scorecard alignment studies, and staffing models—not headline 100% coverage alone.","operatingModelImplications":"Keep humans reviewing fails and conducting coaching; use AI to prioritize which calls enter the human QA queue.","sourceVerification":{"responseId":"curl-prepared-stjohns-20260928T220614","openedUrl":"https://www.prepared911.com/case-studies/st-johns-county-911-went-from-reviewing-2-of-calls-to-every-single-one","referenceExcerpt":"Time spent on manual review is down 90%.","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}