{"resourceId":"adams-triple-blind-evaluation-ai-assisted-police-reports-crimrxiv-20260508","versions":[{"version":"external-5b5a8020d946eed5cfaa7e7ce8d37d902b3c645771ea2ec1ce32bfd2a09cc911","resource":{"id":"adams-triple-blind-evaluation-ai-assisted-police-reports-crimrxiv-20260508","title":"Triple-blind expert review finds Axon Draft One-assisted police reports rated lower on accuracy, with identical approval rates","organization":"University of South Carolina, Clemson University, University of Utah and Manchester (N.H.) Police Department researchers","sector":"Policing / AI-assisted report writing evaluation","geography":"United States (single agency)","publishedAt":"May. 8, 2026","publicationDate":"2026-05-08","eventDate":"2026-05-08","sourceName":"CrimRxiv","sourceLabel":"Triple-blind expert review finds Axon Draft One-assisted police reports rated lo","sourceUrl":"https://assets.pubpub.org/ccb6ab371-fc70-4ef3-b2e5-d0d0580b4d37/pee18b482-ea71-472d-a3be-1a08daf09234/exports/2cd771c54c670003.pdf","evidenceClass":"academic-research","outcomeClass":"cautionary","topics":["knowledge-work","governance-procurement","accessibility-workforce","operating-model"],"finding":"Why now: San Diego's AI report-writing RFI coverage cites this study as the key independent evidence. In a triple-blind design, 92 supervisors produced 354 evaluations of 80 reports from a randomized trial (20 Draft One-assisted). AI-assisted reports were rated lower on accuracy (p = 0.038), showed no composite quality difference, had identical approve-as-is rates (21% vs. 22%), and reviewers could not tell them apart (AUC 0.50).","sledRelevance":"Gives buyers independent evidence that supervisor review alone may not catch AI report quality problems, which matters as agencies such as SDPD move toward procurement.","evidence":"Opened original source from CrimRxiv and drew findings from its text. Preprint; single agency and single tool (Axon Draft One); 20/60 imbalance; measures perceived quality, not factual accuracy; adequate power only for medium effects.","architectureImplications":"Assess whether drafts come only from audio transcripts or include structured officer input, and how AI authorship is stored in the records system.","governanceImplications":"Interpretation: Revise supervisor review rubrics and training, add structural AI-use flags, and set up footage-based audit sampling.","securityPrivacyImplications":"Record AI involvement as tamper-evident metadata and keep source audio for audit.","caveats":"Perceived quality from one department's reports; generalization to other tools and agencies is untested.","streamIds":["public-safety"],"roles":{"sales":"Sales framing: this is the strongest independent evidence available on AI report quality, and it cuts against vendor quality claims: experienced supervisors rated Draft One-assisted reports lower on accuracy and could not identify them. For agencies like SDPD weighing a purchase, the useful offer is an evaluation protocol modeled on this design, not a promise of better reports. It supports no claim of improved quality or time savings and limits generalization to one agency and tool.","engineering":"Engineering framing: the authors tie the completeness gap to the tool's transcript-only architecture, so what does not appear in body-camera audio cannot appear in the draft. Pre-sales engineering should ask how a candidate tool handles non-audible facts, whether active officer input is required, and how AI authorship is recorded structurally, since reviewers cannot detect it. Proof of value should replicate a blind review on the agency's own reports rather than rely on vendor studies.","delivery":"Delivery framing: the study shows human-in-the-loop review is not working as assumed, so delivery must change the review itself. That means retraining supervisors on accuracy and completeness for non-specialist readers, flagging AI-assisted reports in metadata rather than relying on detection, and sampling reports for audit against footage. Acceptance criteria should be agency-measured accuracy, not approval rates, which were identical across conditions in this study."},"roleDetails":{"sales":{"customerProblem":"Agencies are buying AI report tools on vendor quality claims that independent review does not support.","stakeholders":["Police chiefs and report-approving supervisors","Prosecutors","Public defenders","Procurement officers","Oversight boards"],"discoveryQuestions":["What evidence did the vendor provide on report accuracy?","How do supervisors currently rate report quality?","Would the agency run a blind review before scaling?","How will AI-assisted reports be flagged for prosecutors?"],"valueHypothesis":"A blind local evaluation can show whether a tool meets the agency's accuracy bar before scale-up; quality gains are not assumed.","potentialEngagement":"Design and run a small blind-review evaluation of AI-assisted versus conventional reports.","unsupportedClaims":["AI improves report quality","Supervisors can detect AI-written reports","Results apply to tools other than Draft One"]},"engineering":{"fit":"Any agency evaluating transcript-based AI report drafting.","architectureIntegration":"Assess whether drafts come only from audio transcripts or include structured officer input, and how AI authorship is stored in the records system.","prerequisites":["Sample of matched reports","Supervisor rater pool","Report quality rubric"],"constraints":["Single-agency, single-tool evidence","Preprint status","Perceived rather than factual accuracy measured"],"security":"Record AI involvement as tamper-evident metadata and keep source audio for audit.","proofOfValue":["Blind supervisor review on local reports","Spot-check AI drafts against footage for omitted facts","Compare readability scores of AI and conventional reports"]},"delivery":{"work":"Revise supervisor review rubrics and training, add structural AI-use flags, and set up footage-based audit sampling.","dependencies":["Training unit","Records system metadata fields","Prosecutor feedback channel"],"ownership":"Patrol division commanders responsible for report approval","skillsAdoption":"Supervisors need training on accuracy and completeness for non-specialist readers, since detection is not reliable.","governanceCheckpoints":["Rubric revision approval","Pilot blind review","Prosecutor and defense feedback","Scale-up decision"],"acceptanceCriteria":["Proposed: AI-assisted reports rated no lower on accuracy than baseline in blind review","Proposed: all AI-assisted reports flagged in metadata","Proposed: monthly footage audit sample completed"],"risks":["Accuracy loss reaching court","Overreliance on approval rates","False assurance from human review","Generalization error from one study"]}},"retrievedAt":"2026-10-13T02:57:19Z","enrichedAt":"2026-10-13T02:59:39Z","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Supervisors need training on accuracy and completeness for non-specialist readers, since detection is not reliable.","procurementImplications":"Interpretation: Design and run a small blind-review evaluation of AI-assisted versus conventional reports.","operatingModelImplications":"Operational owner: Patrol division commanders responsible for report approval.","sourceVerification":{"openedUrl":"https://assets.pubpub.org/ccb6ab371-fc70-4ef3-b2e5-d0d0580b4d37/pee18b482-ea71-472d-a3be-1a08daf09234/exports/2cd771c54c670003.pdf","referenceExcerpt":"Reviewers cannot distinguish AI-assisted from conventional reports, with accuracy rates equal to a coin flip (AUC = 0.50).","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}