From the Public Safety edition of October 12, 2026
Triple-blind expert review finds Axon Draft One-assisted police reports rated lower on accuracy, with identical approval rates
University of South Carolina, Clemson University, University of Utah and Manchester (N.H.) Police Department researchers · Policing / AI-assisted report writing evaluation · United States (single agency)
- Publisher
- CrimRxiv
- Original publication
- May. 8, 2026
- Source retrieved
- 2026-10-13
- Event date
- 2026-05-08
What happened
Why now: San Diego's AI report-writing RFI coverage cites this study as the key independent evidence. In a triple-blind design, 92 supervisors produced 354 evaluations of 80 reports from a randomized trial (20 Draft One-assisted). AI-assisted reports were rated lower on accuracy (p = 0.038), showed no composite quality difference, had identical approve-as-is rates (21% vs. 22%), and reviewers could not tell them apart (AUC 0.50).
Why it matters
Gives buyers independent evidence that supervisor review alone may not catch AI report quality problems, which matters as agencies such as SDPD move toward procurement.
Evidence and measured results
Opened original source from CrimRxiv and drew findings from its text. Preprint; single agency and single tool (Axon Draft One); 20/60 imbalance; measures perceived quality, not factual accuracy; adequate power only for medium effects.
Limitations and uncertainty
Perceived quality from one department's reports; generalization to other tools and agencies is untested.
Put this evidence to work
Lighthouse Advisory interpretation, grounded in this source. Enriched 2026-10-13; this does not change the original publication date. Each role follows the same checklist so takeaways compare across sources.
Sales
Role takeaway
- Customer problem
- Agencies are buying AI report tools on vendor quality claims that independent review does not support.
- Stakeholders
- Police chiefs and report-approving supervisors
- Prosecutors
- Public defenders
- Procurement officers
- Oversight boards
- Discovery questions
- What evidence did the vendor provide on report accuracy?
- How do supervisors currently rate report quality?
- Would the agency run a blind review before scaling?
- How will AI-assisted reports be flagged for prosecutors?
- Value hypothesis
- A blind local evaluation can show whether a tool meets the agency's accuracy bar before scale-up; quality gains are not assumed.
- Potential engagement
- Design and run a small blind-review evaluation of AI-assisted versus conventional reports.
- Unsupported claims
- AI improves report quality
- Supervisors can detect AI-written reports
- Results apply to tools other than Draft One
Pre-sales engineering
Role takeaway
- Fit
- Any agency evaluating transcript-based AI report drafting.
- Architecture and integration
- Assess whether drafts come only from audio transcripts or include structured officer input, and how AI authorship is stored in the records system.
- Prerequisites
- Sample of matched reports
- Supervisor rater pool
- Report quality rubric
- Constraints
- Single-agency, single-tool evidence
- Preprint status
- Perceived rather than factual accuracy measured
- Security
- Record AI involvement as tamper-evident metadata and keep source audio for audit.
- Proof of value
- Blind supervisor review on local reports
- Spot-check AI drafts against footage for omitted facts
- Compare readability scores of AI and conventional reports
Delivery
Role takeaway
- Work
- Revise supervisor review rubrics and training, add structural AI-use flags, and set up footage-based audit sampling.
- Dependencies
- Training unit
- Records system metadata fields
- Prosecutor feedback channel
- Ownership
- Patrol division commanders responsible for report approval
- Skills and adoption
- Supervisors need training on accuracy and completeness for non-specialist readers, since detection is not reliable.
- Governance checkpoints
- Rubric revision approval
- Pilot blind review
- Prosecutor and defense feedback
- Scale-up decision
- Acceptance criteria
- Proposed: AI-assisted reports rated no lower on accuracy than baseline in blind review
- Proposed: all AI-assisted reports flagged in metadata
- Proposed: monthly footage audit sample completed
- Risks
- Accuracy loss reaching court
- Overreliance on approval rates
- False assurance from human review
- Generalization error from one study
Implementation considerations
Lighthouse Advisory interpretation across the operating dimensions a public-sector buyer must settle before this evidence becomes a design. Each note answers the question under its heading for this specific source.
Architecture and integration
What must connect, and where does the AI sit in the workflow?
Assess whether drafts come only from audio transcripts or include structured officer input, and how AI authorship is stored in the records system.
Governance
Who approves, reviews and stays accountable for outcomes?
Revise supervisor review rubrics and training, add structural AI-use flags, and set up footage-based audit sampling.
Security and privacy
What data, permissions and controls need testing?
Record AI involvement as tamper-evident metadata and keep source audio for audit.
Accessibility and workforce
Who is affected, and what skills or accommodations follow?
Supervisors need training on accuracy and completeness for non-specialist readers, since detection is not reliable.
Procurement
What should contracts, pricing and exit terms secure?
Design and run a small blind-review evaluation of AI-assisted versus conventional reports.
Operating model
Which teams own the service once it runs?
Operational owner: Patrol division commanders responsible for report approval.
Publication history
- 2026-10-12Public Safety · Issue 354 resources
Stable resource ID: adams-triple-blind-evaluation-ai-assisted-police-reports-crimrxiv-20260508