From the Public Safety edition of October 27, 2026
Analysis of Tasmania Police’s released Draft One evaluation: time saved measured against a baseline, but safety-critical errors measured once, unweighted, with no category for unsupported assertions
Tasmania Police (Department of Police, Fire and Emergency Management); analysis by AI & GPP · Policing / AI-drafted incident reports · Tasmania, Australia
- Publisher
- AI & GPP
- Original publication
- AI & GPP, Oct. 26, 2026
- Source retrieved
- 2026-10-28
- Event date
- 2026-10-06
What happened
AI & GPP analyses the 112-page Right to Information release (RTI 557/26, disclosure dated Oct. 6, 2026) behind the ABC’s Oct. 23 report on Tasmania Police’s Axon Draft One family-violence trial, which is already archived. Per the analysis: Phase One (six sergeants, 148 summaries from historic footage) found 21 (14%) with an error or omission judged critical to decision-making, but classified only 12 (6 name/role errors, 6 missed threat, assault or risk factors); the tool also surfaced risk factors absent from 10 human reports (7%). Time was measured against a 24-week human-only baseline (79 minutes per report) versus 42 minutes with the tool, about 38 minutes saved, but timings were self-reported and only 54% of trained testers exported a draft (20% never used it). The end-to-end rate of critical errors reaching a submitted report was measured once, 6% across 51 reports; errors were unweighted; free-text error explanations were summarised by Copilot; and 20% of frontline respondents thought safety had decreased, which the report notes is at odds with expert review. The author proposes six evaluation fixes, including a severity-weighted taxonomy, end-to-end miss rates and a pre-agreed stopping rule.
Why it matters
Gives agencies running AI report-writing pilots a concrete evaluation design and shows how time-saving claims can outrun harm measurement.
Evidence and measured results
Opened and read the AI & GPP article in full. Also opened the linked RTI PDF on police.tas.gov.au (112 pages, HTTP 200), but it is a scanned image without a text layer, so the figures could not be text-verified and are cited as the analysis reports them.
Limitations and uncertainty
Secondary analysis by an independent publication; primary RTI figures could not be text-verified by this run. Australian context; Tasmania Police says future use remains under review. Not a peer-reviewed study.
Put this evidence to work
Lighthouse Advisory interpretation, grounded in this source. Enriched 2026-10-28; this does not change the original publication date. Each role follows the same checklist so takeaways compare across sources.
Sales
Role takeaway
- Customer problem
- Agencies piloting AI report drafting lack an evaluation that measures safety-critical errors with the same rigour as time saved.
- Stakeholders
- Police executives and evaluation units
- Family-violence and specialist sergeants
- Prosecutors
- Police unions or associations
- Oversight bodies
- Discovery questions
- Was a human-only baseline collected before the pilot?
- How are errors classified and weighted by severity?
- What share of trained users actually used the tool?
- What written stopping rule applies?
- Value hypothesis
- Interpretation: a stronger evaluation design could prevent an unsafe rollout or support a defensible one; benefits depend on measured results.
- Potential engagement
- A bounded evaluation design for one agency’s AI report-drafting pilot.
- Unsupported claims
- Guaranteed time savings
- That supervisor review catches critical errors
- That Tasmania’s findings transfer directly to U.S. agencies
Pre-sales engineering
Role takeaway
- Fit
- Relevant to any Draft One or similar pilot, especially for high-risk report types.
- Architecture and integration
- Requires per-draft logging of officer edits and retention of first drafts alongside final reports.
- Prerequisites
- First-draft retention
- Edit logging
- Matched human-only baseline cases
- Constraints
- Australian jurisdiction
- Family-violence reports only
- Primary figures not text-verified
- Security
- Body-camera audio of sensitive incidents is processed by the vendor; data-handling terms must be confirmed.
- Proof of value
- Measure critical-error rate at model output and end to end
- Report adoption rates alongside time saved
- Test judge-stability if a model scores drafts
Delivery
Role takeaway
- Work
- Design and run a severity-weighted, baseline-controlled evaluation with a stopping rule.
- Dependencies
- Vendor logging capabilities
- Supervisor time for review
- Agreed error taxonomy
- Ownership
- Agency evaluation or business-improvement unit with frontline supervisors
- Skills and adoption
- Officers trained to film for AI and to review drafts against recordings, not memory.
- Governance checkpoints
- Pre-registered thresholds
- Four-weekly safety review
- Independent review before rollout decision
- Acceptance criteria
- Severity-weighted error rates reported separately from time saved (proposed)
- End-to-end critical-error rate below a pre-registered threshold (proposed)
- Risks
- Automation bias growing over time
- Self-timed figures overstating savings
- Unclassified error types escaping measurement
Implementation considerations
Lighthouse Advisory interpretation across the operating dimensions a public-sector buyer must settle before this evidence becomes a design. Each note answers the question under its heading for this specific source.
Architecture and integration
What must connect, and where does the AI sit in the workflow?
Treats safety as a property of the composed workflow (model, officer, supervisor, specialist review) rather than the model alone, and proposes per-draft decision logs recording what reviewers kept, changed or rejected.
Governance
Who approves, reviews and stays accountable for outcomes?
Evaluators recommended phased deployment only after ethical-use policies, training and funded support; no legislative change needed. The analysis argues for a written stopping rule agreed before any pilot.
Security and privacy
What data, permissions and controls need testing?
Family-violence body-camera recordings were processed by the tool; the analysis does not address data handling.
Accessibility and workforce
Who is affected, and what skills or accommodations follow?
Low adoption (20% never used the tool) means time-saving figures describe only willing users; frontline and expert views on safety diverged.
Procurement
What should contracts, pricing and exit terms secure?
Time-saving estimates (A$434,187 a year in wages, per the analysis) could drive licensing decisions while the harm side remains thinly measured.
Operating model
Which teams own the service once it runs?
Sergeant-level validation remains the safeguard, but its effectiveness was measured on 51 reports only.
Publication history
- 2026-10-27Public Safety · Issue 496 resources
Stable resource ID: tasmania-draft-one-rti-evaluation-method-critique-harm-measurement-20261026