{"resourceId":"tasmania-draft-one-rti-evaluation-method-critique-harm-measurement-20261026","versions":[{"version":"external-5023707be5f576459b8c1443bac607d603a606a318a810fd38fb7e3ba63d41f3","resource":{"id":"tasmania-draft-one-rti-evaluation-method-critique-harm-measurement-20261026","title":"Analysis of Tasmania Police’s released Draft One evaluation: time saved measured against a baseline, but safety-critical errors measured once, unweighted, with no category for unsupported assertions","organization":"Tasmania Police (Department of Police, Fire and Emergency Management); analysis by AI & GPP","sector":"Policing / AI-drafted incident reports","geography":"Tasmania, Australia","publishedAt":"AI & GPP, Oct. 26, 2026","publicationDate":"2026-10-26","eventDate":"2026-10-06","sourceName":"AI & GPP","sourceLabel":"What a Police AI Trial Measured, and What It Missed","sourceUrl":"https://aiandgpp.com/2026/10/26/what-a-police-ai-trial-measured-and-what-it-missed/","evidenceClass":"independent-research","outcomeClass":"mixed","topics":["knowledge-work","governance-procurement","operating-model"],"finding":"AI & GPP analyses the 112-page Right to Information release (RTI 557/26, disclosure dated Oct. 6, 2026) behind the ABC’s Oct. 23 report on Tasmania Police’s Axon Draft One family-violence trial, which is already archived. Per the analysis: Phase One (six sergeants, 148 summaries from historic footage) found 21 (14%) with an error or omission judged critical to decision-making, but classified only 12 (6 name/role errors, 6 missed threat, assault or risk factors); the tool also surfaced risk factors absent from 10 human reports (7%). Time was measured against a 24-week human-only baseline (79 minutes per report) versus 42 minutes with the tool, about 38 minutes saved, but timings were self-reported and only 54% of trained testers exported a draft (20% never used it). The end-to-end rate of critical errors reaching a submitted report was measured once, 6% across 51 reports; errors were unweighted; free-text error explanations were summarised by Copilot; and 20% of frontline respondents thought safety had decreased, which the report notes is at odds with expert review. The author proposes six evaluation fixes, including a severity-weighted taxonomy, end-to-end miss rates and a pre-agreed stopping rule.","sledRelevance":"Gives agencies running AI report-writing pilots a concrete evaluation design and shows how time-saving claims can outrun harm measurement.","evidence":"Opened and read the AI & GPP article in full. Also opened the linked RTI PDF on police.tas.gov.au (112 pages, HTTP 200), but it is a scanned image without a text layer, so the figures could not be text-verified and are cited as the analysis reports them.","architectureImplications":"Treats safety as a property of the composed workflow (model, officer, supervisor, specialist review) rather than the model alone, and proposes per-draft decision logs recording what reviewers kept, changed or rejected.","governanceImplications":"Evaluators recommended phased deployment only after ethical-use policies, training and funded support; no legislative change needed. The analysis argues for a written stopping rule agreed before any pilot.","securityPrivacyImplications":"Family-violence body-camera recordings were processed by the tool; the analysis does not address data handling.","caveats":"Secondary analysis by an independent publication; primary RTI figures could not be text-verified by this run. Australian context; Tasmania Police says future use remains under review. Not a peer-reviewed study.","streamIds":["public-safety"],"roles":{"sales":"Sales framing: Tasmania’s released evaluation shows a common pilot gap: minutes saved were measured carefully, while harm was measured with one yes/no question and once end to end. Agencies weighing AI report drafting need an evaluation they can defend to prosecutors and the public. A bounded engagement could design a pilot evaluation with a human-only baseline and severity-weighted errors. The evidence cannot support guaranteed time savings, since timings were self-reported by the 54% who used the tool.","engineering":"Engineering framing: the analysis separates model accuracy from the safety of the whole review chain, which was measured at 6% across 51 reports. Pre-sales should ask whether the platform can log, per draft, what the officer kept, changed or rejected, and retain first drafts. A proof of value would measure critical errors at model output and after supervisor review on matched cases. The primary RTI document is scanned, so figures here rely on the analysis.","delivery":"Delivery framing: the work is running an evaluation that measures harm as rigorously as time: a baseline, a severity-weighted error taxonomy, end-to-end miss rates and a stopping rule agreed in writing. The operational owner is the agency’s evaluation or business-improvement unit with frontline supervisors. Acceptance criteria are proposals, such as no consecutive four-week windows above a pre-registered critical-error threshold. Risks include low adoption skewing results and reviewers trusting drafts more over time."},"roleDetails":{"sales":{"customerProblem":"Agencies piloting AI report drafting lack an evaluation that measures safety-critical errors with the same rigour as time saved.","stakeholders":["Police executives and evaluation units","Family-violence and specialist sergeants","Prosecutors","Police unions or associations","Oversight bodies"],"discoveryQuestions":["Was a human-only baseline collected before the pilot?","How are errors classified and weighted by severity?","What share of trained users actually used the tool?","What written stopping rule applies?"],"valueHypothesis":"Interpretation: a stronger evaluation design could prevent an unsafe rollout or support a defensible one; benefits depend on measured results.","potentialEngagement":"A bounded evaluation design for one agency’s AI report-drafting pilot.","unsupportedClaims":["Guaranteed time savings","That supervisor review catches critical errors","That Tasmania’s findings transfer directly to U.S. agencies"]},"engineering":{"fit":"Relevant to any Draft One or similar pilot, especially for high-risk report types.","architectureIntegration":"Requires per-draft logging of officer edits and retention of first drafts alongside final reports.","prerequisites":["First-draft retention","Edit logging","Matched human-only baseline cases"],"constraints":["Australian jurisdiction","Family-violence reports only","Primary figures not text-verified"],"security":"Body-camera audio of sensitive incidents is processed by the vendor; data-handling terms must be confirmed.","proofOfValue":["Measure critical-error rate at model output and end to end","Report adoption rates alongside time saved","Test judge-stability if a model scores drafts"]},"delivery":{"work":"Design and run a severity-weighted, baseline-controlled evaluation with a stopping rule.","dependencies":["Vendor logging capabilities","Supervisor time for review","Agreed error taxonomy"],"ownership":"Agency evaluation or business-improvement unit with frontline supervisors","skillsAdoption":"Officers trained to film for AI and to review drafts against recordings, not memory.","governanceCheckpoints":["Pre-registered thresholds","Four-weekly safety review","Independent review before rollout decision"],"acceptanceCriteria":["Severity-weighted error rates reported separately from time saved (proposed)","End-to-end critical-error rate below a pre-registered threshold (proposed)"],"risks":["Automation bias growing over time","Self-timed figures overstating savings","Unclassified error types escaping measurement"]}},"retrievedAt":"2026-10-28T02:58:59Z","enrichedAt":"2026-10-28T03:05:06Z","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Low adoption (20% never used the tool) means time-saving figures describe only willing users; frontline and expert views on safety diverged.","procurementImplications":"Time-saving estimates (A$434,187 a year in wages, per the analysis) could drive licensing decisions while the harm side remains thinly measured.","operatingModelImplications":"Sergeant-level validation remains the safeguard, but its effectiveness was measured on 51 reports only.","sourceVerification":{"openedUrl":"https://aiandgpp.com/2026/10/26/what-a-police-ai-trial-measured-and-what-it-missed/","referenceExcerpt":"21 of 148 AI-drafted summaries (14 per cent) carried an error or omission the reviewing sergeant judged critical to decision-making","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}