From the OpenAI edition of November 10, 2026
Study: eight GPT-4o setups classifying 1,200 parliamentary questions scored 68-70% accuracy, yet 24.3% of items changed labels; review flags ranged 0.9-41%
Louise Yorke (arXiv preprint) · State and local government (AI-produced evidence for policy and service delivery) · United Kingdom and European Union (international; methods transfer to U.S. agencies)
- Publisher
- arXiv
- Original publication
- Submitted November 4, 2026
- Source retrieved
- 2026-11-11
What happened
The preprint compares eight GPT-4o configurations classifying 1,200 UK Prime Minister's Questions into 21 policy topics against human-coded labels. Accuracy ranges from 68.2% to 70.3%, yet 24.3% of questions receive different labels across configurations, changing topic shares and rankings. Even with unchanged settings at temperature zero, runs disagree on 2.78% of labels on average. Configurations agree more with one another than with human coding. The share flagged for human review ranges from 0.9% to 41.0% under the score-extraction procedure. A review of 37 UK and EU policy documents finds reporting expectations do not consistently cover changing answers, shared errors or review routing.
Why it matters
Agencies using OpenAI models to classify, summarize or triage public input (comments, 311 requests, case notes) should not rely on aggregate accuracy alone. The study gives concrete checks (repeat runs, configuration comparison, review-routing tests) that U.S. state and local AI governance can adopt.
Evidence and measured results
arXiv preprint (not peer reviewed), abstract page and HTML full text opened via curl. One task, one model family (GPT-4o, with a Claude comparison), small configuration set; correctness is agreement with human labels that may themselves be wrong. The author states the experiment is exploratory.
Limitations and uncertainty
Preprint; single task and model; human labels as ground truth; score extraction may not reflect confidence under chain-of-thought.
Put this evidence to work
Lighthouse Advisory interpretation, grounded in this source. Enriched 2026-11-11; this does not change the original publication date. Each role follows the same checklist so takeaways compare across sources.
Sales
Role takeaway
- Customer problem
- Agencies want to use AI to classify and summarize public input but cannot show the outputs are reliable enough for decisions.
- Stakeholders
- Chief data or AI officer
- Policy analysis units
- Records and audit staff
- Program managers using AI-produced evidence
- Discovery questions
- Which decisions will rely on AI-produced classifications or summaries?
- Do you test repeat-run stability and configuration sensitivity?
- Who signs off that AI evidence is sufficient?
- Value hypothesis
- An assurance-first evaluation can reduce the risk of misleading AI evidence; interpretation only.
- Potential engagement
- A bounded evaluation of one agency classification workflow covering accuracy, stability and review routing.
- Unsupported claims
- That GPT-4o is reliable for agency classification tasks
- That results generalize to newer models
- That human review catches all errors
Pre-sales engineering
Role takeaway
- Fit
- Applies to any LLM classification, triage or summarization pipeline in government.
- Architecture and integration
- Add versioned prompts, repeat runs, configuration comparison and logged outputs to the pipeline.
- Prerequisites
- Human-labeled validation sample
- Logging of model, prompt and settings
- Defined review threshold
- Constraints
- One task and model studied
- Ground-truth labels are judgments
- Score extraction may misstate confidence with reasoning
- Security
- Logging outputs for re-testing must follow records and privacy rules for the input data.
- Proof of value
- Run five repeats and measure label disagreement
- Compare at least two configurations on the same sample
- Test whether review flags capture known errors
Delivery
Role takeaway
- Work
- Embed assurance testing and sign-off into AI evidence workflows.
- Dependencies
- Labeled data
- Evaluation expertise
- Governance owner
- Ownership
- Agency AI governance lead with the program owner.
- Skills and adoption
- Staff need to interpret stability and agreement metrics, not just accuracy.
- Governance checkpoints
- Pre-deployment evaluation review
- Re-test on model or prompt change
- Periodic audit of reviewed versus unreviewed outputs
- Acceptance criteria
- Proposed: run-to-run disagreement below an agreed threshold
- Proposed: documented configuration and rationale
- Proposed: review routing captures a defined share of known errors
- Risks
- Aggregate accuracy hiding unstable labels
- Shared errors across configurations
- Model updates invalidating tests
Implementation considerations
Lighthouse Advisory interpretation across the operating dimensions a public-sector buyer must settle before this evidence becomes a design. Each note answers the question under its heading for this specific source.
Architecture and integration
What must connect, and where does the AI sit in the workflow?
Pipeline choices (output format, decoding, examples, reasoning) change classifications; temperature zero did not make outputs deterministic. Stability testing must be part of the architecture.
Governance
Who approves, reviews and stays accountable for outcomes?
Argues public bodies should justify reliance with checks beyond accuracy, clear responsibility, access to expertise and scrutiny of results.
Security and privacy
What data, permissions and controls need testing?
No privacy content; data were public parliamentary questions.
Accessibility and workforce
Who is affected, and what skills or accommodations follow?
Staff need access to evaluation expertise; no accessibility content.
Procurement
What should contracts, pricing and exit terms secure?
Buyers can require stability and review-routing evidence in AI contracts, not only accuracy claims.
Operating model
Which teams own the service once it runs?
Makes AI evidence assurance an ongoing operational duty with named responsibility.
Publication history
- 2026-11-10OpenAI · Issue 138 resources
Stable resource ID: arxiv-beyond-accuracy-ai-assurance-public-services-gpt4o-20261104