From the Anthropic edition of November 10, 2026
UK Defra transparency record: Claude Sonnet 4.6 guidance assistant met 4 of 6 pre-launch pass marks, missing rule-quoting and grounding thresholds
Department for Environment, Food and Rural Affairs (Defra), UK · National government (UK) · United Kingdom
- Publisher
- GOV.UK Algorithmic Transparency Records
- Original publication
- Published October 27, 2026 (page metadata modified November 9, 2026)
- Source retrieved
- 2026-11-11
- Event date
- 2026-10-27
What happened
Defra's transparency record for Ask the toolkit, a guidance assistant on Claude Sonnet 4.6 via Amazon Bedrock in London, reports that before launch it met four of six pass marks but missed the two requiring every answer to pass: quoting the expected rule (16 to 19 of 22) and staying grounded in named pages (87 to 90 of 100). An AI judge (Claude Opus 4.6) scored tests, and two human reviewers agreed with only 38 and 27 of 40 judge decisions. The service sends the whole toolkit with each question, verifies quotes against pages, has no tools or web access, caps use at 600 model calls a day, and live answers are not routinely reviewed.
Why it matters
Why now: the record (opened Oct. 27, page modified Nov. 9) was not previously archived and is a rare, detailed public evaluation of a Claude-based government guidance assistant, including its failures. State and local agencies building policy or benefits-guidance assistants can copy the design (whole-corpus context, quote verification, refusal paths, usage caps) and the transparency format, while noting that even a narrow 40-page corpus did not reach full grounding and that AI-judge scoring needs human calibration.
Evidence and measured results
Government-published pre-launch test results with counts, dates and method (AI judge plus human agreement check, manual prompt-attack rounds, load test of 20 simultaneous questions). Small test sets; no live outcome data.
Limitations and uncertainty
Pre-launch evaluation by the operating team using an AI judge with weak human agreement; no live usage data at writing; UK national context; delivered via AWS Bedrock, so model behavior is Anthropic's but hosting terms are AWS's. Cross-tagged state-government for transferability of the evaluation and transparency method; UK national context limits direct applicability. Anthropic model delivered via Amazon Bedrock; kept in the Anthropic stream because the evidence concerns Claude model behavior.
Put this evidence to work
Lighthouse Advisory interpretation, grounded in this source. Enriched 2026-11-11; this does not change the original publication date. Each role follows the same checklist so takeaways compare across sources.
Sales
Role takeaway
- Customer problem
- Agencies want staff or public guidance assistants but lack a tested pattern for proving answers stay within official policy.
- Stakeholders
- State or county AI program lead
- Policy or guidance owners
- Privacy officer
- Accessibility coordinator
- Discovery questions
- What accuracy bar must answers meet before launch?
- Will you publish evaluation results?
- How large is the guidance corpus?
- Who reviews live answers?
- Value hypothesis
- A grounded, narrowly scoped assistant with published testing could help staff find rules faster while making limits visible.
- Potential engagement
- A design-and-evaluation engagement producing a test set, pass marks and a transparency record for one guidance corpus.
- Unsupported claims
- Full grounding accuracy
- Fit for decisions about individuals
- Usage or time-savings results (none yet)
Pre-sales engineering
Role takeaway
- Fit
- Strong fit for small, stable guidance corpora and staff-facing Q&A.
- Architecture and integration
- Single model call with full corpus, structured output, quote verification, guardrail, pinned model, daily cap.
- Prerequisites
- Curated guidance corpus
- Fixed test set and pass marks
- Human calibration of AI judge
- Constraints
- Corpus size limits whole-context approach
- UK context
- Pre-launch evidence only
- Security
- Guardrail masks some personal data; platform logs retain raw questions 30 days; no tools or web access.
- Proof of value
- Report rule-selection and grounding rates on agency test set
- Measure human-judge agreement
- Run two rounds of prompt-attack tests
Delivery
Role takeaway
- Work
- Build, evaluate and publish a grounded guidance assistant.
- Dependencies
- Guidance owners maintaining pages
- Privacy and security approvals
- Accessibility testing
- Ownership
- Agency AI team with policy owners.
- Skills and adoption
- Team learns evaluation design and judge calibration.
- Governance checkpoints
- Pass marks agreed before build
- Privacy screening
- Publication of transparency record
- Acceptance criteria
- Proposed: grounding at or above agreed threshold on test set
- Proposed: human-judge agreement measured and reported
- Proposed: transparency record published at launch
- Risks
- Fluent incomplete answers
- Personal data in logs
- No live answer review
Implementation considerations
Lighthouse Advisory interpretation across the operating dimensions a public-sector buyer must settle before this evidence becomes a design. Each note answers the question under its heading for this specific source.
Architecture and integration
What must connect, and where does the AI sit in the workflow?
Single model call per question with the full ~40-page corpus (no retrieval), fixed output schema, quote-verification step, Bedrock guardrail, pinned model version, and a daily call cap. The record notes retrieval may be needed if the corpus grows.
Governance
Who approves, reviews and stays accountable for outcomes?
Publishes missed thresholds and limits openly; DPIA screened out because no personal data is sought; security and platform readiness approvals recorded; no formal fairness assessment.
Security and privacy
What data, permissions and controls need testing?
Bedrock guardrail masks some personal data before the model, but platform logs retain questions as typed (including any personal data) for 30 days in a restricted store; AWS processes prompts, and per Amazon documentation model makers cannot see them.
Accessibility and workforce
Who is affected, and what skills or accommodations follow?
Automated WCAG 2.2 AA testing found no failures; Defra's accessibility team will test with users after launch; Welsh and a representative public sample not tested.
Procurement
What should contracts, pricing and exit terms secure?
Model accessed through AWS under the UK One Government Value Agreement, with no direct Anthropic contract; agencies can follow a similar cloud-marketplace path but must read Bedrock third-party model terms.
Operating model
Which teams own the service once it runs?
AI Capability and Enablement team maintains the service, retests a fixed question set before any change, and can stop it by configuration change and redeployment.
Publication history
- 2026-11-10Anthropic · Issue 1210 resources
Stable resource ID: defra-ask-the-toolkit-atrs-claude-sonnet-4-6-missed-grounding-thresholds-20261110