From the Health & Human Services edition of October 29, 2026
What AI gets wrong about SNAP, and what we tried to fix it
Nava PBC (Nava Labs) · SNAP / benefits policy knowledge tools · United States (California CalFresh scenarios)
- Publisher
- Nava Labs Field Notes
- Original publication
- 2026-10-08
- Source retrieved
- 2026-10-30
What happened
On a 500-question SNAP (CalFresh) benchmark keyed by SNAP experts (dataset from Better Government Lab), the best model scored 92.2% at baseline; errors clustered in self-employment, transitional benefits, SSN and citizenship/noncitizenship questions and came from subtle misreadings and outdated policy rather than hallucinated rules. Web search helped most on recently changed citizenship rules; a revised system prompt raised accuracy to 98.3% across three runs, but the author warns the guidance was tailored to this question set and may overfit.
Why it matters
State and county SNAP agencies considering LLM policy assistants for caseworkers or residents need topic-level error analysis, not a headline accuracy score; the categories that failed are the ones that drive eligibility errors.
Evidence and measured results
Benchmark of 500 scenario questions with an expert answer key; matched runs across models; three runs for the revised-prompt result. Not peer reviewed; prompt tuning was done on the same test set; the author notes they are not a policy expert.
Limitations and uncertainty
Single-state scenario set; tuned prompt likely overfits; small categories swing widely; models named are specific to the test date; no evidence of effects on real caseloads.
Put this evidence to work
Lighthouse Advisory interpretation, grounded in this source. Enriched 2026-10-29; this does not change the original publication date. Each role follows the same checklist so takeaways compare across sources.
Sales
Role takeaway
- Customer problem
- Agencies want LLM help with complex SNAP policy but cannot tell if vendor accuracy holds in the error-prone topics that drive payment errors.
- Stakeholders
- SNAP policy unit
- Quality control / payment accuracy staff
- Caseworker supervisors
- Agency AI governance lead
- Discovery questions
- Which SNAP QC categories generate most of your payment errors?
- Do you have an expert-keyed test set for your state's rules?
- Will the tool face residents, caseworkers or both?
- How will you re-test after policy changes?
- Value hypothesis
- Bounded hypothesis: category-level testing on held-out questions lowers the chance of deploying a tool that fails on high-risk eligibility topics.
- Potential engagement
- A short evaluation engagement that builds or adapts a state-specific scenario test set and reports accuracy by QC category for a candidate tool.
- Unsupported claims
- 98% accuracy in production
- LLMs can make final eligibility decisions
- Results transfer from CalFresh to other states without retesting
Pre-sales engineering
Role takeaway
- Fit
- Fits policy-assistant and caseworker-support use cases; not decision automation.
- Architecture and integration
- Retrieval over current state policy manuals plus targeted search for recently changed rules; structured outputs flagging when review is required.
- Prerequisites
- Expert-keyed scenario test set
- Current state policy corpus
- Prompt and model version control
- Constraints
- Prompt tuning on the test set overfits
- State-specific overrides of federal rules
- Rapid federal policy changes (e.g., noncitizen eligibility)
- Security
- Keep benchmark data synthetic; production use needs PII controls and logging of AI-assisted answers.
- Proof of value
- Held-out test set accuracy by QC category
- Regression test after each prompt change
- Re-test after each policy update
Delivery
Role takeaway
- Work
- Build test sets, run evaluations, maintain prompt guidance with SMEs, and report category-level accuracy over time.
- Dependencies
- Policy SME time
- Access to current state guidance
- Model/vendor stability
- Ownership
- State SNAP policy unit with QC staff; IT owns tool configuration.
- Skills and adoption
- SMEs learn to log and classify AI errors; caseworkers learn when to ask the tool for clarification versus consult policy.
- Governance checkpoints
- Category-level accuracy review before go-live
- Change control for prompts
- Re-validation after policy changes
- Acceptance criteria
- Proposed: no QC category below an agreed threshold on held-out questions
- Proposed: regression suite run on every prompt change
- Proposed: policy updates reflected and re-tested within a set number of days
- Risks
- Overfitting to the benchmark
- Regressions from broad guidance
- Staff over-reliance on confident answers
Implementation considerations
Lighthouse Advisory interpretation across the operating dimensions a public-sector buyer must settle before this evidence becomes a design. Each note answers the question under its heading for this specific source.
Architecture and integration
What must connect, and where does the AI sit in the workflow?
Retrieval of current state-specific guidance (e.g., CDSS) and web search for fast-changing rules matter as much as model choice; tools should ask clarifying questions when gross/net income or household role is ambiguous.
Governance
Who approves, reviews and stays accountable for outcomes?
Supports reporting AI accuracy by SNAP quality-control category and expert review of the hardest errors before deployment; distinguishes public-facing from caseworker-facing behavior.
Security and privacy
What data, permissions and controls need testing?
Benchmark used hypothetical scenarios; no real client data described. Production tools would handle PII.
Accessibility and workforce
Who is affected, and what skills or accommodations follow?
Caseworker-facing tools need more precise verification guidance than resident-facing tools; staff remain decision makers.
Procurement
What should contracts, pricing and exit terms secure?
Agencies can require vendors to report accuracy by SNAP QC category on a held-out set rather than an overall score.
Operating model
Which teams own the service once it runs?
Expect an ongoing tuning cycle with program experts, which is a staffing commitment rather than a one-time setup.
Publication history
- 2026-10-29Health & Human Services · Issue 018 resources
Stable resource ID: nava-llm-snap-policy-error-analysis