{"resourceId":"nava-llm-snap-policy-error-analysis","versions":[{"version":"external-ee945a20aeb0d45544a3c9c5dc285bcfdcff072329e6cfd4e0722f115581099c","resource":{"id":"nava-llm-snap-policy-error-analysis","title":"What AI gets wrong about SNAP, and what we tried to fix it","organization":"Nava PBC (Nava Labs)","sector":"SNAP / benefits policy knowledge tools","geography":"United States (California CalFresh scenarios)","publishedAt":"2026-10-08","publicationDate":"2026-10-08","eventDate":null,"sourceName":"Nava Labs Field Notes","sourceLabel":"Nava Labs Field Notes","sourceUrl":"https://fieldnotes.navapbc.com/p/what-ai-gets-wrong-about-snap-and","evidenceClass":"independent-research","outcomeClass":"mixed","topics":["knowledge-work","developers-agents","governance-procurement"],"finding":"On a 500-question SNAP (CalFresh) benchmark keyed by SNAP experts (dataset from Better Government Lab), the best model scored 92.2% at baseline; errors clustered in self-employment, transitional benefits, SSN and citizenship/noncitizenship questions and came from subtle misreadings and outdated policy rather than hallucinated rules. Web search helped most on recently changed citizenship rules; a revised system prompt raised accuracy to 98.3% across three runs, but the author warns the guidance was tailored to this question set and may overfit.","sledRelevance":"State and county SNAP agencies considering LLM policy assistants for caseworkers or residents need topic-level error analysis, not a headline accuracy score; the categories that failed are the ones that drive eligibility errors.","evidence":"Benchmark of 500 scenario questions with an expert answer key; matched runs across models; three runs for the revised-prompt result. Not peer reviewed; prompt tuning was done on the same test set; the author notes they are not a policy expert.","architectureImplications":"Retrieval of current state-specific guidance (e.g., CDSS) and web search for fast-changing rules matter as much as model choice; tools should ask clarifying questions when gross/net income or household role is ambiguous.","governanceImplications":"Supports reporting AI accuracy by SNAP quality-control category and expert review of the hardest errors before deployment; distinguishes public-facing from caseworker-facing behavior.","securityPrivacyImplications":"Benchmark used hypothetical scenarios; no real client data described. Production tools would handle PII.","caveats":"Single-state scenario set; tuned prompt likely overfits; small categories swing widely; models named are specific to the test date; no evidence of effects on real caseloads.","streamIds":["health-human-services"],"roles":{"sales":"Sales framing: agencies will hear vendor accuracy claims above 90%; this evidence shows the remaining errors fall in the highest-risk eligibility topics. Offer a bounded evaluation that tests a candidate policy assistant against QC categories with program experts. Do not claim near-perfect production accuracy; the 98.3% figure came from prompts tuned on the test set itself. Interpretation: evaluation capability is a differentiator in this market.","engineering":"Engineering framing: architecture should combine retrieval of current state policy, document or web search for recently changed rules, and clarifying-question behavior for ambiguous inputs. Hold out a test set separate from prompt tuning to avoid overfitting, track accuracy per QC category, and version prompts. Configure resident-facing and caseworker-facing modes separately, since the right answer differs by audience.","delivery":"Delivery framing: plan continuous expert review cycles rather than a single launch. A policy SME and a QC analyst should own the error log and prompt guidance, under change control, because fixes in one category can regress another. Set acceptance thresholds per category, and make policy updates such as noncitizen eligibility changes trigger re-testing."},"roleDetails":{"sales":{"customerProblem":"Agencies want LLM help with complex SNAP policy but cannot tell if vendor accuracy holds in the error-prone topics that drive payment errors.","stakeholders":["SNAP policy unit","Quality control / payment accuracy staff","Caseworker supervisors","Agency AI governance lead"],"discoveryQuestions":["Which SNAP QC categories generate most of your payment errors?","Do you have an expert-keyed test set for your state's rules?","Will the tool face residents, caseworkers or both?","How will you re-test after policy changes?"],"valueHypothesis":"Bounded hypothesis: category-level testing on held-out questions lowers the chance of deploying a tool that fails on high-risk eligibility topics.","potentialEngagement":"A short evaluation engagement that builds or adapts a state-specific scenario test set and reports accuracy by QC category for a candidate tool.","unsupportedClaims":["98% accuracy in production","LLMs can make final eligibility decisions","Results transfer from CalFresh to other states without retesting"]},"engineering":{"fit":"Fits policy-assistant and caseworker-support use cases; not decision automation.","architectureIntegration":"Retrieval over current state policy manuals plus targeted search for recently changed rules; structured outputs flagging when review is required.","prerequisites":["Expert-keyed scenario test set","Current state policy corpus","Prompt and model version control"],"constraints":["Prompt tuning on the test set overfits","State-specific overrides of federal rules","Rapid federal policy changes (e.g., noncitizen eligibility)"],"security":"Keep benchmark data synthetic; production use needs PII controls and logging of AI-assisted answers.","proofOfValue":["Held-out test set accuracy by QC category","Regression test after each prompt change","Re-test after each policy update"]},"delivery":{"work":"Build test sets, run evaluations, maintain prompt guidance with SMEs, and report category-level accuracy over time.","dependencies":["Policy SME time","Access to current state guidance","Model/vendor stability"],"ownership":"State SNAP policy unit with QC staff; IT owns tool configuration.","skillsAdoption":"SMEs learn to log and classify AI errors; caseworkers learn when to ask the tool for clarification versus consult policy.","governanceCheckpoints":["Category-level accuracy review before go-live","Change control for prompts","Re-validation after policy changes"],"acceptanceCriteria":["Proposed: no QC category below an agreed threshold on held-out questions","Proposed: regression suite run on every prompt change","Proposed: policy updates reflected and re-tested within a set number of days"],"risks":["Overfitting to the benchmark","Regressions from broad guidance","Staff over-reliance on confident answers"]}},"retrievedAt":"2026-10-29T22:05:10-05:00","enrichedAt":"2026-10-29T22:20:00-05:00","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Caseworker-facing tools need more precise verification guidance than resident-facing tools; staff remain decision makers.","procurementImplications":"Agencies can require vendors to report accuracy by SNAP QC category on a held-out set rather than an overall score.","operatingModelImplications":"Expect an ongoing tuning cycle with program experts, which is a staffing commitment rather than a one-time setup.","peerReview":"unknown","sourceVerification":{"openedUrl":"https://fieldnotes.navapbc.com/p/what-ai-gets-wrong-about-snap-and","referenceExcerpt":"GPT-6 Astra with the revised prompt averaged 491.7/500 (98.3%) across three runs, up from 461/500 (92.2%) at baseline.","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}