Utah tax pilot improves benchmark scores, with generalization and production limits

Explore the tax service owner's actual bottleneck with call-center managers, content specialists, shared IT, finance and procurement. Ask whether staff lose time locating guidance, checking answers or correcting outdated material. A bounded comparative evaluation could establish which workflow deserves investment. The value hypothesis is more reliable access to approved guidance, subject to local testing. Offer a test plan and vendor comparison with a documented stop decision. Do not promise the published benchmark gain, lower staffing needs or a production return. Ask who can supply representative questions and release expert reviewers without impairing service.

Fit is internal assistance over controlled tax content. Map the knowledge source, retrieval index, staff interface and approved hosting boundary before integration. Prerequisites include versioned authoritative documents, expert reviewers and a held-out question set. Keep tuning questions separate from acceptance cases, test missing or contradictory guidance, and record abstention and citation correctness. Compare full human verification time with the current search process. Validate permission boundaries and prompt retention independently of factual accuracy. Test deployment in the intended environment before commitment; no cloud-versus-on-premises performance advantage or autonomous-agent authority is established here.

Assign tax operations the acceptance decision, content experts the knowledge refresh process and shared IT the runtime. First baseline the current search workflow, then configure retrieval, train reviewers and run a controlled pilot. Dependencies include specialist time, hosting readiness and support funding.

Proposed acceptance criteria

every sampled answer has a review outcome, all critical misleading-answer failures are resolved, and quality plus verification time meet pre-agreed thresholds on untouched cases. Require documented approval before expansion and repeat tests after content or model changes. Risks include test-set overfitting, stale guidance and hidden support effort.