Zoning benchmark exposes retrieval and useful-answer limitations

Ask the planning director, resident-service manager and CIO whether repeated zoning questions or misunderstood requirements consume material staff effort. What can residents resolve unaided today, and which questions require professional judgment? Offer a bounded discovery and benchmark engagement on one ordinance area. The value hypothesis is fewer avoidable handoffs with reliable answers, subject to local measurement. Include community representatives when defining useful responses. Do not sell a general model as a substitute for planning advice or promise fewer housing delays. Establish the baseline and escalation workload before estimating value; a technically correct refusal can still leave the resident's problem unresolved.

Start with a read-only assistant over approved public ordinances. Map document extraction, section relationships, retrieval, generation and citation display; verify version synchronization with the authoritative source. Prerequisites include a planner-reviewed question set and expected supporting passages. Test multi-section questions, obsolete rules, unsupported assumptions and malicious document instructions. Compare retrieval recall, answer correctness and useful abstention independently. Current cloud, on-premises and hybrid options require local cost and privacy assessment; this exercise does not rank them. Keep agents from submitting applications during validation. Use blinded expert scoring and repeated runs before allowing resident access.

Assign a planning-service owner and a technical indexing owner, then build the test corpus with frontline staff. Budget ordinance cleanup, accessible user testing and correction handling before rollout. Train staff to inspect citations and preserve conventional advice channels. Proposed acceptance criteria are improvement in resident task completion, no deterioration in expert-scored accuracy and successful refresh tests after a code change; set numerical thresholds before testing. These are future criteria, not observed results. Review misses and refusals weekly during the pilot. Stop expansion if maintenance or escalation demand exceeds funded capacity, and require renewed approval after retrieval or model changes.