From the State Government edition of October 5, 2026
Utah's AI policy office signs six independent evaluators so pilot results don't rest on company self-reports, and says attestation receipts prove only that a check ran
Utah Department of Commerce, Office of Artificial Intelligence Policy · State AI regulation / regulatory sandbox oversight · Utah statewide
- Publisher
- Utah Department of Commerce (commerce.utah.gov)
- Original publication
- 2026-10-05 (page modified); MOUs signed Aug 10 to Oct 2, 2026
- Source retrieved
- 2026-10-06
- Event date
- 2026-10-05
What happened
Utah's Office of AI Policy says companies testing AI in its AI Learning Laboratory report on their own performance and that the office 'does not want to rely on those reports alone.' It has signed memoranda of understanding with six organizations (Coalition for Health AI, Stanford Clinical Excellence Research Center, Clarion AI Partners, mpathic, Vega Health, Glacis Technologies; signed between Aug 10 and Oct 2, 2026) that can review a proposal before approval and audit samples of a pilot's real activity while it runs. The agreements cover clinical and technical evaluation, an evaluator network, and runtime attestation, in which selected companies must produce a tamper-evident receipt each time a specified safety check runs. The company pays the evaluator directly; evaluators must certify independence, report any patient safety concern, and accept audit; a certification that does not match findings is grounds to pause, roll back or terminate a pilot. Evaluation criteria are still being finalized.
Why it matters
A state regulator is building an independent-verification layer for AI it authorizes, with explicit limits on what attestation proves. The design transfers to state agencies that buy or permit AI and currently rely on vendor self-reports. The page was updated Oct 5 alongside the governor's health care AI announcement.
Evidence and measured results
Opened the Office of AI Policy page over HTTPS (curl, UA SLED-research-bot/1.0, BeautifulSoup extraction); article:modified_time is 2026-10-05. Confirmed the six MOUs and signing dates, the three agreement types, the payment and independence safeguards, and the statement that criteria are still being finalized. MOU PDFs were not opened. Cross-checked against Fox13 and Deseret News coverage of the Oct 5 announcement.
Limitations and uncertainty
Framework announcement, not results: criteria are still being finalized and no evaluation has been reported. Company-pays funding raises independence questions the page addresses only through certification and audit. Health-sector sandbox; transferability to general agency procurement is interpretation.
Put this evidence to work
Lighthouse Advisory interpretation, grounded in this source. Enriched 2026-10-05; this does not change the original publication date. Each role follows the same checklist so takeaways compare across sources.
Sales
Role takeaway
- Customer problem
- State agencies authorizing or buying AI rely on vendor-reported performance with little independent check during operation.
- Stakeholders
- State AI policy office or CAIO
- Agency program owner
- Procurement and contracts
- Privacy officer
- Independent evaluators
- Discovery questions
- How do you verify vendor AI performance after go-live?
- Who can pause an AI pilot, and on what evidence?
- Do contracts require logs an outside party can audit?
- Who pays for evaluation, and how is independence protected?
- Value hypothesis
- Writing independent evaluation and audit rights into one AI pilot can give the agency evidence beyond vendor reports before scaling.
- Potential engagement
- Two-to-four week design of an evaluation plan, audit clauses and pause criteria for a single AI pilot or contract.
- Unsupported claims
- That independent evaluation makes an AI system safe
- That Utah's evaluators have produced results
- That runtime attestation proves legal compliance
- That this health-sector model transfers unchanged to other agencies
Pre-sales engineering
Role takeaway
- Fit
- Audit-ready pilot design: logging, sampling and attestation, not the AI product.
- Architecture and integration
- Emit structured logs of AI actions and safety-check events to agency logging, sign or hash them for tamper evidence, and give evaluators scoped read access.
- Prerequisites
- Defined safety checks to attest
- Logging of real AI activity
- Evaluator access agreement
- Records-retention rules
- Constraints
- Evaluation criteria not yet final
- MOU terms not inspected
- Health-sector context
- Third-party access to sensitive data
- Security
- Limit evaluator access to sampled, minimized data under agreement; protect attestation records from alteration; log evaluator access.
- Proof of value
- Evaluator can complete a sample audit from logs alone
- Attestation records verify as untampered
- Pause decision can be made from evaluator report within an agreed time
Delivery
Role takeaway
- Work
- Select an evaluator, agree on sampling and reporting, set pause and rollback triggers, and confirm logging before launch.
- Dependencies
- Vendor logging capability
- Evaluator MOU or contract
- Legal review of data sharing
- Ownership
- The authorizing state office (Utah's Office of AI Policy in this case) keeps decision authority.
- Skills and adoption
- Program staff need to interpret evaluator findings and act on them; vendors need to support audit requests.
- Governance checkpoints
- Evaluation plan approved before launch
- Periodic evaluator reports
- Independence certification
- Pause or rollback review on mismatch
- Acceptance criteria
- Proposed: evaluator report delivered each agreed period
- Proposed: sample audits cover an agreed share of activity
- Proposed: every mismatch gets a documented decision
- Risks
- Company-pays model undermines perceived independence
- Criteria lag pilots already running
- Attestation mistaken for proof of safety
Implementation considerations
Lighthouse Advisory interpretation across the operating dimensions a public-sector buyer must settle before this evidence becomes a design. Each note answers the question under its heading for this specific source.
Architecture and integration
What must connect, and where does the AI sit in the workflow?
Runtime attestation requires pilots to emit tamper-evident records each time a safety check runs, and sample audits need access to logs of real AI activity. Both need logging designed in before launch.
Governance
Who approves, reviews and stays accountable for outcomes?
Separates who evaluates (independent organizations) from who decides (the state office keeps start, continue and stop authority), and makes a false certification grounds for pausing or ending a pilot.
Security and privacy
What data, permissions and controls need testing?
Evaluators check privacy safeguards before approval and may audit samples of real activity, which means patient or user data access for third parties that must be governed by the MOUs.
Accessibility and workforce
Who is affected, and what skills or accommodations follow?
The state office needs staff who can read evaluator reports and act on them; the page does not describe capacity.
Procurement
What should contracts, pricing and exit terms secure?
Shows a model in which the regulated party funds independent evaluation and can choose among approved evaluators; state buyers could write similar evaluation and attestation requirements into AI contracts.
Operating model
Which teams own the service once it runs?
The regulator acts as decision-maker over a network of qualified evaluators rather than doing technical evaluation itself.
Publication history
- 2026-10-05State Government · Issue 305 resources
Stable resource ID: utah-oaip-third-party-ai-evaluators-20261005