{"resourceId":"utah-oaip-third-party-ai-evaluators-20261005","versions":[{"version":"external-651a5f68508ec8d2aec4922d3c727e06daf93934b14e0a1e982a3eab73979413","resource":{"id":"utah-oaip-third-party-ai-evaluators-20261005","title":"Utah's AI policy office signs six independent evaluators so pilot results don't rest on company self-reports, and says attestation receipts prove only that a check ran","organization":"Utah Department of Commerce, Office of Artificial Intelligence Policy","sector":"State AI regulation / regulatory sandbox oversight","geography":"Utah statewide","publishedAt":"2026-10-05 (page modified); MOUs signed Aug 10 to Oct 2, 2026","publicationDate":"2026-10-05","eventDate":"2026-10-05","sourceName":"Utah Department of Commerce (commerce.utah.gov)","sourceLabel":"AI Third-Party Evaluators, Office of Artificial Intelligence Policy","sourceUrl":"https://commerce.utah.gov/ai/third-party-evaluators/","evidenceClass":"standards-guidance","outcomeClass":"emerging","topics":["governance-procurement","data-security","operating-model"],"finding":"Utah's Office of AI Policy says companies testing AI in its AI Learning Laboratory report on their own performance and that the office 'does not want to rely on those reports alone.' It has signed memoranda of understanding with six organizations (Coalition for Health AI, Stanford Clinical Excellence Research Center, Clarion AI Partners, mpathic, Vega Health, Glacis Technologies; signed between Aug 10 and Oct 2, 2026) that can review a proposal before approval and audit samples of a pilot's real activity while it runs. The agreements cover clinical and technical evaluation, an evaluator network, and runtime attestation, in which selected companies must produce a tamper-evident receipt each time a specified safety check runs. The company pays the evaluator directly; evaluators must certify independence, report any patient safety concern, and accept audit; a certification that does not match findings is grounds to pause, roll back or terminate a pilot. Evaluation criteria are still being finalized.","sledRelevance":"A state regulator is building an independent-verification layer for AI it authorizes, with explicit limits on what attestation proves. The design transfers to state agencies that buy or permit AI and currently rely on vendor self-reports. The page was updated Oct 5 alongside the governor's health care AI announcement.","evidence":"Opened the Office of AI Policy page over HTTPS (curl, UA SLED-research-bot/1.0, BeautifulSoup extraction); article:modified_time is 2026-10-05. Confirmed the six MOUs and signing dates, the three agreement types, the payment and independence safeguards, and the statement that criteria are still being finalized. MOU PDFs were not opened. Cross-checked against Fox13 and Deseret News coverage of the Oct 5 announcement.","architectureImplications":"Runtime attestation requires pilots to emit tamper-evident records each time a safety check runs, and sample audits need access to logs of real AI activity. Both need logging designed in before launch.","governanceImplications":"Separates who evaluates (independent organizations) from who decides (the state office keeps start, continue and stop authority), and makes a false certification grounds for pausing or ending a pilot.","securityPrivacyImplications":"Evaluators check privacy safeguards before approval and may audit samples of real activity, which means patient or user data access for third parties that must be governed by the MOUs.","caveats":"Framework announcement, not results: criteria are still being finalized and no evaluation has been reported. Company-pays funding raises independence questions the page addresses only through certification and audit. Health-sector sandbox; transferability to general agency procurement is interpretation.","streamIds":["state-government"],"roles":{"sales":"The key point is verification, not trust: Utah's AI office says it won't rely on company self-reports and has signed six independent evaluators who can audit pilots while they run. Discovery should ask how a prospect checks vendor AI performance claims after go-live. A bounded engagement is drafting evaluation and audit clauses for one AI contract or pilot. This page cannot support claims that independent evaluation makes AI safe; Utah itself says an attestation receipt shows only that a check ran.","engineering":"Fit is designing pilots so they can be audited: log real AI activity in a form an evaluator can sample, and produce tamper-evident records each time a defined safety check runs. Integrate those records with the agency's logging and records-retention systems and give evaluators scoped access. Constraint: criteria are not final and MOUs were not opened, so proof of value is an audit-ready logging design tested against a sample request, not a safety score.","delivery":"Delivery work is setting up evaluation before launch: name the evaluator, agree on the sampling and reporting plan, and set the conditions for pause or rollback. Ownership stays with the state office that authorizes the pilot. Acceptance criteria are proposals: evaluator report received on schedule, sample audits completed, mismatches acted on. Risks include the company-pays model weakening independence and criteria lagging behind pilots that are already running."},"roleDetails":{"sales":{"customerProblem":"State agencies authorizing or buying AI rely on vendor-reported performance with little independent check during operation.","stakeholders":["State AI policy office or CAIO","Agency program owner","Procurement and contracts","Privacy officer","Independent evaluators"],"discoveryQuestions":["How do you verify vendor AI performance after go-live?","Who can pause an AI pilot, and on what evidence?","Do contracts require logs an outside party can audit?","Who pays for evaluation, and how is independence protected?"],"valueHypothesis":"Writing independent evaluation and audit rights into one AI pilot can give the agency evidence beyond vendor reports before scaling.","potentialEngagement":"Two-to-four week design of an evaluation plan, audit clauses and pause criteria for a single AI pilot or contract.","unsupportedClaims":["That independent evaluation makes an AI system safe","That Utah's evaluators have produced results","That runtime attestation proves legal compliance","That this health-sector model transfers unchanged to other agencies"]},"engineering":{"fit":"Audit-ready pilot design: logging, sampling and attestation, not the AI product.","architectureIntegration":"Emit structured logs of AI actions and safety-check events to agency logging, sign or hash them for tamper evidence, and give evaluators scoped read access.","prerequisites":["Defined safety checks to attest","Logging of real AI activity","Evaluator access agreement","Records-retention rules"],"constraints":["Evaluation criteria not yet final","MOU terms not inspected","Health-sector context","Third-party access to sensitive data"],"security":"Limit evaluator access to sampled, minimized data under agreement; protect attestation records from alteration; log evaluator access.","proofOfValue":["Evaluator can complete a sample audit from logs alone","Attestation records verify as untampered","Pause decision can be made from evaluator report within an agreed time"]},"delivery":{"work":"Select an evaluator, agree on sampling and reporting, set pause and rollback triggers, and confirm logging before launch.","dependencies":["Vendor logging capability","Evaluator MOU or contract","Legal review of data sharing"],"ownership":"The authorizing state office (Utah's Office of AI Policy in this case) keeps decision authority.","skillsAdoption":"Program staff need to interpret evaluator findings and act on them; vendors need to support audit requests.","governanceCheckpoints":["Evaluation plan approved before launch","Periodic evaluator reports","Independence certification","Pause or rollback review on mismatch"],"acceptanceCriteria":["Proposed: evaluator report delivered each agreed period","Proposed: sample audits cover an agreed share of activity","Proposed: every mismatch gets a documented decision"],"risks":["Company-pays model undermines perceived independence","Criteria lag pilots already running","Attestation mistaken for proof of safety"]}},"retrievedAt":"2026-10-05T22:19:01-05:00","enrichedAt":"2026-10-05T22:19:01-05:00","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"The state office needs staff who can read evaluator reports and act on them; the page does not describe capacity.","procurementImplications":"Shows a model in which the regulated party funds independent evaluation and can choose among approved evaluators; state buyers could write similar evaluation and attestation requirements into AI contracts.","operatingModelImplications":"The regulator acts as decision-maker over a network of qualified evaluators rather than doing technical evaluation itself.","sourceVerification":{"responseId":"curl-ut-evaluators-20261005","openedUrl":"https://commerce.utah.gov/ai/third-party-evaluators/","referenceExcerpt":"A receipt shows the check ran. It does not show that the AI is safe or that the company followed the law.","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}