Twelve-state pilot converts AI oversight into a risk-tiered regulatory evidence request

Problem and stakeholders: State insurance regulators, examiners, model-risk specialists, and entity liaisons need proportionate evidence requests for diverse AI uses.

Discovery

Can reviewers distinguish models from systems, identify direct consumer or financial impact, and locate support for questionnaire answers?

Value hypothesis

Materiality-scoped inventory could direct examiner effort toward consequential uses while reducing repetitive requests.

Potential engagement

A bounded examination-workflow pilot with training and evidence-traceability review.

Evidence boundary

Version 5.0 was an exposure draft and the 12-state pilot remained underway in the archive. The record does not establish effectiveness, consistent scoring, or a mandatory requirement everywhere. Applicability beyond insurance remains limited until materiality concepts are translated and tested.

Fit

Structure assurance evidence rather than treating the supplement as a model-validation product.

Architecture

Link system, model, use-case, dataset, third-party, limitation, and document-page records, including data-to-model relationships.

Prerequisites

Agreed materiality and inherent-risk definitions, entity evidence, and a common examination rubric.

Constraints

Jurisdictional tailoring and draft revisions can change fields; preserve version provenance.

Security

Restrict consumer data, protected attributes, and confidential model documentation while allowing authorized review.

Proposed validation

Have trained reviewers independently assess the same examples, compare escalation decisions, and verify that cited pages support responses. Measure usability, training gaps, and reviewer agreement locally; the archived pilot had not yet demonstrated those results or regulatory effectiveness.

Work and dependencies: Map questions into existing examinations, define evidence submission rules, and pilot bounded use cases.

Ownership

Regulatory leads set scope; examiners assess evidence; entity owners explain models and data; security staff protect submissions.

Skills and adoption

Provide common training on systems versus models, materiality, limitations, and third-party oversight, then capture reviewer and entity feedback.

Governance checkpoints

Approve local adaptations and revisit procedures as the draft changes.

Proposed acceptance

Traceable document references, consistent handling of sample risk tiers, recorded reviewer disagreements, and an actionable training-gap log.

Risks

Inconsistent thresholds, excessive questionnaire burden, confidential evidence exposure, and premature transfer to non-insurance services can undermine a proportionate process even if every field is completed.