{"resourceId":"ny-state-ai-governance-audit-2023-s-50","versions":[{"version":"external-6046a5e99ec2cab53c5d7f84f5399d62fc0dd769b7e5caa9234fd7f064ade2da","resource":{"id":"ny-state-ai-governance-audit-2023-s-50","title":"New York audit documents the gap between AI policy and verifiable operating procedures","organization":"New York State Office of the State Comptroller","sector":"State government","geography":"New York, United States","publishedAt":"April 3, 2025","publicationDate":"2025-04-03","eventDate":null,"sourceName":"Office of the New York State Comptroller","sourceLabel":"Report 2023-S-50","sourceUrl":"https://www.osc.ny.gov/files/state-agencies/audits/pdf/sga-2025-23s50.pdf","evidenceClass":"government-audit","outcomeClass":"cautionary","topics":["data-security","governance-procurement","accessibility-workforce","operating-model"],"finding":"The auditor found inconsistent governance and insufficient evidence of testing procedures across a judgmental agency sample.","sledRelevance":"State shared-service oversight and agency accountability are central. Use-case examples support that governance analysis; no policing or education cross-tags are added.","evidence":"Audit period: January 2019–November 2024. Methods included policy review, interviews, vendor demonstrations and documentation checks for four judgmentally selected agencies and one selected use case each. DMV disputed testing-related findings; the auditor said supplied documentation did not establish the requested procedures.","architectureImplications":"Interpretation: trace inventories to actual configurations and retained evaluation artifacts across hosted and internally managed systems. This governance audit supplies no model or hardware benchmark.","governanceImplications":"Interpretation: resolve AI-definition disputes and require evidence that agency review responsibilities operate in practice.","securityPrivacyImplications":"Interpretation: map data ownership, vendor reuse and retention to enforceable terms and system behavior.","caveats":"Non-statistical sample cannot be projected to all agencies. Historical audit is not a September 2026 compliance assessment. It identifies assurance gaps rather than quantifying population harm; auditee disagreement is preserved.","streamIds":["state-government"],"roles":{"sales":"Interpretation: Qualify the assurance workload with central IT, agency program owners, procurement and internal audit. Ask whether a reviewer can trace policy requirements to executed tests, who settles classification disputes, and which supplier terms constrain evidence access. A bounded engagement could reconcile one agency's inventory and inspect a sample of approval and monitoring records. The credible value hypothesis is clearer accountability and less effort assembling defensible evidence. Do not claim that this older audit describes the customer's present posture or proves actual discrimination. Establish current remediation status independently and include the agency's explanation when documenting any disputed control gap.","engineering":"Interpretation: Fit is evidence integration across inventory, configuration, contracts and test records. Prerequisites include named system owners, an agreed AI definition and access to supplier documentation. Build a trace from each material risk to its test, result and approval, including exceptions. Validate output accuracy and subgroup behavior using use-case-specific criteria; generic security reviews cannot establish these properties. Test whether a model or vendor change triggers renewed review. A proposed proof of value should reconstruct one complete approval history and repeat the relevant evaluation. Deployment may span cloud, on-premises and hybrid environments; the audit supports a governance approach, not selection of a hosting model.","delivery":"Interpretation: Reconcile existing systems, document required procedures and assign recurring review to operational owners. Central IT supplies common definitions and templates, while agency leaders accept residual risks. Dependencies include procurement cooperation, specialist evaluation skills and access to legacy records. Train staff on how to identify AI features and report uncertain classifications. Proposed acceptance criteria: every sampled system has a named owner, documented data terms, retained test evidence and an approved disposition for exceptions. Exercise a supplier change and verify that review occurs. Risks include stale inventories and treating a policy document as operational proof; report disputed findings and remediation evidence separately."},"retrievedAt":"2026-09-10T03:02:13Z","enrichedAt":"2026-09-10T03:02:13Z","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Interpretation: distinguish tool-use training from the skills needed to detect bias and inaccurate outputs; include affected users in validation.","procurementImplications":"Interpretation: make supplier testing access and data-use obligations explicit, with accountable exception approval.","operatingModelImplications":"Interpretation: central IT provides usable procedures; agency owners retain risk acceptance and regular performance review.","updateExplanation":"Full archive contains separate NYC and SUNY audits, but not this statewide 2023-S-50 report. Historical source newly included for the policy-to-procedure distinction and documented auditee disagreement, not as a fresh audit.","sourceVerification":{"openedUrl":"https://www.osc.ny.gov/files/state-agencies/audits/pdf/sga-2025-23s50.pdf","referenceExcerpt":"We used a non-statistical sampling approach","promptVersion":"sled-research-v3.1","model":null,"basis":"agent-reported inspection"}}}]}