{"resourceId":"moghadasi-agent-benchmark-disclosure-audit-20260520","versions":[{"version":"external-a54edf6d5f648b68aa6cfc7cc28e23bb9ce270f499c6d1a3085a8b179e6eb1c2","resource":{"id":"moghadasi-agent-benchmark-disclosure-audit-20260520","title":"Pilot audit: agent benchmarks average 0.38 disclosure; none report inference cost or pinned eval images","organization":"BrightMind AI / Texas Tech University and University of Texas at Arlington (arXiv preprint)","sector":"AI evaluation methodology / research agent benchmarking","geography":"United States (author affiliations); applies to widely cited agent benchmarks","publishedAt":"May 20, 2026 (arXiv v1)","publicationDate":"2026-05-20","eventDate":null,"sourceName":"arXiv","sourceLabel":"Independent academic pilot audit — agent benchmark disclosure","sourceUrl":"https://arxiv.org/html/2605.21404","evidenceClass":"independent-research","outcomeClass":"cautionary","topics":["governance-procurement","developers-agents","operating-model"],"finding":"Moghadasi and Ghaderi score twelve canonical benchmark papers on disclosure of how evaluations were run. Eight agent benchmarks average 0.38/1.0 versus 0.66 for four classical static benchmarks; none of the eight disclose inference cost in any form; none fully disclose a content-addressed container image of the evaluation environment. Authors score disclosure only and call for multi-rater follow-up after a single-auditor pass.","sledRelevance":"Why now: Older than last completed run, but newly relevant because this edition's LabAgent/Chronicle claims intensify campus pressure to trust agent scores—while this still-unarchived audit shows common agent benchmarks remain under-disclosed on cost and environments.","evidence":"Five-field schema; mean scores 0.38 vs 0.66; zero inference-cost disclosure among eight agent papers; missing content-addressed images; single-auditor limitation stated.","architectureImplications":"Campus eval pipelines should require pinned environment digests and cost metering beside accuracy tables.","governanceImplications":"RFPs and internal scorecards should treat missing disclosure fields as incomplete evidence.","securityPrivacyImplications":"Environment images and cost logs must be published without embedding secrets.","caveats":"May 2026 single-auditor pilot; disclosure ≠ correctness per authors; does not directly audit LabAgent/Chronicle/RASER.","streamIds":["research"],"roles":{"sales":"Sales framing: Why now—September agent harness and regression-test papers will be sold on scores, yet Moghadasi/Ghaderi's pilot audit shows canonical agent benchmarks average 0.38 disclosure and omit inference cost and content-addressed environments. Use this to insist RFPs and campus evaluations require disclosure fields, not just accuracy claims. Unsupported: that disclosure equals correctness (authors deny this) or that any specific new agent fails the audit.","engineering":"Engineering framing: Fit is an evaluation-governance checklist (identity, harness, inference settings, cost, failure breakdown) plus pinned content-addressed images—not a new agent. Integrate into campus AI evaluation pipelines and vendor scorecard templates. Prerequisites: agreement on schema fields and ability to publish image digests. Constraints: single-auditor study; May 2026 vintage; disclosure≠trust. Security/privacy: cost and environment pins must not leak credentials inside images.","delivery":"Delivery framing: Adopt the open scoring schema for any research-agent benchmark a campus cites; require cost lines and image digests before accepting comparative claims. Dependencies: research computing and procurement. Ownership: evaluation owners. Governance: multi-rater review to address the paper's single-auditor limit. Acceptance: every cited agent benchmark scored on the five fields; risks include performative disclosure without correctness testing."},"roleDetails":{"sales":{"customerProblem":"Agent benchmark scores used in campus decisions often cannot be re-run because papers omit cost and pinned environments.","stakeholders":["Research computing evaluation leads","Procurement","Faculty comparing agents","Internal audit/risk"],"discoveryQuestions":["Do cited benchmarks publish inference cost?","Is there a content-addressed eval image?","Who scored disclosure?","What failure breakdown exists?"],"valueHypothesis":"Requiring disclosure fields (cost, harness digest, failure breakdown) reduces unfalsifiable agent claims even when accuracy looks strong.","potentialEngagement":"Scorecard workshop applying the open schema to the campus's top cited agent benchmarks.","unsupportedClaims":["Disclosure equals model trustworthiness","All agent benchmarks are invalid","Guaranteed multi-rater agreement"]},"engineering":{"fit":"Evaluation disclosure schema and pinned environment requirements for agent benchmarks.","architectureIntegration":"Add schema validation to internal eval reports; store image digests beside score tables.","prerequisites":["Agreed field codebook","Container registry for digests","Cost metering hooks"],"constraints":["Single-auditor original study","Older May 2026 paper","Does not measure task correctness"],"security":"Ensure published digests and cost logs do not embed secrets or restricted research data.","proofOfValue":["Fraction of cited benchmarks with cost lines","Presence of content-addressed images","Completed five-field scoresheets"]},"delivery":{"work":"Apply schema to current campus agent evaluations; remediate missing fields; schedule multi-rater re-score.","dependencies":["Eval owners","Container registry","Vendor documentation"],"ownership":"Research computing evaluation function owns the scorecard.","skillsAdoption":"Train staff to score disclosure separately from accuracy.","governanceCheckpoints":["Multi-rater calibration","Annual re-audit of cited benchmarks"],"acceptanceCriteria":["All in-use agent benchmarks scored","No production decision cites cost-omitting papers without waiver","Written waiver path"],"risks":["Performative checklists","Vendor pushback","Confusing disclosure with scientific validity"]}},"retrievedAt":"2026-09-19T03:12:28Z","enrichedAt":"2026-09-19T03:12:28Z","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Creates evaluation-literacy work for staff who must score disclosure separately from accuracy.","procurementImplications":"Write disclosure-field requirements into agent-benchmark and platform procurements; waive only with documented risk acceptance.","operatingModelImplications":"Separates evaluation governance from research-agent feature delivery.","sourceVerification":{"openedUrl":"https://arxiv.org/html/2605.21404","referenceExcerpt":"The mean audit score across the eight agent-benchmark papers is 0.38","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}