From the Research Universities edition of September 18, 2026
Pilot audit: agent benchmarks average 0.38 disclosure; none report inference cost or pinned eval images
BrightMind AI / Texas Tech University and University of Texas at Arlington (arXiv preprint) · AI evaluation methodology / research agent benchmarking · United States (author affiliations); applies to widely cited agent benchmarks
- Publisher
- arXiv
- Original publication
- May 20, 2026 (arXiv v1)
- Source retrieved
- 2026-09-19
What happened
Moghadasi and Ghaderi score twelve canonical benchmark papers on disclosure of how evaluations were run. Eight agent benchmarks average 0.38/1.0 versus 0.66 for four classical static benchmarks; none of the eight disclose inference cost in any form; none fully disclose a content-addressed container image of the evaluation environment. Authors score disclosure only and call for multi-rater follow-up after a single-auditor pass.
Why it matters
Why now: Older than last completed run, but newly relevant because this edition's LabAgent/Chronicle claims intensify campus pressure to trust agent scores—while this still-unarchived audit shows common agent benchmarks remain under-disclosed on cost and environments.
Evidence and measured results
Five-field schema; mean scores 0.38 vs 0.66; zero inference-cost disclosure among eight agent papers; missing content-addressed images; single-auditor limitation stated.
Limitations and uncertainty
May 2026 single-auditor pilot; disclosure ≠ correctness per authors; does not directly audit LabAgent/Chronicle/RASER.
Put this evidence to work
Lighthouse Advisory interpretation, grounded in this source. Enriched 2026-09-19; this does not change the original publication date. Each role follows the same checklist so takeaways compare across sources.
Sales
Role takeaway
- Customer problem
- Agent benchmark scores used in campus decisions often cannot be re-run because papers omit cost and pinned environments.
- Stakeholders
- Research computing evaluation leads
- Procurement
- Faculty comparing agents
- Internal audit/risk
- Discovery questions
- Do cited benchmarks publish inference cost?
- Is there a content-addressed eval image?
- Who scored disclosure?
- What failure breakdown exists?
- Value hypothesis
- Requiring disclosure fields (cost, harness digest, failure breakdown) reduces unfalsifiable agent claims even when accuracy looks strong.
- Potential engagement
- Scorecard workshop applying the open schema to the campus's top cited agent benchmarks.
- Unsupported claims
- Disclosure equals model trustworthiness
- All agent benchmarks are invalid
- Guaranteed multi-rater agreement
Pre-sales engineering
Role takeaway
- Fit
- Evaluation disclosure schema and pinned environment requirements for agent benchmarks.
- Architecture and integration
- Add schema validation to internal eval reports; store image digests beside score tables.
- Prerequisites
- Agreed field codebook
- Container registry for digests
- Cost metering hooks
- Constraints
- Single-auditor original study
- Older May 2026 paper
- Does not measure task correctness
- Security
- Ensure published digests and cost logs do not embed secrets or restricted research data.
- Proof of value
- Fraction of cited benchmarks with cost lines
- Presence of content-addressed images
- Completed five-field scoresheets
Delivery
Role takeaway
- Work
- Apply schema to current campus agent evaluations; remediate missing fields; schedule multi-rater re-score.
- Dependencies
- Eval owners
- Container registry
- Vendor documentation
- Ownership
- Research computing evaluation function owns the scorecard.
- Skills and adoption
- Train staff to score disclosure separately from accuracy.
- Governance checkpoints
- Multi-rater calibration
- Annual re-audit of cited benchmarks
- Acceptance criteria
- All in-use agent benchmarks scored
- No production decision cites cost-omitting papers without waiver
- Written waiver path
- Risks
- Performative checklists
- Vendor pushback
- Confusing disclosure with scientific validity
Implementation considerations
Lighthouse Advisory interpretation across the operating dimensions a public-sector buyer must settle before this evidence becomes a design. Each note answers the question under its heading for this specific source.
Architecture and integration
What must connect, and where does the AI sit in the workflow?
Campus eval pipelines should require pinned environment digests and cost metering beside accuracy tables.
Governance
Who approves, reviews and stays accountable for outcomes?
RFPs and internal scorecards should treat missing disclosure fields as incomplete evidence.
Security and privacy
What data, permissions and controls need testing?
Environment images and cost logs must be published without embedding secrets.
Accessibility and workforce
Who is affected, and what skills or accommodations follow?
Creates evaluation-literacy work for staff who must score disclosure separately from accuracy.
Procurement
What should contracts, pricing and exit terms secure?
Write disclosure-field requirements into agent-benchmark and platform procurements; waive only with documented risk acceptance.
Operating model
Which teams own the service once it runs?
Separates evaluation governance from research-agent feature delivery.
Publication history
- 2026-09-18Research Universities · Issue 134 resources
Stable resource ID: moghadasi-agent-benchmark-disclosure-audit-20260520