From the Research Universities edition of September 26, 2026
RECLAIM: best agent reproduces 41% / 27% / 15% of NeurIPS’25 claims by artifact tier (Run/Retrain/Reimplement)
University of Illinois Urbana-Champaign / National Center for Supercomputing Applications (NCSA) · Higher education research computing / ML evaluation · United States (Illinois)
- Publisher
- arXiv 2609.28850
- Original publication
- September 23, 2026 (arXiv)
- Source retrieved
- 2026-09-27
What happened
RECLAIM benchmarks whether agents can reproduce pinned central claims from 100 NeurIPS 2025 papers using author-released artifacts, graded by a separate LLM auditor on execution evidence (not agent self-reports). Best agent per tier reproduces only 41% of Run-tier (code+data+weights), 27% Retrain (no weights), and 15% Reimplement (no code). Failed attempts use on average 29% of GPU budget. Common failure: reimplement without checking against paper numbers (63/400 runs). Authors release harness, rubric, and graded transcripts; human–auditor F1 0.82 on reproduced verdict.
Why it matters
Research computing and research-integrity leaders deploying AI research agents need independent, execution-based reproduction rates by artifact availability—not leaderboard demos—before treating agent 'papers' as scientific progress.
Evidence and measured results
arXiv abs 2609.28850 opened via HTTPS; tiered reproduction rates, budget underuse, and failure-mode counts taken from abstract text (submitted 23 Sep 2026).
Limitations and uncertainty
Preprint; one venue/year; at most one run per cell; no human reproduction on same cluster for misses; possible contamination on released-code tiers; closed frontier models omitted for cost.
Put this evidence to work
Lighthouse Advisory interpretation, grounded in this source. Enriched 2026-09-26; this does not change the original publication date. Each role follows the same checklist so takeaways compare across sources.
Sales
Role takeaway
- Customer problem
- Campuses are pressured to adopt research agents without knowing how often agents actually recover published claims from real artifacts.
- Stakeholders
- VPR / research integrity
- Research computing / NCSA-class centers
- CS/ML faculty
- Library / reproducibility services
- Discovery questions
- Do current agent pilots grade execution evidence or agent reports?
- What share of target papers release code, data, and weights?
- Is GPU budget for agents metered and audited?
- Who owns failure analysis when reproduction misses?
- Value hypothesis
- Independent, artifact-tiered reproduction benchmarks can bound agent claims and drive artifact-release and audit requirements before scale-up.
- Potential engagement
- Pilot RECLAIM-style evaluation on a campus paper sample with frozen rubric and human spot-checks.
- Unsupported claims
- Agents reliably reproduce science end-to-end
- Leaderboard scores equal scientific validity
- More GPU alone fixes Reimplement failures
- Self-reported agent success
Pre-sales engineering
Role takeaway
- Fit
- HPC-hosted agent loop with pinned match targets, artifact links, and separate LLM auditor on run evidence.
- Architecture and integration
- Sandbox workspaces, Slurm GPU grants, evidence directories, and auditor pipeline disconnected from agent self-assessment.
- Prerequisites
- Pinned targets and tolerances
- Artifact availability audit
- GPU accounting
- Frozen grading rubric
- Human audit sample
- Constraints
- Preprint benchmark
- Single run per cell
- Venue/year scope
- Possible code contamination
- Security
- Contain untrusted agent code; redact secrets in published transcripts; control model API keys.
- Proof of value
- Audited reproduction rate by tier
- Agreement with human graders
- Budget utilization vs grant
- Failure-mode taxonomy coverage
Delivery
Role takeaway
- Work
- Stand up harness, select paper sample, run agents, audit, publish institutional baseline, set artifact policy.
- Dependencies
- HPC allocation
- Model serving or API budget
- Librarian/reproducibility partners
- Faculty paper owners
- Ownership
- Research computing owns harness ops; research integrity owns policy; PIs own paper artifacts.
- Skills and adoption
- Train staff to interpret auditor flags and to coach artifact release (code/data/weights).
- Governance checkpoints
- Rubric freeze
- Contamination review
- Publication of baseline rates
- Acceptance criteria
- Tiered rates reported with auditor method
- Human spot-check agreement documented
- Agent self-report not used as sole success metric
- Risks
- Overfitting harness to one model family
- Ignoring Reimplement-tier failures
- Publishing agent papers without audit
- Budget underuse misread as efficiency
Implementation considerations
Lighthouse Advisory interpretation across the operating dimensions a public-sector buyer must settle before this evidence becomes a design. Each note answers the question under its heading for this specific source.
Architecture and integration
What must connect, and where does the AI sit in the workflow?
Thin ReAct harness with GPU grant via scheduler; separate auditor reads logs/outputs; artifact-tiered tasks; labels/targets pinned before agent runs.
Governance
Who approves, reviews and stays accountable for outcomes?
Require provenance-audited grading and artifact release tiers in institutional agent research evaluation; agent self-assessment is untrusted (agents claimed 128 reproductions vs 73 audited).
Security and privacy
What data, permissions and controls need testing?
Benchmark uses academic papers/artifacts; campus deployments still need isolation for untrusted agent code and GPU job containment.
Accessibility and workforce
Who is affected, and what skills or accommodations follow?
Shows reproduction skill and artifact literacy matter as much as GPU access; underused budgets suggest process failures, not only compute scarcity.
Procurement
What should contracts, pricing and exit terms secure?
Require independent evaluation harnesses and artifact-tier reporting in agent platform RFPs; fund audit compute separately from agent demos.
Operating model
Which teams own the service once it runs?
Institutional AI-for-science programs should treat claim reproduction as a first-class service with auditors, not an optional afterthought to agent authoring.
Publication history
- 2026-09-26Research Universities · Issue 204 resources
Stable resource ID: reclaim-agent-ml-paper-reproduction-20260923