{"resourceId":"reclaim-agent-ml-paper-reproduction-20260923","versions":[{"version":"external-141b06f08e475be9d0aba4a671d4459560a81fbb03a2ed0b04461b056e88de6e","resource":{"id":"reclaim-agent-ml-paper-reproduction-20260923","title":"RECLAIM: best agent reproduces 41% / 27% / 15% of NeurIPS’25 claims by artifact tier (Run/Retrain/Reimplement)","organization":"University of Illinois Urbana-Champaign / National Center for Supercomputing Applications (NCSA)","sector":"Higher education research computing / ML evaluation","geography":"United States (Illinois)","publishedAt":"September 23, 2026 (arXiv)","publicationDate":"2026-09-23","eventDate":null,"sourceName":"arXiv 2609.28850","sourceLabel":"Academic preprint (arXiv)","sourceUrl":"https://arxiv.org/abs/2609.28850","evidenceClass":"academic-research","outcomeClass":"cautionary","topics":["developers-agents","governance-procurement","knowledge-work"],"finding":"RECLAIM benchmarks whether agents can reproduce pinned central claims from 100 NeurIPS 2025 papers using author-released artifacts, graded by a separate LLM auditor on execution evidence (not agent self-reports). Best agent per tier reproduces only 41% of Run-tier (code+data+weights), 27% Retrain (no weights), and 15% Reimplement (no code). Failed attempts use on average 29% of GPU budget. Common failure: reimplement without checking against paper numbers (63/400 runs). Authors release harness, rubric, and graded transcripts; human–auditor F1 0.82 on reproduced verdict.","sledRelevance":"Research computing and research-integrity leaders deploying AI research agents need independent, execution-based reproduction rates by artifact availability—not leaderboard demos—before treating agent 'papers' as scientific progress.","evidence":"arXiv abs 2609.28850 opened via HTTPS; tiered reproduction rates, budget underuse, and failure-mode counts taken from abstract text (submitted 23 Sep 2026).","architectureImplications":"Thin ReAct harness with GPU grant via scheduler; separate auditor reads logs/outputs; artifact-tiered tasks; labels/targets pinned before agent runs.","governanceImplications":"Require provenance-audited grading and artifact release tiers in institutional agent research evaluation; agent self-assessment is untrusted (agents claimed 128 reproductions vs 73 audited).","securityPrivacyImplications":"Benchmark uses academic papers/artifacts; campus deployments still need isolation for untrusted agent code and GPU job containment.","caveats":"Preprint; one venue/year; at most one run per cell; no human reproduction on same cluster for misses; possible contamination on released-code tiers; closed frontier models omitted for cost.","streamIds":["research"],"roles":{"sales":"Sales framing: Buyers of 'AI scientist' or research-agent platforms need cautionary evidence that even with code, data, and weights, the strongest tested agent recovers only 41% of pinned claims—and 15% when code is absent. RECLAIM supports discovery conversations about artifact release tiers, independent auditors, and GPU-budget governance on campus HPC. Do not pitch guaranteed paper reproduction or equate agent self-reports with independently verified results.","engineering":"Engineering framing: Fit is an evaluation harness with pinned targets, artifact tiers, scheduler GPU grants, and a separate auditor that grades execution evidence rather than agent narratives. Integrate with campus HPC identity, sandboxed workspaces, and metered GPU allocations. Proof-of-value is audited reproduction rate by tier, not agent-claimed success. Constraints include single-run variance, preprint status, and possible contamination on released-code tiers.","delivery":"Delivery framing: Deploy reproduction benchmarks before production AI-scientist workflows; retain full transcripts and auditor verdicts; and train researchers to release code, data, and weights while distrusting self-graded agent papers. Risks include treating a 41% Run-tier rate as success, ignoring Reimplement-tier failures, and funding agent seats without artifact-release and independent audit policy. This interpretation is bounded to the retrieved source evidence and should not be generalized beyond stated limits."},"roleDetails":{"sales":{"customerProblem":"Campuses are pressured to adopt research agents without knowing how often agents actually recover published claims from real artifacts.","stakeholders":["VPR / research integrity","Research computing / NCSA-class centers","CS/ML faculty","Library / reproducibility services"],"discoveryQuestions":["Do current agent pilots grade execution evidence or agent reports?","What share of target papers release code, data, and weights?","Is GPU budget for agents metered and audited?","Who owns failure analysis when reproduction misses?"],"valueHypothesis":"Independent, artifact-tiered reproduction benchmarks can bound agent claims and drive artifact-release and audit requirements before scale-up.","potentialEngagement":"Pilot RECLAIM-style evaluation on a campus paper sample with frozen rubric and human spot-checks.","unsupportedClaims":["Agents reliably reproduce science end-to-end","Leaderboard scores equal scientific validity","More GPU alone fixes Reimplement failures","Self-reported agent success"]},"engineering":{"fit":"HPC-hosted agent loop with pinned match targets, artifact links, and separate LLM auditor on run evidence.","architectureIntegration":"Sandbox workspaces, Slurm GPU grants, evidence directories, and auditor pipeline disconnected from agent self-assessment.","prerequisites":["Pinned targets and tolerances","Artifact availability audit","GPU accounting","Frozen grading rubric","Human audit sample"],"constraints":["Preprint benchmark","Single run per cell","Venue/year scope","Possible code contamination"],"security":"Contain untrusted agent code; redact secrets in published transcripts; control model API keys.","proofOfValue":["Audited reproduction rate by tier","Agreement with human graders","Budget utilization vs grant","Failure-mode taxonomy coverage"]},"delivery":{"work":"Stand up harness, select paper sample, run agents, audit, publish institutional baseline, set artifact policy.","dependencies":["HPC allocation","Model serving or API budget","Librarian/reproducibility partners","Faculty paper owners"],"ownership":"Research computing owns harness ops; research integrity owns policy; PIs own paper artifacts.","skillsAdoption":"Train staff to interpret auditor flags and to coach artifact release (code/data/weights).","governanceCheckpoints":["Rubric freeze","Contamination review","Publication of baseline rates"],"acceptanceCriteria":["Tiered rates reported with auditor method","Human spot-check agreement documented","Agent self-report not used as sole success metric"],"risks":["Overfitting harness to one model family","Ignoring Reimplement-tier failures","Publishing agent papers without audit","Budget underuse misread as efficiency"]}},"retrievedAt":"2026-09-26T21:59:35-05:00","enrichedAt":"2026-09-26T22:08:00-05:00","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Shows reproduction skill and artifact literacy matter as much as GPU access; underused budgets suggest process failures, not only compute scarcity.","procurementImplications":"Require independent evaluation harnesses and artifact-tier reporting in agent platform RFPs; fund audit compute separately from agent demos.","operatingModelImplications":"Institutional AI-for-science programs should treat claim reproduction as a first-class service with auditors, not an optional afterthought to agent authoring.","sourceVerification":{"openedUrl":"https://arxiv.org/abs/2609.28850","referenceExcerpt":"the best agent in each tier reproduces only 41% of Run-tier papers, 27% at Retrain, and 15% at Reimplement","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}