{"resourceId":"paperbenchx-ai-agents-reproduce-science-14-percent-20261029","versions":[{"version":"external-58dda9bb001082f1ff4b98388a94703e7f5ca15e88008d374652b4a1563ab2b8","resource":{"id":"paperbenchx-ai-agents-reproduce-science-14-percent-20261029","title":"PaperBenchX: the best of 10 frontier agent setups fully reproduced only 14% of 93 cross-discipline paper experiments","organization":"Ruoyu Wu and 16 co-authors (arXiv preprint)","sector":"Academic research on AI agents for scientific reproduction","geography":"International (author affiliations not verified)","publishedAt":"Submitted 29 October 2026 (arXiv v1)","publicationDate":"2026-10-29","eventDate":null,"sourceName":"arXiv","sourceLabel":"Wu et al., \"PAPERBENCHX: Can AI Reproduce Science Across Disciplines?\", arXiv:2610.39744, 29 Oct 2026","sourceUrl":"https://arxiv.org/abs/2610.39744","evidenceClass":"academic-research","outcomeClass":"cautionary","topics":["developers-agents","knowledge-work","operating-model"],"finding":"PaperBenchX is a benchmark of 93 expert-built paper-reproduction tasks across 12 research areas in electromagnetics, photonics, chemistry, materials science, biology and robotics, each with the source paper, a public task specification and a domain-native environment with specialized scientific software. Agents must reconstruct a key experiment, run and diagnose it, and deliver a runnable workflow. Submissions are replayed from scratch in a clean container with network disabled, and only regenerated evidence is scored against expert criteria. Across 10 frontier agent configurations (two using Codex, the rest Claude Code), the best fully reproduced only 14% of tasks at the 90% threshold; mean rubric scores ranged from 27.53% to 59.08%. Task construction took more than 1,000 hours of doctoral-level expert work.","sledRelevance":"Independent evidence for research leaders and research computing groups that coding agents doing domain-science work often produce partial or plausible-looking results that do not reproduce. It supports requiring clean-replay checks and expert review before agent-generated analyses enter publications or grant reports.","evidence":"Read the abstract, introduction, Figure 1 caption, task construction and evaluation protocol. Preprint, not peer reviewed. Per-agent tables were skimmed; appendices not read in full. Model versions and costs per run were not checked.","architectureImplications":"Clean replay in a fresh container with no network and fixed resource limits is a practical verification pattern for agent-produced workflows on shared research computing.","governanceImplications":"Rubric scores far above full-reproduction rates show that partial credit can overstate agent reliability; research policies should distinguish the two.","securityPrivacyImplications":"Network-disabled replay limits data leakage during evaluation; not otherwise addressed.","caveats":"Preprint by the benchmark authors, who also designed the scoring. Tasks are limited to 12 physical-science and engineering areas with available software. Results depend on agent harness and model versions at test time and may change quickly.","streamIds":["research"],"roles":{"sales":"Lead with verification, not speed, when researchers adopt coding agents. A 29 October arXiv preprint found the best of 10 frontier agent setups fully reproduced only 14% of 93 published experiments across 12 fields, while partial rubric scores reached 59%. Research leaders funding agent tools need a way to check outputs. Offer a bounded design of replay and review checks for agent-produced analyses. Do not claim agents cannot do research; this is one preprint benchmark.","engineering":"Frame fit around clean-replay verification on shared clusters. PaperBenchX restores only the agent's submitted package into a fresh container, disables the network, enforces resource limits and scores regenerated evidence. Prerequisites are containerized domain software with valid licenses, reproducible job scripts and stored provenance. Validate by replaying a sample of agent-produced workflows from scratch and comparing outputs with the original run and expert expectations before accepting them.","delivery":"Treat delivery as adding a verification step to agent-assisted research. Work covers container templates, a replay job, a review checklist and training for labs using coding agents. Ownership sits with research computing, with principal investigators signing off on results. Governance checkpoints cover provenance capture and expert review before publication. Acceptance criteria are proposals, such as a set share of agent-produced workflows passing clean replay, not results the authors reported for any institution."},"roleDetails":{"sales":{"customerProblem":"Researchers adopting coding agents get plausible results that may not reproduce, risking errors in publications and grant reports.","stakeholders":["Vice president for research","Research integrity officer","Research computing director","Principal investigators using agents"],"discoveryQuestions":["Which labs use coding agents for analysis today?","How are agent-produced results checked before publication?","Can your clusters replay a workflow from scratch?"],"valueHypothesis":"Replay and review checks may catch non-reproducible agent work before it reaches publications, at the cost of added staff time.","potentialEngagement":"Design of a clean-replay and expert-review process for agent-produced analyses in one department.","unsupportedClaims":["Agents are unusable for research","Results generalize to all fields","Specific agent rankings stay stable"]},"engineering":{"fit":"Fits research computing groups with container support that serve labs using coding agents.","architectureIntegration":"Container-based clean replay with network disabled, resource limits and provenance capture on shared HPC.","prerequisites":["Containerized domain software","Valid software licenses in containers","Provenance storage"],"constraints":["Commercial software licensing","Preprint evidence only","Fast-changing agent versions"],"security":"Run replays with network disabled and isolate evaluation data from agent access.","proofOfValue":["Replay a sample of agent workflows from scratch","Compare regenerated evidence with original claims","Track full versus partial reproduction rates"]},"delivery":{"work":"Build container templates, a replay job, a review checklist and training for labs using agents.","dependencies":["Research computing container platform","Domain software licenses","Faculty reviewers"],"ownership":"Research computing with principal investigators signing off on results.","skillsAdoption":"Lab members learn to package agent work reproducibly and read replay reports.","governanceCheckpoints":["Provenance captured for every agent run","Expert review before publication","Periodic audit of replay outcomes"],"acceptanceCriteria":["Target share of agent workflows pass clean replay (proposal)","Replay turnaround within an agreed time","Review checklist adopted by pilot labs"],"risks":["Reviewer time not funded","Licensing blocks containerization","False confidence from partial scores"]}},"retrievedAt":"2026-10-31T21:59:27-05:00","enrichedAt":"2026-10-31T21:59:28-05:00","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Expert time to build and judge tasks (about ten hours per task) shows verification itself needs skilled staff.","procurementImplications":"Commercial coding-agent products (Codex, Claude Code) were the harnesses tested; licensing of domain software inside containers is a practical constraint.","operatingModelImplications":"Suggests agent use in research should include replay and expert sign-off steps, which add staff time.","peerReview":"preprint","sourceVerification":{"openedUrl":"https://arxiv.org/abs/2610.39744","referenceExcerpt":"Across 10 frontier agent configurations, the best configuration fully reproduces only 14% tasks under the 90% threshold.","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}