From the Research Universities edition of October 31, 2026
PaperBenchX: the best of 10 frontier agent setups fully reproduced only 14% of 93 cross-discipline paper experiments
Ruoyu Wu and 16 co-authors (arXiv preprint) · Academic research on AI agents for scientific reproduction · International (author affiliations not verified)
- Publisher
- arXiv
- Original publication
- Submitted 29 October 2026 (arXiv v1)
- Source retrieved
- 2026-11-01
What happened
PaperBenchX is a benchmark of 93 expert-built paper-reproduction tasks across 12 research areas in electromagnetics, photonics, chemistry, materials science, biology and robotics, each with the source paper, a public task specification and a domain-native environment with specialized scientific software. Agents must reconstruct a key experiment, run and diagnose it, and deliver a runnable workflow. Submissions are replayed from scratch in a clean container with network disabled, and only regenerated evidence is scored against expert criteria. Across 10 frontier agent configurations (two using Codex, the rest Claude Code), the best fully reproduced only 14% of tasks at the 90% threshold; mean rubric scores ranged from 27.53% to 59.08%. Task construction took more than 1,000 hours of doctoral-level expert work.
Why it matters
Independent evidence for research leaders and research computing groups that coding agents doing domain-science work often produce partial or plausible-looking results that do not reproduce. It supports requiring clean-replay checks and expert review before agent-generated analyses enter publications or grant reports.
Evidence and measured results
Read the abstract, introduction, Figure 1 caption, task construction and evaluation protocol. Preprint, not peer reviewed. Per-agent tables were skimmed; appendices not read in full. Model versions and costs per run were not checked.
Limitations and uncertainty
Preprint by the benchmark authors, who also designed the scoring. Tasks are limited to 12 physical-science and engineering areas with available software. Results depend on agent harness and model versions at test time and may change quickly.
Put this evidence to work
Lighthouse Advisory interpretation, grounded in this source. Enriched 2026-10-31; this does not change the original publication date. Each role follows the same checklist so takeaways compare across sources.
Sales
Role takeaway
- Customer problem
- Researchers adopting coding agents get plausible results that may not reproduce, risking errors in publications and grant reports.
- Stakeholders
- Vice president for research
- Research integrity officer
- Research computing director
- Principal investigators using agents
- Discovery questions
- Which labs use coding agents for analysis today?
- How are agent-produced results checked before publication?
- Can your clusters replay a workflow from scratch?
- Value hypothesis
- Replay and review checks may catch non-reproducible agent work before it reaches publications, at the cost of added staff time.
- Potential engagement
- Design of a clean-replay and expert-review process for agent-produced analyses in one department.
- Unsupported claims
- Agents are unusable for research
- Results generalize to all fields
- Specific agent rankings stay stable
Pre-sales engineering
Role takeaway
- Fit
- Fits research computing groups with container support that serve labs using coding agents.
- Architecture and integration
- Container-based clean replay with network disabled, resource limits and provenance capture on shared HPC.
- Prerequisites
- Containerized domain software
- Valid software licenses in containers
- Provenance storage
- Constraints
- Commercial software licensing
- Preprint evidence only
- Fast-changing agent versions
- Security
- Run replays with network disabled and isolate evaluation data from agent access.
- Proof of value
- Replay a sample of agent workflows from scratch
- Compare regenerated evidence with original claims
- Track full versus partial reproduction rates
Delivery
Role takeaway
- Work
- Build container templates, a replay job, a review checklist and training for labs using agents.
- Dependencies
- Research computing container platform
- Domain software licenses
- Faculty reviewers
- Ownership
- Research computing with principal investigators signing off on results.
- Skills and adoption
- Lab members learn to package agent work reproducibly and read replay reports.
- Governance checkpoints
- Provenance captured for every agent run
- Expert review before publication
- Periodic audit of replay outcomes
- Acceptance criteria
- Target share of agent workflows pass clean replay (proposal)
- Replay turnaround within an agreed time
- Review checklist adopted by pilot labs
- Risks
- Reviewer time not funded
- Licensing blocks containerization
- False confidence from partial scores
Implementation considerations
Lighthouse Advisory interpretation across the operating dimensions a public-sector buyer must settle before this evidence becomes a design. Each note answers the question under its heading for this specific source.
Architecture and integration
What must connect, and where does the AI sit in the workflow?
Clean replay in a fresh container with no network and fixed resource limits is a practical verification pattern for agent-produced workflows on shared research computing.
Governance
Who approves, reviews and stays accountable for outcomes?
Rubric scores far above full-reproduction rates show that partial credit can overstate agent reliability; research policies should distinguish the two.
Security and privacy
What data, permissions and controls need testing?
Network-disabled replay limits data leakage during evaluation; not otherwise addressed.
Accessibility and workforce
Who is affected, and what skills or accommodations follow?
Expert time to build and judge tasks (about ten hours per task) shows verification itself needs skilled staff.
Procurement
What should contracts, pricing and exit terms secure?
Commercial coding-agent products (Codex, Claude Code) were the harnesses tested; licensing of domain software inside containers is a practical constraint.
Operating model
Which teams own the service once it runs?
Suggests agent use in research should include replay and expert sign-off steps, which add staff time.
Publication history
- 2026-10-31Research Universities · Issue 533 resources
Stable resource ID: paperbenchx-ai-agents-reproduce-science-14-percent-20261029