From the Research Universities edition of September 16, 2026
Faraday improves figure-replication win-rates vs Claude/Codex; authors still call replication early-stage
Inherent (Faraday / Replica preprint) · AI-assisted scientific research evaluation · International lab preprint (methods transferable; not a campus deployment)
- Publisher
- arXiv
- Original publication
- August 13, 2026, arXiv abs/html 2608.13331
- Source retrieved
- 2026-09-17
What happened
Post-training a 27B research agent improves rubric win-rates on figure replication versus strong coding agents, yet authors warn failures persist and humans must keep inspecting results.
Why it matters
Newly relevant alongside AutoResearchEval after recent fidelity coverage: shows training can raise replication scores without authorizing unattended scientific acceptance on campuses.
Evidence and measured results
Replica: 310 figure-replication tasks from 100 ML/AI-for-science papers. Faraday (Qwen3.6-27B + coding-agent tools; turn-level GRPO) outperforms Claude Opus 4.8 and GPT-5.5 on 73% of in-distribution ML tasks and 60% of held-out AI-for-science tasks; +6% vs Claude and +8% vs Codex average on test; human preference 29/41 rollouts. Limitations section: early-stage; Faraday failed some rigorous originals; do not treat agent non-replication as indictment of originals; humans must retain skills and inspect outputs.
Limitations and uncertainty
Lab/vendor preprint; auto-generated rubric judge; figure-replication scope narrower than full papers or discovery; compute environment may not match campus clusters. Covered for mixed training gains vs process-failure scrutiny.
Put this evidence to work
Lighthouse Advisory interpretation, grounded in this source. Enriched 2026-09-17; this does not change the original publication date. Each role follows the same checklist so takeaways compare across sources.
Sales
Role takeaway
- Customer problem
- Teams need evidence whether trained research agents actually improve replication quality versus strong general coding agents.
- Stakeholders
- Research computing leads
- Research integrity officers
- Principal investigators
- Research software engineers
- Discovery questions
- What replication tasks matter locally?
- What win-rate would change policy?
- Who inspects agent failures?
- How are rubric judges calibrated?
- Value hypothesis
- Relative win-rate pilots under fixed rubrics can inform whether specialized training is worth adopting—without replacing human inspection.
- Potential engagement
- Bounded figure-replication pilot on 10–20 permitted tasks with human adjudication.
- Unsupported claims
- 73%/60% wins equal autonomous science
- Agent failure proves original papers are wrong
- Elimination of human replication skill
Pre-sales engineering
Role takeaway
- Fit
- Trained research agents using coding-agent tools for figure/paper replication evaluation.
- Architecture and integration
- Sandbox with mediated tools; rubric judge; trace store; compare against strong baseline agents on the same harness.
- Prerequisites
- Pinned containers
- Rubric samples
- GPU budget for multi-rollout eval
- Constraints
- Figure-replication scope
- Judge noise
- Environment mismatch with campus clusters
- Security
- Mediate coding-agent credentials; keep paper corpora access-controlled; log tool calls.
- Proof of value
- Local win-rate vs Claude-class baseline
- Human agreement study
- Failure case review where originals are trusted
Delivery
Role takeaway
- Work
- Select tasks, run shadowed comparisons, and document human dispositions on successes and failures.
- Dependencies
- Compute quota
- Domain expert reviewers
- Policy on agent outputs in manuscripts
- Ownership
- RSE maintains harness; PI interprets science; integrity sets inspection policy.
- Skills and adoption
- Train reviewers to inspect agent traces and avoid deskilling human replication.
- Governance checkpoints
- Pre-register rubrics
- Mandatory human inspection
- Prohibit automatic misconduct inferences from agent failure
- Acceptance criteria
- Traces retained
- Win-rates and human ratings both reported
- Failure dispositions documented
- Risks
- Over-trusting relative wins
- Judge gaming
- Misreading non-replication as fraud
Implementation considerations
Lighthouse Advisory interpretation across the operating dimensions a public-sector buyer must settle before this evidence becomes a design. Each note answers the question under its heading for this specific source.
Architecture and integration
What must connect, and where does the AI sit in the workflow?
If adopting trained replication agents, keep coding-agent tools mediated, retain rubrics/traces, and separate win-rate dashboards from manuscript release gates.
Governance
Who approves, reviews and stays accountable for outcomes?
Relative win-rates are not acceptance criteria; require human inspection and prohibit treating agent failure as automatic fraud findings on originals.
Security and privacy
What data, permissions and controls need testing?
Confine paper PDFs and proprietary code to approved sandboxes; control tool credentials used by coding-agent helpers.
Accessibility and workforce
Who is affected, and what skills or accommodations follow?
Training gains do not remove the need to cultivate human replication skills and inspection capacity.
Procurement
What should contracts, pricing and exit terms secure?
Evaluate trained agents with contractual trace export and local pilots—not vendor win-rate claims alone.
Operating model
Which teams own the service once it runs?
Keep evaluation harness ownership separate from scientific acceptance and integrity determinations.
Publication history
- 2026-09-16Research Universities · Issue 114 resources
Stable resource ID: faraday-replica-replication-training-2026