{"resourceId":"faraday-replica-replication-training-2026","versions":[{"version":"external-311657e349efee4cacf00e29ee3061d0601e50b9a167d2ae5258c554be940621","resource":{"id":"faraday-replica-replication-training-2026","title":"Faraday improves figure-replication win-rates vs Claude/Codex; authors still call replication early-stage","organization":"Inherent (Faraday / Replica preprint)","sector":"AI-assisted scientific research evaluation","geography":"International lab preprint (methods transferable; not a campus deployment)","publishedAt":"August 13, 2026, arXiv abs/html 2608.13331","publicationDate":"2026-08-13","eventDate":null,"sourceName":"arXiv","sourceLabel":"Author preprint; abs, html, and PDF text inspected","sourceUrl":"https://arxiv.org/html/2608.13331","evidenceClass":"academic-research","outcomeClass":"mixed","topics":["developers-agents","knowledge-work","governance-procurement","operating-model"],"finding":"Post-training a 27B research agent improves rubric win-rates on figure replication versus strong coding agents, yet authors warn failures persist and humans must keep inspecting results.","sledRelevance":"Newly relevant alongside AutoResearchEval after recent fidelity coverage: shows training can raise replication scores without authorizing unattended scientific acceptance on campuses.","evidence":"Replica: 310 figure-replication tasks from 100 ML/AI-for-science papers. Faraday (Qwen3.6-27B + coding-agent tools; turn-level GRPO) outperforms Claude Opus 4.8 and GPT-5.5 on 73% of in-distribution ML tasks and 60% of held-out AI-for-science tasks; +6% vs Claude and +8% vs Codex average on test; human preference 29/41 rollouts. Limitations section: early-stage; Faraday failed some rigorous originals; do not treat agent non-replication as indictment of originals; humans must retain skills and inspect outputs.","architectureImplications":"Interpretation: if adopting trained replication agents, keep coding-agent tools mediated, retain rubrics/traces, and separate win-rate dashboards from manuscript release gates.","governanceImplications":"Interpretation: relative win-rates are not acceptance criteria; require human inspection and prohibit treating agent failure as automatic fraud findings on originals.","securityPrivacyImplications":"Interpretation: confine paper PDFs and proprietary code to approved sandboxes; control tool credentials used by coding-agent helpers.","caveats":"Lab/vendor preprint; auto-generated rubric judge; figure-replication scope narrower than full papers or discovery; compute environment may not match campus clusters. Covered for mixed training gains vs process-failure scrutiny.","streamIds":["research"],"roles":{"sales":"Interpretation: Engage research computing and integrity leaders comparing trained replication agents to frontier coding agents. Ask what win-rate would change policy and who inspects failures. Offer a bounded replication pilot on permitted figures with human adjudication. Value hypothesis is clearer relative performance under a fixed rubric. Do not sell 73%/60% wins as autonomous science. Unsupported: guaranteed replication of campus papers, reduced review staffing, or fraud detection against originals.","engineering":"Interpretation: Fit is sandboxed replication agents with coding tools and rubric judges. Integrate turn-level training or eval harnesses only where traces export cleanly. Prerequisites: pinned environments and human rubric samples. Constraints: figure-level scope and judge calibration. Security: mediate tools and prevent data leakage. Proof: local win-rates vs Claude-class baselines plus human agreement on a held-out set.","delivery":"Interpretation: RSE owns harnesses; PIs accept scientific meaning; integrity sets inspection rules. Work: select figures, run Faraday-like agents shadowed, and review failures. Dependencies: GPU quota and domain experts. Train staff not to equate agent non-replication with original misconduct. Checkpoints: pre-register rubrics; mandatory human inspection. Acceptance: traces retained and human disposition recorded. Risks: over-trusting win-rates and deskilling human replication."},"roleDetails":{"sales":{"customerProblem":"Teams need evidence whether trained research agents actually improve replication quality versus strong general coding agents.","stakeholders":["Research computing leads","Research integrity officers","Principal investigators","Research software engineers"],"discoveryQuestions":["What replication tasks matter locally?","What win-rate would change policy?","Who inspects agent failures?","How are rubric judges calibrated?"],"valueHypothesis":"Relative win-rate pilots under fixed rubrics can inform whether specialized training is worth adopting—without replacing human inspection.","potentialEngagement":"Bounded figure-replication pilot on 10–20 permitted tasks with human adjudication.","unsupportedClaims":["73%/60% wins equal autonomous science","Agent failure proves original papers are wrong","Elimination of human replication skill"]},"engineering":{"fit":"Trained research agents using coding-agent tools for figure/paper replication evaluation.","architectureIntegration":"Sandbox with mediated tools; rubric judge; trace store; compare against strong baseline agents on the same harness.","prerequisites":["Pinned containers","Rubric samples","GPU budget for multi-rollout eval"],"constraints":["Figure-replication scope","Judge noise","Environment mismatch with campus clusters"],"security":"Mediate coding-agent credentials; keep paper corpora access-controlled; log tool calls.","proofOfValue":["Local win-rate vs Claude-class baseline","Human agreement study","Failure case review where originals are trusted"]},"delivery":{"work":"Select tasks, run shadowed comparisons, and document human dispositions on successes and failures.","dependencies":["Compute quota","Domain expert reviewers","Policy on agent outputs in manuscripts"],"ownership":"RSE maintains harness; PI interprets science; integrity sets inspection policy.","skillsAdoption":"Train reviewers to inspect agent traces and avoid deskilling human replication.","governanceCheckpoints":["Pre-register rubrics","Mandatory human inspection","Prohibit automatic misconduct inferences from agent failure"],"acceptanceCriteria":["Traces retained","Win-rates and human ratings both reported","Failure dispositions documented"],"risks":["Over-trusting relative wins","Judge gaming","Misreading non-replication as fraud"]}},"retrievedAt":"2026-09-17T03:03:35Z","enrichedAt":"2026-09-17T03:07:50Z","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Interpretation: training gains do not remove the need to cultivate human replication skills and inspection capacity.","procurementImplications":"Interpretation: evaluate trained agents with contractual trace export and local pilots—not vendor win-rate claims alone.","operatingModelImplications":"Interpretation: keep evaluation harness ownership separate from scientific acceptance and integrity determinations.","sourceVerification":{"openedUrl":"https://arxiv.org/html/2608.13331","referenceExcerpt":"In distribution, Faraday outperforms both Claude and Codex on 73% of tasks.","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}