{"resourceId":"reprobridge-ml-artifact-reproduction-agents-20261105","versions":[{"version":"external-c626d753a38ac80e94a56d83198f425dc15a5f723ab2828d64630558aee2bcfe","resource":{"id":"reprobridge-ml-artifact-reproduction-agents-20261105","title":"ReproBridge: Building Reproducibility into ML Research Artifacts","organization":"University of Waterloo; Google","sector":"University / industry research (ML computational reproducibility with coding agents)","geography":"Canada / United States (international ML research; transferable to research universities releasing code)","publishedAt":"5 November 2026 (arXiv 2611.08452v1)","publicationDate":"2026-11-05","eventDate":null,"sourceName":"arXiv","sourceLabel":"Wang, Y., Meng, R., Nam, J., Pfister, T. & Yoon, J., \"ReproBridge: Building Reproducibility into ML Research Artifacts,\" arXiv:2611.08452, 5 Nov 2026","sourceUrl":"https://arxiv.org/abs/2611.08452","evidenceClass":"academic-research","outcomeClass":"effective","topics":["developers-agents","knowledge-work","operating-model"],"finding":"ReproBridge defines refined reproducible artifacts (refined code, environment setup, experiment launchers, reproduction guide) built by an agent workflow from papers and raw repos. On ReproBridge-Bench—1,029 tasks from 387 experiments across 36 papers with five human-verified reference scores per task—four coding-agent models raise mean reproduction success from 76.4% (raw code) to 92.6% (refined artifacts) and cut downstream reproducer API spending about 19.7% on average. Cross-model success spread narrows from 21.6 to 3.9 percentage points; ablations show removing any component hurts, especially code and environment. An Auditor checks execution evidence because matching numbers alone can hide substituted measurements.","sledRelevance":"Directly relevant to research universities institutionalizing computational reproducibility and evaluating coding agents as research support tools: shows measured gains from preserving recovered procedures, not only releasing raw code.","evidence":"arXiv preprint with large agent evaluation, human-verified references, Auditor integrity checks, and component ablations; not yet peer-reviewed journal publication.","architectureImplications":"Institutions should treat reproducibility as an artifact contract (code+env+launchers+guide) beside the paper, with sandboxed agent runners and hash-pinned releases.","governanceImplications":"Research integrity and library policies can require RRA-like packages for agent-assisted reproduction claims; Auditors should be independent of the reproducing agent.","securityPrivacyImplications":"Agent reproduction executes third-party code—needs sandboxing, secret hygiene, and pinned environments; paper does not detail campus SOC controls.","caveats":"ML-paper benchmark; Auditor detection imperfect; API-cost savings are evaluation-time measurements; preprint.","streamIds":["research"],"roles":{"sales":"Evaluation framing: ReproBridge shows coding agents reproduce ML experiments far more reliably when authors ship refined artifacts (code, environment, launchers, guide) instead of raw repos—mean success 76.4% to 92.6% across four models, with ~19.7% lower reproducer API spend. Use this with VPR, research computing, and library research-data services that are buying agent-assisted reproducibility tooling. Pitch a bounded artifact-standard pilot on a campus paper corpus, not a guarantee that every historical repo becomes reproducible without curator effort.","engineering":"Architecture framing: treat reproduction as a four-part artifact contract—refined codebase, environment setup, experiment launchers, and a reproduction guide—plus an independent Auditor that checks execution evidence, not numbers alone. Integrate with campus Git, container registries, and HPC job launchers so agents inherit known-good environments rather than rediscovering brittle dependency graphs. Prerequisites include human-verified reference scores and compute for re-runs; layout-specific or GPU-heavy repos still constrain transfer across labs. Instrument hash-pinned RRAs so later agents cannot silently swap measurements.","delivery":"Delivery framing: stand up an RRA pipeline that turns accepted ML papers into curated artifacts, then measure agent success and API cost against a raw-code baseline on a fixed task set. Owners are typically the research data librarian plus research computing; PIs supply reference runs. Acceptance should require Auditor-checked executions and ablations showing each component still matters—do not declare victory from matching table numbers alone."},"roleDetails":{"sales":{"customerProblem":"Campus teams adopting coding agents for paper reproduction still burn API budget reconstructing missing procedures from raw repos.","stakeholders":["Vice President for Research","Research computing director","Library research-data services","ML faculty PIs"],"discoveryQuestions":["What share of recent campus ML papers ship runnable environments today?","Do you score agent reproduction with an independent execution auditor?","What API/compute budget is spent re-deriving launch procedures?","Who owns the refined-artifact standard after acceptance?"],"valueHypothesis":"Standardizing refined reproducible artifacts may lift agent reproduction success and cut redo API spend, as measured on ReproBridge-Bench.","potentialEngagement":"A 36-paper-style pilot that converts a campus ML subset into RRAs and benchmarks agent success versus raw repos.","unsupportedClaims":["Guaranteed reproducibility of all historical campus repos","Elimination of human curators","Transfer of ReproBridge percentages to every discipline without local evaluation"]},"engineering":{"fit":"Strong for ML/CS groups releasing code-heavy papers; weaker for wet-lab-only or closed data studies.","architectureIntegration":"Paper + raw repo ingest → ReproBridge-style agent → RRA (code, env, launchers, guide) stored beside DOI; Auditor service validates execution evidence.","prerequisites":["Author code release","Container or env lockfiles","Reference score observations","GPU/CPU quota for re-runs"],"constraints":["Benchmark is ML-paper centric","Auditor models can miss some invalidations","Fresh sessions still need compute"],"security":"Treat Auditor and reproducer agents as untrusted code runners; sandbox executions and pin artifact hashes.","proofOfValue":["Raw vs RRA success on a fixed campus task set","API spend delta","Ablation removing each RRA component"]},"delivery":{"work":"Define RRA schema; Convert pilot papers; Run agent evaluations; Publish Auditor reports","dependencies":["PI cooperation","Research computing queues","Library metadata/DOI hooks"],"ownership":"Research data services with research computing co-ownership.","skillsAdoption":"Staff shift from ad-hoc help tickets toward artifact curation and audit review.","governanceCheckpoints":["No public RRA without Auditor pass","Hash-pinned artifact versions","Periodic re-eval after model upgrades"],"acceptanceCriteria":["Proposed: mean agent success ≥ agreed threshold vs raw baseline","Proposed: documented API-cost reduction","Proposed: ablation report retained"],"risks":["Curator bottleneck","False confidence from number-matching without Auditor","Model-specific gains that shrink on newer agents"]}},"retrievedAt":"2026-11-06T21:58:50-06:00","enrichedAt":"2026-11-06T22:03:30-06:00","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"May reduce repeated reconstruction labor for students/staff reproducing papers, while shifting effort toward artifact curation skills.","procurementImplications":"When buying reproducibility or coding-agent platforms, require raw-vs-refined benchmarks and Auditor evidence—not demo notebooks alone.","operatingModelImplications":"Supports a shared-service model where research data services publish RRAs and research computing hosts audited re-runs.","peerReview":"preprint","sourceVerification":{"openedUrl":"https://arxiv.org/abs/2611.08452","referenceExcerpt":"refined artifacts raise mean reproduction success from 76.4% to 92.6% and reduce downstream API spending by 19.7% on average.","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}