Coding-agent reproduction gains coexist with bias from expected answers

Engage research integrity staff, librarians, principal investigators and research software engineers around the effort required to rerun deposited analyses. Ask whether packages contain usable data, which languages dominate and how discrepancies are currently reviewed. A bounded evaluation could compare assisted and manual reproduction on approved local packages, including deliberately incomplete cases. The value hypothesis is lower verification effort with preserved honesty about missing evidence. Do not sell the benchmark as proof of autonomous discovery or promise the reported accuracy for other models. Keep model selection contingent on local tests and include expert review effort in any proposed benefit calculation.

Implement a reproducibility harness with immutable input packages, restricted working directories and captured commands, environment changes and outputs. Establish verified reference cases before evaluating assistance. Test manuscript access as a separate condition, since answer context changes what successful reproduction means. Include unavailable-data controls and inspect whether reported numbers actually originate in executed artifacts. Use matched prompts and budgets for comparisons, documenting any adaptation. Review dependency installation and outbound connectivity against institutional data rules. Proposed validation should score correct results, truthful non-completion, unauthorized analytical changes and reviewer effort separately. Good aggregate accuracy must not hide fabricated completion on impossible cases.

Research integrity leadership should own the decision to accept a reproduction, with software staff maintaining the harness and domain researchers adjudicating discrepancies. Dependencies include permitted replication materials, reviewers fluent in the relevant methods and a versioned environment. Train users to request observed results and preserve disagreements instead of demanding confirmation. Gate rollout on a reviewed pilot, then rerun controls after model or prompt changes. Proposed acceptance criteria include traceable execution for every accepted result, explicit failure on all designated missing-data controls and no unreviewed specification changes. Track correction time and accessibility of logs. Risks include result copying and accidental exposure of restricted data.