SciAgentArena shows task-dependent gains; evaluator disclosures remain incomplete

Discuss unreliable analysis handoffs with biomedical investigators, research software engineers and computing directors. Ask which tasks have defensible reference outputs, who checks invalid premises, and how much correction work current assistants create. Offer a bounded benchmark adaptation for one approved workflow, with explicit success and failure cases. The value hypothesis is a better purchasing decision based on usable outputs and review effort. This paper cannot support clinical deployment claims, a universal agent ranking or a guaranteed productivity improvement. Disclose the unfinished conflict statement and seek clarification before using comparative results in a customer presentation.

Build a versioned evaluation package containing input hashes, agent settings, package locks, tool permissions and expected artifacts. Keep the scorer outside the agent's writable environment and include unsupported-task cases that should stop. Use approved synthetic data first, then obtain authorization for any restricted dataset. Compare candidate systems under a declared resource budget and record failed executions as well as successful ones. Validate the scorer with expert review of a sample because string agreement can differ from scientific correctness. The study is a test-design input; local performance and contemporary configurations still require measurement.

Assign a disciplinary investigator and research software engineer to define the task, with an independent reviewer approving the rubric. Dependencies include permitted data, stable environments and capacity to investigate mismatches. Train users to distinguish a completed script from an accepted scientific result. Gate wider use on reviewed failure cases and a reproducible evidence package. Proposed acceptance criteria are complete artifact retention, documented expert disposition of every pilot error, and comparison of total researcher effort against the existing workflow. Maintain a regression set after model or package changes. Risks include scorer shortcuts, configuration drift and insufficient review capacity.