Discovery benchmark exposes scrutiny gaps while leaving its own validation incomplete

Discuss evaluation credibility with research leaders, scientific software teams and procurement reviewers. Ask whether an offered benchmark tests a locally relevant research decision, whether its artifacts are inspectable, and who adjudicates disagreement with automated scoring. A bounded evaluation-design engagement could define claim-level acceptance for one research domain. The value hypothesis is better selection evidence and fewer unsupported automation commitments. Do not use the narrow score ordering as a product ranking or infer frontier-model performance. Establish demand for reviewable evidence before proposing a platform replacement.

Create a held-out task with known provenance and a separately managed evaluator. Record agent settings, network permissions, budgets and failed attempts so a second team can rerun the process. Pair automated scoring with blinded domain-expert review and measure disagreement before relying on the judge. Use repeated runs to distinguish system behavior from sampling variation. Validate that tools cannot read evaluation assets or upload restricted data. A proof of value should report sensitivity to evaluator settings and task design, with explicit non-completion handling and no automatic promotion based on one aggregate score.

The research evaluation lead should own a versioned test protocol, with a domain scientist, research software engineer and data steward. Implement task curation, artifact retention and an adjudication queue before onboarding agent users. Dependencies include suitable validation data, permitted reuse and staff trained in evaluation methods. Proposed acceptance criteria are an independently rerunnable test package, documented human-versus-judge disagreement and no unresolved high-impact scoring dispute. Set repeat-run thresholds before testing. Risks include benchmark contamination, hidden execution settings, evaluator drift and mistaking better scores for new scientific knowledge.