{"resourceId":"autoscibench-generated-scientific-agent-benchmarks-20261004","versions":[{"version":"external-f6ea084be1b56054480b67e8c570090f022d251ab4cbb3e43d51f4361a4ccc9f","resource":{"id":"autoscibench-generated-scientific-agent-benchmarks-20261004","title":"AutoSciBench (Genentech/KAIST): agent-generated scientific benchmarks cut solver accuracy by 22.4 and 25.5 points versus human-curated sets in computational biology and materials science, with quality rated by LLM judges","organization":"Genentech and KAIST (Kim, Lee, Hajiramezanali and colleagues)","sector":"Industry–university research / evaluation of AI agents for science","geography":"United States and South Korea","publishedAt":"arXiv:2610.05140v1, submitted 4 October 2026","publicationDate":"2026-10-04","eventDate":null,"sourceName":"arXiv (cs.AI)","sourceLabel":"Academic preprint (not peer-reviewed)","sourceUrl":"https://arxiv.org/abs/2610.05140","evidenceClass":"academic-research","outcomeClass":"emerging","topics":["developers-agents","knowledge-work"],"finding":"AutoSciBench builds scientific-agent tasks from a high-level concept and a low-level recipe, then revises them for up to 10 rounds using solver trajectories and judge feedback to close shortcuts. It generated 50 tasks per domain in computational biology, materials science and clinical imaging, using Claude Opus 4.8 (Claude Code CLI) and GPT-5.6 Sol (Codex CLI) as generators that also served as solvers during refinement; solvers ran three trials graded by exact match. Generated benchmarks lowered average solver accuracy by 22.4 and 25.5 points versus the human-curated seed benchmarks; the clinical imaging human baseline was omitted because it uses LLM grading. In materials science, experience-guided generation cut solver success from 66.7% to 35.0% and raised hard tasks from 15.0% to 50.0%. Newer models scored much higher on generated sets (Opus 5 +19.5 points, GPT-6 Astra +15.1 points).","sledRelevance":"University labs and research computing groups choosing AI agents rely on benchmarks that saturate quickly. This work suggests evaluation sets can be regenerated from local data and literature, but its quality claims rest on AI judges, so campuses would still need expert review.","evidence":"Opened the arXiv abstract page and full HTML (v1) via HTTPS (curl, text extracted); read the method, experimental setup, results tables, quality assessment, ablations and appendix implementation details. Results are the authors' own; no independent replication.","architectureImplications":"Benchmark generation runs agent CLIs with one-hour stage timeouts, three solver trials per task and exact-match graders; campuses could run similar loops on local datasets inside research computing sandboxes.","governanceImplications":"Interpretation: institutions should not accept AI-judged benchmark quality as sufficient; require domain-expert review of generated tasks before using them for tool selection.","securityPrivacyImplications":"Clinical imaging tasks reused benchmark images or retrieved real medical images from the web; campus versions using local clinical data need privacy review.","caveats":"Preprint; 50 tasks per domain; final benchmarks select the hardest task per refinement run, which biases accuracy downward; quality ratings come from two LLM judges from the same model families as the generators, and no human expert audit of generated tasks was found in the sections read; newer models already gain 15–20 points, so generated sets may also saturate.","streamIds":["research"],"roles":{"sales":"Sales framing: research leaders choosing AI agents for science struggle because public benchmarks saturate and may not match local work. AutoSciBench shows agents can generate harder tasks from existing benchmarks, cutting solver accuracy by about 22–26 points. A bounded offer is a pilot that generates a small local evaluation set, reviewed by faculty. Do not claim generated tasks are expert-quality; the paper's quality scores come from AI judges.","engineering":"Engineering framing: treat benchmark generation as a pipeline with planning, task construction, multiple solver trials, exact-match grading and refinement. Pin generator and solver versions, keep reference answers away from solvers, and log trajectories. Because generators also solved during refinement and judges share model families, add an independent solver and a human review step before relying on results. Clinical imaging tasks need privacy review when they draw on real images.","delivery":"Delivery framing: use generated tasks to refresh a lab's agent evaluation, with faculty reviewing a sample for correctness and scientific value. Research computing owns the pipeline; PIs own task acceptance. Re-run on each model upgrade, since newer models closed much of the gap within the authors' own tests. Start with one lab and one domain where curated data and willing reviewers already exist. Interpretation: the criteria below are proposals, not observed results."},"roleDetails":{"sales":{"customerProblem":"Labs selecting AI agents for scientific analysis rely on public benchmarks that saturate quickly and may not reflect their own data.","stakeholders":["Research computing director","Computational biology and materials PIs","Core facility leads","Research IT security"],"discoveryQuestions":["Which benchmarks do you use today to choose AI agents for analysis?","Do you have curated local datasets with known answers?","Who could review a sample of generated tasks for correctness?"],"valueHypothesis":"Locally generated, expert-reviewed tasks may give a more discriminating comparison of agents than saturated public benchmarks; no deployment benefit is reported.","potentialEngagement":"Six-week pilot generating 20–30 tasks from one lab's datasets, with faculty review and a comparison of two or three agents.","unsupportedClaims":["Generated tasks are expert-quality","Results transfer to all scientific domains","Generated benchmarks will stay unsaturated"]},"engineering":{"fit":"Labs and research computing groups evaluating coding agents for data analysis in biology, materials or imaging.","architectureIntegration":"Agent CLIs (Claude Code, Codex) run generation and solving stages with stage timeouts; exact-match graders; experience memory of refinement histories; local datasets in a sandbox.","prerequisites":["Seed benchmark or curated local dataset","Agent CLI access approved for research data","Sandbox compute","Faculty reviewers"],"constraints":["Same-family LLM judges for quality","Difficulty-biased task selection","Clinical data requires privacy review","Model upgrades shift results quickly"],"security":"Keep reference answers outside solver sandboxes, restrict web retrieval of sensitive images, and review generated data before sharing.","proofOfValue":["Faculty agreement rate on a sample of generated tasks","Accuracy gap between public and generated sets for local agents","Accuracy with an independent solver not used in generation"]},"delivery":{"work":"Run a small generation pilot on one lab's data, have faculty review tasks, and add accepted tasks to the lab's agent evaluation suite.","dependencies":["Curated datasets","Agent API approvals","Faculty review time","Version pinning"],"ownership":"Research computing owns the generation pipeline; PIs own task acceptance and use of results.","skillsAdoption":"Train lab staff to read solver trajectories and spot shortcuts rather than relying on scores.","governanceCheckpoints":["Expert review of generated tasks before use","Privacy review for clinical data","Re-evaluation on each model upgrade"],"acceptanceCriteria":["Proposed: at least 80% of sampled generated tasks judged correct by faculty","Proposed: evaluation suite refreshed after each major model upgrade","Proposed: independent solver included in every comparison"],"risks":["AI-judged quality masking flawed tasks","Rapid saturation by newer models","Leakage of sensitive data into generated tasks"]}},"retrievedAt":"2026-10-07T21:57:02-05:00","enrichedAt":"2026-10-07T22:10:00-05:00","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Benchmark construction shifts from expert authoring to expert review, which still needs domain specialists' time.","procurementImplications":"Agent selection should rest on refreshed, locally relevant tests rather than vendor-reported public benchmark scores.","operatingModelImplications":"Evaluation becomes a recurring service that research computing runs with faculty review rather than a one-time tool selection.","sourceVerification":{"openedUrl":"https://arxiv.org/abs/2610.05140","referenceExcerpt":"Generated benchmarks reduce average solver accuracy by 22.4 and 25.5 percentage points relative to the human-curated benchmarks in computational biology and materials science","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}