{"resourceId":"eurekabench-scientific-insight-discovery-20260930","versions":[{"version":"external-d75f29e31e507350f0a263bae857d39e0fc4320558141c5b596528935418393d","resource":{"id":"eurekabench-scientific-insight-discovery-20260930","title":"EurekaBench: agents near human predictive accuracy but far behind on scientific insights across 26 expert tasks","organization":"Carnegie Mellon University (LTI), Stanford, Yale, Princeton, MIT, Columbia, Flatiron Institute, Engram — arXiv preprint (cs.AI)","sector":"Higher education AI-for-science / autonomous research agent evaluation","geography":"United States (multi-institution authors); methods transferable to campus AI-scientist evaluation programs","publishedAt":"September 30, 2026 (arXiv 2610.00492v1)","publicationDate":"2026-09-30","eventDate":null,"sourceName":"arXiv 2610.00492","sourceLabel":"Academic preprint (arXiv)","sourceUrl":"https://arxiv.org/abs/2610.00492","evidenceClass":"academic-research","outcomeClass":"cautionary","topics":["developers-agents","operating-model","governance-procurement","knowledge-work"],"finding":"EurekaBench (arXiv 2610.00492, submitted 30 Sep 2026) evaluates whether AI agents can discover mechanisms that yield scientific insights—not just fit observations. It contains 26 expert-verified long-horizon tasks across neuroscience, geophysics, plasma physics, astrophysics, computer science, and chemistry, with 306 insight questions. Seven frontier agents run under a 1 H100 / 1,000-iteration / 4-hour budget. Best conditioned score is Claude Fable 5.1 at 29.8% versus 63.7% for human-discovered mechanisms. Predictive accuracy for Fable 5.1 and GPT 6 Astra reaches 47.4% versus human 48.8%, but scientific-insight scores are only 42.4% and 29.4% versus human 69.7%. Except GPT 6 Astra (10/26), agents fail scientific-constraint tests on more than half of problems. Authors conclude agents over-fixate on predictive optimization and under-explore uncertainty.","sledRelevance":"Campus research computing and research administration should not equate leaderboard predictive accuracy or paper-fluency demos with scientific insight discovery; require insight/constraint metrics before expanding autonomous research agent pilots.","evidence":"arXiv abs HTML and PDF retrieved via HTTPS curl (User-Agent SLED-research-bot/1.0); Table 2 conditioned/PA/SI scores confirmed in pdftotext; check_sources showed not archived.","architectureImplications":"Evaluation harnesses for research agents need separate scientific-constraint, predictive-accuracy, and insight axes with environment access for judges—not a single optimization score.","governanceImplications":"Gate autonomy claims on insight and constraint pass rates; treat PA-near-human results as insufficient for unsupervised discovery authority.","securityPrivacyImplications":"Long-horizon experiment loops with simulators still need credential isolation and blocked lookup of ground-truth papers during evaluation; production use needs stronger data-access controls.","caveats":"Preprint; agentic judges for SI/SC; possible pretraining exposure to source research; not a multi-campus wet-lab deployment study.","streamIds":["research"],"roles":{"sales":"Sales framing: predictive accuracy on scientific agent demos is not the same as discovering usable insights. EurekaBench (CMU-led; arXiv 2610.00492, 30 Sep 2026) scores 26 expert tasks on constraints, prediction, and insight. Top agents nearly match humans on predictive accuracy (47.4% vs 48.8%) but trail on scientific insights (best 42.4% vs human 69.7%), with conditioned scores under 30% versus human 63.7%. Pitch supervised insight-gated pilots—not unconstrained AI-scientist rollout.","engineering":"Engineering framing: instrument research-agent sandboxes with three-axis scoring (scientific constraints, predictive accuracy, insight questions) and environment-backed judges. Expect agents to optimize PA while leaving uncertainty untested. Prefer harnesses that force mechanism.md + executable mechanism.py + experiment logs over chat-only outputs. Do not ship a single composite score that hides SI failures.","delivery":"Delivery framing: run a campus EurekaBench-style bake-off on a small domain subset before any unattended discovery loop. Dependencies include domain experts for insight rubrics, GPU budgets, and blocked ground-truth web access during eval. Acceptance should require SI and constraint rates—not PA alone. Risks include judge generosity, pretraining leakage, and mistaking optimization for mechanism discovery."},"roleDetails":{"sales":{"customerProblem":"Vendors sell AI scientists on predictive or paper metrics that do not show insight discovery under expert rubrics.","stakeholders":["VPR / research computing","Domain science PIs","Research integrity offices","AI platform owners"],"discoveryQuestions":["Which metrics currently gate agent autonomy?","Do we score scientific constraints and insights separately from prediction?","What SI/constraint floor would block unsupervised runs?"],"valueHypothesis":"Insight- and constraint-gated evaluation can prevent over-buying PA-optimized agents that fail scientific common-sense tests.","potentialEngagement":"Six-week supervised EurekaBench-style pilot on 3–5 campus-relevant tasks with PA/SI/SC scorecards.","unsupportedClaims":["Human-level autonomous discovery","PA gains imply insight gains","Ready for unsupervised publication"]},"engineering":{"fit":"Cross-domain long-horizon mechanism-discovery benchmark with SC/PA/SI scoring and Apptainer environments.","architectureIntegration":"Add task specs, CLI environments, deliverable contracts (mechanism.md/py, experiment.log), and dual agentic judges for constraints and insights.","prerequisites":["GPU sandbox","Domain expert rubric owners","Environment images","Judge model budgets"],"constraints":["Preprint","Agentic SI judges","26 tasks not campus-specific","Possible pretraining leakage"],"security":"Block source-paper retrieval during evaluation; isolate simulator credentials and keep judge write access off the mechanism under test.","proofOfValue":["Conditioned score vs human baseline","SI score gap vs PA","SC failure count","Expert spot-check of judge Yes verdicts"]},"delivery":{"work":"Select campus-relevant EurekaBench tasks or analogs, freeze rubrics with PIs, run matched agent harnesses, review SI/SC failures weekly.","dependencies":["Compute allocation","Domain experts","Blocked-web eval network","Scorecard dashboard"],"ownership":"Research computing owns harness; domain PIs own insight rubrics and scientific acceptance.","skillsAdoption":"Train RSEs on three-axis scoring; train PIs to reject PA-only agent reports.","governanceCheckpoints":["Rubric approval","Mid-pilot SC failure review","Go/no-go on autonomy expansion"],"acceptanceCriteria":["Documented SI and SC rates","No SC pass without human review for pilot","PA not used alone for go-live"],"risks":["Judge over-credit","Task overfit","Scope creep to unsupervised discovery"]}},"retrievedAt":"2026-10-02T22:05:20-05:00","enrichedAt":"2026-10-02T22:12:00-05:00","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Insight-focused evaluation creates demand for domain experts and RSEs who can write rubrics and interpret experiment logs—not only ML engineers tuning PA.","procurementImplications":"Require separate SI/constraint reporting in AI-scientist RFPs; refuse PA-only demos as proof of scientific discovery capability.","operatingModelImplications":"Operate discovery agents under supervised insight gates; treat PA-near-human results as research-progress signals, not autonomy authorization.","sourceVerification":{"openedUrl":"https://arxiv.org/abs/2610.00492","referenceExcerpt":"falling substantially short in deriving scientific insights","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}