From the Research Universities edition of October 2, 2026
EurekaBench: agents near human predictive accuracy but far behind on scientific insights across 26 expert tasks
Carnegie Mellon University (LTI), Stanford, Yale, Princeton, MIT, Columbia, Flatiron Institute, Engram — arXiv preprint (cs.AI) · Higher education AI-for-science / autonomous research agent evaluation · United States (multi-institution authors); methods transferable to campus AI-scientist evaluation programs
- Publisher
- arXiv 2610.00492
- Original publication
- September 30, 2026 (arXiv 2610.00492v1)
- Source retrieved
- 2026-10-03
What happened
EurekaBench (arXiv 2610.00492, submitted 30 Sep 2026) evaluates whether AI agents can discover mechanisms that yield scientific insights—not just fit observations. It contains 26 expert-verified long-horizon tasks across neuroscience, geophysics, plasma physics, astrophysics, computer science, and chemistry, with 306 insight questions. Seven frontier agents run under a 1 H100 / 1,000-iteration / 4-hour budget. Best conditioned score is Claude Fable 5.1 at 29.8% versus 63.7% for human-discovered mechanisms. Predictive accuracy for Fable 5.1 and GPT 6 Astra reaches 47.4% versus human 48.8%, but scientific-insight scores are only 42.4% and 29.4% versus human 69.7%. Except GPT 6 Astra (10/26), agents fail scientific-constraint tests on more than half of problems. Authors conclude agents over-fixate on predictive optimization and under-explore uncertainty.
Why it matters
Campus research computing and research administration should not equate leaderboard predictive accuracy or paper-fluency demos with scientific insight discovery; require insight/constraint metrics before expanding autonomous research agent pilots.
Evidence and measured results
arXiv abs HTML and PDF retrieved via HTTPS curl (User-Agent SLED-research-bot/1.0); Table 2 conditioned/PA/SI scores confirmed in pdftotext; check_sources showed not archived.
Limitations and uncertainty
Preprint; agentic judges for SI/SC; possible pretraining exposure to source research; not a multi-campus wet-lab deployment study.
Put this evidence to work
Lighthouse Advisory interpretation, grounded in this source. Enriched 2026-10-02; this does not change the original publication date. Each role follows the same checklist so takeaways compare across sources.
Sales
Role takeaway
- Customer problem
- Vendors sell AI scientists on predictive or paper metrics that do not show insight discovery under expert rubrics.
- Stakeholders
- VPR / research computing
- Domain science PIs
- Research integrity offices
- AI platform owners
- Discovery questions
- Which metrics currently gate agent autonomy?
- Do we score scientific constraints and insights separately from prediction?
- What SI/constraint floor would block unsupervised runs?
- Value hypothesis
- Insight- and constraint-gated evaluation can prevent over-buying PA-optimized agents that fail scientific common-sense tests.
- Potential engagement
- Six-week supervised EurekaBench-style pilot on 3–5 campus-relevant tasks with PA/SI/SC scorecards.
- Unsupported claims
- Human-level autonomous discovery
- PA gains imply insight gains
- Ready for unsupervised publication
Pre-sales engineering
Role takeaway
- Fit
- Cross-domain long-horizon mechanism-discovery benchmark with SC/PA/SI scoring and Apptainer environments.
- Architecture and integration
- Add task specs, CLI environments, deliverable contracts (mechanism.md/py, experiment.log), and dual agentic judges for constraints and insights.
- Prerequisites
- GPU sandbox
- Domain expert rubric owners
- Environment images
- Judge model budgets
- Constraints
- Preprint
- Agentic SI judges
- 26 tasks not campus-specific
- Possible pretraining leakage
- Security
- Block source-paper retrieval during evaluation; isolate simulator credentials and keep judge write access off the mechanism under test.
- Proof of value
- Conditioned score vs human baseline
- SI score gap vs PA
- SC failure count
- Expert spot-check of judge Yes verdicts
Delivery
Role takeaway
- Work
- Select campus-relevant EurekaBench tasks or analogs, freeze rubrics with PIs, run matched agent harnesses, review SI/SC failures weekly.
- Dependencies
- Compute allocation
- Domain experts
- Blocked-web eval network
- Scorecard dashboard
- Ownership
- Research computing owns harness; domain PIs own insight rubrics and scientific acceptance.
- Skills and adoption
- Train RSEs on three-axis scoring; train PIs to reject PA-only agent reports.
- Governance checkpoints
- Rubric approval
- Mid-pilot SC failure review
- Go/no-go on autonomy expansion
- Acceptance criteria
- Documented SI and SC rates
- No SC pass without human review for pilot
- PA not used alone for go-live
- Risks
- Judge over-credit
- Task overfit
- Scope creep to unsupervised discovery
Implementation considerations
Lighthouse Advisory interpretation across the operating dimensions a public-sector buyer must settle before this evidence becomes a design. Each note answers the question under its heading for this specific source.
Architecture and integration
What must connect, and where does the AI sit in the workflow?
Evaluation harnesses for research agents need separate scientific-constraint, predictive-accuracy, and insight axes with environment access for judges—not a single optimization score.
Governance
Who approves, reviews and stays accountable for outcomes?
Gate autonomy claims on insight and constraint pass rates; treat PA-near-human results as insufficient for unsupervised discovery authority.
Security and privacy
What data, permissions and controls need testing?
Long-horizon experiment loops with simulators still need credential isolation and blocked lookup of ground-truth papers during evaluation; production use needs stronger data-access controls.
Accessibility and workforce
Who is affected, and what skills or accommodations follow?
Insight-focused evaluation creates demand for domain experts and RSEs who can write rubrics and interpret experiment logs—not only ML engineers tuning PA.
Procurement
What should contracts, pricing and exit terms secure?
Require separate SI/constraint reporting in AI-scientist RFPs; refuse PA-only demos as proof of scientific discovery capability.
Operating model
Which teams own the service once it runs?
Operate discovery agents under supervised insight gates; treat PA-near-human results as research-progress signals, not autonomy authorization.
Publication history
- 2026-10-02Research Universities · Issue 253 resources
Stable resource ID: eurekabench-scientific-insight-discovery-20260930