{"resourceId":"mechhypobench-ai-scientist-mechanistic-hypotheses-20261004","versions":[{"version":"external-852eb9ab880e2da8f7c741f634b6226072813841cea5015e51041a502264e1ae","resource":{"id":"mechhypobench-ai-scientist-mechanistic-hypotheses-20261004","title":"MechHypoBench: across 300 tasks in 14 fields, best agent reaches 64.9%/50.8% mechanism-consequence agreement vs 87.6%/82.9% for reference mechanisms; AI-scientist workflows do not help","organization":"Computer Network Information Center, Chinese Academy of Sciences, with National University of Singapore and Sichuan University","sector":"Research AI evaluation / AI-assisted hypothesis generation","geography":"China and Singapore authors; benchmark built from papers and public datasets across 14 fields; transferable to U.S. research evaluation with caution","publishedAt":"arXiv:2610.05197v1, submitted 4 October 2026","publicationDate":"2026-10-04","eventDate":null,"sourceName":"arXiv (cs.AI)","sourceLabel":"Academic preprint","sourceUrl":"https://arxiv.org/abs/2610.05197","evidenceClass":"academic-research","outcomeClass":"cautionary","topics":["developers-agents","knowledge-work"],"finding":"MechHypoBench compiles mechanisms from 244 papers in 14 fields into fixed generating graphs, applies them to 150 independently collected datasets (17.98 million records in the source pool) and asks agents for open-form mechanistic hypotheses, scored by how a fixed GPT-5.6 Terra interpreter reads their implications for hidden matched comparisons (MCA, MCNLL). On 300 tasks (120 easy, 180 hard), the best configuration, Codex with GPT-5.6 Sol and skills, reached 64.9%/50.8% MCA versus 87.6%/82.9% for the qualitative reference mechanism and 96.8%/97.5% for the parameterized one. AI-Scientist-v2 scored 59.2%/43.8%; AutoDiscovery 27.5%/27.1% on a 40-task subset against 47.1%/35.8% for Codex. Scientific-skill access cut runtime 10.1–18.1% for two backends without consistent quality gains; precomputed evidence summaries raised MCA 10.0 points, structured self-records only about 2 points. Only 214–289 of 300 tasks produced valid outputs per general-agent configuration.","sledRelevance":"Mechanistic explanation, not pattern fitting, is what research groups expect from 'AI scientist' tools. This benchmark shows current agents and dedicated AI-scientist workflows fall well short and that harness choice changes results, which matters for campuses selecting tools.","evidence":"Opened the arXiv abstract page and full HTML text via HTTPS (curl, text extracted); read abstract, benchmark construction, human evaluation (table 2), results (table 3) and sections 4.2–4.5. Results are the authors' own; no independent replication.","architectureImplications":"Harness and skill routing materially affect cost and output; pilots should hold the model fixed while comparing harnesses, budget for invalid outputs, and consider supplying curated evidence summaries, which helped more than agent self-notes.","governanceImplications":"Interpretation: dedicated AI-scientist products should not be assumed to outperform a general coding agent on the same model; institutions should require task-relevant evaluation before endorsing them for hypothesis generation.","securityPrivacyImplications":"Limited direct security content; the benchmark uses public datasets and replaces source terminology to avoid memorization cues, a technique that also limits leakage of identifiable domain context.","caveats":"Semi-synthetic: responses are generated by paper-derived mechanisms on real dataset conditions, not real experimental outcomes; scoring relies on one LLM interpreter; invalid outputs scored as empty controls; AutoDiscovery tested on 40 tasks only; no explicit limitations section; preprint.","streamIds":["research"],"roles":{"sales":"Sales framing: buyers of AI-scientist tools should expect hypothesis generation to trail reference mechanisms by a wide margin. MechHypoBench found the best agent 23 to 32 points below a qualitative reference and dedicated AI-scientist workflows no better than a general coding agent. A bounded offer is a tool-selection bake-off on a department's own data that compares harnesses on the same model. Avoid promising discovery acceleration; the benchmark measures directional consequences only.","engineering":"Engineering framing: evaluate harness, model and skills separately. The benchmark shows harness choice changing accuracy and runtime with the model fixed, skills shortening runs without better hypotheses, and curated evidence summaries adding about ten points. Plan for 4–29% invalid outputs per configuration, pin interpreter and model versions, and hold out conditions so hypotheses are judged on consequences rather than wording. Validate on real experimental outcomes, since the benchmark's responses are mechanism-generated.","delivery":"Delivery framing: treat AI-generated mechanistic hypotheses as candidates needing expert review, not findings. Research computing can own a reusable bake-off harness, PIs own domain judgment, and the research office owns guidance on citing agent-generated hypotheses. Measure held-out consequence agreement, invalid-output rate and runtime for each configuration. Interpretation: the proposed targets below are planning aids; the paper reports only benchmark results on semi-synthetic tasks."},"roleDetails":{"sales":{"customerProblem":"Departments evaluating AI-scientist products lack evidence on whether agents can turn their observational data into correct mechanistic explanations.","stakeholders":["Department chairs and PIs","Research computing director","Research development office","Graduate program directors"],"discoveryQuestions":["Which AI-scientist or coding-agent tools are researchers already using for hypothesis generation?","How are agent-generated hypotheses checked against held-out conditions or experiments?","Has any comparison held the model fixed while changing the harness or workflow?"],"valueHypothesis":"A structured bake-off can steer funds away from tools that do not improve on a general agent and toward curated evidence preparation, which helped more here; no cost savings are evidenced.","potentialEngagement":"Four-week harness and model comparison on 20–40 department tasks with held-out conditions, reporting consequence agreement, invalid-output rate and runtime.","unsupportedClaims":["AI scientists discover mechanisms reliably","Dedicated AI-scientist workflows beat general agents","Results transfer to real wet-lab outcomes"]},"engineering":{"fit":"Research groups with tabular or observational datasets who want agents to propose mechanisms, and research computing teams supporting agent harnesses.","architectureIntegration":"Agent harness (Codex or Claude Code style) with optional skill router; data pipeline that produces evidence summaries; held-out comparison set; fixed LLM interpreter for scoring with version pinning.","prerequisites":["Held-out conditions or experiments","Pinned model and interpreter versions","Compute budget for repeated runs (AutoDiscovery averaged 111.6 LLM calls per task)","Domain reviewers"],"constraints":["Semi-synthetic benchmark responses","Single LLM interpreter as scorer","Invalid outputs in 4–29% of tasks per configuration","Cloud model access may be restricted for sensitive data"],"security":"Keep sensitive datasets out of hosted model APIs unless approved, log agent tool calls, and separate hypothesis generation from scoring data.","proofOfValue":["Run two harnesses on the same model and compare consequence agreement","Measure gain from curated evidence summaries versus none","Track invalid-output rate per configuration","Test top hypotheses against real held-out experiments"]},"delivery":{"work":"Build a reusable hypothesis-evaluation harness with held-out conditions, run periodic bake-offs, and document which configurations are approved for exploratory use.","dependencies":["Curated datasets with held-out conditions","Model and harness licenses","Domain reviewer time","Research computing allocation"],"ownership":"Research computing owns the evaluation harness; PIs own scientific acceptance of hypotheses; research development office owns tool guidance.","skillsAdoption":"Train researchers to prepare evidence summaries, read consequence-based scores and treat agent hypotheses as candidates for testing.","governanceCheckpoints":["Bake-off before institutional endorsement","Expert review before hypotheses enter proposals","Re-test on model upgrades","Disclosure of agent-generated hypotheses"],"acceptanceCriteria":["Proposed: consequence agreement reported per configuration on held-out conditions","Proposed: invalid-output rate under 10% for approved configurations","Proposed: at least one hypothesis per pilot tested experimentally","Proposed: harness comparison documented with model held fixed"],"risks":["Overconfident hypotheses entering grant narratives","Tool selection driven by branding rather than results","High LLM-call costs for search-heavy workflows","Benchmark-to-practice gap"]}},"retrievedAt":"2026-10-05T22:07:51-05:00","enrichedAt":"2026-10-05T22:12:00-05:00","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Curated evidence summaries helped most, shifting effort to data stewards and domain experts who prepare and review evidence.","procurementImplications":"Require vendors to report results with the model held fixed across harnesses and to disclose invalid-output rates and LLM-call costs.","operatingModelImplications":"Hypothesis-generation agents fit as exploratory assistants inside expert-reviewed workflows rather than autonomous discovery services.","sourceVerification":{"openedUrl":"https://arxiv.org/abs/2610.05197","referenceExcerpt":"Experiments with general agents and AI scientists reveal a substantial gap between generated hypotheses and the underlying mechanisms.","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}