{"resourceId":"compmat-bench-agents-computational-materials-20260930","versions":[{"version":"external-2ace17f7b1c6639cb900eba88123223ffee05127c6e16afd093bf844deb4f6d1","resource":{"id":"compmat-bench-agents-computational-materials-20260930","title":"CompMat-Bench (Rice): agents pass 66.0–91.5% of 94 single materials-research steps with full guidance, but long workflows with reduced guidance drop to 50–75%; most workflow failures are scientific errors","organization":"Rice University (Zhang, Yakobson and colleagues)","sector":"University research / evaluation of AI agents for computational materials science","geography":"United States (Houston); tasks drawn from 15 published studies in Physical Review B, Nano Letters and npj Computational Materials","publishedAt":"arXiv:2610.00636v1, submitted 30 September 2026","publicationDate":"2026-09-30","eventDate":null,"sourceName":"arXiv (cs.AI)","sourceLabel":"Academic preprint (not peer-reviewed)","sourceUrl":"https://arxiv.org/abs/2610.00636","evidenceClass":"academic-research","outcomeClass":"mixed","topics":["developers-agents","infrastructure","knowledge-work"],"finding":"CompMat-Bench turns 15 recent computational-materials studies into 94 tasks (30 input-preparation, 29 output-analysis, 35 run-and-analyze) across 12 codes including VASP, Quantum ESPRESSO and LAMMPS, with expensive simulations precomputed and fixed-rule grading instead of an LLM judge. With full guidance, single-task pass rates were 66.0% (DeepSeek V4.1 Flash), 89.4% (Gemini 3.8 Flash), 90.4% (GPT-5.6 Sol in the MatClaw harness) and 91.5% (GPT-5.6 Sol in Codex CLI); input preparation was hardest (36.7–76.7%). One 94-task run cost 3.86, 27.00 and 106.43 USD for DeepSeek, GPT-5.6 Sol and Gemini. In eight multi-stage workflows run on one 128-core node within five hours, reduced guidance plus long workflows gave 50%, 54% and 75%. 38 of 50 failed workflow runs were scientific errors; in nine failing CrN runs the agent's own Wannier90 output showed the anomaly but the agent continued.","sledRelevance":"Campus research computing groups are being asked to support agents that set up and analyze HPC simulations. This benchmark shows agents handle well-specified single steps but lose reliability when they must plan multi-step calculations, and miss anomalies an experienced researcher would catch.","evidence":"Opened the arXiv abstract page and full HTML (v1) via HTTPS (curl, text extracted); read task construction, evaluation setup, results sections 5.1–5.3 and the limitations section. Results are the authors' own; no independent replication.","architectureImplications":"Evaluation can reuse precomputed simulation results to avoid rerunning costly jobs; workflow runs need a real compute node (two 64-core EPYC 7763, 239 GiB) and offline code manuals inside an isolated container with only model-API network access.","governanceImplications":"Interpretation: research groups should not treat single-step pass rates as evidence that agents can run a study end to end; require expert review of physical choices (for example dipole or van der Waals corrections) before agent-produced results are used.","securityPrivacyImplications":"Agents ran in isolated containers with no internet except the model API and no access to reference answers, a pattern campuses can reuse for agent sandboxes on licensed codes such as VASP.","caveats":"Preprint; 94 tasks from 15 studies in one field; workflow and reduced-guidance conditions have few runs (24 per agent and condition); fixed-rule grading can reject correct answers; studies proposing new concepts excluded; single runs on most tasks, with a 20-task repeat subset; published 30 Sep, before the prior run, but not previously archived and included because it extends this week's AI-scientist evaluation thread to HPC simulation work.","streamIds":["research"],"roles":{"sales":"Sales framing: research computing leaders want to know whether agents can safely run simulation work for labs. CompMat-Bench shows agents passing most well-specified single materials steps but dropping to 50–75% when they must plan multi-stage workflows with less guidance, mostly through scientific mistakes. A bounded offer is an agent-readiness assessment on a few of a lab's own simulation workflows. Do not claim agents can replace a computational scientist; the evidence covers one field and few workflow runs.","engineering":"Engineering framing: evaluate simulation agents the way this benchmark does. Precompute expensive runs so tests are cheap, grade with fixed rules rather than an LLM judge, isolate agents with offline manuals and model-API-only network access, and separate single-step from workflow tests. Expect input-preparation errors that never surface because the code is not run, and budget for cost differences: one model cost about four times another for a similar pass rate.","delivery":"Delivery framing: introduce simulation agents as assistants for well-specified steps, with expert checkpoints between stages. Research computing owns the sandbox and code licensing, PIs own physical correctness, and facilitators own user guidance. Track pass rates on local reference tasks and anomaly catches by reviewers. Interpretation: the proposed criteria below are planning aids; the paper reports benchmark results only, not deployments."},"roleDetails":{"sales":{"customerProblem":"Labs and research computing centers lack evidence on when AI agents can be trusted to prepare and analyze expensive HPC simulations without introducing physics errors.","stakeholders":["Research computing director","Computational materials PIs","HPC facilitation staff","Software licensing office (VASP and similar codes)"],"discoveryQuestions":["Which simulation steps are researchers already delegating to coding agents?","Do you have reference inputs and outputs from published local studies that could serve as test tasks?","Who reviews physical choices such as corrections and convergence settings in agent-prepared inputs?"],"valueHypothesis":"Scoping agents to well-specified single steps with expert review may save researcher time while avoiding the workflow-level failures this benchmark measured; no time or cost savings are reported.","potentialEngagement":"Four-week agent-readiness assessment: convert 10–20 steps from a lab's published studies into graded tasks and report pass rates by task type and guidance level.","unsupportedClaims":["Agents can run complete materials studies unsupervised","Results generalize to other scientific domains","Any measured researcher time savings"]},"engineering":{"fit":"Campus HPC sites supporting DFT, molecular dynamics and related codes where researchers use coding agents for input preparation and output analysis.","architectureIntegration":"Isolated container per run with offline code manuals; precomputed simulation outputs for single-step tests; full compute node for workflow tests; fixed-rule graders; harness choice (MatClaw or Codex CLI) evaluated separately.","prerequisites":["Reference tasks with known-correct inputs and outputs","Licensed simulation codes available in the sandbox","Model API access approved for research data","Node allocation for workflow tests"],"constraints":["Few workflow runs per condition","Single-field benchmark","Fixed-rule graders can reject valid answers","Code licenses may restrict agent use or containerization"],"security":"Allow only model-API egress, keep graders and reference answers outside the agent container, and review generated job scripts before they run on shared nodes.","proofOfValue":["Compare single-step and workflow pass rates on local tasks","Measure how often agents flag anomalies in their own outputs","Record API cost per task set per model","Re-test with reduced guidance to mimic real user prompts"]},"delivery":{"work":"Build a local test suite from published campus studies, approve agent use for specific simulation steps, and add expert checkpoints between workflow stages.","dependencies":["PI-provided reference studies","Code licensing review","Sandbox and node allocation","Model and harness version pinning"],"ownership":"Research computing owns the sandbox and test suite; PIs own scientific acceptance; facilitators own user guidance on approved uses.","skillsAdoption":"Train users to give explicit method guidance, check physical settings and read raw code output for anomalies instead of relying on agent summaries.","governanceCheckpoints":["Test-suite pass before approving a model or harness","Expert review between workflow stages","Re-test on each model upgrade","Disclosure of agent-prepared inputs in publications"],"acceptanceCriteria":["Proposed: local single-step pass rate above 85% before approval for that task type","Proposed: workflow use only with stage-level expert sign-off","Proposed: cost per task recorded for every approved configuration","Proposed: zero unreviewed agent-generated jobs on shared nodes"],"risks":["Silent input errors when codes are not run","Physically wrong but plausible results","Large cost differences between models","Users treating single-step success as workflow reliability"]}},"retrievedAt":"2026-10-06T22:05:23-05:00","enrichedAt":"2026-10-06T22:10:00-05:00","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Failures stem from implicit expert knowledge rarely written down, so experienced computational scientists remain essential reviewers and trainers.","procurementImplications":"Compare models on cost per passed task, not list price per token; one model reached a similar pass rate at a quarter of another's cost.","operatingModelImplications":"Agents fit as step-level assistants inside expert-reviewed simulation workflows rather than autonomous study runners.","sourceVerification":{"openedUrl":"https://arxiv.org/abs/2610.00636","referenceExcerpt":"Failure analysis attributes most failures to scientific errors rather than to errors in software usage.","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}