{"resourceId":"arxiv-sciexam-enso-agents-build-climate-models-hidden-graders-20261021","versions":[{"version":"external-4f03683b3f1b25d5ecf5a37176de9c9e2801ea44a27d4dc2edb55c505e2a2a48","resource":{"id":"arxiv-sciexam-enso-agents-build-climate-models-hidden-graders-20261021","title":"SciExam for ENSO (v2, 21 Oct): hidden-grader benchmark where AI agents build El Nino climate models from real observations in six hours; six of twelve agent systems beat a published reference model","organization":"Yinling Zhang, Langchen Liu, Dongbin Xiu, Xueyan Zou, Xu Kuang, Mengdi Wang, Shilong Liu (The Ohio State University, Yale, UC San Diego, Stanford, Princeton)","sector":"Academic research (AI for climate science, agent evaluation)","geography":"United States","publishedAt":"arXiv v1 7 October 2026; revised v2 21 October 2026 (version inspected)","publicationDate":"2026-10-21","eventDate":"2026-10-21","sourceName":"arXiv","sourceLabel":"arXiv:2610.10513v2 [cs.AI], Zhang et al., \"SciExam for ENSO: Can AI Agents Build Climate Models?\", revised 21 October 2026","sourceUrl":"https://arxiv.org/abs/2610.10513","evidenceClass":"academic-research","outcomeClass":"emerging","topics":["developers-agents","knowledge-work","infrastructure"],"finding":"SciExam for ENSO is a benchmark in which language-model agents build low-order stochastic models of the El Nino-Southern Oscillation from real observations within a six-hour budget. Agents write their own diagnostics, which are frozen, and develop models using only that feedback; hidden graders then test whether the model reproduces ENSO statistics (1950-2019), recovers unobserved variables (1980-2014) and forecasts held-out years (2015-2024), scoring a published reference model the same way. Across twelve agent systems (Claude, GPT and Gemini models in their vendors' scaffolds, others via OpenRouter, each in an offline container), six produced final models scoring above the published model, mainly through better reconstruction and forecasting; eight would exceed it on best checkpoints. Simplified forms of stronger models align with one of two competing explanations of ENSO warm-cold asymmetry, and controlled runs suggest the top system was not recalling the dated record. The cost appendix shows per-run API costs from about $2 to $132 for metered systems.","sledRelevance":"Shows a way for research groups and research-computing units to evaluate AI research agents where no answer is known, using hidden, independent scientific tests rather than rubrics or LLM judges. Useful to universities piloting AI agents in climate and Earth-system science, and to funders assessing claims about autonomous research, while showing such evaluations need careful harness design.","evidence":"Opened the arXiv /abs/ page showing v1 (7 Oct 2026) and v2 (21 Oct 2026, 6,660 KB, 28 pages, 5 figures, 8 tables) and the v2 HTML full text: evaluation design (section 3.3), experimental setup (section 3.4: twelve systems, offline containers, five-minute checkpoints), conclusion (six of twelve exceed the reference) and Appendix D cost table (e.g. Qwen3-Max $132.37, DeepSeek V4.1 Flash $2.43; subscription systems at fixed fee).","architectureImplications":"Agents ran in isolated offline containers with checkpointing and hidden graders, a pattern research-computing groups can reuse for safe, reproducible agent evaluation on shared clusters.","governanceImplications":"Evaluation design (frozen self-written diagnostics, hidden graders, held-out years, information-control runs) is itself a governance control against overclaiming or memorisation.","securityPrivacyImplications":"Offline containers limit data leakage and external tool use; public climate data, no personal data.","caveats":"Preprint, not peer reviewed; single phenomenon and a low-order model class; results compare final checkpoints and depend on the harness. Beating a published reference on these tests does not establish scientific validity or acceptance by the field. The v2 revision date is used as the publication date because the inspected version was released 21 Oct; v1 dates from 7 Oct. Model names reflect vendor products at the time.","streamIds":["research"],"roles":{"sales":"Lead with rigorous evaluation of research agents. SciExam for ENSO tested twelve AI agent systems building El Nino models from real data in six hours; six beat a published reference model on hidden tests. Research VPs, climate-science centres and research-computing leaders evaluating agent pilots want credible evidence. A bounded engagement designs a hidden-grader evaluation for one local research problem. Interpretation: this is a single-domain preprint, so it cannot support claims that agents produce accepted science or replace researchers.","engineering":"Frame the fit as a reproducible agent-evaluation harness on research computing. The study ran each agent in an offline container with five-minute checkpoints, frozen diagnostics and hidden graders, and reports per-run costs from about $2 to $132 for metered systems. Pre-sales should size container isolation, checkpoint storage and API or subscription budgets on the cluster. Interpretation: results depend on harness design, so proof-of-value must replicate the controls, including information-varied runs.","delivery":"Treat delivery as standing up an agent-evaluation service for a research group. Work covers selecting a problem with independent tests, building hidden graders with domain scientists, running agents in isolated containers, and reviewing candidate models with experts. A research group lead owns it with research computing. Interpretation: acceptance should use pre-registered hidden tests and expert review, since the source shows promise on one phenomenon, not general capability."},"roleDetails":{"sales":{"customerProblem":"Research leaders lack trustworthy ways to judge whether AI research agents produce valid science rather than plausible output.","stakeholders":["Vice president for research","Climate or Earth-system science centre director","Research computing director","Principal investigators piloting agents"],"discoveryQuestions":["Which research problems have independent tests you could hide from an agent?","How are you evaluating agent pilots today?","What budget model (subscription vs. metered API) applies?"],"valueHypothesis":"A hidden-grader evaluation could tell a research group whether agents add value on its own problems before scaling use.","potentialEngagement":"A six- to eight-week pilot building a hidden-grader evaluation for one local research problem and comparing several agent systems.","unsupportedClaims":["Agents produce peer-accepted science","Results generalise beyond ENSO","Agents can replace domain scientists"]},"engineering":{"fit":"Good fit for groups with well-defined physical models, public data and access to containerised compute.","architectureIntegration":"Run agents in isolated offline containers on the cluster with checkpointing, frozen diagnostics and a separate hidden-grader service.","prerequisites":["Container platform on research computing","Domain-built hidden tests","Model API or subscription access"],"constraints":["Single-domain preprint","Harness-dependent results","Variable per-run API costs"],"security":"Offline containers prevent external lookups and data leakage; keep graders and held-out data inaccessible to agents.","proofOfValue":["Replicate the benchmark harness on local infrastructure","Run information-controlled variants to test memorisation","Compare agent models with a human-built baseline"]},"delivery":{"work":"Select a problem, build hidden graders, run agents in isolated containers and convene expert review of candidate models.","dependencies":["Domain scientists to design tests","Research computing container support","Budget for API usage"],"ownership":"Principal investigator or research group lead, with research computing.","skillsAdoption":"Researchers need skills in agent harness design and in interpreting agent-generated model structures.","governanceCheckpoints":["Pre-registration of hidden tests before agent runs","Expert review before any agent-derived model is published","Cost review after pilot runs"],"acceptanceCriteria":["Proposed: agent models evaluated only on pre-registered hidden tests","Proposed: at least one agent result reviewed by two domain experts","Proposed: per-run costs within the agreed budget"],"risks":["Overfitting to the harness","Memorised data inflating scores","Overclaiming results in publications"]}},"retrievedAt":"2026-10-23T05:32:51-05:00","enrichedAt":"2026-10-23T05:32:52-05:00","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Domain scientists remain essential to design graders and interpret model structures; the benchmark depends on expert-built tests.","procurementImplications":"Costs vary widely across systems (fixed-fee subscriptions vs. metered API from a few dollars to over $100 per run), so pilots should budget per-run costs and compare scaffolds, not only models.","operatingModelImplications":"Suggests a research-group workflow where agents propose candidate models and humans evaluate them through pre-registered hidden tests.","sourceVerification":{"openedUrl":"https://arxiv.org/abs/2610.10513","referenceExcerpt":"Across twelve agent systems, six produce models that score higher than the published model, mainly through better reconstruction and forecasting.","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}