{"resourceId":"physclosurebench-agents-research-closure-gap-202610","versions":[{"version":"external-d4902eb0918c6156a6426258d33959d95a0e02e0db00eb6460b954fa0398eaaa","resource":{"id":"physclosurebench-agents-research-closure-gap-202610","title":"PhysClosureBench: frontier LLM agents reach full research closure on only 32.7% of physics research attempts","organization":"Mao et al. (arXiv, cs.AI)","sector":"Higher education and research agents","geography":"International (author affiliations not stated on the abstract page)","publishedAt":"15 October 2026","publicationDate":"2026-10-15","eventDate":null,"sourceName":"arXiv","sourceLabel":"arXiv preprint (v1)","sourceUrl":"https://arxiv.org/abs/2610.21767","evidenceClass":"academic-research","outcomeClass":"cautionary","topics":["knowledge-work","operating-model"],"finding":"A 50-task, expert-verified benchmark of goal-specified physics research tasks scores seven frontier LLMs on a 20-criterion Atomic Score and on Research Closure (RC0–RC4). The best model's mean Atomic Score is 6.29/10 but it reaches RC4 in only 32.7% of attempts; 36.2% of attempts scoring at least 8/10 still fall short of RC4, and every model reaches RC4 on fewer tasks in all three runs than in at least one run.","sledRelevance":"Research offices and research-computing teams are being asked to support agentic research tools. This is a controlled benchmark in one narrow field (strong-field and attosecond physics), so it bounds how much unsupervised closure to expect rather than describing campus deployments.","evidence":"Abstract of the arXiv v1 preprint submitted 15 Oct 2026: 50 tasks, seven frontier LLMs, three runs, Atomic Score and RC0–RC4 closure levels. Only the abstract page was opened; the full paper's methods and tables were not reviewed.","architectureImplications":"Agent workflows should record the evidence the agent generated and the scope conditions it checked, so a reviewer can see whether closure criteria were met rather than only whether sub-steps succeeded.","governanceImplications":"Acceptance of agent-produced research outputs should test closure (indispensable evidence and applicability limits), not just local accuracy; inconsistency across repeated runs argues for repeat-run checks before relying on a result.","securityPrivacyImplications":"Not addressed by the source. The benchmark does not evaluate data handling, access control or confidentiality.","caveats":"Single-domain benchmark, preprint not yet peer reviewed, scores come from the authors' own rubric, and the abstract does not give task-level baselines or human comparison. Results for physics may not transfer to other fields or to campus agent deployments.","streamIds":["research"],"roles":{"sales":"Lead with the gap between local success and complete closure. Research-office and CIO buyers hear many 'AI scientist' claims; this preprint gives a measured counterpoint (32.7% full closure for the best model, 36.2% of high-scoring attempts still incomplete). Position any engagement as evaluation and verification design, not autonomous research. Interpretation: it supports discovery conversations about acceptance criteria, but it is one physics benchmark and cannot support productivity or savings claims.","engineering":"Treat closure as an explicit acceptance check. The benchmark separates sub-task scores from whether indispensable evidence and scope conditions are satisfied, and repeated runs disagree. Interpretation: a proof of value should log evidence artifacts, run each task several times, and score closure with a domain expert. Fit is limited: tasks are physics-specific and the abstract does not describe tooling, sandboxing or compute setup.","delivery":"Frame delivery as building review and closure checkpoints around agent-assisted research. The source shows high partial scores can mask incomplete answers, so reviewers need explicit criteria and time. Interpretation: assign an operational owner in the research office or core facility, define closure levels per project type, and keep human sign-off. Limited applicability: no deployment, cost or staffing data are reported."},"roleDetails":{"sales":{"customerProblem":"Research leaders cannot tell how far to trust agent-generated research outputs and lack an acceptance standard.","stakeholders":["Vice president or associate vice president for research","Research computing director","Research integrity officer","Department chairs"],"discoveryQuestions":["How are agent-assisted analyses currently reviewed before results are used?","Who decides whether a research question has been fully resolved?","Are repeated runs compared today?"],"valueHypothesis":"A bounded closure-review process may reduce rework from plausible but incomplete agent outputs; the benchmark does not quantify any saving.","potentialEngagement":"A short assessment that defines closure criteria and a repeat-run test for two or three pilot research workflows.","unsupportedClaims":["Guaranteed productivity or cost savings","That results in physics apply to other disciplines","That any model is reliable for unsupervised research"]},"engineering":{"fit":"Fits teams evaluating agent tools for analysis-heavy research; limited to what a benchmark of expert-verified tasks can tell.","architectureIntegration":"Add evidence capture and a rubric-scoring step to agent runs, and store repeat-run results next to the outputs.","prerequisites":["Domain experts to write task constraints and rubrics","Run logging for agent actions and generated evidence","Budget for repeated runs"],"constraints":["Abstract-only inspection; method details unverified","Physics-specific tasks","Model versions change quickly"],"security":"Not covered by the source; keep sandboxing and data-access reviews separate from this evidence.","proofOfValue":["Score three runs per task and report variance, mirroring the run-to-run inconsistency noted in the abstract","Compare Atomic-style partial scores with a closure judgment by an expert","Track which scope conditions agents skip"]},"delivery":{"work":"Define closure levels, review checkpoints and expert sign-off for pilot agent-assisted research workflows.","dependencies":["Domain expert availability","Agent run logs","Agreed rubric"],"ownership":"Research office or core facility lead, with domain faculty as accountable reviewers.","skillsAdoption":"Researchers need training to read agent evidence trails and judge applicability limits, not just final answers.","governanceCheckpoints":["Approve closure criteria before the pilot","Review a sample of repeat runs","Record human sign-off on each accepted output"],"acceptanceCriteria":["Proposed: every accepted output has an expert-confirmed closure level","Proposed: repeat-run disagreement is reported for each task","Proposed: unresolved scope conditions are listed in the output"],"risks":["Reviewers over-trust high partial scores","Closure rubric becomes a box-checking exercise","Benchmark findings are over-generalised"]}},"retrievedAt":"2026-10-18T05:24:22-05:00","enrichedAt":"2026-10-18T05:24:22-05:00","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Moves researcher effort from generating analyses toward verifying closure, which needs expert time that benchmarks do not cost.","procurementImplications":"Ask vendors for closure-style, repeat-run evaluation evidence in the buyer's domain, not only headline accuracy.","operatingModelImplications":"Agent-assisted research needs a named reviewer and explicit closure criteria in the project workflow.","sourceVerification":{"openedUrl":"https://arxiv.org/abs/2610.21767","referenceExcerpt":"reveal a persistent gap between AI agents' local scientific accomplishments and their ability to achieve complete research closure","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}