{"resourceId":"frontierphysics-end-to-end-physics-research-agents-20261104","versions":[{"version":"external-c626d753a38ac80e94a56d83198f425dc15a5f723ab2828d64630558aee2bcfe","resource":{"id":"frontierphysics-end-to-end-physics-research-agents-20261104","title":"FrontierPhysics: Evaluating Agents for End-to-End Physics Research","organization":"Multi-institution (incl. UC Berkeley, Caltech, Harvard, UW, Stanford, UCSD, LBNL, ORNL, Purdue, et al.; BenchFlow)","sector":"University / national-lab research (AI agents for physics research evaluation)","geography":"United States-led international collaboration (transferable caution for any research university deploying research agents)","publishedAt":"4 November 2026 (arXiv 2611.06910v1)","publicationDate":"2026-11-04","eventDate":null,"sourceName":"arXiv","sourceLabel":"You, B. et al., \"FrontierPhysics: Evaluating Agents for End-to-End Physics Research,\" arXiv:2611.06910, 4 Nov 2026","sourceUrl":"https://arxiv.org/abs/2611.06910","evidenceClass":"academic-research","outcomeClass":"cautionary","topics":["developers-agents","knowledge-work","governance-procurement"],"finding":"FrontierPhysics is an end-to-end benchmark of 56 physicist-authored tasks spanning nine domains of theoretical, computational, and experimental physics. Agents must review literature, plan, execute research/experiments, and produce structured outputs plus manuscripts; scoring combines deterministic verifiers with expert rubrics for correctness and publishability. Across 22 models in 26 configurations, no configuration passes more than 46 of 168 rollouts (best reported overall pass rate 27.4% for Claude Opus 5.5 in the leaderboard framing). Manual trajectory review finds main weaknesses in insufficiently rigorous reasoning and limited expert intuition—not missing physics knowledge or routine execution errors—including circular self-verification.","sledRelevance":"Highly relevant scrutiny for research universities evaluating 'AI scientist' tools: separates engineering competence from publishable scientific judgment and supplies a failure-mode taxonomy campuses can reuse.","evidence":"Large multi-institution preprint with expert-authored tasks, verifiers, rubrics, multi-model rollouts, and human failure-mode audit; authors note some grading noise/false fails.","architectureImplications":"Research-agent stacks need sandboxes, literature tools, verifiers, and human rubric review—not chat UIs alone—if used for physics research support.","governanceImplications":"VPR/integrity offices should require end-to-end publishability evaluations before endorsing unsupervised agent research claims.","securityPrivacyImplications":"Agent code execution and web tools need isolation; paper focuses on scientific failure modes more than enterprise security controls.","caveats":"Preprint; physics-centric tasks; expert grading can err; pass rates are harness/model specific.","streamIds":["research"],"roles":{"sales":"Caution framing: FrontierPhysics, built by 42 physicists across many universities, finds no tested agent configuration passes more than 46 of 168 end-to-end physics research rollouts (best ~27.4%). Failures center on rigorous reasoning and expert intuition, not missing textbook facts. Use this to set honest expectations with VPR and physics departments buying 'AI scientist' platforms—sell evaluation harnesses and human review gates, not unsupervised discovery.","engineering":"Architecture framing: end-to-end physics agents need literature search, planning, experiment or code execution, manuscript generation, deterministic verifiers, and expert rubric blockers—not chat Q&A alone. Integrate sandboxed compute, paper retrieval, trajectory logging, and human domain review into one harness. Treat circular self-verification and weak physical intuition as first-class failure modes to instrument in logs, because FrontierPhysics shows those dominate over missing textbook facts. Keep instrument or cluster credentials behind policy checks the LLM cannot bypass.","delivery":"Delivery framing: before production 'AI physicist' rollouts, run a FrontierPhysics-like local task set authored by campus physicists with verifiers plus blocker rubrics. Ownership sits with the department research computing lead and faculty task authors who define what publishable means. Acceptance should require pass-rate reporting with a failure-mode taxonomy—not demo notebooks that skip publishability checks. Budget time for human audits of trajectories, since grading noise can create false fails that still need expert adjudication."},"roleDetails":{"sales":{"customerProblem":"Vendors pitch AI agents that 'do physics research,' but universities lack end-to-end evidence those agents produce correct, publishable work.","stakeholders":["Physics department chair","VPR / research integrity office","Research computing","Lab PIs"],"discoveryQuestions":["Which physics workflows would an agent own end-to-end versus assist?","Do you grade agent manuscripts with expert rubrics, not only unit tests?","What pass rate on faculty-authored tasks is acceptable before wider use?","Who reviews agent literature synthesis and intuition calls?"],"valueHypothesis":"Adopting physicist-authored end-to-end benchmarks can expose reasoning/intuition gaps before campuses fund unsupervised research agents.","potentialEngagement":"A scoped evaluation engagement that ports a subset of campus physics tasks into verifier+rubric harnesses and scores current agents.","unsupportedClaims":["Frontier agents are ready to replace physics graduate researchers","High homework-benchmark scores imply publishable research","Any single vendor pass rate transfers without local tasks"]},"engineering":{"fit":"Strong for theory/computation/experiment groups evaluating research agents; not a deployment recipe for wet-lab robots alone.","architectureIntegration":"Agent harness with web/literature tools, sandboxed compute, structured outputs, deterministic verifiers, reviewer-agent plus human audit on failures.","prerequisites":["Faculty-authored tasks","Verifier tests","Rubric blockers","Trajectory logging"],"constraints":["56-task / 9-domain sample","Expert grading noise (some false fails)","Preprint, not peer-reviewed journal article"],"security":"Sandbox all agent code execution; restrict credentialed instrument APIs until pass gates clear.","proofOfValue":["Pass@1 under full verifier+blocker rules","Failure-mode labels on trajectories","Cost/time versus pass-rate curves"]},"delivery":{"work":"Author local tasks; Wire verifiers; Score agents; Publish failure taxonomy to stakeholders","dependencies":["Faculty task authors","HPC/sandbox capacity","Optional external model APIs"],"ownership":"Physics research computing lead with faculty steering.","skillsAdoption":"Faculty learn rubric writing; students learn to audit agent trajectories rather than trust finals.","governanceCheckpoints":["No unsupervised manuscript submission","Human sign-off on blocker failures","Re-run after model upgrades"],"acceptanceCriteria":["Proposed: documented pass rate with CI","Proposed: ≥N trajectories human-audited","Proposed: explicit ban list for auto-actuation"],"risks":["Overfitting harnesses to one vendor","False fails from noisy rubrics","Scope creep beyond physics"]}},"retrievedAt":"2026-11-06T21:58:52-06:00","enrichedAt":"2026-11-06T22:03:30-06:00","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Implies continued need for physicist judgment in research loops; agents may absorb engineering grunt work without replacing expert intuition.","procurementImplications":"Procurement of research agents should demand faculty-authored end-to-end tasks and published pass/fail rubrics, not homework-style leaderboards alone.","operatingModelImplications":"Favors a supervised research-assistant operating model with mandatory human review of agent manuscripts and plans.","peerReview":"preprint","sourceVerification":{"openedUrl":"https://arxiv.org/abs/2611.06910","referenceExcerpt":"no configuration passes more than 46 of its 168 rollouts.","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}