{"resourceId":"apexa-aps-execution-integrity-20260921","versions":[{"version":"external-bee6995e436cecc00c11859ae6b27d41649bb088478a73ca0cfffc95ec22b9af","resource":{"id":"apexa-aps-execution-integrity-20260921","title":"APEXA at APS: tool-layer execution integrity; motor safety 0/200 vs 15/200 prompt-only across four models","organization":"Argonne National Laboratory — Advanced Photon Source / X-ray Science Division (arXiv preprint)","sector":"National laboratory user-facility scientific computing / AI agents for synchrotron data reduction","geography":"United States (APS, Argonne National Laboratory, Lemont, IL); transferable methods with facility-control and simulation limits","publishedAt":"September 21, 2026 (arXiv; revised September 26, 2026 v2)","publicationDate":"2026-09-21","eventDate":null,"sourceName":"arXiv 2609.24165","sourceLabel":"Academic preprint (arXiv) — DOE national laboratory authors","sourceUrl":"https://arxiv.org/abs/2609.24165","evidenceClass":"academic-research","outcomeClass":"mixed","topics":["developers-agents","infrastructure","data-security","operating-model"],"finding":"APEXA is a deployed multi-agent framework coordinating 61 tools for synchrotron data reduction at APS. A deterministic execution-integrity guard refuses to surface results for tool calls that did not execute; the same tool layer enforces motor safety. Across 50 adversarial scenarios and four models, tool-layer enforcement produced 0/200 violations against a simulated EPICS controller versus 15/200 for a prompt-only baseline. Authors introduce APEXA-Bench (58 facility tasks labeled by scientific correctness and physical consequence). On real APS beamline data, one natural-language prompt recovered detector geometry and integrated a full attenuation/exposure sweep; opened full text reports recovered detector distance stable to ±0.02 mm and autonomous rejection of nonphysical calibrations. Authors argue trustworthy scientific agents require deterministic validation of execution and scientific output rather than model-side assurances; framework, benchmark, and traces are released.","sledRelevance":"University and user-facility research programs connecting LLM agents to instruments need evidence that safety and execution integrity must be enforced in tool-layer code—prompt-only safeguards failed in 15/200 adversarial trials here.","evidence":"arXiv abs 2609.24165 opened via HTTPS; HTML full text confirmed metrics. Submission 21 Sep 2026; revised 26 Sep 2026 v2. Findings and excerpt from opened abstract/HTML.","architectureImplications":"Place deterministic guards between stochastic agents and tools/instruments: execution-integrity checks, format-drift-tolerant parsers, and hard motor/safety validators in server-side code—not only system prompts.","governanceImplications":"Facility AI-agent deployments should require published adversarial safety suites and consequence taxonomies (compute vs beam time vs equipment damage) before expanding motor or analysis authority.","securityPrivacyImplications":"Institutional LLM gateways can silently drop tool calls; treat model narration as untrusted and require executed-tool attestation before accepting scientific results or issuing hardware commands.","caveats":"Preprint; motor suite on simulated EPICS IOC; large-scale model scoring deferred; refinement accuracy not assessed; facility-specific tool set.","streamIds":["research"],"roles":{"sales":"Sales framing: User-facility and research-computing buyers wiring LLM agents to analysis or instrument APIs need deterministic execution integrity—not prompt-only safety. APEXA supports a bounded pitch that tool-layer guards can hold motor-safety violations to 0/200 across four models where a prompt-only baseline allowed 15/200, while running real APS calibration/integration from natural language. Do not sell proven live-motor safety or complete refinement accuracy—the adversarial suite used a simulated EPICS IOC and refinement scoring is deferred.","engineering":"Engineering framing: Fit is MCP/tool-server mediation for scientific pipelines where a stochastic agent must not invent results or bypass instrument limits. Integrate execution-integrity attestation, five-class motor/safety checks before any hardware command, and consequence-labeled evaluation harnesses. Prerequisites include simulated IOC testbeds, tool manifests, and gateway-failure handling. Proof-of-value is 0 unsafe actions on an adversarial suite plus end-to-end analysis on real datasets with nonphysical-result rejection.","delivery":"Delivery framing: Deploy tool-layer guards first on analysis-only paths, then expand motor authority only after simulated adversarial suites pass; release traces and benchmarks as continuous integration for tools. Ownership: facility controls/beamline ops own safety gates; data/analysis teams own scientific validators; AI platform owners own agent runtime. Acceptance requires published 0-violation adversarial results on the mock IOC and documented real-data calibration stability. Risks include mistaking simulated safety for live-motor proof and over-trusting task completion without scientific constraint."},"roleDetails":{"sales":{"customerProblem":"LLM agents can narrate scientific results or issue unsafe instrument commands when reliability lives only in prompts.","stakeholders":["User-facility directors","Beamline scientists","Research computing / AI platforms","ES&H / controls engineers","DOE program sponsors"],"discoveryQuestions":["Is execution attested in code or only requested in prompts?","Do you have an adversarial motor/analysis suite?","How are gateway-dropped tool calls detected?","What consequence classes label agent tasks?"],"valueHypothesis":"Deterministic tool-layer integrity can prevent fabricated analyses and block unsafe commands that prompt-only safeguards miss.","potentialEngagement":"Six-week pilot: mock-IOC adversarial suite + analysis-only agent path with trace review on one beamline workflow.","unsupportedClaims":["Live-motor 0-incident proof from this paper alone","Guaranteed refinement scientific accuracy","Model-agnostic perfection without tool-layer code","Full replacement of beamline experts"]},"engineering":{"fit":"Tool-mediated scientific agent with execution-integrity and hard safety checks between LLM and instruments/analysis engines.","architectureIntegration":"Insert deterministic guards in MCP/tool servers: executed-call attestation, format-drift parsing, soft/hard limit checks before caput-like commands.","prerequisites":["Tool inventory with manifests","Simulated IOC/test harness","Gateway observability","Dataset permissions for real-data trials"],"constraints":["Simulated motor evaluation","Deferred large-scale scoring","Facility-specific tools","Preprint"],"security":"Treat model output as untrusted; never allow hardware commands that bypass tool-layer checks; log attestations.","proofOfValue":["0/N unsafe actions on adversarial mock suite","Prompt-only baseline contrast","Real-data calibration stability band","Explicit non-result on unexecuted tools"]},"delivery":{"work":"Implement guards, run mock adversarial suite, pilot analysis-only NL workflows, gate any motor expansion, publish traces.","dependencies":["Controls/EPICS access policy","LLM gateway","Analysis engines (e.g., MIDAS)","ES&H approval"],"ownership":"Controls/ES&H own safety gates; beamline scientists own scientific acceptance; platform team owns agent runtime.","skillsAdoption":"Train beamline staff to read attestations/traces and reject narrated-but-unexecuted results.","governanceCheckpoints":["Safety architecture review","Mock-suite sign-off","Analysis-only go-live","Motor-authority expansion review"],"acceptanceCriteria":["Adversarial mock suite 0 unsafe","Attested real-data end-to-end run","Documented nonphysical rejection behavior"],"risks":["Simulated≠live motors","Task-completion without scientific constraint","Gateway silent drops","Scope creep to unsafe authority"]}},"retrievedAt":"2026-09-30T22:02:14-05:00","enrichedAt":"2026-09-30T22:08:00-05:00","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Natural-language access can broaden who can run facility pipelines, but staff must be trained to demand execution attestations and understand consequence-labeled refusals.","procurementImplications":"Require tool-layer safety/execution-integrity features and adversarial test evidence in any agent-for-instruments RFP; reject prompt-only safety claims.","operatingModelImplications":"Facility agent ops should treat tool servers as the trust boundary, with continuous adversarial CI and staged authority expansion.","sourceVerification":{"openedUrl":"https://arxiv.org/abs/2609.24165","referenceExcerpt":"it produced 0/200 violations against a simulated EPICS controller, versus 15/200 for a prompt-only baseline","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}