{"resourceId":"arxiv-evoriskbench-workspace-agent-runtime-risks-2026","versions":[{"version":"external-d893fcfbc32769abd54dcee8d71d1acd375113551006db25d13c282d8efec19a","resource":{"id":"arxiv-evoriskbench-workspace-agent-runtime-risks-2026","title":"EvoRiskBench: An Evolving Benchmark for Runtime Security Risks in Workspace Agents","organization":"Shiyi Kuang et al. (arXiv preprint)","sector":"Academic research; AI agent runtime security","geography":"Not specified (benchmark study; transferability to SLED deployments untested)","publishedAt":"2026-10-02","publicationDate":"2026-10-02","eventDate":null,"sourceName":"arXiv","sourceLabel":"Preprint, not peer reviewed (arXiv:2610.03153)","sourceUrl":"https://arxiv.org/abs/2610.03153","evidenceClass":"academic-research","outcomeClass":"cautionary","topics":["developers-agents","data-security"],"finding":"Preprint, not peer reviewed. EvoRiskBench provides 450 adversarial tasks across six scenarios and evaluates nine configurations combining three models (GPT-5.6 Sol, DeepSeek-V4-Pro-0813, Claude Opus 5) with three harnesses (Claude Code, Codex, OpenClaw), verifying outcomes from runtime traces and environment state. The authors report substantial vulnerabilities in all systems, with Codex plus DeepSeek-V4-Pro-0813 reaching a 68.44% attack success rate; success varied more by model than by harness. Benchmark cases will be released after safety and reproducibility checks.","sledRelevance":"Two of the evaluated harnesses (Claude Code and Codex) are among those CrowdStrike says Falcon Guardian agent hooks support, so this independent work describes the runtime risk those controls target. For universities and agencies allowing coding agents, it suggests harness configuration alone is not a sufficient control. It does not test any CrowdStrike product.","evidence":"Multi-author preprint with 450 tasks and trace-based verification; a 20-participant study supports the taxonomy. Dataset not yet released, limiting replication.","architectureImplications":"Runtime monitoring of what agents actually execute is needed because model and harness choices leave high attack success rates.","governanceImplications":"Approvals for coding-agent use should consider model choice, since risk varied more by model than by harness.","securityPrivacyImplications":"Attack success up to 68.44% in the worst configuration indicates real exposure when agents act on files and systems.","caveats":"Preprint, not peer reviewed; dataset unreleased; lab scenarios may not reflect institutional environments; no endpoint security product tested.","streamIds":["crowdstrike"],"roles":{"sales":"Independent risk evidence is the framing point: this preprint found coding-agent configurations with attack success rates up to 68.44%, and two of the tested harnesses are ones Falcon Guardian supports. For SLED institutions allowing Claude Code or Codex, the bounded conversation is about runtime controls and model approval. Interpretation: use the study to frame the problem, not to prove any product works. It tests no security tool, so it cannot support claims about blocking these attacks.","engineering":"Model-driven risk is the framing point: attack success varied more by model than by harness, and harness differences depended on the model. Interpretation: pre-sales should ask which models the customer's agents call, not just which agent tool is installed, and test runtime controls against representative adversarial tasks once the benchmark is released. Trace-based verification in this study is a useful template for validating endpoint agent monitoring.","delivery":"Approval scope is the framing point: approving an agent tool without approving its model leaves risk unmanaged. Interpretation: delivery should add model selection to coding-agent approval workflows and plan a runtime-control test when the dataset becomes available. Ownership sits with the AI governance lead and security engineering. Proposed acceptance criteria include an approved model-and-harness list and a recorded attack-success rate on a sample of tasks with controls enabled."},"roleDetails":{"sales":{"customerProblem":"Coding agents with file and system access are vulnerable to runtime attacks regardless of harness configuration.","stakeholders":["AI governance lead","CISO","Research computing director","Developer platform owner"],"discoveryQuestions":["Which models do your coding agents use?","Are agents allowed to modify files or run commands on managed devices?","How is agent activity monitored at runtime?"],"valueHypothesis":"Runtime monitoring plus model approval may reduce exposure shown in this benchmark; unmeasured for any product.","potentialEngagement":"A coding-agent risk review covering model and harness approvals and runtime monitoring.","unsupportedClaims":["That any security product blocks these attacks","Peer-reviewed status","Equivalence of lab and campus environments"]},"engineering":{"fit":"Institutions permitting workspace or coding agents.","architectureIntegration":"Runtime traces and environment-state checks as validation for endpoint agent monitoring.","prerequisites":["Agent and model inventory","Isolated test environment","Access to benchmark when released"],"constraints":["Dataset not yet public","Rapid model version churn","Varied harness configurations"],"security":"Restrict agent permissions and monitor executed actions, not just prompts.","proofOfValue":["Run sample adversarial tasks with and without controls","Compare attack success by model","Verify trace capture"]},"delivery":{"work":"Add model approval to agent governance and plan a controlled runtime test.","dependencies":["Benchmark release","Test environment","Governance body"],"ownership":"AI governance lead with security engineering.","skillsAdoption":"Staff need to run adversarial agent tests safely.","governanceCheckpoints":["Approved model-and-harness list","Test plan approval","Results review"],"acceptanceCriteria":["Proposed: approved model-and-harness list published","Proposed: attack success measured with controls on","Proposed: retest after model changes"],"risks":["Unsafe test execution","Outdated model list","False assurance from lab results"]}},"retrievedAt":"2026-10-30T22:13:23-05:00","enrichedAt":"2026-10-30T22:21:24-05:00","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"No workforce data.","procurementImplications":"Supports asking vendors of agent controls for results against adversarial runtime benchmarks.","operatingModelImplications":"Agent governance must cover models as well as tools.","peerReview":"preprint","sourceVerification":{"openedUrl":"https://arxiv.org/abs/2610.03153","referenceExcerpt":"The most vulnerable configuration, Codex with DeepSeek-V4-Pro-0813, reaches a 68.44% attack success rate (ASR)","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}