From the CrowdStrike edition of October 30, 2026
EvoRiskBench: An Evolving Benchmark for Runtime Security Risks in Workspace Agents
Shiyi Kuang et al. (arXiv preprint) · Academic research; AI agent runtime security · Not specified (benchmark study; transferability to SLED deployments untested)
- Publisher
- arXiv
- Original publication
- 2026-10-02
- Source retrieved
- 2026-10-31
What happened
Preprint, not peer reviewed. EvoRiskBench provides 450 adversarial tasks across six scenarios and evaluates nine configurations combining three models (GPT-5.6 Sol, DeepSeek-V4-Pro-0813, Claude Opus 5) with three harnesses (Claude Code, Codex, OpenClaw), verifying outcomes from runtime traces and environment state. The authors report substantial vulnerabilities in all systems, with Codex plus DeepSeek-V4-Pro-0813 reaching a 68.44% attack success rate; success varied more by model than by harness. Benchmark cases will be released after safety and reproducibility checks.
Why it matters
Two of the evaluated harnesses (Claude Code and Codex) are among those CrowdStrike says Falcon Guardian agent hooks support, so this independent work describes the runtime risk those controls target. For universities and agencies allowing coding agents, it suggests harness configuration alone is not a sufficient control. It does not test any CrowdStrike product.
Evidence and measured results
Multi-author preprint with 450 tasks and trace-based verification; a 20-participant study supports the taxonomy. Dataset not yet released, limiting replication.
Limitations and uncertainty
Preprint, not peer reviewed; dataset unreleased; lab scenarios may not reflect institutional environments; no endpoint security product tested.
Put this evidence to work
Lighthouse Advisory interpretation, grounded in this source. Enriched 2026-10-30; this does not change the original publication date. Each role follows the same checklist so takeaways compare across sources.
Sales
Role takeaway
- Customer problem
- Coding agents with file and system access are vulnerable to runtime attacks regardless of harness configuration.
- Stakeholders
- AI governance lead
- CISO
- Research computing director
- Developer platform owner
- Discovery questions
- Which models do your coding agents use?
- Are agents allowed to modify files or run commands on managed devices?
- How is agent activity monitored at runtime?
- Value hypothesis
- Runtime monitoring plus model approval may reduce exposure shown in this benchmark; unmeasured for any product.
- Potential engagement
- A coding-agent risk review covering model and harness approvals and runtime monitoring.
- Unsupported claims
- That any security product blocks these attacks
- Peer-reviewed status
- Equivalence of lab and campus environments
Pre-sales engineering
Role takeaway
- Fit
- Institutions permitting workspace or coding agents.
- Architecture and integration
- Runtime traces and environment-state checks as validation for endpoint agent monitoring.
- Prerequisites
- Agent and model inventory
- Isolated test environment
- Access to benchmark when released
- Constraints
- Dataset not yet public
- Rapid model version churn
- Varied harness configurations
- Security
- Restrict agent permissions and monitor executed actions, not just prompts.
- Proof of value
- Run sample adversarial tasks with and without controls
- Compare attack success by model
- Verify trace capture
Delivery
Role takeaway
- Work
- Add model approval to agent governance and plan a controlled runtime test.
- Dependencies
- Benchmark release
- Test environment
- Governance body
- Ownership
- AI governance lead with security engineering.
- Skills and adoption
- Staff need to run adversarial agent tests safely.
- Governance checkpoints
- Approved model-and-harness list
- Test plan approval
- Results review
- Acceptance criteria
- Proposed: approved model-and-harness list published
- Proposed: attack success measured with controls on
- Proposed: retest after model changes
- Risks
- Unsafe test execution
- Outdated model list
- False assurance from lab results
Implementation considerations
Lighthouse Advisory interpretation across the operating dimensions a public-sector buyer must settle before this evidence becomes a design. Each note answers the question under its heading for this specific source.
Architecture and integration
What must connect, and where does the AI sit in the workflow?
Runtime monitoring of what agents actually execute is needed because model and harness choices leave high attack success rates.
Governance
Who approves, reviews and stays accountable for outcomes?
Approvals for coding-agent use should consider model choice, since risk varied more by model than by harness.
Security and privacy
What data, permissions and controls need testing?
Attack success up to 68.44% in the worst configuration indicates real exposure when agents act on files and systems.
Accessibility and workforce
Who is affected, and what skills or accommodations follow?
No workforce data.
Procurement
What should contracts, pricing and exit terms secure?
Supports asking vendors of agent controls for results against adversarial runtime benchmarks.
Operating model
Which teams own the service once it runs?
Agent governance must cover models as well as tools.
Publication history
- 2026-10-30CrowdStrike · Issue 0212 resources
Stable resource ID: arxiv-evoriskbench-workspace-agent-runtime-risks-2026