From the NVIDIA edition of October 21, 2026
Independent test of NVIDIA OpenShell v0.1.2: default policy blocked every exfiltration path tried and cut an agent-driven secret leak from 10/10 to 0/10, but operator settings such as auto-approval and audit mode let data out
Sorami (Australian cyber security consultancy) · Independent security research (AI agent sandboxing) · Australia (lab test; mapped to Australian ISM AI agent controls)
- Publisher
- Sorami Research
- Original publication
- 2026-09-29
- Source retrieved
- 2026-10-22
- Event date
- 2026-09-29
What happened
Sorami tested OpenShell v0.1.2 (released Sep 28, 2026) under a pre-registered plan: 35 deterministic test IDs over 123 trials plus agent tests at N=10 per condition, with public logs. Default deny held for HTTP, HTTPS, raw TCP, direct IP, external DNS and unlisted binaries (3/3 each); a malicious setup script run by a local agent leaked a canary secret 10/10 without OpenShell and 0/10 under the default policy. Operator-enabled settings changed the result: read-write rules leaked 10/10; a GET-only rule still let data out via query string or header; leaving enforcement unset (audit mode, the default for L7 rules) let a forbidden POST through while logging it as allowed; automatic approval granted new public hosts with no human in 12/12 trials, including drafts from blocked connections. The policy prover reported GraphQL, MCP, WebSocket and JSON-RPC rules as unsupported while the loader accepted them. No bypass of a documented control was found; three logging gaps appeared on the VM driver. Newly relevant because NVIDIA's Oct 21 OpenShell Reference Architecture builds on this runtime; the page metadata also shows a modification dated Oct 22 (UTC).
Why it matters
One of the few independent, reproducible tests of an agent sandbox. For SLED security teams evaluating NVIDIA's new reference architecture, it shows the runtime's defaults hold but that common convenience settings (auto-approval, audit mode, broad rules) defeat containment, and that MCP traffic is not covered by the policy prover. It also maps controls to a government framework (Australia's ISM).
Evidence and measured results
Pre-registered test plan with deviation log and public evidence repository; Wilson 95% intervals for agent tests. Limits stated by the authors: one release, one VM driver on one Apple Silicon laptop without an NVIDIA GPU; Docker, Podman and Kubernetes drivers untested; a small local model (qwen3:8b), not frontier agents; AI assistants helped build the harness; no independent specialist review; repeated runs are not independent replication. The author is a consultancy that sells agent security reviews.
Limitations and uncertainty
Tested v0.1.2, which may differ from the runtime shipped with the Oct 21 reference architecture; single host and driver; small agent; consultancy author with commercial interest.
Put this evidence to work
Lighthouse Advisory interpretation, grounded in this source. Enriched 2026-10-22; this does not change the original publication date. Each role follows the same checklist so takeaways compare across sources.
Sales
Role takeaway
- Customer problem
- Security teams cannot tell whether an agent sandbox actually stops data leaving.
- Stakeholders
- CISO
- Security architecture
- Agent pilot owners
- Internal audit
- Discovery questions
- Which OpenShell driver and version would you run?
- Who approves new network access for agents?
- Are MCP tools in scope for your agents?
- Value hypothesis
- Interpretation: disciplined configuration of an open-source sandbox can block common exfiltration paths; outcomes depend on operator settings.
- Potential engagement
- Agent sandbox configuration review and canary-based validation on the customer's stack.
- Unsupported claims
- That OpenShell is safe under any configuration
- Results for Docker, Kubernetes or BlueField enforcement
- Protection against frontier-model agents specifically
Pre-sales engineering
Role takeaway
- Fit
- Teams piloting coding or operations agents with shell and API access.
- Architecture and integration
- OpenShell sandbox with deny-by-default egress, L7 enforce rules, CI policy proofs and log export to SIEM.
- Prerequisites
- Chosen driver and version
- Policy repository
- Log pipeline
- Constraints
- Prover does not cover MCP, GraphQL, WebSocket or JSON-RPC
- Logging gaps on the tested driver
- Results limited to v0.1.2
- Security
- Disable automatic approval, enforce L7 rules, avoid read-write rules, inject credentials outside the sandbox.
- Proof of value
- Replicate canary exfiltration tests on the target driver
- Verify audit mode is not left as default
- Confirm blocked events reach the SIEM
Delivery
Role takeaway
- Work
- Implement policy CI, approval workflow and validation tests before agents go live.
- Dependencies
- Security operations integration
- Policy owners
- Test harness
- Ownership
- Security architecture team, with platform team support.
- Skills and adoption
- Staff need to learn policy authoring, prover use and canary testing.
- Governance checkpoints
- Pre-production canary test
- Change approval for each policy widening
- Periodic re-test after upgrades
- Acceptance criteria
- Proposed: 0 canary leaks under production policy across repeated trials
- Proposed: auto-approval disabled and verified in configuration audit
- Proposed: all denied connections visible in SIEM
- Risks
- Convenience settings re-enabled under deadline pressure
- Untested drivers behaving differently
- Unprovable protocol rules accepted silently
Implementation considerations
Lighthouse Advisory interpretation across the operating dimensions a public-sector buyer must settle before this evidence becomes a design. Each note answers the question under its heading for this specific source.
Architecture and integration
What must connect, and where does the AI sit in the workflow?
Results apply to the software runtime only, not BlueField/Sentry enforcement or other drivers; MCP, GraphQL, WebSocket and JSON-RPC rules lack prover coverage.
Governance
Who approves, reviews and stays accountable for outcomes?
Supports manual approval, enforce-mode L7 rules, avoiding read-write rules and proving policies in CI as baseline governance controls.
Security and privacy
What data, permissions and controls need testing?
Data can leave through any allowed endpoint, through parameters on read-only rules, and silently in audit mode; logging gaps can hide blocked DNS attempts.
Accessibility and workforce
Who is affected, and what skills or accommodations follow?
Shows policy authoring and review skill is the deciding factor, not the tool alone.
Procurement
What should contracts, pricing and exit terms secure?
Buyers can require proof that auto-approval is off, L7 rules are enforced and policies are proven in CI before accepting agent deployments.
Operating model
Which teams own the service once it runs?
Containment is an operating discipline: approval workflow, enforcement mode and log export must be configured and monitored.
Publication history
- 2026-10-21NVIDIA · Issue 469 resources
Stable resource ID: sorami-openshell-v012-egress-policy-empirical-evaluation-20260929