{"resourceId":"sorami-openshell-v012-egress-policy-empirical-evaluation-20260929","versions":[{"version":"external-610d1e0cdc4df420a7a6bd5894dd4d31926222252cf11e93fbcb1218f19c96ff","resource":{"id":"sorami-openshell-v012-egress-policy-empirical-evaluation-20260929","title":"Independent test of NVIDIA OpenShell v0.1.2: default policy blocked every exfiltration path tried and cut an agent-driven secret leak from 10/10 to 0/10, but operator settings such as auto-approval and audit mode let data out","organization":"Sorami (Australian cyber security consultancy)","sector":"Independent security research (AI agent sandboxing)","geography":"Australia (lab test; mapped to Australian ISM AI agent controls)","publishedAt":"2026-09-29","publicationDate":"2026-09-29","eventDate":"2026-09-29","sourceName":"Sorami Research","sourceLabel":"Independent technical report","sourceUrl":"https://sorami.com.au/research/nvidia-openshell-agent-sandbox-test/","evidenceClass":"independent-research","outcomeClass":"mixed","topics":["developers-agents","data-security","governance-procurement"],"finding":"Sorami tested OpenShell v0.1.2 (released Sep 28, 2026) under a pre-registered plan: 35 deterministic test IDs over 123 trials plus agent tests at N=10 per condition, with public logs. Default deny held for HTTP, HTTPS, raw TCP, direct IP, external DNS and unlisted binaries (3/3 each); a malicious setup script run by a local agent leaked a canary secret 10/10 without OpenShell and 0/10 under the default policy. Operator-enabled settings changed the result: read-write rules leaked 10/10; a GET-only rule still let data out via query string or header; leaving enforcement unset (audit mode, the default for L7 rules) let a forbidden POST through while logging it as allowed; automatic approval granted new public hosts with no human in 12/12 trials, including drafts from blocked connections. The policy prover reported GraphQL, MCP, WebSocket and JSON-RPC rules as unsupported while the loader accepted them. No bypass of a documented control was found; three logging gaps appeared on the VM driver. Newly relevant because NVIDIA's Oct 21 OpenShell Reference Architecture builds on this runtime; the page metadata also shows a modification dated Oct 22 (UTC).","sledRelevance":"One of the few independent, reproducible tests of an agent sandbox. For SLED security teams evaluating NVIDIA's new reference architecture, it shows the runtime's defaults hold but that common convenience settings (auto-approval, audit mode, broad rules) defeat containment, and that MCP traffic is not covered by the policy prover. It also maps controls to a government framework (Australia's ISM).","evidence":"Pre-registered test plan with deviation log and public evidence repository; Wilson 95% intervals for agent tests. Limits stated by the authors: one release, one VM driver on one Apple Silicon laptop without an NVIDIA GPU; Docker, Podman and Kubernetes drivers untested; a small local model (qwen3:8b), not frontier agents; AI assistants helped build the harness; no independent specialist review; repeated runs are not independent replication. The author is a consultancy that sells agent security reviews.","architectureImplications":"Results apply to the software runtime only, not BlueField/Sentry enforcement or other drivers; MCP, GraphQL, WebSocket and JSON-RPC rules lack prover coverage.","governanceImplications":"Supports manual approval, enforce-mode L7 rules, avoiding read-write rules and proving policies in CI as baseline governance controls.","securityPrivacyImplications":"Data can leave through any allowed endpoint, through parameters on read-only rules, and silently in audit mode; logging gaps can hide blocked DNS attempts.","caveats":"Tested v0.1.2, which may differ from the runtime shipped with the Oct 21 reference architecture; single host and driver; small agent; consultancy author with commercial interest.","streamIds":["nvidia"],"roles":{"sales":"Sales framing: independent evidence that the sandbox works by default and fails through configuration gives a credible, non-hype message to public-sector security buyers. Interpretation: offer policy hardening and validation rather than the runtime itself as the value. The test is one version on one laptop with a small model, so do not present it as certification of NVIDIA's full reference architecture or of production deployments.","engineering":"Engineering framing: keep approval manual, set L7 rules to enforce, avoid read-write rules, and do not rely on the prover for MCP or GraphQL traffic. Pre-sales engineers should test the customer's chosen driver (Docker, Kubernetes) because isolation code differs from the tested VM driver, and confirm log export works, since OCSF export and DNS block events were missing in the tested setup.","delivery":"Delivery framing: build a policy pipeline with CI proofs, manual approvals and log forwarding, and rerun Sorami-style canary tests on the customer's actual stack before go-live. Assign an owner for approving access proposals so auto-approval is never used as a shortcut. Tests should be repeated after every runtime upgrade. Acceptance criteria below are proposals modelled on the study's tests, not results observed in any SLED deployment."},"roleDetails":{"sales":{"customerProblem":"Security teams cannot tell whether an agent sandbox actually stops data leaving.","stakeholders":["CISO","Security architecture","Agent pilot owners","Internal audit"],"discoveryQuestions":["Which OpenShell driver and version would you run?","Who approves new network access for agents?","Are MCP tools in scope for your agents?"],"valueHypothesis":"Interpretation: disciplined configuration of an open-source sandbox can block common exfiltration paths; outcomes depend on operator settings.","potentialEngagement":"Agent sandbox configuration review and canary-based validation on the customer's stack.","unsupportedClaims":["That OpenShell is safe under any configuration","Results for Docker, Kubernetes or BlueField enforcement","Protection against frontier-model agents specifically"]},"engineering":{"fit":"Teams piloting coding or operations agents with shell and API access.","architectureIntegration":"OpenShell sandbox with deny-by-default egress, L7 enforce rules, CI policy proofs and log export to SIEM.","prerequisites":["Chosen driver and version","Policy repository","Log pipeline"],"constraints":["Prover does not cover MCP, GraphQL, WebSocket or JSON-RPC","Logging gaps on the tested driver","Results limited to v0.1.2"],"security":"Disable automatic approval, enforce L7 rules, avoid read-write rules, inject credentials outside the sandbox.","proofOfValue":["Replicate canary exfiltration tests on the target driver","Verify audit mode is not left as default","Confirm blocked events reach the SIEM"]},"delivery":{"work":"Implement policy CI, approval workflow and validation tests before agents go live.","dependencies":["Security operations integration","Policy owners","Test harness"],"ownership":"Security architecture team, with platform team support.","skillsAdoption":"Staff need to learn policy authoring, prover use and canary testing.","governanceCheckpoints":["Pre-production canary test","Change approval for each policy widening","Periodic re-test after upgrades"],"acceptanceCriteria":["Proposed: 0 canary leaks under production policy across repeated trials","Proposed: auto-approval disabled and verified in configuration audit","Proposed: all denied connections visible in SIEM"],"risks":["Convenience settings re-enabled under deadline pressure","Untested drivers behaving differently","Unprovable protocol rules accepted silently"]}},"retrievedAt":"2026-10-22T03:00:13Z","enrichedAt":"2026-10-22T03:10:04Z","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Shows policy authoring and review skill is the deciding factor, not the tool alone.","procurementImplications":"Buyers can require proof that auto-approval is off, L7 rules are enforced and policies are proven in CI before accepting agent deployments.","operatingModelImplications":"Containment is an operating discipline: approval workflow, enforcement mode and log export must be configured and monitored.","sourceVerification":{"openedUrl":"https://sorami.com.au/research/nvidia-openshell-agent-sandbox-test/","referenceExcerpt":"Automatic approval granted new public hosts with no human, 12/12, including drafts from blocked connections with the policy advisor off.","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}