{"resourceId":"devto-openshell-ocsf-audit-export-zero-records-agent-seatbelt-20261023","versions":[{"version":"external-37f249f6f32382192fab1a1c8640af6abc2507e2aef0b02f4eeaee6596d7ff6e","resource":{"id":"devto-openshell-ocsf-audit-export-zero-records-agent-seatbelt-20261023","title":"Independent test (dev.to, Oct 23): NVIDIA OpenShell 0.1.5's OCSF audit export wrote zero records on a CI runner; author builds an external evidence layer","organization":"Independent developer (dev.to author techdevfun); NVIDIA OpenShell","sector":"AI agent security and audit evidence (cross-sector, applicable to public agencies)","geography":"Not location-specific (GitHub Actions runner)","publishedAt":"2026-10-23 (dev.to)","publicationDate":"2026-10-23","eventDate":"2026-10-23","sourceName":"DEV Community (dev.to)","sourceLabel":"Independent hands-on test","sourceUrl":"https://dev.to/techdevfun/i-measured-an-ai-sandboxs-audit-export-it-contained-zero-records-so-i-built-the-layer-that-46n4","evidenceClass":"independent-reporting","outcomeClass":"cautionary","topics":["data-security","developers-agents","governance-procurement"],"finding":"The author cites an independent evaluation of NVIDIA OpenShell that found blocking held up but evidence did not: a blocked DNS query left no event, the OCSF audit export silently wrote nothing, and some events existed only on the VM console. The author then booted OpenShell 0.1.5 with the docker driver on a GitHub Actions runner and reports that the documented OCSF-JSON export produced zero records in both runs with ocsf_json_enabled set as documented, and that the supervisor file sink sits on a tmpfs external tools cannot read. Their own external layer (Agent Seatbelt, OPA/Rego policies plus a hash-chained ledger) reported 11/11 (clean) and 13/13 (planted) coverage, five planted violations blocked and recorded, no false blocks, and p95 decision latency of 27.5 ms. NVIDIA's OpenShell documentation separately says OCSF delivery is best-effort and retention depends on the runtime and external collector.","sledRelevance":"Why now: Onetera's Oct 25 city platform cites OpenShell for 'full audit logging'. For agencies subject to public-records, audit and due-process requirements, this test shows audit evidence from OpenShell must be verified in the actual deployment configuration.","evidence":"Single-author hands-on test with published method (version, driver, runner) and two runs, building on a cited prior evaluation; NVIDIA documentation (opened) confirms best-effort delivery. Small sample, one configuration, not peer reviewed; the author also promotes their own tool.","architectureImplications":"Treat the sandbox's own export as one input; ship events to an external, durable collector and verify completeness with test actions.","governanceImplications":"Claims of 'full audit logging' in procurement should be contract terms with a verification test, not marketing statements.","securityPrivacyImplications":"Missing audit records undermine incident investigation and records-retention compliance for agent actions on resident data.","caveats":"One configuration (0.1.5, Docker driver, CI runner); results may differ on other drivers or later versions; author has a commercial interest in the add-on layer.","streamIds":["nvidia"],"roles":{"sales":"Interpretation: this is a useful caution to raise before a customer does. When a vendor promises full audit logging through OpenShell, propose a simple verification test as part of the pilot. Doing so positions the engagement as rigorous with security officers and auditors, and avoids overstating what the open component guarantees, since NVIDIA's own documentation calls delivery best-effort. It also gives a concrete, low-cost pilot deliverable that security teams value.","engineering":"Interpretation: design audit capture outside the sandbox. Forward OpenShell events to an external collector, run scripted agent actions that should each produce an event, and compare counts. Pin versions and drivers, because the test covered only 0.1.5 with Docker on a CI runner. Measure any latency added by extra evidence layers in the agency's own environment. Note that the supervisor's file sink sits on a tmpfs that external tools cannot read, so plan an export path that does not depend on it.","delivery":"Interpretation: make audit completeness a go-live gate. Proposed acceptance criteria are that a scripted test run produces a matching external log record for every agent action, and that gaps raise an alert. Assign an owner for log retention under the agency's records schedule, and repeat the test after each OpenShell upgrade. Record the OpenShell version and driver in the deployment record so later tests compare like with like."},"roleDetails":{"sales":{"customerProblem":"Agencies need defensible records of what AI agents did with resident data, but sandbox audit exports may be incomplete.","stakeholders":["CISO / security officer","Records manager","Internal auditor","Agent platform vendor"],"discoveryQuestions":["Where do agent audit events land today, and who reviews them?","Has anyone tested that every agent action is logged?","What retention period applies to agent logs?"],"valueHypothesis":"Interpretation: verified audit evidence may reduce audit findings and make agent pilots easier to approve.","potentialEngagement":"Audit-evidence verification test for an agent pilot before go-live.","unsupportedClaims":["That OpenShell always fails to export (one configuration tested)","That the author's add-on works in other environments"]},"engineering":{"fit":"Any OpenShell-based agent deployment handling records or consequential decisions.","architectureIntegration":"OpenShell sandbox events forwarded to an external collector or SIEM, with scripted completeness checks.","prerequisites":["Agency log platform","Test harness of scripted agent actions"],"constraints":["Single test configuration","Best-effort delivery per NVIDIA docs"],"security":"Durable, tamper-evident external log storage; alert on missing events.","proofOfValue":["Scripted run with expected event count compared with collected records","Repeat after upgrade"]},"delivery":{"work":"Set up external log forwarding, completeness tests and retention for agent events.","dependencies":["Security operations","Records management","Platform vendor"],"ownership":"Agency security team owns log completeness; records manager owns retention.","skillsAdoption":"Security staff learn agent event schemas (OCSF) and testing.","governanceCheckpoints":["Audit-completeness test before go-live","Retest after each platform upgrade"],"acceptanceCriteria":["Proposed: 100% of scripted agent actions recorded externally","Proposed: alert fires within a defined time when events are missing"],"risks":["Silent log loss","Version drift","Unretained evidence for appeals"]}},"retrievedAt":"2026-10-26T03:03:09Z","enrichedAt":"2026-10-26T03:14:13Z","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Security and records staff need skills to test agent logging, not just review dashboards.","procurementImplications":"Add acceptance tests requiring that every agent action in a scripted run appears in the agency's log system.","operatingModelImplications":"Audit evidence becomes an operated service with monitoring for gaps, owned by the agency or its provider.","sourceVerification":{"openedUrl":"https://dev.to/techdevfun/i-measured-an-ai-sandboxs-audit-export-it-contained-zero-records-so-i-built-the-layer-that-46n4","referenceExcerpt":"the OCSF audit export silently wrote nothing, and some events existed only on the VM console.","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}