{"resourceId":"eclypsium-trusted-ai-infrastructure-runtime-weight-tampering-report-20261029","versions":[{"version":"external-dfac8c4ff9400a7ccc4669beccdf436d264ceecca0c75c6a490dfe8081a24a94","resource":{"id":"eclypsium-trusted-ai-infrastructure-runtime-weight-tampering-report-20261029","title":"Eclypsium report: on a GH200 test system with IOMMU passthrough, a compromised BlueField DPU changed model weights in memory while the model file, hash and signature stayed valid","organization":"Eclypsium (research coordinated with NVIDIA)","sector":"AI infrastructure security (vendor research report)","geography":"Global (U.S.-based researcher)","publishedAt":"2026-10-29","publicationDate":"2026-10-29","eventDate":null,"sourceName":"Eclypsium Research","sourceLabel":"Trusted AI Requires Trusted Infrastructure (technical report, PDF)","sourceUrl":"https://eclypsium.com/wp-content/uploads/Trusted-AI-Requires-Trusted-Infrastructure.pdf","evidenceClass":"vendor-claim","outcomeClass":"cautionary","topics":["data-security","infrastructure","governance-procurement","operating-model"],"finding":"Eclypsium's 15-page report describes a proof of concept on a controlled NVIDIA GH200 system with a BlueField DPU. Using DMA from the DPU, with the IOMMU in passthrough mode and permissive PCIe isolation, the researchers scanned host and GPU memory at about 3 GB/s and overwrote selected model tensors, generally in under a minute, after load-time checks had passed. The report says these are not default Linux settings and that with stronger DMA isolation and IOMMU configuration the same path was unavailable. It lists checks operators can run (IOMMU translation, PCIe ACS, ATS) and recommends inventorying BMC, DPU and GPU firmware as first-class assets.","sledRelevance":"State agencies, universities and research computing centers now run their own GPU servers for AI inference and research. The report shows that a valid model hash does not prove the running model is unchanged, and it gives concrete configuration checks that agency and campus infrastructure teams can verify on existing NVIDIA systems.","evidence":"Company research report describing a single laboratory demonstration with methods, configuration preconditions and mitigations; coordinated with NVIDIA; no independent replication, success rates or production incidents reported.","architectureImplications":"IOMMU translation, PCIe ACS and ATS settings become part of the AI security boundary; DPUs and BMCs must be treated as privileged computers; confidential computing on Hopper, Blackwell and Vera Rubin reduces the components that must be trusted.","governanceImplications":"Performance-tuning choices such as IOMMU passthrough should be documented as risk acceptances with time limits and measured benefit, as the report recommends.","securityPrivacyImplications":"A privileged infrastructure component could also read weights, adapters, system prompts and runtime context from memory, which matters for agencies hosting models that process sensitive data.","caveats":"Vendor research by a company that sells infrastructure-assurance products and partners with NVIDIA; one controlled test platform (GH200) with weakened isolation and IPMI enabled; not demonstrated on every accelerator; no production exploitation reported.","streamIds":["nvidia"],"roles":{"sales":"Frame this for CISOs and research computing directors who run their own NVIDIA GPU servers. The point is not that NVIDIA hardware is unsafe but that a model check at load time does not prove what is running later. The demonstration needed weakened isolation settings, so the credible conversation is about verifying configuration. Offer a bounded configuration review of existing GPU servers against the report's checks, and avoid claiming the attack is common or that any product fully prevents it.","engineering":"Engineering should check every GPU host for IOMMU translation, PCIe ACS on DPU ports and ATS settings, since the demonstrated path needed IOMMU passthrough and permissive isolation. Inventory BMC, DPU and GPU firmware versions with owners and change history. Disable IPMI where Redfish is available and segment management networks. If performance tuning requires passthrough, benchmark the actual gain and document the exception. Evaluate confidential computing on Hopper or Blackwell for workloads with sensitive models or data.","delivery":"Delivery should turn the report into a repeatable hardening checklist for GPU servers, applied at deployment, after firmware updates, after incident recovery and before reassigning hardware. Assign an owner for BMC and DPU firmware alongside host patching. Record any approved exception to memory-isolation defaults with an expiry date. Add management-plane logs to security monitoring. Report status to the CISO as a configuration compliance measure rather than as proof that the attack cannot happen."},"roleDetails":{"sales":{"customerProblem":"Agencies and campuses running their own GPU servers assume a verified model file means the running model is trustworthy.","stakeholders":["CISO","Research computing director","Infrastructure manager","AI program lead"],"discoveryQuestions":["Which GPU servers run with IOMMU passthrough or relaxed PCIe isolation?","Who owns BMC and DPU firmware patching?","Are management interfaces segmented and is IPMI disabled?"],"valueHypothesis":"A configuration review against the report's checks can close the specific path demonstrated and improve evidence for auditors.","potentialEngagement":"Configuration and firmware-inventory assessment of existing GPU servers against the report's checks.","unsupportedClaims":["That the attack is common or has occurred in production","That any single product prevents all runtime tampering","That the finding applies identically to every platform"]},"engineering":{"fit":"Organizations operating on-premises NVIDIA GPU servers with DPUs or BMCs.","architectureIntegration":"IOMMU translation, PCIe ACS/ATS controls and segmented management networks sit beneath the inference stack; confidential computing where supported.","prerequisites":["Host and device inventory","Access to kernel boot parameters and lspci output","Management network segmentation"],"constraints":["Demonstrated on one GH200 configuration with weakened isolation","Some performance tuning relies on passthrough","Not shown on other accelerators"],"security":"Keep IOMMU translation enabled, verify ACS and ATS control bits, segment BMC/DPU networks and monitor firmware drift.","proofOfValue":["Audit each GPU host's IOMMU and ACS/ATS state","Benchmark the performance effect of strict isolation on a representative workload"]},"delivery":{"work":"Build and apply a GPU-server hardening checklist across the hardware lifecycle.","dependencies":["Infrastructure operations","Security operations","Hardware vendor guidance"],"ownership":"Infrastructure operations, with oversight from the CISO.","skillsAdoption":"Train server administrators on IOMMU, PCIe and BMC/DPU firmware management.","governanceCheckpoints":["Checklist applied before production","Re-check after firmware updates and incident recovery","Exceptions approved with expiry"],"acceptanceCriteria":["Proposed: all GPU hosts report IOMMU translation enabled or a documented exception","Proposed: BMC and DPU firmware inventoried with named owners"],"risks":["Performance regressions from strict isolation","Unpatched management firmware","False assurance from configuration checks alone"]}},"retrievedAt":"2026-10-30T03:02:55Z","enrichedAt":"2026-10-30T03:10:14Z","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Requires infrastructure staff who understand firmware, PCIe and IOMMU configuration, skills that many public-sector IT teams lack.","procurementImplications":"Agencies buying GPU servers or AI cloud capacity can ask vendors for secure-mode configuration profiles, firmware inventories and attestation evidence for BMCs and DPUs.","operatingModelImplications":"Shifts AI assurance from a one-time model check to continuous monitoring of the hardware and firmware beneath the model.","sourceVerification":{"openedUrl":"https://eclypsium.com/wp-content/uploads/Trusted-AI-Requires-Trusted-Infrastructure.pdf","referenceExcerpt":"We have not demonstrated the same implementation on every accelerator architecture","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}