Strategic PartnersIssue 547 records
NVIDIA
An AI-infrastructure security and compute-efficiency edition. On Oct 29 NVIDIA and Eclypsium published research showing that on a GH200 test system, a compromised BlueField DPU could change a running model's weights in memory in under a minute while the model file, hash and signature stayed valid, skewing answers on national security and medical questions. The path required network access to a DPU or BMC and weakened, non-default memory isolation; NVIDIA told CNN nothing new was identified, two outside experts called it credible, and no production incident is reported. For agencies and universities running GPU servers, the practical response is configuration checks and treating BMCs and DPUs as privileged assets, consistent with CISA and NSA's 2023 BMC guidance. Separately, a Princeton preprint found most jobs on a 319-GPU campus cluster used a small fraction of GPU capacity, and NVIDIA/EPFL and ETH Zurich report software emulation speedups of 1.5-3.9x on existing GPUs. No new U.S. state, local or education NVIDIA deployment was found for Oct 28-29.
- Evidence records
- 7
- Cross-source patterns
- 3
- Evidence classes
- 3 vendor claim2 academic research1 independent reporting1 standards or public-body guidance
- Outcomes
- 3 cautionary3 emerging1 effective
- Source freshness
- 6 new this fortnight1 older, newly relevant
- Research completed
- 2026-10-30
Choose a role to see its takeaway beside every record in the ledger.
Patterns across the evidence
Synthesis · Lighthouse Advisory interpretation
A valid model file, hash or signature does not prove what model is running, but the demonstrated attack needed weakened, non-default isolation
Eclypsium's report shows that on a GH200 test system, a compromised BlueField DPU changed selected model weights in memory in under a minute while the stored model, hash and signature remained valid. NVIDIA's blog agrees that load-time checks and boot measurements alone are insufficient. Both sources and CNN's reporting stress the preconditions: network access to a DPU or BMC, IOMMU passthrough and permissive PCIe isolation, which the report says are not default Linux settings. NVIDIA told CNN nothing new was identified, two outside experts called the findings credible, and no production incident is reported.
Operating questionFor each GPU server running agency or campus AI models, can the team show that IOMMU translation and PCIe isolation are enforced, or that any relaxation is a documented, time-limited exception?
Supporting evidenceEclypsium (research coordinated with NVIDIA)NVIDIACNN Wire / Scripps News
BMCs and DPUs on AI servers need the same inventory, patching, segmentation and monitoring as any privileged system
The Eclypsium report and NVIDIA's blog both recommend treating each DPU, GPU and management controller as an asset with its own firmware, configuration and owner, adding BMC and Redfish logs to security monitoring, and reassessing after incident response and before hardware is reassigned. The report cites legacy IPMI weaknesses and OpenBMC vulnerabilities as entry points. CISA and NSA's 2023 guidance already warns that hardened credentials, firmware updates and network segmentation are often overlooked on BMCs, giving agencies an existing federal baseline to apply now.
Operating questionAre every GPU server's BMC and DPU in the asset inventory with a named owner, current firmware and a segmented management network?
Supporting evidenceEclypsium (research coordinated with NVIDIA)NVIDIACybersecurity and Infrastructure Security Agency (CISA) and National Security Agency (NSA)
Software and scheduling may extract more from existing NVIDIA GPUs before buying more, but each gain needs local validation
A Princeton preprint measuring a 319-GPU university cluster found over 70% of jobs used less than 20% of GPU memory and half averaged below 5% SM utilization, pointing to better scheduling and GPU sharing. NVIDIA and EPFL report that cuBLAS FP32 emulation sped molecular dynamics steps by up to 1.7x on Blackwell and 1.5x on A100 with one environment variable, and ETH Zurich reports 3.9x faster training with 58% less memory using integer emulation on GH200. All three are preprints or vendor posts, based on single clusters or model families, and the ETH code is not yet released.
Operating questionBefore the next GPU purchase, what measured utilization and software-optimization results from the institution's own cluster will inform the size of the request?
Supporting evidencePrinceton University, Department of Computer ScienceNVIDIA with EPFL (Michele Ceriotti)ETH Zurich, D-ITET
Evidence ledger
Full record · every source keeps its link and limitations
Eclypsium report: on a GH200 test system with IOMMU passthrough, a compromised BlueField DPU changed model weights in memory while the model file, hash and signature stayed valid
Eclypsium's 15-page report describes a proof of concept on a controlled NVIDIA GH200 system with a BlueField DPU. Using DMA from the DPU, with the IOMMU in passthrough mode and permissive PCIe isolation, the researchers scanned host and GPU memory at about 3 GB/s and overwrote selected model tensors, generally in under a minute, after load-time checks had passed. The report says these are not default Linux settings and that with stronger DMA isolation and IOMMU configuration the same path was unavailable. It lists checks operators can run (IOMMU translation, PCIe ACS, ATS) and recommends inventorying BMC, DPU and GPU firmware as first-class assets.
Why it matters, evidence and limitations
- Why it matters
- State agencies, universities and research computing centers now run their own GPU servers for AI inference and research. The report shows that a valid model hash does not prove the running model is unchanged, and it gives concrete configuration checks that agency and campus infrastructure teams can verify on existing NVIDIA systems.
- Evidence and measured results
- Company research report describing a single laboratory demonstration with methods, configuration preconditions and mitigations; coordinated with NVIDIA; no independent replication, success rates or production incidents reported.
- Limitations and uncertainty
- Vendor research by a company that sells infrastructure-assurance products and partners with NVIDIA; one controlled test platform (GH200) with weakened isolation and IPMI enabled; not demonstrated on every accelerator; no production exploitation reported.
NVIDIA blog on its Eclypsium collaboration: BlueField's configurable trust boundaries, DMA protections and treating DPUs, GPUs and BMCs as first-class security assets
NVIDIA's Oct 29 post says it partnered with Eclypsium to demonstrate how compromised infrastructure could change a running model while the model file, hash and signature stayed valid, and that the challenge is not unique to any vendor. It describes BlueField's authenticated boot, isolation, attestation and rollback protection, and stresses that security depends on configuration: operators choose which interfaces to expose. It recommends treating each DPU, GPU and management controller as an asset with its own firmware, configuration and owner, adding BMC and Redfish logs to monitoring, and reassessing devices after incident response and before reassignment.
Why it matters, evidence and limitations
- Why it matters
- Gives public-sector infrastructure teams NVIDIA's own position that BlueField and GPU servers must be configured and monitored, not assumed secure, which is useful when agencies or universities write hardening standards for AI hardware.
- Evidence and measured results
- Vendor blog; describes capabilities and recommended practices, with no measured results or independent testing.
- Limitations and uncertainty
- Vendor-authored and promotional; NVIDIA's spokesperson told CNN nothing new was identified. Capability descriptions are not evidence that defaults are secure.
CNN (via Scripps News): researchers tamper with AI hardware to skew chatbot answers; NVIDIA says nothing new, independent experts call findings credible
CNN reports that Eclypsium altered model weights on an NVIDIA platform to raise a model's estimated probability of a Chinese invasion of Taiwan and to tell a simulated chest-pain patient to stay home. The attacker generally needs network access to a DPU or BMC, and certain security configurations must be disabled. An NVIDIA spokesperson said the company did not identify anything the industry was not previously aware of and that strong memory protections block the path. Two independent hardware security experts called the findings credible. Eclypsium said it briefed NSA and CISA; neither had commented.
Why it matters, evidence and limitations
- Why it matters
- Health, public safety and benefits agencies piloting AI assistants need to know that harmful output can come from the infrastructure layer, not only from the model or prompts. The piece also gives balanced viewpoints for briefing agency leadership without overstating the risk.
- Evidence and measured results
- Independent news reporting with comments from the vendor under test and two outside experts; no new testing by CNN.
- Limitations and uncertainty
- Exclusive access story based on vendor research; examples are demonstrations, not real incidents; federal agencies had not commented at publication; Scripps republishes CNN Wire content.
CISA and NSA joint guidance on hardening baseboard management controllers, newly relevant as BMCs become an AI-server attack path
This 2023 CISA alert announces joint CISA and NSA guidance on hardening baseboard management controllers. It explains that BMCs run separately from the operating system and allow remote management even when a system is shut down, and that hardened credentials, firmware updates and network segmentation are often overlooked, leaving BMCs vulnerable. It encourages all organizations managing servers to apply the recommended actions. It is newly relevant because the Oct 29 Eclypsium research uses BMC and DPU access as an entry point to alter AI model weights in memory.
Why it matters, evidence and limitations
- Why it matters
- Gives state and local agencies an existing federal baseline they can apply right away to GPU servers, without waiting for AI-specific guidance. Agencies can cite it in hardening standards and contracts.
- Evidence and measured results
- Federal guidance from CISA and NSA; recommended practices rather than measured outcomes.
- Limitations and uncertainty
- Published in 2023 and not AI-specific; filed as newly relevant context, not as a new announcement. The linked full guidance document was not retrievable (403).
Princeton preprint: fine-grained DCGM monitoring of a 319-GPU university cluster finds most jobs use a small fraction of GPU memory and compute
Preprint, not peer reviewed. Princeton researchers built a lightweight monitoring tool on NVIDIA's DCGM library that samples GPU activity every 100 ms, and deployed it on a university research cluster with 89 nodes and 319 GPUs. In a 10-day window covering 7,399 jobs from 150 users in 12 departments, over 70% of jobs used less than 20% of GPU memory and half averaged below 5% SM utilization. Eighty percent of jobs had a large gap between requested and actual run time, limiting scheduler effectiveness. The authors argue better scheduling and GPU sharing could yield substantial gains without new hardware. The monitoring tool is open source.
Why it matters, evidence and limitations
- Why it matters
- Public universities and state research computing centers are buying expensive NVIDIA GPU capacity. This study gives them a measurement method and evidence that utilization, not only capacity, should drive investment and allocation decisions before expansion.
- Evidence and measured results
- Academic measurement study on one university cluster with an open-source tool; preprint, not peer reviewed.
- Limitations and uncertainty
- Preprint, not peer reviewed. Originally written in fall 2024 on A100 GPUs; detailed job-level analysis covers only 10 days on one cluster, and results are aggregated across departments; transferability to other campuses and newer GPUs is untested.
NVIDIA and EPFL: cuBLAS FP32 emulation on Tensor Cores speeds machine-learning interatomic potentials by up to 1.7x on Blackwell and 1.5x on A100 with one environment variable
NVIDIA's Oct 29 technical post, co-authored with EPFL's Michele Ceriotti, reports that cuBLAS FP32 emulation can run single-precision matrix math on Tensor Cores without code changes, enabled by setting CUBLAS_EMULATE_SINGLE_PRECISION=1. On Blackwell, BF16x9 emulation sped large GEMMs in the PET model by 2.2-2.5x for a 17,280-atom system and the full molecular dynamics step by up to 1.7x. On A100, TF32x3 sped GEMMs by 1.7-1.9x and the MD step by up to 1.5x. The authors say end-to-end gains depend on system size and GEMM share, and TF32x3 should be validated against accuracy requirements.
Why it matters, evidence and limitations
- Why it matters
- University and national-lab-affiliated materials science groups on existing A100 and Blackwell clusters may get faster simulations without new hardware, extending the value of public research computing investments.
- Evidence and measured results
- Vendor technical blog with co-author benchmarks on specific models and systems; no independent replication.
- Limitations and uncertainty
- Vendor-authored with benchmarks on a single model family; gains vary with system size; TF32x3 trades some accuracy and must be validated; combined gains with other optimizations were not measured.
ETH Zurich preprint: FlashSO(2) uses integer Ozaki-scheme emulation on GH200 to train a materials model 3.9x faster with 58% less memory
Preprint, not peer reviewed. ETH Zurich researchers introduce FlashSO(2), which emulates higher-precision math with low-precision integer operations (the Ozaki scheme) to accelerate equivariant graph neural network layers used in materials models. On an NVIDIA GH200 they report up to 236 effective Tflop/s, 3.5x the FP32 peak. Training the MALOQ model ran 3.9x faster than pure FP32 while cutting peak memory by 58% with no accuracy loss, and molecular dynamics with UMA ran 3.3x faster with 51% less memory. Code will be released upon publication.
Why it matters, evidence and limitations
- Why it matters
- Independent academic evidence, alongside NVIDIA's own post, that precision emulation can stretch existing GPU capacity for public research. It is relevant to university and state-funded HPC centers supporting materials and energy research, though the work is Swiss and not yet reproducible.
- Evidence and measured results
- Academic preprint with benchmarks on GH200; code not yet released; not peer reviewed.
- Limitations and uncertainty
- Preprint, not peer reviewed; Swiss academic setting, so transferability to U.S. public institutions' clusters and workloads is untested; code not yet released; authors disclose using Claude to improve grammar and readability.
How to read this edition
Source findings, measured results and limitations come from the cited publications. Patterns, operating questions, role takeaways and implementation considerations are Lighthouse Advisory interpretation, stated as questions to validate locally rather than guaranteed outcomes. Vendor and operator claims are labeled as claims. Full research method.
- Vendor claim
- A supplier-provided assertion that has not been upgraded to independent evidence.
- Academic research
- Research produced through an academic institution or peer-reviewed venue.
- Independent reporting
- Independent reporting with attributable sources but without a formal evaluation design.
- Standards or public-body guidance
- Normative or advisory guidance from a standards body or public institution.