{"resourceId":"nvidia-gpu-operator-26-7-1-notes-2026","versions":[{"version":"external-8818e74dd04fbba1000b551c8ed227ba233e239d008e3af3f4c323e24a328254","resource":{"id":"nvidia-gpu-operator-26-7-1-notes-2026","title":"GPU Operator 26.7.1: CUDA memory limits, driver 615.71.09, H100 MIG upgrade known issue","organization":"NVIDIA","sector":"Kubernetes GPU infrastructure","geography":"Global","publishedAt":"GPU Operator 26.7.1 release notes (page undated; inspected 2026-09-24)","publicationDate":null,"eventDate":null,"sourceName":"NVIDIA GPU Operator Docs","sourceLabel":"Vendor standards / release notes","sourceUrl":"https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/26.7/release-notes.html","evidenceClass":"standards-guidance","outcomeClass":"cautionary","topics":["infrastructure","developers-agents","data-security"],"finding":"GPU Operator 26.7.1 release notes add soft/hard per-GPU CUDA memory limits via NVIDIA_GPU_MEMORY_REQUEST and NVIDIA_GPU_MEMORY_LIMIT (MiB; requires R615+ driver), support for Data Center driver 615.71.09, Confidential Computing Manager v0.4.4, and operand wait-for-driver-validation fixes. A known issue warns that on H100 nodes, upgrading drivers from 595.91.07 to 615.71.09 without reboot or GPU reset can break subsequent MIG reconfiguration (NVML ERROR_UNKNOWN / Xid 119), with documented reboot or nvidia-smi -r workarounds. 26.7.0 previously introduced GPUCluster DRA management mutually exclusive with ClusterPolicy. Newly relevant beside NVCRE blogs that list GPU Operator as a prerequisite.","sledRelevance":"Campus/agency Kubernetes GPU fleets need memory-limit controls and must plan H100 MIG driver upgrade downtime—operational constraint evidence, not a SLED outcome study.","evidence":"Primary release notes detailing 26.7.1 features/fixes and H100 MIG known issue steps.","architectureImplications":"Enable memory request/limit env only with R615+; treat GPUCluster vs ClusterPolicy as exclusive modes; schedule MIG-sensitive upgrades with reset windows.","governanceImplications":"Change control for driver upgrades must include MIG validation gates on H100.","securityPrivacyImplications":"CC Manager bump is incremental; memory limits aid noisy-neighbor control but are not a TEE.","caveats":"Vendor docs; publicationDate not stated on page; behavior depends on cluster runtime/CDI configuration.","streamIds":["nvidia"],"roles":{"sales":"Position 26.7.1 as ops hygiene: memory limits for multi-tenant GPUs and a concrete H100 MIG upgrade footgun. Tie to NVCRE prerequisites without overselling DRA. Bound engagement to upgrade runbook review.","engineering":"Fit for K8s GPU platforms on 26.7 line. Architecture: pick ClusterPolicy or GPUCluster, not both; require R615+ for memory limits. Constraints: H100 MIG known issue. Proof: post-upgrade MIG reconfig succeeds; memory limits enforced in test pods.","delivery":"Work: stage driver upgrade with reboot/reset procedure; enable memory env in policies; regression-test MIG. Ownership: platform SRE. Risks: silent MIG breakage after driver bump."},"roleDetails":{"sales":{"customerProblem":"Multi-tenant GPU clusters need memory controls and safe driver upgrades.","stakeholders":["platform SRE","research computing","security"],"discoveryQuestions":["H100 MIG in use?","Driver branch today?","Need per-GPU memory caps?"],"valueHypothesis":"26.7.1 controls reduce neighbor risk if upgrade runbooks include MIG checks.","potentialEngagement":"Upgrade readiness review for one H100 pool.","unsupportedClaims":["Memory limits equal confidential computing","DRA GPUCluster is always preferred"]},"engineering":{"fit":"Operational guidance for 26.7.x fleets.","architectureIntegration":"Operator-managed drivers/operands; exclusive ClusterPolicy vs GPUCluster.","prerequisites":["R615+ for memory limits","CDI-compatible runtime for some DRA paths"],"constraints":["H100 MIG upgrade known issue","Mutual exclusion of management modes"],"security":"Memory caps are isolation aids, not TEEs; CC Manager is separate.","proofOfValue":["MIG reconfig after upgrade","Memory limit rejected oversize alloc","Operands healthy"]},"delivery":{"work":"Apply 26.7.1; execute H100 upgrade SOP; validate MIG and memory limits.","dependencies":["node maintenance windows","workload owners"],"ownership":"Kubernetes platform team.","skillsAdoption":"Train on nvidia-smi reset and MIG state labels.","governanceCheckpoints":["CAB for driver upgrade","Post-change MIG test","Rollback driver pin"],"acceptanceCriteria":["No Xid 119 on MIG ops","Memory envs honored","ClusterPolicy ready"],"risks":["Skipped reboot","Mixed MIG/full-GPU scheduler issues on older drivers"]}},"retrievedAt":"2026-09-25T03:02:43Z","enrichedAt":"2026-09-25T03:05:00Z","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Ops procedure complexity; limited end-user accessibility impact.","procurementImplications":"Specify supported Operator/driver pairs and maintenance windows in contracts.","operatingModelImplications":"Driver/MIG changes need formal change control tied to AI capacity calendars.","sourceVerification":{"openedUrl":"https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/26.7/release-notes.html","referenceExcerpt":"On nodes with NVIDIA H100 GPUs, upgrading the driver from 595.91.07 to 615.71.09 without rebooting","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}