{"resourceId":"rice-umn-arxiv-mps-fault-resilience-multi-tenant-gpu-20261027","versions":[{"version":"external-746933e6694b1700656200a0ec34642fb8dda1c78962dda27581be376307d648","resource":{"id":"rice-umn-arxiv-mps-fault-resilience-multi-tenant-gpu-20261027","title":"Rice/Minnesota study finds a fault in one NVIDIA MPS process can terminate all co-running processes, and prototypes isolation and fast recovery","organization":"Rice University; University of Minnesota","sector":"Higher education research (GPU systems)","geography":"United States","publishedAt":"2026-05-26","publicationDate":"2026-05-26","eventDate":null,"sourceName":"arXiv","sourceLabel":"Characterization-Guided GPU Fault Resilience in NVIDIA MPS (arXiv 2605.26461v1)","sourceUrl":"https://arxiv.org/html/2605.26461v1","evidenceClass":"academic-research","outcomeClass":"cautionary","topics":["infrastructure","data-security"],"finding":"The authors characterize MPS fault propagation and state that MPS has weak fault resilience because one process's fault can terminate all co-running processes, limiting adoption in multi-tenant GPU clusters. Their research prototype, evaluated on NVIDIA L40 and H100 GPUs, contained all propagating MMU fault combinations with zero throughput overhead and cut SM-fault recovery from 73.4-1359.1 ms to 31.1-80.6 ms, with under 1% fault-free overhead.","sledRelevance":"Newly relevant because NVIDIA's Oct 27 CUDA 13.4 post offers an SM-sharing mode with a shared error domain; campus clusters weighing that mode need independent evidence of how faults propagate between co-located jobs.","evidence":"Academic preprint with a characterization study and prototype evaluation on L40 and H100 GPUs; not peer reviewed as inspected.","architectureImplications":"Shared-SM sharing modes inherit fault-propagation risk; mitigation in the paper relies on research mechanisms (including driver-level changes) rather than shipping product features.","governanceImplications":"Supports a policy of placing only mutually trusted, fault-tolerant workloads in shared-error-domain pools.","securityPrivacyImplications":"A fault in one tenant can terminate others, an availability risk for multi-tenant research clusters; it does not on its own show data leakage.","caveats":"Preprint; the paper's mechanisms are research prototypes, and the inspected text does not say they ship in NVIDIA software. It studies MPS directly; the relationship to CUDA 13.4 SM-sharing is inferred from NVIDIA's own shared-error-domain description.","streamIds":["nvidia"],"roles":{"sales":"This study gives sales an independent, academic basis for a candid conversation about GPU sharing on campus clusters. When a research computing leader asks about packing more jobs per GPU, explain that shared-error-domain modes can let one job's fault end others, per Rice and Minnesota researchers. That reframes the opportunity as designing tenancy tiers and resilience practices, not just enabling a feature. It also positions the team as evidence-led. Do not cite the paper's prototype recovery numbers as product capabilities; they come from research mechanisms, not shipped NVIDIA software.","engineering":"Engineering should treat shared-error-domain modes, including MPS and CUDA 13.4 SM-sharing, as availability trade-offs. The paper shows faults propagating across co-running processes on L40 and H100 GPUs. Its mitigations are research prototypes, so plan for jobs being killed by a co-tenant: require checkpointing, configure automatic requeue and keep long or critical jobs in time-sliced or whole-GPU pools. Reproduce a fault-injection test on campus hardware before production. Track which co-tenant caused a fault, because NVIDIA's MPS documentation says fatal faults are reported to all clients without identifying the cause.","delivery":"Delivery should translate the paper into operating rules: which queues may use shared-error-domain modes, what checkpoint interval users must adopt, and how requeues are handled after a co-tenant fault. Build an incident procedure that captures logs across co-located jobs, since attributing a fault may require correlation. Include a fault-injection drill in pilot acceptance and communicate the trade-off clearly in user documentation. Revisit the policy if NVIDIA ships fault-containment improvements, and record the decision rationale for the research computing governance committee."},"roleDetails":{"sales":{"customerProblem":"Consolidating jobs onto shared GPUs can let one faulty job terminate others, undermining service reliability.","stakeholders":["Research computing director","HPC user-services lead","Faculty PIs running long jobs"],"discoveryQuestions":["Which jobs tolerate being restarted?","Do users checkpoint today?","What uptime commitments exist for shared queues?"],"valueHypothesis":"Tenancy tiers and resilience rules let institutions gain sharing benefits without surprising researchers with lost jobs.","potentialEngagement":"Shared-GPU resilience design workshop and fault-injection test.","unsupportedClaims":["Prototype recovery results are not product features.","No direct study of CUDA 13.4 SM-sharing was found."]},"engineering":{"fit":"Clusters considering MPS or SM-sharing for inference or small training jobs.","architectureIntegration":"Policy-based placement across time-sliced, shared-SM and whole-GPU pools with checkpoint/requeue.","prerequisites":["Checkpoint support in user workflows","Scheduler requeue policies","Fault-injection capability"],"constraints":["Fault in one process can terminate co-running processes","Mitigations are research prototypes"],"security":"Treat shared-error-domain pools as an availability risk; keep critical jobs isolated.","proofOfValue":["Fault-injection drill on campus GPUs","Requeue success rate after injected faults"]},"delivery":{"work":"Define shared-pool rules, checkpoint guidance, incident procedure and a fault drill.","dependencies":["Scheduler configuration","User communication channels"],"ownership":"Research computing operations.","skillsAdoption":"Teach users checkpointing and what a co-tenant fault looks like.","governanceCheckpoints":["Governance approval of shared-pool rules","Drill completed before production"],"acceptanceCriteria":["Requeue works after injected fault","User documentation published"],"risks":["Lost research time from co-tenant faults","Hard-to-attribute incidents"]}},"retrievedAt":"2026-10-28T03:04:47Z","enrichedAt":"2026-10-28T03:12:33Z","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Researchers sharing GPUs should be told that a co-tenant's fault may end their job and should checkpoint accordingly.","procurementImplications":"Buyers should ask vendors what fault-containment guarantees apply to shared-SM modes before committing to consolidation-based capacity plans.","operatingModelImplications":"Shared pools need checkpointing expectations, job-retry policy and incident attribution procedures.","sourceVerification":{"openedUrl":"https://arxiv.org/html/2605.26461v1","referenceExcerpt":"MPS has weak fault resilience: a fault in one process can terminate all co-running processes","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}