From the NVIDIA edition of October 27, 2026
Rice/Minnesota study finds a fault in one NVIDIA MPS process can terminate all co-running processes, and prototypes isolation and fast recovery
Rice University; University of Minnesota · Higher education research (GPU systems) · United States
- Publisher
- arXiv
- Original publication
- 2026-05-26
- Source retrieved
- 2026-10-28
What happened
The authors characterize MPS fault propagation and state that MPS has weak fault resilience because one process's fault can terminate all co-running processes, limiting adoption in multi-tenant GPU clusters. Their research prototype, evaluated on NVIDIA L40 and H100 GPUs, contained all propagating MMU fault combinations with zero throughput overhead and cut SM-fault recovery from 73.4-1359.1 ms to 31.1-80.6 ms, with under 1% fault-free overhead.
Why it matters
Newly relevant because NVIDIA's Oct 27 CUDA 13.4 post offers an SM-sharing mode with a shared error domain; campus clusters weighing that mode need independent evidence of how faults propagate between co-located jobs.
Evidence and measured results
Academic preprint with a characterization study and prototype evaluation on L40 and H100 GPUs; not peer reviewed as inspected.
Limitations and uncertainty
Preprint; the paper's mechanisms are research prototypes, and the inspected text does not say they ship in NVIDIA software. It studies MPS directly; the relationship to CUDA 13.4 SM-sharing is inferred from NVIDIA's own shared-error-domain description.
Put this evidence to work
Lighthouse Advisory interpretation, grounded in this source. Enriched 2026-10-28; this does not change the original publication date. Each role follows the same checklist so takeaways compare across sources.
Sales
Role takeaway
- Customer problem
- Consolidating jobs onto shared GPUs can let one faulty job terminate others, undermining service reliability.
- Stakeholders
- Research computing director
- HPC user-services lead
- Faculty PIs running long jobs
- Discovery questions
- Which jobs tolerate being restarted?
- Do users checkpoint today?
- What uptime commitments exist for shared queues?
- Value hypothesis
- Tenancy tiers and resilience rules let institutions gain sharing benefits without surprising researchers with lost jobs.
- Potential engagement
- Shared-GPU resilience design workshop and fault-injection test.
- Unsupported claims
- Prototype recovery results are not product features.
- No direct study of CUDA 13.4 SM-sharing was found.
Pre-sales engineering
Role takeaway
- Fit
- Clusters considering MPS or SM-sharing for inference or small training jobs.
- Architecture and integration
- Policy-based placement across time-sliced, shared-SM and whole-GPU pools with checkpoint/requeue.
- Prerequisites
- Checkpoint support in user workflows
- Scheduler requeue policies
- Fault-injection capability
- Constraints
- Fault in one process can terminate co-running processes
- Mitigations are research prototypes
- Security
- Treat shared-error-domain pools as an availability risk; keep critical jobs isolated.
- Proof of value
- Fault-injection drill on campus GPUs
- Requeue success rate after injected faults
Delivery
Role takeaway
- Work
- Define shared-pool rules, checkpoint guidance, incident procedure and a fault drill.
- Dependencies
- Scheduler configuration
- User communication channels
- Ownership
- Research computing operations.
- Skills and adoption
- Teach users checkpointing and what a co-tenant fault looks like.
- Governance checkpoints
- Governance approval of shared-pool rules
- Drill completed before production
- Acceptance criteria
- Requeue works after injected fault
- User documentation published
- Risks
- Lost research time from co-tenant faults
- Hard-to-attribute incidents
Implementation considerations
Lighthouse Advisory interpretation across the operating dimensions a public-sector buyer must settle before this evidence becomes a design. Each note answers the question under its heading for this specific source.
Architecture and integration
What must connect, and where does the AI sit in the workflow?
Shared-SM sharing modes inherit fault-propagation risk; mitigation in the paper relies on research mechanisms (including driver-level changes) rather than shipping product features.
Governance
Who approves, reviews and stays accountable for outcomes?
Supports a policy of placing only mutually trusted, fault-tolerant workloads in shared-error-domain pools.
Security and privacy
What data, permissions and controls need testing?
A fault in one tenant can terminate others, an availability risk for multi-tenant research clusters; it does not on its own show data leakage.
Accessibility and workforce
Who is affected, and what skills or accommodations follow?
Researchers sharing GPUs should be told that a co-tenant's fault may end their job and should checkpoint accordingly.
Procurement
What should contracts, pricing and exit terms secure?
Buyers should ask vendors what fault-containment guarantees apply to shared-SM modes before committing to consolidation-based capacity plans.
Operating model
Which teams own the service once it runs?
Shared pools need checkpointing expectations, job-retry policy and incident attribution procedures.
Publication history
- 2026-10-27NVIDIA · Issue 5212 resources
Stable resource ID: rice-umn-arxiv-mps-fault-resilience-multi-tenant-gpu-20261027