From the NVIDIA edition of November 10, 2026
Start and Restore GPU Workloads in Seconds with NVIDIA Snapshot
NVIDIA · AI inference infrastructure software · Global
- Publisher
- NVIDIA Technical Blog
- Original publication
- November 10, 2026
- Source retrieved
- 2026-11-11
- Event date
- 2026-11-10
What happened
NVIDIA Snapshot is an open source checkpoint and restore system for GPU workloads on Kubernetes. On NVIDIA B200 GPUs with VAST NFS storage, NVIDIA reports restores of 3.6 to 42.7 seconds, 8x to 36x faster than cold starts of about 87 seconds to 18.5 minutes across vLLM, SGLang and TensorRT LLM, including models with checkpoints up to about 1 TB over eight GPUs. Checkpoint creation took 15 seconds to 5.5 minutes, and checkpoint plus restore was shorter than one cold start in every tested configuration. Design partners report Crusoe cutting GLM-5.3 readiness from about eight minutes to 31 seconds and Saturn Cloud cutting vLLM startup from 2 to 8.5 minutes to 6 to 34 seconds, both on HGX B300.
Why it matters
Limited and indirect. Faster inference scale-up and recovery matters to operators of shared GPU services, such as university research computing or state-run AI platforms, but the post offers no SLED deployment evidence.
Evidence and measured results
Vendor post with detailed tables for one GPU type (B200), one storage backend (VAST NFS) and specified engine versions; excludes container image pull and provisioning from the measured path. Partner numbers are summarized without test conditions in the post.
Limitations and uncertainty
Vendor-run benchmarks on one GPU model and one storage backend; same-GPU-and-driver restore requirement; workloads must meet compatibility requirements; image pull and provisioning time excluded; not independently reproduced.
Put this evidence to work
Lighthouse Advisory interpretation, grounded in this source. Enriched 2026-11-10; this does not change the original publication date. Each role follows the same checklist so takeaways compare across sources.
Sales
Role takeaway
- Customer problem
- Inference capacity is slow to come online after demand spikes or failures even when GPUs are free.
- Stakeholders
- Research computing or AI platform lead
- Infrastructure architect
- Security reviewer
- Budget owner
- Discovery questions
- How long does a replica take to become ready today?
- Which engines, models and storage do you use?
- How often do scale events or failures occur?
- Value hypothesis
- Restoring initialized replicas may cut startup and recovery time and allow less idle capacity, if conditions resemble the tested setup.
- Potential engagement
- Time-boxed Snapshot pilot on one model with baseline and restore measurements.
- Unsupported claims
- That speedups apply to other GPUs or storage
- That Snapshot lowers cost
- Partner results under unstated test conditions
Pre-sales engineering
Role takeaway
- Fit
- Fits Kubernetes inference fleets using vLLM, SGLang or TensorRT LLM on identical GPU and driver combinations.
- Architecture and integration
- PodSnapshot and SnapshotJob resources, nvidia.com/restore-from annotation, Dynamo integration, shared storage for artifacts.
- Prerequisites
- Same GPU model and driver on restore nodes
- Fast shared storage
- Compatible inference engine workload
- Kubernetes cluster
- Constraints
- Tested on B200 with VAST NFS only in the tables
- Checkpoint sizes up to about 1 TB
- Image pull and provisioning excluded
- Security
- Artifacts contain CPU and GPU memory; assess for sensitive content, encrypt and control access.
- Proof of value
- Measure end-to-end ready time cold versus restore
- Test restore on a different node with identical GPU and driver
- Test failure recovery under load
Delivery
Role takeaway
- Work
- Pilot checkpoint and restore on one inference service with measured baselines.
- Dependencies
- Compatible GPUs and drivers
- Shared storage
- Kubernetes admin access
- Ownership
- Platform team owns artifacts and restore runbook.
- Skills and adoption
- Kubernetes and inference-engine operations skills required.
- Governance checkpoints
- Security review of snapshot contents
- Artifact retention policy
- Pilot results review
- Acceptance criteria
- Proposed: end-to-end ready time compared with baseline
- Proposed: successful restore after simulated failure
- Risks
- Results not transferable
- Artifact sensitivity
- Driver or GPU drift
Implementation considerations
Lighthouse Advisory interpretation across the operating dimensions a public-sector buyer must settle before this evidence becomes a design. Each note answers the question under its heading for this specific source.
Architecture and integration
What must connect, and where does the AI sit in the workflow?
Uses Kubernetes custom resources (PodSnapshot, SnapshotJob) and a restore annotation; restore requires nodes with the same GPU model and driver; workloads need a small pause and resume handshake; examples exist for vLLM, SGLang and TensorRT LLM. Storage speed and GPU state copy dominate large restores.
Governance
Who approves, reviews and stays accountable for outcomes?
Checkpoint artifacts of hundreds of GB are new data assets that need retention, access control and versioning decisions.
Security and privacy
What data, permissions and controls need testing?
Snapshots capture CPU and GPU memory; the post does not discuss whether artifacts could contain sensitive prompts, keys or user data. This needs separate review before use with regulated workloads.
Accessibility and workforce
Who is affected, and what skills or accommodations follow?
Benefits accrue mainly to teams already operating inference platforms; it does not directly change accessibility, and adoption needs Kubernetes and inference-ops skills.
Procurement
What should contracts, pricing and exit terms secure?
Open source reduces license friction, but value depends on GPU, driver and storage choices already purchased; ask vendors for reproducible benchmarks on your configuration rather than relying on the published B200 table.
Operating model
Which teams own the service once it runs?
Introduces snapshot artifacts as managed assets and shifts capacity planning from keeping warm replicas toward restoring on demand, which needs clear ownership and monitoring.
Publication history
- 2026-11-10NVIDIA · Issue 665 resources
Stable resource ID: nvidia-snapshot-gpu-checkpoint-restore-benchmarks-20261110