{"resourceId":"arxiv-dynamo-gms-recovery-260925451","versions":[{"version":"external-a773ab94e74681c35080945cb66d6ea5beab0f76a39e43b49a61d6b378069255","resource":{"id":"arxiv-dynamo-gms-recovery-260925451","title":"arXiv:2609.25451 — Dynamo GMS decouples GPU memory lifetime for fast LLM recovery","organization":"arXiv (NVIDIA-affiliated authors)","sector":"Academic preprint / systems research","geography":"Global (research preprint)","publishedAt":"2026-09-21T00:00:00Z","publicationDate":"2026-09-21","eventDate":"2026-09-21","sourceName":"arXiv","sourceLabel":"Academic preprint","sourceUrl":"https://arxiv.org/abs/2609.25451","evidenceClass":"academic-research","outcomeClass":"emerging","topics":["infrastructure","developers-agents","operating-model"],"finding":"Submitted Sep 21, 2026, arXiv:2609.25451 (NVIDIA-affiliated authors) presents fast recovery for Dynamo by arguing most production failures are device-preserving: the engine process dies while GPU-resident allocations remain. Snapshots capture initialized engine state; the GPU Memory Service (GMS) decouples device-memory ownership from engine processes so replacements can reattach instead of reloading weights. Analysis cites 18 weeks of Dynamo cluster failures. Why now: pairs with the still-pinned Dynamo Enterprise 1.4.2 support boundary—research and OSS recovery advances should not be assumed Enterprise-ready without catalog movement.","sledRelevance":"Informs SLED platform SLOs for inference restart windows, while remaining a preprint (not a support commitment) and NVIDIA-authored.","evidence":"arXiv abs page dated Submitted on 21 Sep 2026; abstract states GMS decoupling and 18-week failure analysis framing.","architectureImplications":"Design serving for process-fail/device-preserve recovery; budget memory for GMS/sidecars; do not equate preprint with Enterprise pin.","governanceImplications":"Treat as research evidence until mirrored in supported Enterprise docs/artifacts.","securityPrivacyImplications":"Shared GPU memory services expand blast radius if isolation is weak—review multi-tenant mapping.","caveats":"NVIDIA-affiliated authors; preprint not peer-reviewed here; abs page lacks full numeric tables (PDF not fully extracted in this pass).","streamIds":["nvidia"],"roles":{"sales":"Use the Sep 21 preprint to discuss inference recovery ambitions without selling them as AI Enterprise features. Open with minute-scale cold restarts hurting SLOs, then cite GMS’s device-preserving insight—and immediately pair with the live Dynamo Enterprise 1.4.2 pin. Bound any commercial talk to a lab evaluation, not a contracted recovery-time objective.","engineering":"Fit for Dynamo lab clusters exploring GMS or shadow-style recovery; weak as an entitled production architecture until Enterprise artifacts document it. Integrate only on OSS or lab pins separate from production Helm. Prerequisites: a Dynamo lab stack with GMS prerequisites per companion docs. Constraints: preprint scope and memory overhead. Proof: measured failover time on buyer models versus cold start.","delivery":"Work: lab enablement, failure injection, measure promote/reattach times, and compare to cold start baselines. Dependencies: lab GPUs and observability. Ownership: inference platform team. Skills: CUDA virtual memory and GMS operational concepts. Governance: keep lab versus production pins strict. Acceptance: documented lab RTO with no production claim. Risks: memory fragmentation and multi-tenant isolation leaks."},"roleDetails":{"sales":{"customerProblem":"Inference worker crashes cause multi-minute capacity loss.","stakeholders":["platform SRE","app owners","vendor management"],"discoveryQuestions":["What is current cold-restart RTO?","Is Enterprise pin required for prod?"],"valueHypothesis":"Device-preserving recovery can shrink RTO in lab if memory budget allows.","potentialEngagement":"Lab failure-injection study with explicit non-prod scope.","unsupportedClaims":["Preprint guarantees Enterprise support","All failures are device-preserving"]},"engineering":{"fit":"Lab Dynamo recovery research; not default prod.","architectureIntegration":"GMS sidecar plus engine reattach; isolate from Enterprise 1.4.2 prod.","prerequisites":["Dynamo lab cluster","failure injection harness"],"constraints":["Preprint","Possible memory overhead","NVIDIA-authored"],"security":"Audit cross-process GPU memory sharing for tenant isolation.","proofOfValue":["Failover seconds vs cold start","Correctness after promote"]},"delivery":{"work":"Stand up lab, inject kills, record metrics, decide wait-for-Enterprise.","dependencies":["OSS Dynamo bits","metrics pipeline"],"ownership":"Inference platform lab owner.","skillsAdoption":"Train on GMS operational failure modes.","governanceCheckpoints":["Lab-only approval","No prod pin change"],"acceptanceCriteria":["RTO report","Isolation test notes"],"risks":["Premature prod enable","OOM from shadows"]}},"retrievedAt":"2026-09-26T03:01:17Z","enrichedAt":"2026-09-26T03:10:00Z","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Faster recovery can reduce user-visible outages for assisted services; not measured here.","procurementImplications":"Do not procure against preprint features; wait for Enterprise packaging.","operatingModelImplications":"Encourages explicit lab-vs-prod recovery roadmaps.","sourceVerification":{"openedUrl":"https://arxiv.org/abs/2609.25451","referenceExcerpt":"The GPU Memory Service (GMS) decouples device-memory ownership from engine processes","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}