{"resourceId":"arxiv-2610-07688-failbench-no-universal-fault-tolerance-mechanism-20261102","versions":[{"version":"external-e1c4c959232b1a856a746defcf560b286b9241c159f8688e8ba5e3e08793e2e3","resource":{"id":"arxiv-2610-07688-failbench-no-universal-fault-tolerance-mechanism-20261102","title":"FailBench study finds no fault-tolerance mechanism is best across distributed training architectures; costs range from 3.7% to 176%","organization":"Khaled Aljbab and Amine Barrak (arXiv preprint)","sector":"Academic research on distributed training reliability","geography":"Global","publishedAt":"2026-10-06","publicationDate":"2026-10-06","eventDate":"2026-10-06","sourceName":"arXiv","sourceLabel":"arXiv:2610.07688 FailBench","sourceUrl":"https://arxiv.org/abs/2610.07688","evidenceClass":"academic-research","outcomeClass":"mixed","topics":["infrastructure"],"finding":"FailBench evaluates 142 combinations of seven distributed training architectures, eight crash-fault-tolerance mechanisms plus a no-fault-tolerance baseline, and single, concurrent and cascading fail-stop failures on an 8xV100 cluster. No mechanism is best everywhere: on one architecture, disk checkpointing has the lowest steady-state overhead (0.5%), in-memory replication restores fastest (~17 ms), and just-in-time checkpointing avoids periodic cost but takes ~0.9 s on failure. In-memory replication overhead ranges from 3.7% to 176% depending on architecture. Gossip training raised sample throughput 17.9% after losing a worker but showed no detectable improvement in loss progress. The authors release a decision framework and the harness.","sledRelevance":"Independent counterweight to vendor recovery tools released Nov 2: university HPC centres should expect the best recovery approach to depend on their workloads, and should measure training progress, not throughput or action latency alone. Published Oct 6; newly relevant because of NVIDIA's Recovery release.","evidence":"Preprint, not peer reviewed; small 8xV100 testbed; fail-stop failures only.","architectureImplications":"Fault-tolerance choice should follow parallelism strategy and checkpoint size.","governanceImplications":"Evaluation criteria should include loss progress, not just throughput.","securityPrivacyImplications":"No security content.","caveats":"Small, older-GPU testbed; results may not transfer to large NVLink clusters; does not evaluate NVIDIA Recovery.","streamIds":["nvidia"],"roles":{"sales":"Use this study to keep recovery conversations honest. It shows no single fault-tolerance approach wins everywhere, and one that raised throughput did not improve training progress. For research-computing clients evaluating NVIDIA Recovery or similar tools, offer to design an evaluation that measures training progress on their own workloads. Do not present it as evidence for or against NVIDIA's tools, which it did not test, and do not generalise its small V100 results to large modern clusters.","engineering":"Read this as a measurement guide. Overheads vary from 3.7% to 176% for the same mechanism across architectures, so test on your own parallelism mix and checkpoint sizes. Track loss progress alongside throughput, since the two diverged for gossip training. The FailBench harness is open and can be adapted to a site's workloads. Limits: an 8xV100 testbed and fail-stop failures only, with no hangs or stragglers, which NVIDIA Recovery targets. Use it to frame proof-of-value metrics for any recovery tool.","delivery":"Fold this into evaluation design. Before adopting cluster-level recovery, the research-computing team should define metrics: steady-state overhead, restore time and training progress after failures, by workload type. Ownership sits with the HPC operations lead together with key research groups. Staff need some benchmarking skill to adapt the harness. Risks include tuning for throughput while training progress stalls, and over-relying on small-testbed results."},"roleDetails":{"sales":{"customerProblem":"Buyers lack independent evidence for picking fault-tolerance approaches.","stakeholders":["HPC operations lead","Research group leads"],"discoveryQuestions":["Which parallelism strategies do your main workloads use?","How do you measure training progress after failures?"],"valueHypothesis":"Workload-specific evaluation avoids adopting a mechanism that adds overhead without helping progress.","potentialEngagement":"Design a site-specific recovery evaluation using open benchmarks.","unsupportedClaims":["Any conclusion about NVIDIA Recovery","Results at large scale"]},"engineering":{"fit":"Informs evaluation of any training recovery tooling.","architectureIntegration":"Open benchmarking harness across seven architectures and eight mechanisms.","prerequisites":["Test cluster time","Representative workloads"],"constraints":["8xV100 testbed","Fail-stop failures only","Preprint"],"security":"Not addressed.","proofOfValue":["Measure loss progress as well as throughput","Benchmark overhead per architecture"]},"delivery":{"work":"Define recovery evaluation metrics and run site benchmarks.","dependencies":["Cluster time","Research group cooperation"],"ownership":"HPC operations lead with research groups.","skillsAdoption":"Benchmarking and metrics skills.","governanceCheckpoints":["Metric definitions agreed before pilot"],"acceptanceCriteria":["Proposed: evaluation reports overhead, restore time and progress for each major workload type"],"risks":["Optimising the wrong metric","Over-generalising small-scale results"]}},"retrievedAt":"2026-11-03T04:00:38Z","enrichedAt":"2026-11-03T04:04:41Z","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Open harness lets academic teams test options themselves.","procurementImplications":"Supports asking vendors for workload-specific evidence rather than generic recovery claims.","operatingModelImplications":"Recovery policy needs per-workload tuning.","sourceVerification":{"openedUrl":"https://arxiv.org/abs/2610.07688","referenceExcerpt":"no mechanism is universally best","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}