{"resourceId":"kai-scheduler-gpu-fractioning-readme-prerequisites-silent-failure-20261027","versions":[{"version":"external-746933e6694b1700656200a0ec34642fb8dda1c78962dda27581be376307d648","resource":{"id":"kai-scheduler-gpu-fractioning-readme-prerequisites-silent-failure-20261027","title":"GPU Fractioning project README lists strict version prerequisites and warns that overriding checks leaves GPU memory unfenced without any report","organization":"KAI Scheduler project (NVIDIA-led open source)","sector":"Research computing / shared GPU clusters (open-source project documentation)","geography":"Global","publishedAt":"Undated project README (retrieved 2026-10-27)","publicationDate":null,"eventDate":null,"sourceName":"GitHub: kai-scheduler/gpu-fractioning","sourceLabel":"gpu-fractioning README","sourceUrl":"https://github.com/kai-scheduler/gpu-fractioning/","evidenceClass":"vendor-claim","outcomeClass":"cautionary","topics":["infrastructure","data-security","operating-model"],"finding":"The project's own README requires Kubernetes 1.28+, containerd 2.0+ with NRI (or CRI-O), GPU Operator v26.7.0+ and the r615 driver branch, which is short-lived and not installed by default by GPU Operator v26.7.1. A container that exceeds its limit is terminated as out-of-memory; disabling the version gates lets daemons roll out on a stack that cannot enforce limits, leaving GPU memory unfenced with nothing reported, and metrics can degrade silently to zeros.","sledRelevance":"Campus and consortium platform teams evaluating the CUDA 13.4 sharing feature need to know that enforcement depends on a narrow, non-default driver and runtime stack, and that misconfiguration fails open rather than loudly.","evidence":"Project documentation; describes behavior and requirements, not measured deployments.","architectureImplications":"Pins clusters to specific runtime (containerd NRI or CRI-O), GPU Operator and a non-default short-lived driver branch, which affects patch cadence and long-term support planning.","governanceImplications":"Change control should forbid skipping version checks in production and require evidence that limits are enforced before declaring a pool 'fenced'.","securityPrivacyImplications":"Fail-open behavior (unfenced memory with no report) means isolation claims must be verified by test, not assumed from configuration.","caveats":"README is undated and may change; it is project documentation rather than an independent evaluation.","streamIds":["nvidia"],"roles":{"sales":"Use this README to set expectations honestly with research computing buyers: the sharing feature is promising but depends on a narrow software stack and can fail silently. That makes a readiness assessment more valuable than a quick enablement. Ask whether the institution can run a non-default, short-lived driver branch under its support agreements. Frame the offer as de-risking: verifying prerequisites, designing tests that prove limits are enforced, and building monitoring so a misconfigured pool cannot look healthy while running unfenced. Avoid implying it is a turnkey production capability today.","engineering":"Engineering must check every prerequisite before piloting: Kubernetes 1.28 or later, containerd 2.0 with NRI or CRI-O, GPU Operator v26.7.0 or later and the r615 driver branch, which v26.7.1 does not install by default. Never set the option that skips stack version checks in production, because the README says memory then goes unfenced with nothing reported. Build an automated test that allocates past the limit and confirms out-of-memory termination. Alert when every reported GPU metric is zero, since the metrics daemon can degrade silently, and document the driver lifecycle plan.","delivery":"Delivery should add a prerequisite gate to the pilot plan, with sign-off from platform and security leads confirming runtime, operator and driver versions. Treat the short-lived driver branch as a lifecycle risk and schedule a review when it is superseded. Include an enforcement test and a telemetry-health alert in acceptance criteria, and run them after every upgrade. Record in the runbook that skipping version checks is prohibited. Keep whole-GPU allocation as the documented fallback if enforcement cannot be demonstrated on the institution's supported stack."},"roleDetails":{"sales":{"customerProblem":"Institutions may assume fractional sharing is enforced once enabled, when enforcement depends on a narrow stack and can fail open.","stakeholders":["Research computing director","Platform engineering lead","Security officer","Hardware support/vendor manager"],"discoveryQuestions":["Can you run a non-default, short-lived driver branch under your support terms?","Which container runtime and version do you run?","How would you detect unfenced memory today?"],"valueHypothesis":"A readiness and verification engagement prevents a pool from running unfenced while appearing healthy.","potentialEngagement":"Prerequisite audit plus enforcement-test and monitoring build.","unsupportedClaims":["No production deployment results are documented.","No statement on long-term support of the r615 branch was found."]},"engineering":{"fit":"Clusters already on recent Kubernetes and containerd with NRI.","architectureIntegration":"GPU Operator v26.7.x, r615 driver, NRI-enabled runtime, KAI Scheduler fractions.","prerequisites":["Kubernetes 1.28+","containerd 2.0+ with NRI or CRI-O","GPU Operator v26.7.0+","r615 driver branch"],"constraints":["r615 not installed by default by GPU Operator v26.7.1","Skipping checks leaves memory unfenced silently","Metrics can degrade silently to zero"],"security":"Verify enforcement by test; alert on zeroed telemetry; prohibit skipping version gates.","proofOfValue":["Automated over-limit allocation test","Telemetry-health alert firing in a drill","Version audit report"]},"delivery":{"work":"Gate the pilot on verified prerequisites and add enforcement and telemetry checks to operations.","dependencies":["Platform team upgrade capacity","Vendor/OEM support confirmation for driver branch"],"ownership":"Platform engineering, with security sign-off.","skillsAdoption":"Runbook training on enforcement tests and telemetry alerts.","governanceCheckpoints":["Prerequisite version sign-off","Post-upgrade enforcement re-test"],"acceptanceCriteria":["Over-limit test terminates the container","Zeroed-metrics alert verified"],"risks":["Driver branch lifecycle churn","Fail-open misconfiguration","Silent monitoring gaps"]}},"retrievedAt":"2026-10-28T03:03:14Z","enrichedAt":"2026-10-28T03:12:33Z","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Requires staff comfortable with container runtime internals (NRI) and driver lifecycle management.","procurementImplications":"Short-lived driver branches can conflict with institutional support or OEM certification requirements; check vendor support terms before adopting.","operatingModelImplications":"Operations must add active monitoring that detects zeroed metrics and periodic enforcement tests to runbooks.","sourceVerification":{"openedUrl":"https://github.com/kai-scheduler/gpu-fractioning/","referenceExcerpt":"A container that exceeds its limit is terminated (out-of-memory), the same way a container exceeding its Kubernetes memory limit is.","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}