From the NVIDIA edition of October 27, 2026
GPU Fractioning project README lists strict version prerequisites and warns that overriding checks leaves GPU memory unfenced without any report
KAI Scheduler project (NVIDIA-led open source) · Research computing / shared GPU clusters (open-source project documentation) · Global
- Publisher
- GitHub: kai-scheduler/gpu-fractioning
- Original publication
- Undated project README (retrieved 2026-10-27)
- Source retrieved
- 2026-10-28
What happened
The project's own README requires Kubernetes 1.28+, containerd 2.0+ with NRI (or CRI-O), GPU Operator v26.7.0+ and the r615 driver branch, which is short-lived and not installed by default by GPU Operator v26.7.1. A container that exceeds its limit is terminated as out-of-memory; disabling the version gates lets daemons roll out on a stack that cannot enforce limits, leaving GPU memory unfenced with nothing reported, and metrics can degrade silently to zeros.
Why it matters
Campus and consortium platform teams evaluating the CUDA 13.4 sharing feature need to know that enforcement depends on a narrow, non-default driver and runtime stack, and that misconfiguration fails open rather than loudly.
Evidence and measured results
Project documentation; describes behavior and requirements, not measured deployments.
Limitations and uncertainty
README is undated and may change; it is project documentation rather than an independent evaluation.
Put this evidence to work
Lighthouse Advisory interpretation, grounded in this source. Enriched 2026-10-28; this does not change the original publication date. Each role follows the same checklist so takeaways compare across sources.
Sales
Role takeaway
- Customer problem
- Institutions may assume fractional sharing is enforced once enabled, when enforcement depends on a narrow stack and can fail open.
- Stakeholders
- Research computing director
- Platform engineering lead
- Security officer
- Hardware support/vendor manager
- Discovery questions
- Can you run a non-default, short-lived driver branch under your support terms?
- Which container runtime and version do you run?
- How would you detect unfenced memory today?
- Value hypothesis
- A readiness and verification engagement prevents a pool from running unfenced while appearing healthy.
- Potential engagement
- Prerequisite audit plus enforcement-test and monitoring build.
- Unsupported claims
- No production deployment results are documented.
- No statement on long-term support of the r615 branch was found.
Pre-sales engineering
Role takeaway
- Fit
- Clusters already on recent Kubernetes and containerd with NRI.
- Architecture and integration
- GPU Operator v26.7.x, r615 driver, NRI-enabled runtime, KAI Scheduler fractions.
- Prerequisites
- Kubernetes 1.28+
- containerd 2.0+ with NRI or CRI-O
- GPU Operator v26.7.0+
- r615 driver branch
- Constraints
- r615 not installed by default by GPU Operator v26.7.1
- Skipping checks leaves memory unfenced silently
- Metrics can degrade silently to zero
- Security
- Verify enforcement by test; alert on zeroed telemetry; prohibit skipping version gates.
- Proof of value
- Automated over-limit allocation test
- Telemetry-health alert firing in a drill
- Version audit report
Delivery
Role takeaway
- Work
- Gate the pilot on verified prerequisites and add enforcement and telemetry checks to operations.
- Dependencies
- Platform team upgrade capacity
- Vendor/OEM support confirmation for driver branch
- Ownership
- Platform engineering, with security sign-off.
- Skills and adoption
- Runbook training on enforcement tests and telemetry alerts.
- Governance checkpoints
- Prerequisite version sign-off
- Post-upgrade enforcement re-test
- Acceptance criteria
- Over-limit test terminates the container
- Zeroed-metrics alert verified
- Risks
- Driver branch lifecycle churn
- Fail-open misconfiguration
- Silent monitoring gaps
Implementation considerations
Lighthouse Advisory interpretation across the operating dimensions a public-sector buyer must settle before this evidence becomes a design. Each note answers the question under its heading for this specific source.
Architecture and integration
What must connect, and where does the AI sit in the workflow?
Pins clusters to specific runtime (containerd NRI or CRI-O), GPU Operator and a non-default short-lived driver branch, which affects patch cadence and long-term support planning.
Governance
Who approves, reviews and stays accountable for outcomes?
Change control should forbid skipping version checks in production and require evidence that limits are enforced before declaring a pool 'fenced'.
Security and privacy
What data, permissions and controls need testing?
Fail-open behavior (unfenced memory with no report) means isolation claims must be verified by test, not assumed from configuration.
Accessibility and workforce
Who is affected, and what skills or accommodations follow?
Requires staff comfortable with container runtime internals (NRI) and driver lifecycle management.
Procurement
What should contracts, pricing and exit terms secure?
Short-lived driver branches can conflict with institutional support or OEM certification requirements; check vendor support terms before adopting.
Operating model
Which teams own the service once it runs?
Operations must add active monitoring that detects zeroed metrics and periodic enforcement tests to runbooks.
Publication history
- 2026-10-27NVIDIA · Issue 5212 resources
Stable resource ID: kai-scheduler-gpu-fractioning-readme-prerequisites-silent-failure-20261027