{"resourceId":"purdue-gautschi-ai-cluster-gpu-limits-burst-qos-20261028","versions":[{"version":"external-8d8fbd0be64a3af294d9d5005951e8994012e2ccff67dd540e2106201868045f","resource":{"id":"purdue-gautschi-ai-cluster-gpu-limits-burst-qos-20261028","title":"Purdue cuts per-group GPU limits on its near-100%-utilized Gautschi AI cluster to 2x purchase and adds a 2x-charged burst queue from 2 November","organization":"Purdue University Rosen Center for Advanced Computing (RCAC)","sector":"University research computing (HPC and GPU clusters)","geography":"United States (Indiana)","publishedAt":"Posted 28 October 2026; changes effective Monday 2 November 2026 (RCAC announcement)","publicationDate":"2026-10-28","eventDate":"2026-11-02","sourceName":"Purdue RCAC","sourceLabel":"Purdue RCAC, \"Gautschi Scheduler Changes Coming November 2: Shorter Waits, New Burst QOS,\" 28 Oct 2026","sourceUrl":"https://rcac.purdue.edu/news/7941","evidenceClass":"vendor-claim","outcomeClass":"mixed","topics":["infrastructure","operating-model","governance-procurement"],"finding":"RCAC says Gautschi's AI partition, in production for 19 months, has crept toward 100% utilization while normal-queue waits grew and preemptible jobs rarely started. Because groups buy GPU hours rather than GPUs and Gautschi cannot use Slurm Fairshare, every group could run up to 24 H100s at once. From 2 November the normal-queue limit becomes twice the GPUs a group purchased (minimum two), maximum walltime drops from two weeks to four days, a new burst queue allows up to eight GPUs for eight hours at twice the GPU-hour charge, one burst job per group, and preemptible jobs get a 10-minute grace period to checkpoint. RCAC modeled options: a 2x limit frees 50.45 GPUs (31.53% of the cluster) for pending jobs versus 5.86% for 4x and 0.54% for a cut to 16; 78.3% of the 97 groups would not notice the new limit.","sledRelevance":"A data-backed example of a campus AI cluster hitting saturation and moving from generous flat limits to purchase-proportional access with paid burst capacity. Research computing directors facing the same demand can reuse the modeling approach and see the trade-offs for heavy users.","evidence":"Read the full announcement including the design-options table. Figures are RCAC's own usage analysis; wait-time data, job counts and outcomes after the change are not yet available.","architectureImplications":"Slurm quality-of-service tiers (normal, preemptible, burst) replace Fairshare for a GPU-hour purchase model; burst jobs target full eight-GPU nodes with fast GPU-to-GPU links.","governanceImplications":"Access rules now tie concurrency to investment, with a single burst job per group to protect fairness.","securityPrivacyImplications":"Not addressed; this is a scheduling and allocation change.","caveats":"Operator announcement with internal data; no independent evaluation and no post-change results. Purdue-specific purchase model limits transfer. Other campuses use Fairshare or national allocations.","streamIds":["research"],"roles":{"sales":"Lead with fair access on saturated AI clusters. Purdue said on 28 October that its Gautschi AI partition is near 100% utilization and, from 2 November, will cap each group at twice its purchased GPUs and add a burst queue charged at twice the rate. Research computing directors with similar queues need defensible rules. Offer a bounded review of allocation and queue policy using their own usage data. Do not promise shorter waits; Purdue has not reported results yet.","engineering":"Frame fit around Slurm QOS design when Fairshare is not an option. Gautschi uses purchase-proportional concurrency limits, four-day walltimes, a burst QOS for whole eight-GPU nodes and a 10-minute GraceTime for preemptible jobs. Prerequisites are per-group usage history, accounting that can apply multiplier charges, and checkpoint-capable workflows. Validate by simulating candidate limits on past job logs, as RCAC did, then tracking wait times and preemptible start rates after the change.","delivery":"Treat delivery as a policy change with user impact. Work covers usage modeling, user consultation, scheduler configuration, burst-charge billing and checkpointing guidance. Ownership sits with the research computing center, with faculty advisory groups reviewing the rules. Governance checkpoints cover advance notice and a post-change review. Acceptance criteria are proposals, such as lower median normal-queue waits and more preemptible starts within one semester, not results Purdue reported."},"roleDetails":{"sales":{"customerProblem":"A shared AI cluster at full utilization gives long waits and unfair access when a few groups use far more than they bought.","stakeholders":["Research computing director","Vice president for research","Faculty advisory committee","Heavy-use research groups"],"discoveryQuestions":["What is your GPU partition utilization and wait time?","How do purchases map to scheduling priority today?","Can your accounting charge a premium for burst use?"],"valueHypothesis":"Purchase-proportional limits and paid burst capacity may cut waits for most groups while keeping large runs possible.","potentialEngagement":"A review of GPU allocation and queue policy using the customer's job logs.","unsupportedClaims":["Shorter waits are guaranteed","Heavy users will not be harmed","The 2x factor fits other campuses"]},"engineering":{"fit":"Fits condo or purchase-based GPU clusters running Slurm where Fairshare does not suit the purchase model.","architectureIntegration":"Slurm QOS tiers with per-account GPU limits, walltime caps, a multiplier-charged burst QOS and preemption grace time.","prerequisites":["Per-group usage history","Accounting with charge multipliers","Checkpoint-capable workloads"],"constraints":["Institution-specific purchase model","No post-change data","Shorter walltimes for long jobs"],"security":"No new security controls; keep accounting and QOS changes under change management.","proofOfValue":["Simulate limits on historical job logs","Track wait times before and after","Measure preemptible job start rates"]},"delivery":{"work":"Model usage, consult users, configure QOS, set up burst billing and publish checkpointing guidance.","dependencies":["Slurm accounting data","Billing system","User support and consultations"],"ownership":"Research computing center with faculty advisory oversight.","skillsAdoption":"Researchers learn checkpointing and when to use burst; staff learn QOS tuning.","governanceCheckpoints":["Advance user notice","Advisory committee review","Post-change review after one term"],"acceptanceCriteria":["Median normal-queue wait falls by an agreed amount (proposal)","Preemptible jobs start regularly","Burst charges billed correctly"],"risks":["Heavy users lose throughput","Burst queue contention","Jobs lost without checkpointing"]}},"retrievedAt":"2026-11-02T08:38:56-06:00","enrichedAt":"2026-11-02T08:38:59-06:00","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Heavy users face shorter walltimes and must checkpoint; RCAC points them to consultations on checkpointing.","procurementImplications":"Encourages groups that need large GPU counts often to buy enough to cover about half their regular usage.","operatingModelImplications":"Shows a shift from growth-phase generosity to rationing by purchase share and surcharge as AI demand saturates a cluster.","peerReview":"unknown","sourceVerification":{"openedUrl":"https://rcac.purdue.edu/news/7941","referenceExcerpt":"When reviewing the average usage of all 97 groups on Gautschi since the start of the semester, 78.3% of groups would not notice this limit","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}