{"resourceId":"gatech-pace-ice-gpu-scheduler-limits-20260922","versions":[{"version":"external-da95b2af97a65ae57e02967c548deea5a7c4f251e27ed968671b63c14c72688f","resource":{"id":"gatech-pace-ice-gpu-scheduler-limits-20260922","title":"Georgia Tech PACE ICE: GPU queue fairness caps (500→50 jobs/user; 32 GPU-hours) after wait-time reports","organization":"Georgia Institute of Technology — Partnership for an Advanced Computing Environment (PACE)","sector":"Higher education research computing operations (instructional ICE cluster)","geography":"United States (Atlanta, GA — Georgia Tech)","publishedAt":"September 22, 2026","publicationDate":"2026-09-22","eventDate":"2026-09-22","sourceName":"Georgia Tech PACE blog","sourceLabel":"University research computing operations notice (PACE)","sourceUrl":"https://sites.gatech.edu/pacewww-dev/2026/09/22/ice-scheduler-improvements/","evidenceClass":"independent-reporting","outcomeClass":"emerging","topics":["infrastructure","operating-model","accessibility-workforce"],"finding":"On 22 September 2026 Georgia Tech PACE announced immediate ICE-cluster scheduler changes after reports of extended GPU queue wait times impacting coursework. Review found a small number of users consuming a disproportionate share of GPUs under prior Slurm settings. Caps reduce maximum GPU jobs per user (running+pending) from 500 to 50 and set a new maximum running GPU capacity of 32 GPU-hours per user; scheduling priorities are adjusted so each user can run at least one GPU job. PACE will monitor effects. Notice also reminds of an Oct 5–8, 2026 maintenance window across Phoenix, Firebird, and ICE.","sledRelevance":"Campus research-computing operators facing GPU contention need concrete fairness controls and wait-time response patterns—capacity expansions alone do not fix disproportionate multi-job occupancy.","evidence":"PACE ICE scheduler improvements page opened via HTTPS; 500→50 job cap, 32 GPU-hour limit, wait-time motivation, and priority change taken from opened notice dated 22 Sep 2026.","architectureImplications":"Slurm fairshare-style limits on jobs-in-queue and concurrent GPU-hour consumption plus priority favoring first GPU job per user; monitor post-change wait distributions.","governanceImplications":"Transparent ops communication and immediate policy change after utilization review; instructional GPU access treated as shared scarce resource requiring equity rules.","securityPrivacyImplications":"Notice focuses on scheduling equity, not isolation; still plan separate controls for CUI/research-sensitive work on other PACE systems (e.g., Firebird) rather than assuming ICE limits address data protection.","caveats":"Institutional self-report; no published before/after wait-time statistics on this page; ICE is instructional-focused though operated by research computing; outcomes of the new caps not yet measured in the notice.","streamIds":["research"],"roles":{"sales":"Sales framing: Research-computing buyers drowning in GPU wait complaints need schedulers and policies that restore equitable access—not only more cards. Georgia Tech PACE’s ICE notice supports a bounded pitch around job-count and GPU-hour caps (500→50 jobs; 32 GPU-hours) plus priority so every user can run at least one GPU job after disproportionate consumption was found. Do not sell guaranteed wait-time SLOs or measured post-change outcomes—the page announces controls and monitoring, not a completed before/after study.","engineering":"Engineering framing: Fit is Slurm policy tuning on a shared instructional/research GPU cluster facing wait-time complaints. Integrate per-user MaxJobs-style limits on running-plus-pending GPU jobs, GPU-hour consumption caps, and priority weights favoring at least one GPU job per user after utilization review. Prerequisites include occupancy telemetry that identifies heavy users and a clear user communications plan. Constraints include possible friction for legitimate multi-job workflows; proof-of-value is reduced wait-time percentiles and fewer single-user monopolies without collapsing useful throughput.","delivery":"Delivery framing: Analyze GPU occupancy, announce caps, deploy scheduler changes, and monitor wait/utilization for several weeks including around maintenance windows. Ownership: PACE/research computing owns scheduler policy; instructors own coursework deadlines; users own right-sizing requests. Acceptance should include published caps, monitoring dashboards, and a review of wait-time complaints. Risks include under-documented outcomes, heavy-user workarounds, and conflating instructional ICE policy with research-production clusters."},"roleDetails":{"sales":{"customerProblem":"Shared campus GPU queues become unusable when a few users hold hundreds of pending/running jobs.","stakeholders":["Research computing / PACE-equivalent","Instructors / TAs on GPU courses","Student researchers","VPR / academic technology","Faculty GPU users"],"discoveryQuestions":["What are current p50/p95 GPU wait times?","What share of GPUs do the top N users hold?","Are job-count and GPU-hour caps enforced today?","How are instructional vs research queues separated?"],"valueHypothesis":"Per-user job and GPU-hour caps with first-job priority can restore equitable GPU access when wait complaints track disproportionate occupancy.","potentialEngagement":"Two-week utilization study plus scheduler-policy pilot with wait-time and occupancy dashboards.","unsupportedClaims":["Guaranteed wait-time SLO from this notice","Measured post-change success already proven here","Same caps fit every research production cluster"]},"engineering":{"fit":"Slurm fairness limits on shared GPU instructional/research cluster.","architectureIntegration":"Configure MaxJobs (running+pending), GPU-hour consumption caps, and priority favoring ≥1 GPU job per user; instrument wait telemetry.","prerequisites":["Utilization audit","Scheduler admin access","User communication channel","Wait-time metrics"],"constraints":["Instructional ICE context","No published before/after stats yet","Possible multi-job workflow friction"],"security":"Scheduler equity changes do not replace isolation for regulated data; keep sensitive work on appropriate partitions.","proofOfValue":["Wait-time percentile change","Top-user GPU share reduction","Users able to obtain ≥1 running GPU job","Ticket volume on wait complaints"]},"delivery":{"work":"Audit occupancy, set caps/priorities, announce, monitor, iterate after maintenance windows.","dependencies":["Slurm controllers","Instructor calendars","Monitoring stack"],"ownership":"Research computing owns scheduler policy; instructors own course expectations; users own job hygiene.","skillsAdoption":"Teach users to right-size job arrays and avoid parking hundreds of GPU jobs.","governanceCheckpoints":["Policy announcement","One-week monitoring review","Post-maintenance reassessment"],"acceptanceCriteria":["Caps enforced in scheduler","Monitoring dashboard live","Wait-complaint trend reviewed"],"risks":["Unmeasured outcomes","Heavy-user circumvention","Misapplied caps on research partitions"]}},"retrievedAt":"2026-09-29T22:01:24-05:00","enrichedAt":"2026-09-29T22:02:46-05:00","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Fairness caps protect instructional and novice GPU users from starvation; support staff still need to help legitimate high-throughput users move to appropriate research partitions.","procurementImplications":"GPU expansions should be paired with scheduler fairness features and monitoring; buying cards without occupancy policy may not fix wait-time complaints.","operatingModelImplications":"Shared campus GPU access is an operating model of transparent equity rules and continuous monitoring, not unlimited per-user pending job sprawl.","sourceVerification":{"openedUrl":"https://sites.gatech.edu/pacewww-dev/2026/09/22/ice-scheduler-improvements/","referenceExcerpt":"reduced from 500 to 50 jobs","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}