From the Research Universities edition of September 29, 2026
Georgia Tech PACE ICE: GPU queue fairness caps (500→50 jobs/user; 32 GPU-hours) after wait-time reports
Georgia Institute of Technology — Partnership for an Advanced Computing Environment (PACE) · Higher education research computing operations (instructional ICE cluster) · United States (Atlanta, GA — Georgia Tech)
- Publisher
- Georgia Tech PACE blog
- Original publication
- September 22, 2026
- Source retrieved
- 2026-09-30
- Event date
- 2026-09-22
What happened
On 22 September 2026 Georgia Tech PACE announced immediate ICE-cluster scheduler changes after reports of extended GPU queue wait times impacting coursework. Review found a small number of users consuming a disproportionate share of GPUs under prior Slurm settings. Caps reduce maximum GPU jobs per user (running+pending) from 500 to 50 and set a new maximum running GPU capacity of 32 GPU-hours per user; scheduling priorities are adjusted so each user can run at least one GPU job. PACE will monitor effects. Notice also reminds of an Oct 5–8, 2026 maintenance window across Phoenix, Firebird, and ICE.
Why it matters
Campus research-computing operators facing GPU contention need concrete fairness controls and wait-time response patterns—capacity expansions alone do not fix disproportionate multi-job occupancy.
Evidence and measured results
PACE ICE scheduler improvements page opened via HTTPS; 500→50 job cap, 32 GPU-hour limit, wait-time motivation, and priority change taken from opened notice dated 22 Sep 2026.
Limitations and uncertainty
Institutional self-report; no published before/after wait-time statistics on this page; ICE is instructional-focused though operated by research computing; outcomes of the new caps not yet measured in the notice.
Put this evidence to work
Lighthouse Advisory interpretation, grounded in this source. Enriched 2026-09-29; this does not change the original publication date. Each role follows the same checklist so takeaways compare across sources.
Sales
Role takeaway
- Customer problem
- Shared campus GPU queues become unusable when a few users hold hundreds of pending/running jobs.
- Stakeholders
- Research computing / PACE-equivalent
- Instructors / TAs on GPU courses
- Student researchers
- VPR / academic technology
- Faculty GPU users
- Discovery questions
- What are current p50/p95 GPU wait times?
- What share of GPUs do the top N users hold?
- Are job-count and GPU-hour caps enforced today?
- How are instructional vs research queues separated?
- Value hypothesis
- Per-user job and GPU-hour caps with first-job priority can restore equitable GPU access when wait complaints track disproportionate occupancy.
- Potential engagement
- Two-week utilization study plus scheduler-policy pilot with wait-time and occupancy dashboards.
- Unsupported claims
- Guaranteed wait-time SLO from this notice
- Measured post-change success already proven here
- Same caps fit every research production cluster
Pre-sales engineering
Role takeaway
- Fit
- Slurm fairness limits on shared GPU instructional/research cluster.
- Architecture and integration
- Configure MaxJobs (running+pending), GPU-hour consumption caps, and priority favoring ≥1 GPU job per user; instrument wait telemetry.
- Prerequisites
- Utilization audit
- Scheduler admin access
- User communication channel
- Wait-time metrics
- Constraints
- Instructional ICE context
- No published before/after stats yet
- Possible multi-job workflow friction
- Security
- Scheduler equity changes do not replace isolation for regulated data; keep sensitive work on appropriate partitions.
- Proof of value
- Wait-time percentile change
- Top-user GPU share reduction
- Users able to obtain ≥1 running GPU job
- Ticket volume on wait complaints
Delivery
Role takeaway
- Work
- Audit occupancy, set caps/priorities, announce, monitor, iterate after maintenance windows.
- Dependencies
- Slurm controllers
- Instructor calendars
- Monitoring stack
- Ownership
- Research computing owns scheduler policy; instructors own course expectations; users own job hygiene.
- Skills and adoption
- Teach users to right-size job arrays and avoid parking hundreds of GPU jobs.
- Governance checkpoints
- Policy announcement
- One-week monitoring review
- Post-maintenance reassessment
- Acceptance criteria
- Caps enforced in scheduler
- Monitoring dashboard live
- Wait-complaint trend reviewed
- Risks
- Unmeasured outcomes
- Heavy-user circumvention
- Misapplied caps on research partitions
Implementation considerations
Lighthouse Advisory interpretation across the operating dimensions a public-sector buyer must settle before this evidence becomes a design. Each note answers the question under its heading for this specific source.
Architecture and integration
What must connect, and where does the AI sit in the workflow?
Slurm fairshare-style limits on jobs-in-queue and concurrent GPU-hour consumption plus priority favoring first GPU job per user; monitor post-change wait distributions.
Governance
Who approves, reviews and stays accountable for outcomes?
Transparent ops communication and immediate policy change after utilization review; instructional GPU access treated as shared scarce resource requiring equity rules.
Security and privacy
What data, permissions and controls need testing?
Notice focuses on scheduling equity, not isolation; still plan separate controls for CUI/research-sensitive work on other PACE systems (e.g., Firebird) rather than assuming ICE limits address data protection.
Accessibility and workforce
Who is affected, and what skills or accommodations follow?
Fairness caps protect instructional and novice GPU users from starvation; support staff still need to help legitimate high-throughput users move to appropriate research partitions.
Procurement
What should contracts, pricing and exit terms secure?
GPU expansions should be paired with scheduler fairness features and monitoring; buying cards without occupancy policy may not fix wait-time complaints.
Operating model
Which teams own the service once it runs?
Shared campus GPU access is an operating model of transparent equity rules and continuous monitoring, not unlimited per-user pending job sprawl.
Publication history
- 2026-09-29Research Universities · Issue 224 resources
Stable resource ID: gatech-pace-ice-gpu-scheduler-limits-20260922