From the NVIDIA edition of October 29, 2026
Princeton preprint: fine-grained DCGM monitoring of a 319-GPU university cluster finds most jobs use a small fraction of GPU memory and compute
Princeton University, Department of Computer Science · Higher education research computing · New Jersey, United States
- Publisher
- arXiv
- Original publication
- 2026-10-26
- Source retrieved
- 2026-10-30
What happened
Preprint, not peer reviewed. Princeton researchers built a lightweight monitoring tool on NVIDIA's DCGM library that samples GPU activity every 100 ms, and deployed it on a university research cluster with 89 nodes and 319 GPUs. In a 10-day window covering 7,399 jobs from 150 users in 12 departments, over 70% of jobs used less than 20% of GPU memory and half averaged below 5% SM utilization. Eighty percent of jobs had a large gap between requested and actual run time, limiting scheduler effectiveness. The authors argue better scheduling and GPU sharing could yield substantial gains without new hardware. The monitoring tool is open source.
Why it matters
Public universities and state research computing centers are buying expensive NVIDIA GPU capacity. This study gives them a measurement method and evidence that utilization, not only capacity, should drive investment and allocation decisions before expansion.
Evidence and measured results
Academic measurement study on one university cluster with an open-source tool; preprint, not peer reviewed.
Limitations and uncertainty
Preprint, not peer reviewed. Originally written in fall 2024 on A100 GPUs; detailed job-level analysis covers only 10 days on one cluster, and results are aggregated across departments; transferability to other campuses and newer GPUs is untested.
Put this evidence to work
Lighthouse Advisory interpretation, grounded in this source. Enriched 2026-10-30; this does not change the original publication date. Each role follows the same checklist so takeaways compare across sources.
Sales
Role takeaway
- Customer problem
- Universities face rising GPU demand and costly expansion without clear utilization data.
- Stakeholders
- Research computing director
- CIO
- Provost's office
- Department chairs
- Discovery questions
- What share of allocated GPU memory and compute is actually used?
- How accurate are users' run-time requests?
- Value hypothesis
- Measured utilization can defer or right-size purchases and shorten queues.
- Potential engagement
- Utilization assessment using open-source DCGM-based monitoring.
- Unsupported claims
- Specific utilization gains for any campus
- That findings hold on newer GPUs
Pre-sales engineering
Role takeaway
- Fit
- Shared university or state research GPU clusters running Slurm or similar schedulers.
- Architecture and integration
- DCGM telemetry feeding monitoring dashboards; MIG/MPS or time-slicing for small jobs.
- Prerequisites
- DCGM access on nodes
- Scheduler accounting data
- Constraints
- One cluster, A100-era data
- Preprint
- Security
- Isolate shared GPUs according to data sensitivity.
- Proof of value
- Compare measured utilization to requests for 30 days
- Pilot GPU sharing for small jobs
Delivery
Role takeaway
- Work
- Run a utilization study and introduce sharing policies.
- Dependencies
- Scheduler administrators
- Department representatives
- Ownership
- Research computing operations.
- Skills and adoption
- User guidance on requesting resources and run times.
- Governance checkpoints
- Baseline report reviewed by leadership
- Sharing policy approved
- Acceptance criteria
- Proposed: utilization and wait-time metrics reported monthly
- Proposed: capacity request informed by measured utilization
- Risks
- User resistance to sharing
- Isolation concerns for sensitive research data
Implementation considerations
Lighthouse Advisory interpretation across the operating dimensions a public-sector buyer must settle before this evidence becomes a design. Each note answers the question under its heading for this specific source.
Architecture and integration
What must connect, and where does the AI sit in the workflow?
Fine-grained DCGM telemetry, GPU sharing (MIG/MPS or time-slicing) and better runtime estimates can raise utilization on existing clusters.
Governance
Who approves, reviews and stays accountable for outcomes?
Allocation policies and chargeback can use measured utilization rather than requested GPUs to manage demand.
Security and privacy
What data, permissions and controls need testing?
Sharing GPUs among users requires isolation controls appropriate to data sensitivity.
Accessibility and workforce
Who is affected, and what skills or accommodations follow?
Researchers may need training on requesting appropriate resources and run times.
Procurement
What should contracts, pricing and exit terms secure?
Before buying more GPUs, institutions can measure actual utilization to justify or right-size purchases.
Operating model
Which teams own the service once it runs?
Research computing teams can add fine-grained utilization reporting and sharing policies to cluster operations.
Publication history
- 2026-10-29NVIDIA · Issue 547 resources
Stable resource ID: princeton-gpu-cluster-fine-grained-monitoring-underutilization-arxiv-20261026