{"resourceId":"princeton-gpu-cluster-fine-grained-monitoring-underutilization-arxiv-20261026","versions":[{"version":"external-dfac8c4ff9400a7ccc4669beccdf436d264ceecca0c75c6a490dfe8081a24a94","resource":{"id":"princeton-gpu-cluster-fine-grained-monitoring-underutilization-arxiv-20261026","title":"Princeton preprint: fine-grained DCGM monitoring of a 319-GPU university cluster finds most jobs use a small fraction of GPU memory and compute","organization":"Princeton University, Department of Computer Science","sector":"Higher education research computing","geography":"New Jersey, United States","publishedAt":"2026-10-26","publicationDate":"2026-10-26","eventDate":null,"sourceName":"arXiv","sourceLabel":"Plenty of Room in the Middle: Revealing Opportunities on a GPU Cluster via Fine-Grained Monitoring (Preprint, not peer reviewed)","sourceUrl":"https://arxiv.org/abs/2610.36329","evidenceClass":"academic-research","outcomeClass":"cautionary","topics":["infrastructure","operating-model","governance-procurement"],"finding":"Preprint, not peer reviewed. Princeton researchers built a lightweight monitoring tool on NVIDIA's DCGM library that samples GPU activity every 100 ms, and deployed it on a university research cluster with 89 nodes and 319 GPUs. In a 10-day window covering 7,399 jobs from 150 users in 12 departments, over 70% of jobs used less than 20% of GPU memory and half averaged below 5% SM utilization. Eighty percent of jobs had a large gap between requested and actual run time, limiting scheduler effectiveness. The authors argue better scheduling and GPU sharing could yield substantial gains without new hardware. The monitoring tool is open source.","sledRelevance":"Public universities and state research computing centers are buying expensive NVIDIA GPU capacity. This study gives them a measurement method and evidence that utilization, not only capacity, should drive investment and allocation decisions before expansion.","evidence":"Academic measurement study on one university cluster with an open-source tool; preprint, not peer reviewed.","architectureImplications":"Fine-grained DCGM telemetry, GPU sharing (MIG/MPS or time-slicing) and better runtime estimates can raise utilization on existing clusters.","governanceImplications":"Allocation policies and chargeback can use measured utilization rather than requested GPUs to manage demand.","securityPrivacyImplications":"Sharing GPUs among users requires isolation controls appropriate to data sensitivity.","caveats":"Preprint, not peer reviewed. Originally written in fall 2024 on A100 GPUs; detailed job-level analysis covers only 10 days on one cluster, and results are aggregated across departments; transferability to other campuses and newer GPUs is untested.","streamIds":["nvidia"],"roles":{"sales":"Bring this to research computing directors and CIOs at public universities facing pressure to buy more GPUs. The message is to measure first: an open-source DCGM-based tool showed most jobs on one campus cluster used little of their allocated GPU. Offer a short utilization assessment that informs whether to expand, share or re-allocate. Do not promise specific gains, since this is a preprint from one cluster using older data.","engineering":"Engineering should deploy fine-grained DCGM telemetry, such as the open-source tool described or dcgm-exporter with Prometheus, on the cluster and compare memory and SM utilization with scheduler requests. Pilot GPU sharing through MIG, MPS or time-slicing for small jobs, with isolation appropriate to data sensitivity. Improve run-time estimates through scheduler defaults or feedback to users. Validate whether results match this study on newer GPUs.","delivery":"Delivery should run a time-boxed utilization study, for example 30 days, with a baseline report for leadership. Define a sharing policy and a user guide for requesting appropriate GPU fractions and run times. Track queue wait times and utilization before and after changes. Use the results to inform the next capacity purchase and chargeback model, and share findings with departments."},"roleDetails":{"sales":{"customerProblem":"Universities face rising GPU demand and costly expansion without clear utilization data.","stakeholders":["Research computing director","CIO","Provost's office","Department chairs"],"discoveryQuestions":["What share of allocated GPU memory and compute is actually used?","How accurate are users' run-time requests?"],"valueHypothesis":"Measured utilization can defer or right-size purchases and shorten queues.","potentialEngagement":"Utilization assessment using open-source DCGM-based monitoring.","unsupportedClaims":["Specific utilization gains for any campus","That findings hold on newer GPUs"]},"engineering":{"fit":"Shared university or state research GPU clusters running Slurm or similar schedulers.","architectureIntegration":"DCGM telemetry feeding monitoring dashboards; MIG/MPS or time-slicing for small jobs.","prerequisites":["DCGM access on nodes","Scheduler accounting data"],"constraints":["One cluster, A100-era data","Preprint"],"security":"Isolate shared GPUs according to data sensitivity.","proofOfValue":["Compare measured utilization to requests for 30 days","Pilot GPU sharing for small jobs"]},"delivery":{"work":"Run a utilization study and introduce sharing policies.","dependencies":["Scheduler administrators","Department representatives"],"ownership":"Research computing operations.","skillsAdoption":"User guidance on requesting resources and run times.","governanceCheckpoints":["Baseline report reviewed by leadership","Sharing policy approved"],"acceptanceCriteria":["Proposed: utilization and wait-time metrics reported monthly","Proposed: capacity request informed by measured utilization"],"risks":["User resistance to sharing","Isolation concerns for sensitive research data"]}},"retrievedAt":"2026-10-30T03:02:32Z","enrichedAt":"2026-10-30T03:10:14Z","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Researchers may need training on requesting appropriate resources and run times.","procurementImplications":"Before buying more GPUs, institutions can measure actual utilization to justify or right-size purchases.","operatingModelImplications":"Research computing teams can add fine-grained utilization reporting and sharing policies to cluster operations.","peerReview":"preprint","sourceVerification":{"openedUrl":"https://arxiv.org/abs/2610.36329","referenceExcerpt":"over 70% of jobs using less than 20% of GPU memory and 50% averaging below 5% Streaming Multiprocessor (SM) utilization","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}