{"resourceId":"harvard-fasrc-coding-agents-hpc-measurement-20260930","versions":[{"version":"external-81e144c199153bc8bc29119b3e3692e46a69fca827ba43ad39efac75e4654eb1","resource":{"id":"harvard-fasrc-coding-agents-hpc-measurement-20260930","title":"Harvard FASRC: agent users 19.5% of population but 55.8% of Slurm submissions, 42.7% of GPU-hours","organization":"Harvard University and Harvard FAS Research Computing — arXiv preprint (cs)","sector":"Higher education research computing / AI-assisted scientific HPC workloads","geography":"United States (Harvard FASRC production cluster); methods transferable to other university Slurm sites","publishedAt":"September 30, 2026 (arXiv 2609.38723v1); SCN summary dated October 2, 2026","publicationDate":"2026-09-30","eventDate":"2026-07-07","sourceName":"arXiv 2609.38723","sourceLabel":"Academic preprint (arXiv)","sourceUrl":"https://arxiv.org/abs/2609.38723","evidenceClass":"academic-research","outcomeClass":"cautionary","topics":["developers-agents","infrastructure","operating-model","data-security","governance-procurement"],"finding":"Towards Efficient HPC Systems for Agents (arXiv 2609.38723, submitted 30 Sep 2026; Harvard CS + FASRC) measures coding-agent use on FASRC from 7 Jul–1 Aug 2026 across >1,740 nodes / 122k CPU cores / 2,000 GPUs (1,611 submitting users; 7.09M jobs). Users running Claude Code, Codex, Cursor, or Copilot were 19.5% of the observed population but filed 55.8% of top-level Slurm submissions, 29.1% of CPU core-hours, and 42.7% of GPU-hours. Agents issued commands at 20.8× the human rate, canceled jobs at 4.4× the human rate, and generated ~28× more file read/edit tool calls per 100 submissions, with metadata-heavy tree walks and recursive searches. Authors argue neither banning agents nor treating them as ordinary users is sustainable and propose co-design (trial-aware scheduling, event-driven status, locality/metadata isolation, federated failure memory with provenance, system-boundary guardrails). Security observations include chmod 777 / curl|sh patterns, 127 exposed API keys over three weeks, and login-tier external fetches by 77.2% of agent users (prompt-injection surface). Measurement is one cluster; metadata-server load is inferred from command counts, not direct MDS telemetry.","sledRelevance":"University research computing centers already host multi-vendor coding agents as first-class load and need agent-aware scheduler, filesystem, memory, and login-node security policies—not human-user assumptions or blanket bans.","evidence":"Opened https://arxiv.org/abs/2609.38723 via WebFetch (full abstract + sections I–VIII); cross-checked SCN Oct 2 facility summary; check_sources: not in archive.","architectureImplications":"Treat agents as facility-aware principals: trial-aware QoS with anti-gaming bounds, event-driven job status instead of tight squeue polls, node-local/NVMe working sets for small I/O, and hooks below the agent (Landlock/seccomp) for dangerous shell actions.","governanceImplications":"Codify account-holder responsibility for agent actions; publish agent AUP with login-node concurrency limits; require disclosure of agent harnesses in allocation reviews; prefer co-design over bans that agents can evade via the same shell interfaces.","securityPrivacyImplications":"Move safety checks beneath the agent; taint fetched README/setup content; redact secrets from argv/history/logs; default-deny non-loopback service binds; expect prompt injection via clones/packages more than search.","caveats":"Single-site (Harvard FASRC) July–Aug 2026 window; agent vs human are different user populations; fixed-rate process sampling can miss short-lived processes; metadata storm risk inferred from command counts without reported MDS load; security issues responsibly disclosed but not a penetration-test corpus.","streamIds":["research"],"roles":{"sales":"Sales framing: campus research computing already runs multi-vendor coding agents as a majority submission load—not a pilot curiosity. Harvard FASRC (arXiv 2609.38723, 30 Sep 2026) measured July–Aug 2026: agent users were 19.5% of the population yet filed 55.8% of Slurm submissions and 42.7% of GPU-hours, with 20.8× command rate and acute scheduler/metadata/security pressure. Pitch an agent-aware facility assessment—trial QoS, event-driven status, login-node guardrails—not seats or a ban that agents can route around.","engineering":"Engineering framing: integrate agents as first-class principals on Slurm and shared filesystems. Instrument harness ancestry on login nodes; add trial-aware QoS with revoke-on-abuse; replace tight squeue polls with events; keep hot trees on local NVMe and rate-limit metadata storms; enforce Landlock/seccomp hooks and secret redaction beneath the agent. Match PoV to submission share, cancel/poll rates, exposed-key counts, and MDS latency—not Terminal-Bench scores alone.","delivery":"Delivery framing: run a 30-day agent telemetry pilot on login + Slurm accounting, publish revised agent AUP, stand up trial QoS and event status hooks, and deploy system-boundary allowlists before scaling autonomous research campaigns. Ownership sits with research computing operations; PIs remain accountable for agent actions under their accounts. Accept only when polling storms and secret-exposure rates fall under agreed floors without blocking legitimate interactive work."},"roleDetails":{"sales":{"customerProblem":"Coding agents already dominate job-submission volume on shared university HPC while straining schedulers, metadata services, and login-node security assumptions built for humans.","stakeholders":["CIO / research computing leadership","HPC operators","PI computational groups","InfoSec / research security","Faculty senate research committee"],"discoveryQuestions":["What share of Slurm submissions and GPU-hours come from known agent harnesses?","How often do agents poll the scheduler relative to site AUP limits?","Which login-node shortcuts (chmod 777, curl|sh, unbound services) are already observed?"],"valueHypothesis":"Agent-aware co-design (trial QoS, event status, metadata isolation, system guardrails) can absorb agent load without bans that forfeit productivity or are unenforceable.","potentialEngagement":"Four-week FASRC-style measurement and policy redesign engagement with telemetry report and draft agent AUP.","unsupportedClaims":["Guaranteed cost savings from banning agents","One-vendor agent monopoly on campus","Zero-risk unattended autonomy on shared login nodes"]},"engineering":{"fit":"Production Slurm + shared parallel filesystem campus cluster with multi-vendor coding agents under user accounts.","architectureIntegration":"Harness-labeled telemetry; trial-aware QoS; event-driven job notifications; local NVMe working sets; kernel/agent hooks for dangerous actions; federated failure memory with provenance boundaries.","prerequisites":["Login-node process monitoring","Slurm accounting export","Agent harness allowlist","Secret-scanning hooks"],"constraints":["Single-site evidence","Agents share human interfaces","Metadata rate limits hard to attribute per tool call","Federated memory privacy risks"],"security":"Enforce controls beneath the agent; taint external fetches; redact credentials from argv/logs; default-deny public binds; disclose and remediate exposed keys.","proofOfValue":["Agent vs human submission/GPU-hour shares","Scheduler query rate per job","Cancel/resubmit trial patterns","Exposed API-key and AUP-violation counts"]},"delivery":{"work":"Deploy telemetry, classify harnesses, revise agent AUP, pilot trial QoS and event status, install system-boundary hooks, report weekly load and security metrics.","dependencies":["Slurm admins","Storage team","IdM/IAM","InfoSec review","PI communication channel"],"ownership":"Research computing operations owns facility controls; account holders own agent behavior; InfoSec owns secret-exposure response.","skillsAdoption":"Train consultants and PIs on agent AUP, polling limits, and non-interactive editing patterns; teach agents via AGENTS.md/site skills.","governanceCheckpoints":["Telemetry ethics/sanitization approval","AUP publication","Trial-QoS anti-gaming review","Secret-exposure incident retrospective"],"acceptanceCriteria":["Agent-labeled share of submissions published monthly","Median scheduler polls/job under site limit for high-volume agent users","Zero unresolved exposed keys older than 7 days","Trial QoS abuse rate below agreed threshold"],"risks":["Misclassification of human vs agent sessions","PI backlash to tighter login-node limits","Agents learning to game trial priority","Federated memory leaking unpublished ideas"]}},"retrievedAt":"2026-10-03T22:05:10-05:00","enrichedAt":"2026-10-03T22:08:30-05:00","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Agent productivity gains concentrate among users who adopt harnesses; facilities must document agent skills and avoid login-node bans that push work underground without training safer patterns.","procurementImplications":"Do not procure solely on agent capability benchmarks; require facility co-design features (event APIs, metadata isolation, secret redaction) in research-computing refresh and vendor agent offerings.","operatingModelImplications":"Shift from human-paced interactive support to agent-aware operations: continuous telemetry, AUP enforcement automation, and joint RC–InfoSec ownership of agent principals.","sourceVerification":{"openedUrl":"https://arxiv.org/abs/2609.38723","referenceExcerpt":"Neither banning agents nor treating them as ordinary users is sustainable.","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}