From the Research Universities edition of October 3, 2026
Harvard FASRC: agent users 19.5% of population but 55.8% of Slurm submissions, 42.7% of GPU-hours
Harvard University and Harvard FAS Research Computing — arXiv preprint (cs) · Higher education research computing / AI-assisted scientific HPC workloads · United States (Harvard FASRC production cluster); methods transferable to other university Slurm sites
- Publisher
- arXiv 2609.38723
- Original publication
- September 30, 2026 (arXiv 2609.38723v1); SCN summary dated October 2, 2026
- Source retrieved
- 2026-10-04
- Event date
- 2026-07-07
What happened
Towards Efficient HPC Systems for Agents (arXiv 2609.38723, submitted 30 Sep 2026; Harvard CS + FASRC) measures coding-agent use on FASRC from 7 Jul–1 Aug 2026 across >1,740 nodes / 122k CPU cores / 2,000 GPUs (1,611 submitting users; 7.09M jobs). Users running Claude Code, Codex, Cursor, or Copilot were 19.5% of the observed population but filed 55.8% of top-level Slurm submissions, 29.1% of CPU core-hours, and 42.7% of GPU-hours. Agents issued commands at 20.8× the human rate, canceled jobs at 4.4× the human rate, and generated ~28× more file read/edit tool calls per 100 submissions, with metadata-heavy tree walks and recursive searches. Authors argue neither banning agents nor treating them as ordinary users is sustainable and propose co-design (trial-aware scheduling, event-driven status, locality/metadata isolation, federated failure memory with provenance, system-boundary guardrails). Security observations include chmod 777 / curl|sh patterns, 127 exposed API keys over three weeks, and login-tier external fetches by 77.2% of agent users (prompt-injection surface). Measurement is one cluster; metadata-server load is inferred from command counts, not direct MDS telemetry.
Why it matters
University research computing centers already host multi-vendor coding agents as first-class load and need agent-aware scheduler, filesystem, memory, and login-node security policies—not human-user assumptions or blanket bans.
Evidence and measured results
Opened https://arxiv.org/abs/2609.38723 via WebFetch (full abstract + sections I–VIII); cross-checked SCN Oct 2 facility summary; check_sources: not in archive.
Limitations and uncertainty
Single-site (Harvard FASRC) July–Aug 2026 window; agent vs human are different user populations; fixed-rate process sampling can miss short-lived processes; metadata storm risk inferred from command counts without reported MDS load; security issues responsibly disclosed but not a penetration-test corpus.
Put this evidence to work
Lighthouse Advisory interpretation, grounded in this source. Enriched 2026-10-03; this does not change the original publication date. Each role follows the same checklist so takeaways compare across sources.
Sales
Role takeaway
- Customer problem
- Coding agents already dominate job-submission volume on shared university HPC while straining schedulers, metadata services, and login-node security assumptions built for humans.
- Stakeholders
- CIO / research computing leadership
- HPC operators
- PI computational groups
- InfoSec / research security
- Faculty senate research committee
- Discovery questions
- What share of Slurm submissions and GPU-hours come from known agent harnesses?
- How often do agents poll the scheduler relative to site AUP limits?
- Which login-node shortcuts (chmod 777, curl|sh, unbound services) are already observed?
- Value hypothesis
- Agent-aware co-design (trial QoS, event status, metadata isolation, system guardrails) can absorb agent load without bans that forfeit productivity or are unenforceable.
- Potential engagement
- Four-week FASRC-style measurement and policy redesign engagement with telemetry report and draft agent AUP.
- Unsupported claims
- Guaranteed cost savings from banning agents
- One-vendor agent monopoly on campus
- Zero-risk unattended autonomy on shared login nodes
Pre-sales engineering
Role takeaway
- Fit
- Production Slurm + shared parallel filesystem campus cluster with multi-vendor coding agents under user accounts.
- Architecture and integration
- Harness-labeled telemetry; trial-aware QoS; event-driven job notifications; local NVMe working sets; kernel/agent hooks for dangerous actions; federated failure memory with provenance boundaries.
- Prerequisites
- Login-node process monitoring
- Slurm accounting export
- Agent harness allowlist
- Secret-scanning hooks
- Constraints
- Single-site evidence
- Agents share human interfaces
- Metadata rate limits hard to attribute per tool call
- Federated memory privacy risks
- Security
- Enforce controls beneath the agent; taint external fetches; redact credentials from argv/logs; default-deny public binds; disclose and remediate exposed keys.
- Proof of value
- Agent vs human submission/GPU-hour shares
- Scheduler query rate per job
- Cancel/resubmit trial patterns
- Exposed API-key and AUP-violation counts
Delivery
Role takeaway
- Work
- Deploy telemetry, classify harnesses, revise agent AUP, pilot trial QoS and event status, install system-boundary hooks, report weekly load and security metrics.
- Dependencies
- Slurm admins
- Storage team
- IdM/IAM
- InfoSec review
- PI communication channel
- Ownership
- Research computing operations owns facility controls; account holders own agent behavior; InfoSec owns secret-exposure response.
- Skills and adoption
- Train consultants and PIs on agent AUP, polling limits, and non-interactive editing patterns; teach agents via AGENTS.md/site skills.
- Governance checkpoints
- Telemetry ethics/sanitization approval
- AUP publication
- Trial-QoS anti-gaming review
- Secret-exposure incident retrospective
- Acceptance criteria
- Agent-labeled share of submissions published monthly
- Median scheduler polls/job under site limit for high-volume agent users
- Zero unresolved exposed keys older than 7 days
- Trial QoS abuse rate below agreed threshold
- Risks
- Misclassification of human vs agent sessions
- PI backlash to tighter login-node limits
- Agents learning to game trial priority
- Federated memory leaking unpublished ideas
Implementation considerations
Lighthouse Advisory interpretation across the operating dimensions a public-sector buyer must settle before this evidence becomes a design. Each note answers the question under its heading for this specific source.
Architecture and integration
What must connect, and where does the AI sit in the workflow?
Treat agents as facility-aware principals: trial-aware QoS with anti-gaming bounds, event-driven job status instead of tight squeue polls, node-local/NVMe working sets for small I/O, and hooks below the agent (Landlock/seccomp) for dangerous shell actions.
Governance
Who approves, reviews and stays accountable for outcomes?
Codify account-holder responsibility for agent actions; publish agent AUP with login-node concurrency limits; require disclosure of agent harnesses in allocation reviews; prefer co-design over bans that agents can evade via the same shell interfaces.
Security and privacy
What data, permissions and controls need testing?
Move safety checks beneath the agent; taint fetched README/setup content; redact secrets from argv/history/logs; default-deny non-loopback service binds; expect prompt injection via clones/packages more than search.
Accessibility and workforce
Who is affected, and what skills or accommodations follow?
Agent productivity gains concentrate among users who adopt harnesses; facilities must document agent skills and avoid login-node bans that push work underground without training safer patterns.
Procurement
What should contracts, pricing and exit terms secure?
Do not procure solely on agent capability benchmarks; require facility co-design features (event APIs, metadata isolation, secret redaction) in research-computing refresh and vendor agent offerings.
Operating model
Which teams own the service once it runs?
Shift from human-paced interactive support to agent-aware operations: continuous telemetry, AUP enforcement automation, and joint RC–InfoSec ownership of agent principals.
Publication history
- 2026-10-03Research Universities · Issue 263 resources
Stable resource ID: harvard-fasrc-coding-agents-hpc-measurement-20260930