{"resourceId":"diamond-agent-federated-hpc-placement-20260905","versions":[{"version":"external-8c37e036a82d81edc2ca5349e3a54a18102c8eeba4a1f21040c32f4fb17a6285","resource":{"id":"diamond-agent-federated-hpc-placement-20260905","title":"Diamond Agent: federated HPC placement cuts median extra completion 42s→4s (10.5×) across four supercomputers","organization":"Rutgers / NCSA / University of Chicago / Argonne / UW–Madison / University of Rochester (Xie et al.)","sector":"University/national research computing / agentic HPC orchestration","geography":"United States (NCSA, TACC, RCAC and related); federation-limited to authorized sites","publishedAt":"September 5, 2026 (arXiv v1)","publicationDate":"2026-09-05","eventDate":null,"sourceName":"arXiv","sourceLabel":"Academic preprint — Diamond Agent federated HPC resources as a service","sourceUrl":"https://arxiv.org/html/2609.06181","evidenceClass":"academic-research","outcomeClass":"effective","topics":["infrastructure","developers-agents","operating-model","governance-procurement"],"finding":"Diamond Agent provides an agent-facing control plane over Diamond/Globus for multi-supercomputer workflows with ResourceEnvelopes, deterministic feasibility, live best-feasible placement, and event-driven continuation. Evaluation reports 27 hours of federation telemetry and 19 matched multi-site submission rounds (83 jobs) across Delta, DeltaAI, Anvil, and Vista; relative to a fixed-Delta baseline, best-feasible reduced median additional completion time versus the fastest observed placement from 42 seconds to 4 seconds (10.5×). Two coding-agent workflows resumed after 37 and 84 minutes with zero model-side status calls.","sledRelevance":"Why now: Unarchived 5 Sep measured multi-site placement outcomes—newly actionable after 2026-09-20 NAIRR/campus capacity stories—show federation telemetry and event-driven agents can cut fixed-site wait penalties without putting long-lived agents on every login node.","evidence":"19 rounds/83 jobs; 42s→4s median additional completion; 10.5×; 543 snapshots/27h; 0 status polls over 37–84 min intervals.","architectureImplications":"Research computing should expose typed federation skills and persistent orchestrators above Slurm/Globus rather than per-login coding agents.","governanceImplications":"Keep site admission/credit authority; treat AI-drafted envelopes as auditable proposals under deterministic rules.","securityPrivacyImplications":"Exclude credentials from prompts; enforce authorization/credit; audit placement and continuation events.","caveats":"Large p95 tails; no user-code sandbox; transfer duration not jointly optimized; four-system study; preprint.","streamIds":["research"],"roles":{"sales":"Sales framing: Multi-allocation research computing users waste wall-clock waiting on a favorite cluster while other authorized sites are freer. Diamond Agent evidence supports a bounded pitch around agent-facing federated HPC control—reporting 27 hours of telemetry and 19 matched multi-site rounds (83 jobs) across four production supercomputers where best-feasible placement cut median additional completion time versus the fastest observed placement from 42 seconds to 4 seconds (10.5×)—not a claim of globally optimal scheduling or NAIRR-wide deployment. Discovery should map Globus endpoints, credits, and login-node agent policies. Unsupported: guaranteed fastest placement every round, sandboxing of user code, or joint transfer+queue optimality.","engineering":"Engineering framing: Fit is an agent gateway over Diamond/Globus Compute with ResourceEnvelope drafting, deterministic feasibility rules, best-feasible placement from live snapshots, and event-driven continuation that reactivates agents on results rather than login-node polling. Architecture integration points are Globus Transfer, site profiles, Task Orchestrator state, and typed skills—not replacing Slurm. Prerequisites: user allocations on ≥2 sites, Globus auth, and persistent control service. Constraints: estimator tails remain large (p95 additional completion 3000s); sizing separated from placement; no user-code isolation; four-system testbed. Security: keep credentials out of prompts; enforce credit/authorization rules. Proof of value is matched-round telemetry plus zero model-side status calls during async intervals.","delivery":"Delivery framing: Pilot Diamond Agent-style federated placement for one multi-site workflow with ResourceEnvelope templates and event-driven continuation. Dependencies: center identity/Globus teams, research computing, and PI workflow owners. Ownership: RC owns control-plane ops; centers retain admission authority. Skills adoption trains users to author envelopes and read placement traces. Governance checkpoints: credit policy, blocked-partition rules, and continuation event audit. Acceptance: measured median extra-completion improvement vs fixed-site baseline on a registered workload set; risks include stale snapshots, soft diversity constraint mistakes, and login-node policy conflicts."},"roleDetails":{"sales":{"customerProblem":"Users with multiple HPC allocations still pin work to one site and burn wait time while other authorized capacity sits freer.","stakeholders":["Research computing directors","Multi-allocation PIs","HPC center operators","Workflow engineers"],"discoveryQuestions":["Which sites/allocations are authorized?","What Globus endpoints exist?","What is the fixed-site baseline wait?","Which jobs can be re-placed when pending?"],"valueHypothesis":"Live-state best-feasible placement across authorized sites can cut typical extra completion time versus a fixed favorite site (authors: 42s→4s median additional completion) when envelopes and deterministic rules are enforced.","potentialEngagement":"Pilot federated placement on one multi-site workflow with matched-round measurement against fixed-site baseline.","unsupportedClaims":["Always selects the absolute fastest site","Sandboxed untrusted code","NAIRR-wide production guarantee"]},"engineering":{"fit":"Agent-facing federated HPC control plane with envelopes, live snapshots, deterministic placement, and event-driven continuation.","architectureIntegration":"Sit above Globus Compute/Transfer and site Slurm dialects; persist tasks/events outside the model context.","prerequisites":["≥2 authorized systems","Globus auth","Site profiles","Persistent orchestrator host"],"constraints":["Large tail estimation error","No joint transfer+queue optimization yet","No code sandbox","Four-system evaluation"],"security":"Never put cluster credentials in prompts; enforce authorization/credit; audit placement traces.","proofOfValue":["Matched multi-site rounds vs fixed baseline","Async continuation with zero status polls","UNSCHEDULABLE rejection of impossible envelopes"]},"delivery":{"work":"Deploy control service, configure site profiles, run pilot workloads with fixed vs best-feasible comparison.","dependencies":["Globus/identity","Center policies","RC platform","PI workflow definition"],"ownership":"RC owns control plane; centers own admission; PI owns scientific workflow.","skillsAdoption":"Train users to write ResourceEnvelopes and interpret decision traces instead of hard-coding sites.","governanceCheckpoints":["Credit/allowlist review","Blocked partition policy","Event/continuation audit"],"acceptanceCriteria":["Documented median extra-completion improvement","Placement traces retained","Written limits on estimator accuracy"],"risks":["Stale federation snapshots","Policy conflicts on login nodes","Overtrusting AI-drafted envelopes"]}},"retrievedAt":"2026-09-22T03:18:53Z","enrichedAt":"2026-09-22T03:18:53Z","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Helps multi-allocation users without login-node agent sprawl; still requires Globus literacy and envelope authoring skills.","procurementImplications":"Procure federation control planes with matched-round measurement requirements against fixed-site baselines—not agent demos alone.","operatingModelImplications":"Operating model separates active agent time from batch lifetime and funds persistent orchestration as shared RC services.","sourceVerification":{"openedUrl":"https://arxiv.org/html/2609.06181","referenceExcerpt":"reduces the median additional completion time relative to the fastest observed placement from 42 seconds to 4 seconds","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}