{"resourceId":"hpc-site-profiles-agentic-discovery-20261001","versions":[{"version":"external-81e144c199153bc8bc29119b3e3692e46a69fca827ba43ad39efac75e4654eb1","resource":{"id":"hpc-site-profiles-agentic-discovery-20261001","title":"HPC site profiles: verified agentic docs+measurement+pilots; field accuracy 0.78–0.91 across Anvil/Stampede3/CRC","organization":"University of Notre Dame (Computer Science / Center for Research Computing) — arXiv preprint; NSF OAC-2411436","sector":"Higher education research computing / scientific workflow portability","geography":"United States (Purdue Anvil, TACC Stampede3, Notre Dame CRC); transferable profile method for other Slurm/HTCondor sites","publishedAt":"October 1, 2026 (arXiv 2610.00971v1)","publicationDate":"2026-10-01","eventDate":null,"sourceName":"arXiv 2610.00971","sourceLabel":"Academic preprint (arXiv)","sourceUrl":"https://arxiv.org/abs/2610.00971","evidenceClass":"academic-research","outcomeClass":"mixed","topics":["developers-agents","infrastructure","operating-model","governance-procurement"],"finding":"The Other Half of Workflow Portability (arXiv 2610.00971, submitted 1 Oct 2026; Notre Dame) argues packages/containers/backpacks capture what a workflow needs but not how each HPC site must be used. Authors automate an evidence-backed HPC site profile via login-node measurement, a bounded read-only LM agent over official docs with deterministic citation checks, and approved pilot jobs; a rule—not the model—decides what enters the profile. Across 108 builds (3 sites × 4 models × 3 retrieval modes × 3 reps), precision is 0.957–0.992, field accuracy 0.783–0.910, and all 5,592 accepted citations pass mechanical integrity. BM25 beats full-corpus on quality/cost; stronger models differ by <1 pp after verification. Preflight correctly planned an Anvil GPU backpack and blocked Stampede3 for insufficient CPU cores per node, but falsely blocked Notre Dame CRC when per-PI /group storage was missing from the profile—showing preflight reliability tracks profile completeness. Measurement vs docs disagreed on walltime caps (scheduler often “unlimited” while docs state limits).","sledRelevance":"Campus research computing deploying portable AI/science workflows needs machine-readable, evidence-backed site profiles with verified provenance—not LLM answers alone or human tribal knowledge—before agents submit across facilities.","evidence":"Opened https://arxiv.org/abs/2610.00971 via WebFetch (full abstract + evaluation tables); check_sources: not archived (neighbor 2610.00980 RAC is archived).","architectureImplications":"Separate measured_config, documented_policy, and demonstrated_capability with explicit precedence; keep LM authority to propose cited fields only; feed pilots from a reviewed registry; preflight to execution plan or explainable failure without submitting.","governanceImplications":"Require explicit approval for live pilots; version profiles with freshness SLAs; treat unresolved fields as abstention rather than guessed safety; use deployment outcomes to drive profile gap closure.","securityPrivacyImplications":"Bound the docs agent to allowed domains and read-only tools; do not let the model submit jobs or write the profile; record conflicts instead of silently preferring the first source found.","caveats":"Preprint; three sites; Floability Backpack preflight only; Notre Dame false failure from missing per-PI storage; field accuracy gaps 9–22% remain; not a measured production agent-load study.","streamIds":["research"],"roles":{"sales":"Sales framing: portable AI/science workflows still fail at the site half—scheduler flags, storage visibility, and network policy that packages do not carry. Notre Dame’s HPC site profiles (arXiv 2610.00971, 1 Oct 2026) build evidence-backed profiles from login measurement, bounded doc agents, and approved pilots, with rules—not the model—admitting fields (precision 0.957–0.992; field accuracy up to 0.910 across Anvil/Stampede3/CRC). Pitch a multi-site preflight service that fails closed with explainable gaps before wasting queue time.","engineering":"Engineering framing: implement a closed schema with source precedence, BM25 retrieval over frozen corpora, deterministic citation validation, and a pilot registry that operators must approve before live probes. Wire preflight to emit scheduler arguments or named incompatibilities without submission. Prefer verified abstention over guessed defaults when documentation and sinfo disagree on walltime caps or GPU counts. Keep the language model read-only: it may propose cited fields but cannot submit jobs or write the profile.","delivery":"Delivery framing: stand up profile builds for each campus or regional resource, set freshness recheck intervals, require operator approval for pilots, and gate cross-site agent submissions on a preflight pass. Own the profile pipeline in research computing; workflow authors own requirement specs. Accept when precision stays high, citation integrity remains perfect on accepted fields, and false blocks from missing fields are tracked as profile debt with remediation owners—not blamed on end users."},"roleDetails":{"sales":{"customerProblem":"Even containerized or backpacked scientific workflows need manual retuning at each HPC site because site policy and resource shape are not portable artifacts.","stakeholders":["Research computing directors","Workflow/platform engineers","Multi-campus PIs","Regional cyberinfrastructure consortia"],"discoveryQuestions":["Which site fields today live only in tribal knowledge or stale docs?","How often do cross-site submissions fail after queue wait?","Who may approve pilot jobs that probe network/storage behavior?"],"valueHypothesis":"Evidence-backed profiles with deterministic verification can preflight workflows into grounded plans or early explainable failures across Slurm/HTCondor sites.","potentialEngagement":"Eight-week profile build and preflight pilot across three campus or regional resources with precision/recall and case-study report.","unsupportedClaims":["Fully automatic multi-site deployment without operator approval","Guaranteed capture of all site knowledge","Model-only profile authority"]},"engineering":{"fit":"Multi-site workflow portability layer for Slurm/HTCondor campuses using backpacks or similar portable specs.","architectureIntegration":"Login probes + bounded docs agent + pilot registry → verified profile → preflight validator; LM proposes cited fields only.","prerequisites":["Allowed-domain doc corpus","Login measurement scripts","Pilot approval workflow","Normalized workflow adapter"],"constraints":["Three-site evaluation","Backpack-only preflight adapter","Missing fields cause false blocks","Docs/measurement conflicts common on limits"],"security":"Read-only agent; no model job submission; domain allowlists; explicit pilot approval; record conflicting evidence.","proofOfValue":["Field precision/recall vs hand profiles","Citation integrity rate","Correct block vs false block case studies","Token/latency vs retrieval mode"]},"delivery":{"work":"Build profiles per site, validate against operator ground truth, wire preflight into submission UX, track profile gaps from false failures, refresh on staleness SLA.","dependencies":["Site operators","Workflow authors","Identity/account mapping","Pilot job quotas"],"ownership":"Research computing owns profile pipeline and pilot approvals; research groups own workflow requirement specs.","skillsAdoption":"Train consultants to read evidence reports and remediate unset fields; teach users to treat preflight unknowns as profile work, not overrides.","governanceCheckpoints":["Pilot approval charter","Profile freshness policy","Conflict-resolution precedence review","False-failure retrospective"],"acceptanceCriteria":["Precision ≥0.95 on agreed field set","100% mechanical citation integrity on accepted fields","Preflight never submits on known incompatibility","Open profile-debt tickets for each false block"],"risks":["Stale documentation","Over-abstention blocking feasible runs","Pilot jobs consuming scarce allocation","Schema gaps for non-backpack engines"]}},"retrievedAt":"2026-10-03T22:06:40-05:00","enrichedAt":"2026-10-03T22:08:30-05:00","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Profiles lower the expertise barrier for multi-site runs if consultants maintain them; otherwise abstention shifts burden back to expert users.","procurementImplications":"When buying workflow platforms or regional AI hubs, require machine-readable site contracts with provenance—not chatbot support alone.","operatingModelImplications":"Make site knowledge a versioned operational product with refresh SLAs and pilot governance, shared across users and agent submitters.","sourceVerification":{"openedUrl":"https://arxiv.org/abs/2610.00971","referenceExcerpt":"Every field is verified against its evidence or discarded, so a rule, not the model, decides what enters the profile.","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}