From the Research Universities edition of October 3, 2026
HPC site profiles: verified agentic docs+measurement+pilots; field accuracy 0.78–0.91 across Anvil/Stampede3/CRC
University of Notre Dame (Computer Science / Center for Research Computing) — arXiv preprint; NSF OAC-2411436 · Higher education research computing / scientific workflow portability · United States (Purdue Anvil, TACC Stampede3, Notre Dame CRC); transferable profile method for other Slurm/HTCondor sites
- Publisher
- arXiv 2610.00971
- Original publication
- October 1, 2026 (arXiv 2610.00971v1)
- Source retrieved
- 2026-10-04
What happened
The Other Half of Workflow Portability (arXiv 2610.00971, submitted 1 Oct 2026; Notre Dame) argues packages/containers/backpacks capture what a workflow needs but not how each HPC site must be used. Authors automate an evidence-backed HPC site profile via login-node measurement, a bounded read-only LM agent over official docs with deterministic citation checks, and approved pilot jobs; a rule—not the model—decides what enters the profile. Across 108 builds (3 sites × 4 models × 3 retrieval modes × 3 reps), precision is 0.957–0.992, field accuracy 0.783–0.910, and all 5,592 accepted citations pass mechanical integrity. BM25 beats full-corpus on quality/cost; stronger models differ by <1 pp after verification. Preflight correctly planned an Anvil GPU backpack and blocked Stampede3 for insufficient CPU cores per node, but falsely blocked Notre Dame CRC when per-PI /group storage was missing from the profile—showing preflight reliability tracks profile completeness. Measurement vs docs disagreed on walltime caps (scheduler often “unlimited” while docs state limits).
Why it matters
Campus research computing deploying portable AI/science workflows needs machine-readable, evidence-backed site profiles with verified provenance—not LLM answers alone or human tribal knowledge—before agents submit across facilities.
Evidence and measured results
Opened https://arxiv.org/abs/2610.00971 via WebFetch (full abstract + evaluation tables); check_sources: not archived (neighbor 2610.00980 RAC is archived).
Limitations and uncertainty
Preprint; three sites; Floability Backpack preflight only; Notre Dame false failure from missing per-PI storage; field accuracy gaps 9–22% remain; not a measured production agent-load study.
Put this evidence to work
Lighthouse Advisory interpretation, grounded in this source. Enriched 2026-10-03; this does not change the original publication date. Each role follows the same checklist so takeaways compare across sources.
Sales
Role takeaway
- Customer problem
- Even containerized or backpacked scientific workflows need manual retuning at each HPC site because site policy and resource shape are not portable artifacts.
- Stakeholders
- Research computing directors
- Workflow/platform engineers
- Multi-campus PIs
- Regional cyberinfrastructure consortia
- Discovery questions
- Which site fields today live only in tribal knowledge or stale docs?
- How often do cross-site submissions fail after queue wait?
- Who may approve pilot jobs that probe network/storage behavior?
- Value hypothesis
- Evidence-backed profiles with deterministic verification can preflight workflows into grounded plans or early explainable failures across Slurm/HTCondor sites.
- Potential engagement
- Eight-week profile build and preflight pilot across three campus or regional resources with precision/recall and case-study report.
- Unsupported claims
- Fully automatic multi-site deployment without operator approval
- Guaranteed capture of all site knowledge
- Model-only profile authority
Pre-sales engineering
Role takeaway
- Fit
- Multi-site workflow portability layer for Slurm/HTCondor campuses using backpacks or similar portable specs.
- Architecture and integration
- Login probes + bounded docs agent + pilot registry → verified profile → preflight validator; LM proposes cited fields only.
- Prerequisites
- Allowed-domain doc corpus
- Login measurement scripts
- Pilot approval workflow
- Normalized workflow adapter
- Constraints
- Three-site evaluation
- Backpack-only preflight adapter
- Missing fields cause false blocks
- Docs/measurement conflicts common on limits
- Security
- Read-only agent; no model job submission; domain allowlists; explicit pilot approval; record conflicting evidence.
- Proof of value
- Field precision/recall vs hand profiles
- Citation integrity rate
- Correct block vs false block case studies
- Token/latency vs retrieval mode
Delivery
Role takeaway
- Work
- Build profiles per site, validate against operator ground truth, wire preflight into submission UX, track profile gaps from false failures, refresh on staleness SLA.
- Dependencies
- Site operators
- Workflow authors
- Identity/account mapping
- Pilot job quotas
- Ownership
- Research computing owns profile pipeline and pilot approvals; research groups own workflow requirement specs.
- Skills and adoption
- Train consultants to read evidence reports and remediate unset fields; teach users to treat preflight unknowns as profile work, not overrides.
- Governance checkpoints
- Pilot approval charter
- Profile freshness policy
- Conflict-resolution precedence review
- False-failure retrospective
- Acceptance criteria
- Precision ≥0.95 on agreed field set
- 100% mechanical citation integrity on accepted fields
- Preflight never submits on known incompatibility
- Open profile-debt tickets for each false block
- Risks
- Stale documentation
- Over-abstention blocking feasible runs
- Pilot jobs consuming scarce allocation
- Schema gaps for non-backpack engines
Implementation considerations
Lighthouse Advisory interpretation across the operating dimensions a public-sector buyer must settle before this evidence becomes a design. Each note answers the question under its heading for this specific source.
Architecture and integration
What must connect, and where does the AI sit in the workflow?
Separate measured_config, documented_policy, and demonstrated_capability with explicit precedence; keep LM authority to propose cited fields only; feed pilots from a reviewed registry; preflight to execution plan or explainable failure without submitting.
Governance
Who approves, reviews and stays accountable for outcomes?
Require explicit approval for live pilots; version profiles with freshness SLAs; treat unresolved fields as abstention rather than guessed safety; use deployment outcomes to drive profile gap closure.
Security and privacy
What data, permissions and controls need testing?
Bound the docs agent to allowed domains and read-only tools; do not let the model submit jobs or write the profile; record conflicts instead of silently preferring the first source found.
Accessibility and workforce
Who is affected, and what skills or accommodations follow?
Profiles lower the expertise barrier for multi-site runs if consultants maintain them; otherwise abstention shifts burden back to expert users.
Procurement
What should contracts, pricing and exit terms secure?
When buying workflow platforms or regional AI hubs, require machine-readable site contracts with provenance—not chatbot support alone.
Operating model
Which teams own the service once it runs?
Make site knowledge a versioned operational product with refresh SLAs and pilot governance, shared across users and agent submitters.
Publication history
- 2026-10-03Research Universities · Issue 263 resources
Stable resource ID: hpc-site-profiles-agentic-discovery-20261001