Lighthouse AdvisorySLED AI Adoption Intelligence

Higher Education & ResearchIssue 134 records

Research Universities

Four newly archived sources advance the research stream past the 2026-09-17 Paper2Agent/ECAS/SciRIGOR edition along lab continuity, HPC agent runtime, and evaluation controls. LabAgent shows a lab-specific skill harness beating Claude Code/Codex on many life-science reproduction entries while still failing most GeneBench-Pro problems. RASER reports ~39% HPC makespan reduction for agentic Slurm workloads with user-space checkpointing. Chronicle (17 Sep) turns recorded agent incidents into cut-point regression tests that catch unsafe-tool mutants stubbed baselines miss. A May disclosure audit (why now) finds agent benchmarks still average 0.38 disclosure with no inference-cost or content-addressed environment pins. One pattern: treat agent research tooling as incomplete without executable verification and disclosure controls.

Evidence records
4
Cross-source patterns
1
Evidence classes
3 academic research1 independent research
Outcomes
2 mixed1 emerging1 cautionary
Source freshness
2 new this fortnight1 recent1 older, newly relevant
Research completed
2026-09-19

Choose a role to see its takeaway beside every record in the ledger.

Patterns across the evidence

Synthesis · Lighthouse Advisory interpretation

1 pattern, each supported by at least two sources

  1. Research agents need executable verification and disclosure—not task completion alone

    LabAgent's verified skills and figure reproduction, Chronicle's cut-point regression tests that catch unsafe mutants stubbed baselines miss, and Moghadasi/Ghaderi's 0.38 agent-benchmark disclosure score jointly show that campus research AI programs should demand locked executables, incident replay, and cost/environment disclosure—not leaderboard completion rates alone.

    Operating questionBefore funding or deploying a research agent stack, which executable verification gates and evaluation-disclosure fields are required evidence?

    Supporting evidenceYale University and collaborators (arXiv preprint)Independent academic authors (arXiv preprint; AgentPlane GitHub)BrightMind AI / Texas Tech University and University of Texas at Arlington (arXiv preprint)

Evidence ledger

Full record · every source keeps its link and limitations

4 records
  1. Academic researchMixedNew this fortnight

    LabAgent verified lab skills beat Claude Code/Codex on many entries but clear only 2/10 GeneBench-Pro problems

    LabAgent converts a laboratory's papers, code, tutorials, and issues into verified executable skills with corrective-experience memory. Against Claude Code (Opus 4.8) and Codex CLI (GPT-5.5), it landed inside published standard deviations on 32 of 65 entries, achieved best mean rank 2.62, and ranked first on 13/65 versus 8 for each commercial harness, while GeneBench-Pro mean partial score was only 31.4 with 2 of 10 binary verdicts.

    Yale University and collaborators (arXiv preprint)United States (author affiliations); life-science benchmark tasksSeptember 11, 2026 (arXiv v1)

    Why it matters, evidence and limitations
    Why it matters
    Why now: 2026-09-17 edition archived Paper2Agent interactive papers and ECAS/SciRIGOR agent-validation limits; LabAgent (11 Sep, still unarchived) adds lab-continuity implementation evidence with measured commercial-harness comparisons and an explicit hard-problem failure mode useful for research-computing delivery.
    Evidence and measured results
    Author comparisons on drug property, biomedical, protein variant, and statistical genetics tasks; figure-reproduction claim; GeneBench-Pro per-problem table with mean 31.4 and 2/10 verdicts.
    Limitations and uncertainty
    Author-evaluated preprint; commercial CLI versions author-pinned; GeneBench-Pro shows large remaining failure modes; not a multi-campus deployment study.
    Academic preprint — LabAgent reproduce-and-discovery harness Role takeaways & implementation
  2. Academic researchMixedRecent

    RASER user-space Slurm agent runtime cuts makespan ~39% vs static partitioning

    RASER extends Slurm with agentic job arrays, filesystem work-stealing queues, application-level checkpointing plus requeue, and Apptainer isolation without image changes. Authors report nearly 39% makespan reduction versus static partitioning (38.8% in the detailed scenario) with near-full CPU utilization and estimated NVMe checkpoint/restore under 30 seconds, exercising resilience with injected preemption.

    TU Dresden / TU Chemnitz collaborators (arXiv preprint)Germany (author institutions); transferable to Slurm campuses with limitsSeptember 3, 2026 (arXiv v1)

    Why it matters, evidence and limitations
    Why it matters
    Why now: Prior research editions covered capacity awards and validation-gated agents; this still-unarchived Sep 2026 runtime paper supplies measured HPC scheduling benefits/constraints for deploying agentic scientific workflows on existing campus clusters.
    Evidence and measured results
    Comparative makespan vs static partitioning; CPU utilization claims; checkpoint overhead estimate; fault injection via scancel SIGTERM; user-space/no-kernel-privilege design.
    Limitations and uncertainty
    Scheduling metrics ≠ scientific outcomes; authors flag FS lock contention at higher concurrency; site Apptainer/Slurm assumptions vary.
    Academic preprint — RASER HPC agent scheduling runtime Role takeaways & implementation
  3. Academic researchEmergingNew this fortnight

    Chronicle cut-point replay turns agent incidents into CI tests that catch unsafe-tool mutants

    Chronicle records LLM-agent runs at non-deterministic boundaries as immutable envelopes and supports cut-point replay so new code can be tested against recorded incidents in CI. On six recorded failures with simulated model boundaries, recording adds 23 µs per crossing, full replay is bit-stable across 20 repetitions with zero model calls, and cut-point tests catch every mutant that lets a recorded unsafe action through while a stub-all-boundaries baseline catches none.

    Independent academic authors (arXiv preprint; AgentPlane GitHub)International authorship; transferable CI practice for research agentsSeptember 17, 2026 (arXiv v1)

    Why it matters, evidence and limitations
    Why it matters
    Published on the research stream's last completed date (2026-09-17) and still unarchived—newly available control evidence for agent reliability complementary to SciRIGOR/ECAS scrutiny already in the archive.
    Evidence and measured results
    Six-incident benchmark; recording overhead; bit-stable replay; mutation study vs stub baseline; public GitHub claim.
    Limitations and uncertainty
    Simulated model boundaries; small incident set; not a multi-institution deployment study; does not prove scientific result correctness.
    Academic preprint — Chronicle agent cut-point replay Role takeaways & implementation
  4. Independent researchCautionaryNewly relevant · May 2026

    Pilot audit: agent benchmarks average 0.38 disclosure; none report inference cost or pinned eval images

    Moghadasi and Ghaderi score twelve canonical benchmark papers on disclosure of how evaluations were run. Eight agent benchmarks average 0.38/1.0 versus 0.66 for four classical static benchmarks; none of the eight disclose inference cost in any form; none fully disclose a content-addressed container image of the evaluation environment. Authors score disclosure only and call for multi-rater follow-up after a single-auditor pass.

    BrightMind AI / Texas Tech University and University of Texas at Arlington (arXiv preprint)United States (author affiliations); applies to widely cited agent benchmarksMay 20, 2026 (arXiv v1)

    Why it matters, evidence and limitations
    Why it matters
    Why now: Older than last completed run, but newly relevant because this edition's LabAgent/Chronicle claims intensify campus pressure to trust agent scores—while this still-unarchived audit shows common agent benchmarks remain under-disclosed on cost and environments.
    Evidence and measured results
    Five-field schema; mean scores 0.38 vs 0.66; zero inference-cost disclosure among eight agent papers; missing content-addressed images; single-auditor limitation stated.
    Limitations and uncertainty
    May 2026 single-auditor pilot; disclosure ≠ correctness per authors; does not directly audit LabAgent/Chronicle/RASER.
    Independent academic pilot audit — agent benchmark disclosure Role takeaways & implementation

How to read this edition

Source findings, measured results and limitations come from the cited publications. Patterns, operating questions, role takeaways and implementation considerations are Lighthouse Advisory interpretation, stated as questions to validate locally rather than guaranteed outcomes. Vendor and operator claims are labeled as claims. Full research method.

Academic research
Research produced through an academic institution or peer-reviewed venue.
Independent research
Research conducted outside the implementing organization.