From the Research Universities edition of September 18, 2026
Chronicle cut-point replay turns agent incidents into CI tests that catch unsafe-tool mutants
Independent academic authors (arXiv preprint; AgentPlane GitHub) · Research software engineering / LLM agent reliability · International authorship; transferable CI practice for research agents
- Publisher
- arXiv
- Original publication
- September 17, 2026 (arXiv v1)
- Source retrieved
- 2026-09-19
What happened
Chronicle records LLM-agent runs at non-deterministic boundaries as immutable envelopes and supports cut-point replay so new code can be tested against recorded incidents in CI. On six recorded failures with simulated model boundaries, recording adds 23 µs per crossing, full replay is bit-stable across 20 repetitions with zero model calls, and cut-point tests catch every mutant that lets a recorded unsafe action through while a stub-all-boundaries baseline catches none.
Why it matters
Published on the research stream's last completed date (2026-09-17) and still unarchived—newly available control evidence for agent reliability complementary to SciRIGOR/ECAS scrutiny already in the archive.
Evidence and measured results
Six-incident benchmark; recording overhead; bit-stable replay; mutation study vs stub baseline; public GitHub claim.
Limitations and uncertainty
Simulated model boundaries; small incident set; not a multi-institution deployment study; does not prove scientific result correctness.
Put this evidence to work
Lighthouse Advisory interpretation, grounded in this source. Enriched 2026-09-19; this does not change the original publication date. Each role follows the same checklist so takeaways compare across sources.
Sales
Role takeaway
- Customer problem
- LLM agent failures are hard to reproduce, so research teams cannot regression-test tool and router changes against real incidents.
- Stakeholders
- Research software engineering
- Research computing
- Security/compliance for tool actions
- PI sponsors of agent workflows
- Discovery questions
- Which tool actions are unsafe if replayed?
- Do you record agent traces today?
- What CI exists for agent code?
- How do you test guards after model upgrades?
- Value hypothesis
- Cut-point replay can turn recorded incidents into CI tests that catch unsafe-tool regressions stubs miss, if boundaries are instrumented.
- Potential engagement
- Instrument one agent, record a failure corpus, and add cut-point tests for tool guards.
- Unsupported claims
- Fixes non-determinism of production LLM APIs
- Guarantees scientific result correctness
- Coverage beyond small incident benchmarks
Pre-sales engineering
Role takeaway
- Fit
- Boundary recording + cut-point replay regression harness for LLM agents.
- Architecture and integration
- Wrap model/tool boundaries; store immutable envelopes; replay selected boundaries live in CI.
- Prerequisites
- Boundary hooks
- Envelope storage
- Assertions on unsafe actions
- CI integration
- Constraints
- Simulated model boundaries in paper benchmark
- Live tool state drift
- Small public incident set
- Security
- Redact secrets from envelopes; control access to recorded research prompts and tool payloads.
- Proof of value
- Bit-stable full replay
- Mutation catch rate on guarded tools
- Stub-baseline comparison
- Recording overhead per crossing
Delivery
Role takeaway
- Work
- Add instrumentation, build incident library, wire cut-point tests into CI, train owners on assertions.
- Dependencies
- Agent codebase access
- CI runners
- Data-handling policy for envelopes
- Ownership
- RSE/platform owns harness; research owners define unsafe-action policy.
- Skills and adoption
- Teach writing assertions and choosing cut-points rather than stubbing every boundary.
- Governance checkpoints
- Envelope retention review
- Secret scanning of records
- Periodic mutation testing
- Acceptance criteria
- Zero model calls on full replay
- All guarded-tool mutants caught in pilot set
- Documented redaction SOP
- Risks
- Envelope leakage
- False confidence from simulated boundaries
- CI flakiness if live tools drift
Implementation considerations
Lighthouse Advisory interpretation across the operating dimensions a public-sector buyer must settle before this evidence becomes a design. Each note answers the question under its heading for this specific source.
Architecture and integration
What must connect, and where does the AI sit in the workflow?
Research agent platforms need boundary recording and selective live replay hooks—not only trace dashboards.
Governance
Who approves, reviews and stays accountable for outcomes?
Define unsafe-action assertions and envelope retention before production agent tools.
Security and privacy
What data, permissions and controls need testing?
Recorded envelopes may contain prompts, tokens, and research data requiring redaction and access control.
Accessibility and workforce
Who is affected, and what skills or accommodations follow?
Adds RSE/CI skills for assertion writing; reduces reliance on irreproducible manual re-runs.
Procurement
What should contracts, pricing and exit terms secure?
Require agent platforms to export boundary records and support cut-point or equivalent regression tests.
Operating model
Which teams own the service once it runs?
Treats agent reliability as continuous integration work owned jointly by RSE and research sponsors.
Publication history
- 2026-09-18Research Universities · Issue 134 resources
Stable resource ID: chronicle-cutpoint-replay-agent-regression-20260917