{"resourceId":"chronicle-cutpoint-replay-agent-regression-20260917","versions":[{"version":"external-a54edf6d5f648b68aa6cfc7cc28e23bb9ce270f499c6d1a3085a8b179e6eb1c2","resource":{"id":"chronicle-cutpoint-replay-agent-regression-20260917","title":"Chronicle cut-point replay turns agent incidents into CI tests that catch unsafe-tool mutants","organization":"Independent academic authors (arXiv preprint; AgentPlane GitHub)","sector":"Research software engineering / LLM agent reliability","geography":"International authorship; transferable CI practice for research agents","publishedAt":"September 17, 2026 (arXiv v1)","publicationDate":"2026-09-17","eventDate":null,"sourceName":"arXiv","sourceLabel":"Academic preprint — Chronicle agent cut-point replay","sourceUrl":"https://arxiv.org/html/2609.20625","evidenceClass":"academic-research","outcomeClass":"emerging","topics":["developers-agents","data-security","operating-model","governance-procurement"],"finding":"Chronicle records LLM-agent runs at non-deterministic boundaries as immutable envelopes and supports cut-point replay so new code can be tested against recorded incidents in CI. On six recorded failures with simulated model boundaries, recording adds 23 µs per crossing, full replay is bit-stable across 20 repetitions with zero model calls, and cut-point tests catch every mutant that lets a recorded unsafe action through while a stub-all-boundaries baseline catches none.","sledRelevance":"Published on the research stream's last completed date (2026-09-17) and still unarchived—newly available control evidence for agent reliability complementary to SciRIGOR/ECAS scrutiny already in the archive.","evidence":"Six-incident benchmark; recording overhead; bit-stable replay; mutation study vs stub baseline; public GitHub claim.","architectureImplications":"Research agent platforms need boundary recording and selective live replay hooks—not only trace dashboards.","governanceImplications":"Define unsafe-action assertions and envelope retention before production agent tools.","securityPrivacyImplications":"Recorded envelopes may contain prompts, tokens, and research data requiring redaction and access control.","caveats":"Simulated model boundaries; small incident set; not a multi-institution deployment study; does not prove scientific result correctness.","streamIds":["research"],"roles":{"sales":"Sales framing: Research and research-computing teams adopting LLM agents face failures that will not reappear on re-run, so 'we fixed it' is hard to prove. Chronicle evidence supports selling cut-point replay regression tests built from recorded incidents—catching unsafe tool mutants that stubbed-boundary tests miss—not autonomous science. Position beside agent platforms as CI quality infrastructure. Unsupported: bitwise model reproducibility in production LLM APIs or guaranteed coverage beyond the six-incident benchmark.","engineering":"Engineering framing: Fit is record-at-nondeterministic-boundaries with immutable envelopes and selective live replay for tool/router changes. Integrate into CI for research agents that call tools with side effects. Prerequisites: boundary instrumentation, storage for envelopes, assertions on unsafe actions. Constraints: authors' simulated model boundaries and small incident set; tool state still drifts if live systems change. Security: treat recorded envelopes as sensitive if they contain tokens, prompts, or research data.","delivery":"Delivery framing: Instrument one research agent, record incidents, add cut-point tests for tool-guard changes, and compare against stub-all baselines. Dependencies: agent platform owners and CI. Ownership: engineering owns replay harness; research owners define unsafe-action assertions. Governance: retain incident envelopes under data policy. Acceptance: bit-stable full replay and mutation catch on guarded tools; risks include false confidence from simulated boundaries and envelope leakage."},"roleDetails":{"sales":{"customerProblem":"LLM agent failures are hard to reproduce, so research teams cannot regression-test tool and router changes against real incidents.","stakeholders":["Research software engineering","Research computing","Security/compliance for tool actions","PI sponsors of agent workflows"],"discoveryQuestions":["Which tool actions are unsafe if replayed?","Do you record agent traces today?","What CI exists for agent code?","How do you test guards after model upgrades?"],"valueHypothesis":"Cut-point replay can turn recorded incidents into CI tests that catch unsafe-tool regressions stubs miss, if boundaries are instrumented.","potentialEngagement":"Instrument one agent, record a failure corpus, and add cut-point tests for tool guards.","unsupportedClaims":["Fixes non-determinism of production LLM APIs","Guarantees scientific result correctness","Coverage beyond small incident benchmarks"]},"engineering":{"fit":"Boundary recording + cut-point replay regression harness for LLM agents.","architectureIntegration":"Wrap model/tool boundaries; store immutable envelopes; replay selected boundaries live in CI.","prerequisites":["Boundary hooks","Envelope storage","Assertions on unsafe actions","CI integration"],"constraints":["Simulated model boundaries in paper benchmark","Live tool state drift","Small public incident set"],"security":"Redact secrets from envelopes; control access to recorded research prompts and tool payloads.","proofOfValue":["Bit-stable full replay","Mutation catch rate on guarded tools","Stub-baseline comparison","Recording overhead per crossing"]},"delivery":{"work":"Add instrumentation, build incident library, wire cut-point tests into CI, train owners on assertions.","dependencies":["Agent codebase access","CI runners","Data-handling policy for envelopes"],"ownership":"RSE/platform owns harness; research owners define unsafe-action policy.","skillsAdoption":"Teach writing assertions and choosing cut-points rather than stubbing every boundary.","governanceCheckpoints":["Envelope retention review","Secret scanning of records","Periodic mutation testing"],"acceptanceCriteria":["Zero model calls on full replay","All guarded-tool mutants caught in pilot set","Documented redaction SOP"],"risks":["Envelope leakage","False confidence from simulated boundaries","CI flakiness if live tools drift"]}},"retrievedAt":"2026-09-19T03:12:22Z","enrichedAt":"2026-09-19T03:12:22Z","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Adds RSE/CI skills for assertion writing; reduces reliance on irreproducible manual re-runs.","procurementImplications":"Require agent platforms to export boundary records and support cut-point or equivalent regression tests.","operatingModelImplications":"Treats agent reliability as continuous integration work owned jointly by RSE and research sponsors.","sourceVerification":{"openedUrl":"https://arxiv.org/html/2609.20625","referenceExcerpt":"cut-point tests catch every mutant that lets the recorded unsafe action through","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}