{"resourceId":"reagent-claim-artifact-audit-20260818","versions":[{"version":"external-ad8ab2150ae571cb5489a4fad0f4793b657db1b62bb097fcc707fcbe1e2ee94c","resource":{"id":"reagent-claim-artifact-audit-20260818","title":"ReAgent: 28/96 agent repos fail static consistency; only 25 both consistent and reproducing","organization":"University of Science and Technology of China","sector":"Academic AI research integrity / autonomous science evaluation","geography":"International (USTC authors; agent-generated research artifact corpus)","publishedAt":"August 18, 2026 (arXiv 2609.22111v1)","publicationDate":"2026-08-18","eventDate":"2026-08-18","sourceName":"arXiv","sourceLabel":"Academic preprint — claim–code–execution audit of agent-written papers","sourceUrl":"https://arxiv.org/html/2609.22111v1","evidenceClass":"academic-research","outcomeClass":"cautionary","topics":["developers-agents","knowledge-work","governance-procurement","data-security"],"finding":"ReAgent (Shen et al., USTC; submitted 18 Aug 2026) statically and dynamically audits whether agent-generated papers’ claims match code and execution evidence. Among 96 auditable repositories, 28 have at least one static consistency issue (including substantive implementation discrepancies and data/label leakage). Thirty-three reproduce reported metrics, but only 25 satisfy both static consistency and successful reproduction; eight reproduce numbers while remaining claim-code inconsistent—so reproduction alone can miss methodology mismatches. Why now: newly urgent after same-day vendor AI-discovery announcements that elevate agent-written scientific claims.","sledRelevance":"Gives research universities a concrete audit pattern for AI-scientist outputs: require claim–artifact consistency, not only regenerated metrics or fluent papers.","evidence":"arXiv HTML/PDF: 96 auditable repos; 28 with static issues; 33 reproduce metrics; 25 both consistent+reproducing; 8 reproduce despite inconsistencies; overall audit accuracy 70.83% vs Codex 35.42%.","architectureImplications":"Add static claim-to-code checkers and dynamic execution evidence capture to research-agent platforms before treating papers as validated.","governanceImplications":"Research integrity offices should treat metric reproduction without methodology fidelity as insufficient for AI-assisted manuscript acceptance.","securityPrivacyImplications":"Audits also surface credential exposure and undocumented configs—operational security issues in agent repos, not only scientific ones.","caveats":"August preprint; auditable-subset bias (many repos excluded for missing assets); not a U.S. campus field deployment study.","streamIds":["research"],"roles":{"sales":"Sales framing: After flashy AI-scientist demos, integrity officers need tools that catch hard-coded metrics and unimplemented methods—not another leaderboard. ReAgent supports a cautionary conversation: 28/96 auditable agent repos had static consistency issues and only 25 were both consistent and reproducing, including eight that reproduced numbers despite claim-code mismatches. Ask how campus AI-writing pilots verify artifacts today. Unsupported: that ReAgent certifies scientific truth, that all agent papers fail, or that August mixed-corpus rates equal your labs’ rates.","engineering":"Engineering framing: Fit is a two-stage auditor—static claim-code consistency plus dynamic execution evidence—against agent document–repo pairs. Integration points are manuscript/claim extraction, repository analysis, and controlled experiment runners. Prerequisites: accessible code/data, sandboxed execution, and human adjudication for edge cases. Constraints: auditable-subset bias; preprint benchmark. Security must sandbox executions and redact credentials found in repos. Proof of value is catching known inconsistency classes (leakage, unimplemented methods) on local agent outputs.","delivery":"Delivery framing: Pilot claim-artifact audits on a sample of AI-assisted manuscripts before any policy mandate. Dependencies: research integrity, RC sandboxes, and lab PIs. Ownership: integrity office sets policy; RC owns audit runtime. Skills adoption trains reviewers to read static/dynamic audit reports. Governance checkpoints: define pass/fail for consistency vs reproduction-only. Acceptance: audited sample with documented 25-style dual-pass criteria; risks include false confidence from metric reproduction alone and selection bias."},"roleDetails":{"sales":{"customerProblem":"AI-generated papers can look complete while code, methods, and reported metrics disagree.","stakeholders":["Research integrity","VPR","Library/reproducibility services","Lab PIs","Research computing"],"discoveryQuestions":["How are AI-assisted manuscripts checked today?","Is metric reproduction enough?","Who owns artifact review?","What sandbox exists for re-runs?"],"valueHypothesis":"Static+dynamic claim-artifact audits (as ReAgent shows many agent repos fail consistency even when some metrics reproduce) can reduce false confidence before campus endorsement of AI-scientist outputs.","potentialEngagement":"Bounded integrity pilot: audit a sample of agent-assisted repos against dual consistency+reproduction criteria.","unsupportedClaims":["ReAgent proves scientific truth","All agent research is fraudulent","Corpus rates equal every campus lab"]},"engineering":{"fit":"Automated static claim-code auditing plus dynamic execution evidence for agent-generated research repos.","architectureIntegration":"Ingest paper+repo; extract claims; static analyze implementations; run targeted experiments; emit structured audit report.","prerequisites":["Repo/data access","Execution sandbox","Claim schema","Human review workflow"],"constraints":["Auditable-subset selection bias","Preprint tool","Compute cost of dynamic audits"],"security":"Sandbox untrusted agent code; detect credential leaks; control data egress during re-runs.","proofOfValue":["Detect known inconsistency classes on local samples","Flag reproduction-without-consistency cases","Reviewer time comparison vs manual-only"]},"delivery":{"work":"Select sample AI-assisted manuscripts, run dual audits, and draft policy language distinguishing reproduction from consistency.","dependencies":["Integrity office","RC sandbox","PI consent","Library repos"],"ownership":"Research integrity owns standards; RC owns audit execution environment.","skillsAdoption":"Train reviewers on interpreting static vs dynamic findings.","governanceCheckpoints":["Dual-pass criteria definition","Appeal/adjudication process","Public communication rules"],"acceptanceCriteria":["Completed sample audit report","Documented dual-pass rule","Sandbox safety review"],"risks":["False confidence from metrics-only reproduction","Selection bias","Adversarial hard-coding evasion"]}},"retrievedAt":"2026-09-23T22:09:51-05:00","enrichedAt":"2026-09-23T22:09:51-05:00","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Audit tooling can protect less-resourced reviewers from fluent but unsupported AI manuscripts if sandboxes and training are shared services.","procurementImplications":"Require AI research-agent procurements to expose artifacts and support third-party claim-consistency audits—not paper text alone.","operatingModelImplications":"Pairs AI-scientist enablement with mandatory artifact audit capacity in integrity/RC operating models.","sourceVerification":{"openedUrl":"https://arxiv.org/html/2609.22111v1","referenceExcerpt":"Among the 96 repositories, 28 contain at least one static consistency issue","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}