{"resourceId":"scirigor-evidence-chain-scientific-analysis-20260905","versions":[{"version":"external-603645f71399153cb28e8beb6dd935f2d74ad304a0b364ddcd2dc8abf2a13718","resource":{"id":"scirigor-evidence-chain-scientific-analysis-20260905","title":"SciRIGOR: agents' claims match faithful and unfaithful results alike; strict evidence chains stay rare","organization":"HKUST / Alibaba author team (Liu, Nie, Du, Li)","sector":"AI-assisted scientific analysis evaluation","geography":"International preprint methods (U.S. SLED transferable for evaluation design)","publishedAt":"September 5, 2026, arXiv 2609.06192v1","publicationDate":"2026-09-05","eventDate":null,"sourceName":"arXiv","sourceLabel":"Author preprint; abs and html results inspected","sourceUrl":"https://arxiv.org/html/2609.06192","evidenceClass":"academic-research","outcomeClass":"cautionary","topics":["developers-agents","knowledge-work","governance-procurement","operating-model"],"finding":"Across 100 scientific analysis cases and 11 agent configurations, claims agree with faithful and unfaithful results at nearly the same rate (~92%), while no system exceeds 18% strict whole-chain evidence success—so coherence is not correctness.","sledRelevance":"Newly relevant scrutiny after 2026-09-16 replication-training coverage: shifts evaluation from endpoint/replication scores to data→claim evidence paths campuses need before trusting analysis agents.","evidence":"SciRigor reconstructs typed evidence graphs; separates artifact fidelity from relational validity; scores claim-support paths. 100 cases from articles (incl. Nature), 6 domains/17 subfields; 11 agent/model configs. Claims agree with faithful vs unfaithful results 91.8% vs 91.0%; max soft evidence-chain 62.6%; max strict whole-chain success 18.0%. Authors: internal coherence insufficient; verify support along complete data-to-claim path.","architectureImplications":"Interpretation: instrument analysis agents to retain code, results, figures, and claim graphs—not only final notebooks—and score relational validity separately from executability.","governanceImplications":"Interpretation: research integrity and computing review should reject 'looks coherent' agent reports without evidence-chain checks that can catch unfaithful results.","securityPrivacyImplications":"Interpretation: keep proprietary or sensitive datasets out of unapproved agent judges; audit for claim–result mismatches that could mask errors.","caveats":"Author preprint with author-defined cases and automated judging; not a campus productivity study; does not test specific remediation orchestration.","streamIds":["research"],"roles":{"sales":"Interpretation: Speak with research integrity, RSE, and VPR offices piloting coding agents for analysis. Ask how they currently decide an agent result is scientifically supported, and whether they measure claim–result linkage. Offer a bounded evidence-chain audit on 10–20 local analyses—not a guarantee agents will pass SciRIGOR rates. Value hypothesis is catching coherent-but-unsupported claims earlier. Unsupported: that any vendor stack already exceeds 18% strict whole-chain success on campus data.","engineering":"Interpretation: Fit is evaluation harnesses for scientific coding agents producing code, figures, and claims. Integrate typed evidence graphs, faithfulness checks, and earliest-break localization beside unit tests. Prerequisites: frozen inputs and reference oracles where available. Constraints: judge noise; domain coverage. Security: isolate evaluation data. Proof: soft vs strict chain scores and faithful/unfaithful agreement gaps on local cases.","delivery":"Interpretation: RSE owns harness; integrity sets release gates; PIs interpret science. Work: sample analyses, run SciRIGOR-style scoring, and require human review when chains break. Dependencies: compute for agent runs and domain experts. Train reviewers to distrust coherence alone. Checkpoints: pre-register scoring rules; block auto-publication of agent claims. Acceptance: chain scores and break points retained. Risks: over-trusting high claim–result agreement when results are unfaithful."},"roleDetails":{"sales":{"customerProblem":"Analysis agents can produce internally coherent reports that are not scientifically supported by the run's own artifacts.","stakeholders":["Research integrity officers","Research software engineers","Principal investigators","Research computing evaluation leads"],"discoveryQuestions":["How do you currently verify agent claims against results and figures?","Do you track faithful vs unfaithful result agreement?","What strict evidence-chain threshold would block release?","Who owns the evaluation harness?"],"valueHypothesis":"Evidence-path scoring can expose coherent-but-unsupported agent claims that endpoint scores miss.","potentialEngagement":"Bounded audit of 10–20 campus analyses with soft and strict evidence-chain scoring plus human adjudication.","unsupportedClaims":["Campus agents already clear 18% strict whole-chain success","Coherent notebooks equal validated science","Automated judges replace domain experts"]},"engineering":{"fit":"Process evaluation for scientific coding agents producing interdependent code, results, figures, and claims.","architectureIntegration":"Capture artifacts per run; build typed evidence graphs; score artifact fidelity and relational validity; localize earliest unsupported edge.","prerequisites":["Artifact retention","Case definitions","Judge calibration samples"],"constraints":["Author-benchmark transfer risk","Multimodal figure scoring difficulty","Compute cost of full-benchmark style runs"],"security":"Control access to evaluation corpora; prevent leakage of unpublished analyses into public judges.","proofOfValue":["Faithful vs unfaithful agreement gap","Soft and strict chain scores","Human agreement on break localization"]},"delivery":{"work":"Select cases, run evidence-chain evaluation, and gate agent-assisted analyses on documented chain outcomes.","dependencies":["Domain expert reviewers","Compute for agent configurations","Policy on agent outputs in manuscripts"],"ownership":"RSE maintains harness; integrity sets gates; PIs own scientific acceptance.","skillsAdoption":"Train reviewers that high claim–result agreement can still accompany unfaithful results.","governanceCheckpoints":["Pre-register scoring protocol","Mandatory review on chain breaks","Prohibit auto-acceptance of coherent reports"],"acceptanceCriteria":["Chain scores retained","Break points documented","Human disposition recorded"],"risks":["Judge gaming","False confidence from soft scores","Ignoring unfaithful-but-agreeing claims"]}},"retrievedAt":"2026-09-18T03:07:38Z","enrichedAt":"2026-09-18T03:10:00Z","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Interpretation: evidence-chain literacy should be taught alongside agent use so students do not equate polished reports with validated science.","procurementImplications":"Interpretation: require vendors to expose artifact graphs and chain-score exports, not only final notebooks.","operatingModelImplications":"Interpretation: keep evaluation harness ownership separate from scientific acceptance decisions.","sourceVerification":{"openedUrl":"https://arxiv.org/html/2609.06192","referenceExcerpt":"The strongest run reaches 62.6% SoftECS but only 18.0% Strict success.","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}