From the Research Universities edition of September 17, 2026
SciRIGOR: agents' claims match faithful and unfaithful results alike; strict evidence chains stay rare
HKUST / Alibaba author team (Liu, Nie, Du, Li) · AI-assisted scientific analysis evaluation · International preprint methods (U.S. SLED transferable for evaluation design)
- Publisher
- arXiv
- Original publication
- September 5, 2026, arXiv 2609.06192v1
- Source retrieved
- 2026-09-18
What happened
Across 100 scientific analysis cases and 11 agent configurations, claims agree with faithful and unfaithful results at nearly the same rate (~92%), while no system exceeds 18% strict whole-chain evidence success—so coherence is not correctness.
Why it matters
Newly relevant scrutiny after 2026-09-16 replication-training coverage: shifts evaluation from endpoint/replication scores to data→claim evidence paths campuses need before trusting analysis agents.
Evidence and measured results
SciRigor reconstructs typed evidence graphs; separates artifact fidelity from relational validity; scores claim-support paths. 100 cases from articles (incl. Nature), 6 domains/17 subfields; 11 agent/model configs. Claims agree with faithful vs unfaithful results 91.8% vs 91.0%; max soft evidence-chain 62.6%; max strict whole-chain success 18.0%. Authors: internal coherence insufficient; verify support along complete data-to-claim path.
Limitations and uncertainty
Author preprint with author-defined cases and automated judging; not a campus productivity study; does not test specific remediation orchestration.
Put this evidence to work
Lighthouse Advisory interpretation, grounded in this source. Enriched 2026-09-18; this does not change the original publication date. Each role follows the same checklist so takeaways compare across sources.
Sales
Role takeaway
- Customer problem
- Analysis agents can produce internally coherent reports that are not scientifically supported by the run's own artifacts.
- Stakeholders
- Research integrity officers
- Research software engineers
- Principal investigators
- Research computing evaluation leads
- Discovery questions
- How do you currently verify agent claims against results and figures?
- Do you track faithful vs unfaithful result agreement?
- What strict evidence-chain threshold would block release?
- Who owns the evaluation harness?
- Value hypothesis
- Evidence-path scoring can expose coherent-but-unsupported agent claims that endpoint scores miss.
- Potential engagement
- Bounded audit of 10–20 campus analyses with soft and strict evidence-chain scoring plus human adjudication.
- Unsupported claims
- Campus agents already clear 18% strict whole-chain success
- Coherent notebooks equal validated science
- Automated judges replace domain experts
Pre-sales engineering
Role takeaway
- Fit
- Process evaluation for scientific coding agents producing interdependent code, results, figures, and claims.
- Architecture and integration
- Capture artifacts per run; build typed evidence graphs; score artifact fidelity and relational validity; localize earliest unsupported edge.
- Prerequisites
- Artifact retention
- Case definitions
- Judge calibration samples
- Constraints
- Author-benchmark transfer risk
- Multimodal figure scoring difficulty
- Compute cost of full-benchmark style runs
- Security
- Control access to evaluation corpora; prevent leakage of unpublished analyses into public judges.
- Proof of value
- Faithful vs unfaithful agreement gap
- Soft and strict chain scores
- Human agreement on break localization
Delivery
Role takeaway
- Work
- Select cases, run evidence-chain evaluation, and gate agent-assisted analyses on documented chain outcomes.
- Dependencies
- Domain expert reviewers
- Compute for agent configurations
- Policy on agent outputs in manuscripts
- Ownership
- RSE maintains harness; integrity sets gates; PIs own scientific acceptance.
- Skills and adoption
- Train reviewers that high claim–result agreement can still accompany unfaithful results.
- Governance checkpoints
- Pre-register scoring protocol
- Mandatory review on chain breaks
- Prohibit auto-acceptance of coherent reports
- Acceptance criteria
- Chain scores retained
- Break points documented
- Human disposition recorded
- Risks
- Judge gaming
- False confidence from soft scores
- Ignoring unfaithful-but-agreeing claims
Implementation considerations
Lighthouse Advisory interpretation across the operating dimensions a public-sector buyer must settle before this evidence becomes a design. Each note answers the question under its heading for this specific source.
Architecture and integration
What must connect, and where does the AI sit in the workflow?
Instrument analysis agents to retain code, results, figures, and claim graphs—not only final notebooks—and score relational validity separately from executability.
Governance
Who approves, reviews and stays accountable for outcomes?
Research integrity and computing review should reject 'looks coherent' agent reports without evidence-chain checks that can catch unfaithful results.
Security and privacy
What data, permissions and controls need testing?
Keep proprietary or sensitive datasets out of unapproved agent judges; audit for claim–result mismatches that could mask errors.
Accessibility and workforce
Who is affected, and what skills or accommodations follow?
Evidence-chain literacy should be taught alongside agent use so students do not equate polished reports with validated science.
Procurement
What should contracts, pricing and exit terms secure?
Require vendors to expose artifact graphs and chain-score exports, not only final notebooks.
Operating model
Which teams own the service once it runs?
Keep evaluation harness ownership separate from scientific acceptance decisions.
Publication history
- 2026-09-17Research Universities · Issue 124 resources
Stable resource ID: scirigor-evidence-chain-scientific-analysis-20260905