From the Research Universities edition of October 20, 2026
SEER-Repro preprint (Stanford and Databricks): a coding agent reproduces 77.4% of 2,190 claims from 20 cancer-registry papers, but nine of ten sampled robustness flags did not hold up on expert review
Stanford University (Computer Science; School of Medicine, Division of Oncology); Databricks AI Research; University of Massachusetts Amherst · Academic research (metascience, biomedical epidemiology) · United States (SEER is the U.S. National Cancer Institute registry; method transferable to other shared research datasets)
- Publisher
- arXiv
- Original publication
- 16 October 2026 (arXiv v1)
- Source retrieved
- 2026-10-21
What happened
The preprint (arXiv v1, 16 October 2026) builds SEER-Repro: 20 published papers using the U.S. SEER cancer registry, 901 findings and 2,190 atomic claims, with an environment of 39 data tables (about 198 million rows) and 40 documentation resources. A coding agent (GPT-5.6 Sol via Codex) reproduced 1,694 claims (77.4%); 76 were close and 420 were not reproduced. Of the failures, 251 (59.8%) traced to data issues such as unavailable or revised records, 47 to underspecified analyses, 28 to unavailable software, and 48 (11.4%) to agent errors. A robustness pipeline flagged potentially consequential analytical choices in 9.5% of fully reproduced findings (55 of 579), but expert review of ten flagged cases found nine did not establish a consequential change and one needed further review.
Why it matters
Research integrity offices, core facilities and journals are looking for scalable ways to check published findings. This is a measured test of agents doing reproduction and robustness checks on a widely used public dataset, and it shows both real capacity and a high false-alarm rate on flags, which argues for agents as triage for expert reviewers rather than as auditors.
Evidence and measured results
Opened the arXiv abstract page (v1 submitted 16 October 2026, nine authors) and the full HTML paper. Read the abstract, benchmark construction, reproduction results (Table 1), failure taxonomy (Table 2), robustness analysis and expert-review results, and the author affiliations.
Limitations and uncertainty
Single unreviewed preprint. One case study domain (SEER) and 20 papers chosen as ten highly cited plus ten recent. Grading and failure diagnosis partly rely on LLM prompts; 20 failures were suspected evaluation errors and 26 had unresolved causes. Expert review covered only ten flagged cases. No cost or runtime figures were found in the sections read. Model names reflect the authors' configuration.
Put this evidence to work
Lighthouse Advisory interpretation, grounded in this source. Enriched 2026-10-20; this does not change the original publication date. Each role follows the same checklist so takeaways compare across sources.
Sales
Role takeaway
- Customer problem
- Research institutions and journals cannot afford manual reproduction checks at scale but need to find fragile findings.
- Stakeholders
- Research integrity officer
- Director of a biostatistics or data core
- Associate vice president for research
- Journal or institutional repository editors
- Discovery questions
- Which shared datasets underlie many of your institution's publications?
- Who would adjudicate agent-raised flags, and how much time do they have?
- How would confirmed issues be communicated to authors?
- Value hypothesis
- Agent-assisted reproduction may let a small expert team review many more findings by focusing on flagged cases.
- Potential engagement
- A six- to eight-week pilot reproducing a sample of papers built on one shared dataset, with expert adjudication and a false-alarm report.
- Unsupported claims
- Agents can certify that research is correct
- Flags indicate misconduct or error
- Results transfer to restricted or identifiable data
Pre-sales engineering
Role takeaway
- Fit
- Fits fields with shared, documented public datasets and standard analyses; weak fit where data or software are unavailable.
- Architecture and integration
- Curated execution environment with versioned data tables and documentation, a reproduction agent, a separate grading step and a review queue.
- Prerequisites
- Versioned historical data releases
- Documentation and variable dictionaries the agent can read
- Access to or substitutes for original analysis software
- Constraints
- Data revisions break reproduction
- LLM-based grading can misjudge claims
- Restricted data may not be usable with external model endpoints
- Security
- Run on approved environments, restrict model endpoints for any non-public data, and keep logs of agent actions and outputs.
- Proof of value
- Reproduce ten local papers and compare agent results with an analyst's
- Have experts adjudicate every flag and report the false-alarm rate
- Measure how many failures trace to data availability rather than agent error
Delivery
Role takeaway
- Work
- Curate the dataset environment, run reproduction on sampled papers, route flags to expert reviewers and log decisions.
- Dependencies
- Data steward access to historical releases
- Statistician reviewer time
- Approved compute and model endpoints
- Ownership
- Research integrity office or data core, with statisticians as adjudicators.
- Skills and adoption
- Training reviewers to read agent evidence logs and to distinguish data-availability failures from analytical problems.
- Governance checkpoints
- Scope and dataset approval before the pilot
- Expert adjudication of every flag before any communication
- Policy review before contacting authors
- Acceptance criteria
- Proposal: every flag adjudicated by a named expert with a recorded outcome
- Proposal: false-alarm rate reported for the pilot sample
- Proposal: reviewer time per flag measured against manual reanalysis
- Risks
- False alarms damaging trust with authors
- Overreading reproduction rates as research quality
- Data revisions making results unstable
Implementation considerations
Lighthouse Advisory interpretation across the operating dimensions a public-sector buyer must settle before this evidence becomes a design. Each note answers the question under its heading for this specific source.
Architecture and integration
What must connect, and where does the AI sit in the workflow?
Reproduction at this scale needs a curated execution environment: versioned data tables, documentation the agent can read, and saved evidence for each claim. Historical data releases matter, since many failures came from records that had changed or were no longer available.
Governance
Who approves, reviews and stays accountable for outcomes?
Supports using agents to direct expert attention, not to issue findings about published work. Any institutional or journal use should require human review of every flag before it is communicated, given nine of ten sampled flags did not hold up.
Security and privacy
What data, permissions and controls need testing?
SEER is de-identified public-use data; applying the approach to restricted or identifiable datasets would need approved environments and controls on which model endpoints can see the data.
Accessibility and workforce
Who is affected, and what skills or accommodations follow?
Shifts epidemiologist and statistician time toward reviewing flagged cases; the expert-review step remains essential and needs dedicated reviewer capacity.
Procurement
What should contracts, pricing and exit terms secure?
Institutions evaluating reproduction or research-audit tools should ask for false-positive rates on flags and data-availability assumptions, not only headline reproduction rates.
Operating model
Which teams own the service once it runs?
Could sit with a research integrity office or a data core as a triage service: agent reproduces and flags, experts adjudicate, and authors are contacted only after adjudication.
Publication history
- 2026-10-20Research Universities · Issue 423 resources
Stable resource ID: seer-repro-ai-reproducibility-robustness-arxiv-2610-23845