From the Research Universities edition of September 22, 2026
PaperDoctor: 70.6% author evidence acceptance on 1,299 items; L3 reproduction mostly blocked
University of Oxford / Stanford University / National University of Singapore (et al.) · Academic AI-for-science / scholarly communication tooling · International academic collaboration (UK/US/Singapore et al.); transfer to U.S. SLED research offices
- Publisher
- arXiv
- Original publication
- September 15, 2026 (arXiv:2609.16995v1)
- Source retrieved
- 2026-09-23
What happened
PaperDoctor evaluates in-progress papers with a hierarchical L1–L3 pipeline (surface screening, typed claim verification, prioritised experiment reproduction) that emits observation–evidence–suggestion triples. In a human study of 30 pre-submission papers (1,299 items), authors accepted PaperDoctor's evidence on 70.6% of items (mean per-paper rate 68.5%; median 71.1%) and gave all-positive holistic scores (mean +1.30). L2 verifiable claims were rejected less often than L1 surface checks (11% vs 27%). On 40 manuscripts with released code, only 18.0% of L3 reproduction plans were initially ready to run; after resolving blockers, 7.7 plans/paper reached execution and 5.1 returned pass or warning; training jobs passed 11.3% once run versus 48.9% for analysis pipelines.
Why it matters
Why now: Still-unarchived 15 Sep measured diagnostic/reproduction package that distinguishes author-accepted feedback quality from end-to-end experimental fidelity—directly complementary to same-day capacity announcements.
Evidence and measured results
30 papers / 1,299 items; 70.6% evidence acceptance; all-positive holistic scores mean +1.30; L2 rejection 11% vs L1 27%; 40-paper L3: 18.0% initially ready; training pass 11.3% vs analysis 48.9%.
Limitations and uncertainty
Preprint; author-judged acceptance; L3 author-approved and compute-limited; not a campus production deployment evaluation.
Put this evidence to work
Lighthouse Advisory interpretation, grounded in this source. Enriched 2026-09-22; this does not change the original publication date. Each role follows the same checklist so takeaways compare across sources.
Sales
Role takeaway
- Customer problem
- AI-accelerated drafting risks flawed claims entering review faster than human advisors can give evidence-linked feedback.
- Stakeholders
- VPR / research integrity
- Graduate advisors
- Research computing
- Library / scholarly communications
- Discovery questions
- Will authors attach code and data?
- Who approves L3 GPU budgets?
- How do findings enter integrity workflows?
- What counts as success beyond acceptance rate?
- Value hypothesis
- Evidence-grounded diagnostic agents can raise auditable pre-submission feedback quality (70.6% author evidence acceptance in the study) if reproduction gates and human triage are funded—without claiming guaranteed publication outcomes.
- Potential engagement
- Bounded pre-submission diagnostic pilot with code-attachment rules and L3 approval policy.
- Unsupported claims
- 70.6% acceptance equals scientific correctness
- Replaces human advisors/referees
- Guaranteed journal acceptance
Pre-sales engineering
Role takeaway
- Fit
- L1–L3 hierarchical paper+code diagnostic agent with selective experiment reproduction.
- Architecture and integration
- Ingest manuscript and repository; route claims to typed verifiers; queue author-approved L3 jobs on campus/cloud GPUs with log capture.
- Prerequisites
- Paper+code packaging
- Claim extraction/routing
- L3 budget approval UX
- Secure execution sandbox
- Constraints
- Preprint study
- Author-judged outcomes
- Many plans blocked before execution
- Security
- Protect unpublished drafts; sandbox reproduction credentials; control dataset egress.
- Proof of value
- Auditable finding triples
- Logged L3 pass/warn/error rates
- Human triage of Error items
Delivery
Role takeaway
- Work
- Stand up pilot intake, code-attachment checklist, L3 approval workflow, and integrity triage board.
- Dependencies
- Research computing GPUs
- Integrity office
- Advisor champions
- Storage for artifacts
- Ownership
- Research integrity owns finding triage; research computing owns execution environments.
- Skills and adoption
- Train authors on runnable repos and responding to evidence-linked suggestions.
- Governance checkpoints
- Error-item human review
- Data-use approval for L3
- Escalation for integrity flags
- Acceptance criteria
- Cohort papers processed with logged acceptance
- L3 jobs only after approval
- Triage SLAs met
- Risks
- Author agreement ≠ correctness
- Underfunded reproduction compute
- False confidence from L1-only runs
Implementation considerations
Lighthouse Advisory interpretation across the operating dimensions a public-sector buyer must settle before this evidence becomes a design. Each note answers the question under its heading for this specific source.
Architecture and integration
What must connect, and where does the AI sit in the workflow?
Research-AI tooling should couple manuscript agents to code/data artifacts and budgeted reproduction executors; paper-only reviewers miss implementation and numeric gaps.
Governance
Who approves, reviews and stays accountable for outcomes?
Treat diagnostic-agent outputs as advisory evidence for human integrity/advisory workflows; require approval gates before consuming GPU for L3 reruns.
Security and privacy
What data, permissions and controls need testing?
Pre-submission systems process unpublished manuscripts and potentially sensitive code/data—need access control, sandboxing, and retention limits.
Accessibility and workforce
Who is affected, and what skills or accommodations follow?
Advisor-style agents may extend feedback access for large graduate cohorts if RSE and integrity staffing absorb triage; otherwise bottlenecks shift to human review of Error items.
Procurement
What should contracts, pricing and exit terms secure?
Procure diagnostic agents on evidence-grounding and reproduction-audit capabilities, not raw finding volume; budget L3 compute separately.
Operating model
Which teams own the service once it runs?
Splits drafting assistance from verification ownership: authors supply artifacts, integrity offices triage, research computing runs approved reproductions.
Publication history
- 2026-09-22Research Universities · Issue 174 resources
Stable resource ID: paperdoctor-presubmission-diagnostics-20260915