{"resourceId":"paperdoctor-presubmission-diagnostics-20260915","versions":[{"version":"external-71a3c0f10c6799519aa6f1b73610087d291f2468da219b988a09aa159dabb753","resource":{"id":"paperdoctor-presubmission-diagnostics-20260915","title":"PaperDoctor: 70.6% author evidence acceptance on 1,299 items; L3 reproduction mostly blocked","organization":"University of Oxford / Stanford University / National University of Singapore (et al.)","sector":"Academic AI-for-science / scholarly communication tooling","geography":"International academic collaboration (UK/US/Singapore et al.); transfer to U.S. SLED research offices","publishedAt":"September 15, 2026 (arXiv:2609.16995v1)","publicationDate":"2026-09-15","eventDate":null,"sourceName":"arXiv","sourceLabel":"Academic preprint — evidence-grounded pre-submission paper diagnostics","sourceUrl":"https://arxiv.org/html/2609.16995v1","evidenceClass":"academic-research","outcomeClass":"mixed","topics":["developers-agents","knowledge-work","governance-procurement","data-security"],"finding":"PaperDoctor evaluates in-progress papers with a hierarchical L1–L3 pipeline (surface screening, typed claim verification, prioritised experiment reproduction) that emits observation–evidence–suggestion triples. In a human study of 30 pre-submission papers (1,299 items), authors accepted PaperDoctor's evidence on 70.6% of items (mean per-paper rate 68.5%; median 71.1%) and gave all-positive holistic scores (mean +1.30). L2 verifiable claims were rejected less often than L1 surface checks (11% vs 27%). On 40 manuscripts with released code, only 18.0% of L3 reproduction plans were initially ready to run; after resolving blockers, 7.7 plans/paper reached execution and 5.1 returned pass or warning; training jobs passed 11.3% once run versus 48.9% for analysis pipelines.","sledRelevance":"Why now: Still-unarchived 15 Sep measured diagnostic/reproduction package that distinguishes author-accepted feedback quality from end-to-end experimental fidelity—directly complementary to same-day capacity announcements.","evidence":"30 papers / 1,299 items; 70.6% evidence acceptance; all-positive holistic scores mean +1.30; L2 rejection 11% vs L1 27%; 40-paper L3: 18.0% initially ready; training pass 11.3% vs analysis 48.9%.","architectureImplications":"Research-AI tooling should couple manuscript agents to code/data artifacts and budgeted reproduction executors; paper-only reviewers miss implementation and numeric gaps.","governanceImplications":"Treat diagnostic-agent outputs as advisory evidence for human integrity/advisory workflows; require approval gates before consuming GPU for L3 reruns.","securityPrivacyImplications":"Pre-submission systems process unpublished manuscripts and potentially sensitive code/data—need access control, sandboxing, and retention limits.","caveats":"Preprint; author-judged acceptance; L3 author-approved and compute-limited; not a campus production deployment evaluation.","streamIds":["research"],"roles":{"sales":"Sales framing: Research offices drowning in AI-drafted manuscripts need diagnostic feedback with evidence pointers—not another opaque review score. PaperDoctor's study supports a bounded conversation about pre-submission agents that authors judged helpful on 30 drafts (70.6% evidence acceptance across 1,299 items; all holistic scores positive)—while also showing reproduction remains hard (only 18.0% of L3 plans initially ready; training passes 11.3%). Discovery should map whether campus workflows will allow code+data attachment, who approves GPU budgets for L3, and how findings enter integrity review. Unsupported: guaranteed publication success, that 70.6% acceptance equals scientific correctness, or replacement of human advisors.","engineering":"Engineering framing: Fit is a hierarchical L1–L3 agent pipeline over paper+code: surface screening, typed claim verifiers, and author-approved prioritised reproduction. Integration points are manuscript repos, code/data artifacts, and optional GPU executors—not a standalone chat UI. Prerequisites include parseable paper/code pairs, claim routing, and budgeted L3 approval. Constraints: preprint evaluation; author-judged acceptance; compute-limited reproduction. Security must protect unpublished manuscripts and credentials in reproduction environments. Proof of value is auditable observation–evidence–suggestion triples and measured reproduction pass/warn/error rates—not reviewer-style prose alone.","delivery":"Delivery framing: Pilot PaperDoctor-style diagnostics on a bounded cohort of pre-submission papers with mandatory code attachment and a written L3 compute-approval gate. Dependencies: research computing, library/research integrity, PI supervisors, and GPU queue policy. Ownership: research integrity/advisory owns triage of findings; research computing owns L3 execution environments. Skills adoption trains authors to supply runnable repos and respond to evidence-linked suggestions. Governance checkpoints: human review of Error-marked items, data-use for reproduction, and escalation paths. Acceptance: N papers with accepted evidence rates logged; risks include mistaking author agreement for truth and underfunding reproduction compute."},"roleDetails":{"sales":{"customerProblem":"AI-accelerated drafting risks flawed claims entering review faster than human advisors can give evidence-linked feedback.","stakeholders":["VPR / research integrity","Graduate advisors","Research computing","Library / scholarly communications"],"discoveryQuestions":["Will authors attach code and data?","Who approves L3 GPU budgets?","How do findings enter integrity workflows?","What counts as success beyond acceptance rate?"],"valueHypothesis":"Evidence-grounded diagnostic agents can raise auditable pre-submission feedback quality (70.6% author evidence acceptance in the study) if reproduction gates and human triage are funded—without claiming guaranteed publication outcomes.","potentialEngagement":"Bounded pre-submission diagnostic pilot with code-attachment rules and L3 approval policy.","unsupportedClaims":["70.6% acceptance equals scientific correctness","Replaces human advisors/referees","Guaranteed journal acceptance"]},"engineering":{"fit":"L1–L3 hierarchical paper+code diagnostic agent with selective experiment reproduction.","architectureIntegration":"Ingest manuscript and repository; route claims to typed verifiers; queue author-approved L3 jobs on campus/cloud GPUs with log capture.","prerequisites":["Paper+code packaging","Claim extraction/routing","L3 budget approval UX","Secure execution sandbox"],"constraints":["Preprint study","Author-judged outcomes","Many plans blocked before execution"],"security":"Protect unpublished drafts; sandbox reproduction credentials; control dataset egress.","proofOfValue":["Auditable finding triples","Logged L3 pass/warn/error rates","Human triage of Error items"]},"delivery":{"work":"Stand up pilot intake, code-attachment checklist, L3 approval workflow, and integrity triage board.","dependencies":["Research computing GPUs","Integrity office","Advisor champions","Storage for artifacts"],"ownership":"Research integrity owns finding triage; research computing owns execution environments.","skillsAdoption":"Train authors on runnable repos and responding to evidence-linked suggestions.","governanceCheckpoints":["Error-item human review","Data-use approval for L3","Escalation for integrity flags"],"acceptanceCriteria":["Cohort papers processed with logged acceptance","L3 jobs only after approval","Triage SLAs met"],"risks":["Author agreement ≠ correctness","Underfunded reproduction compute","False confidence from L1-only runs"]}},"retrievedAt":"2026-09-22T22:17:16-05:00","enrichedAt":"2026-09-22T22:17:16-05:00","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Advisor-style agents may extend feedback access for large graduate cohorts if RSE and integrity staffing absorb triage; otherwise bottlenecks shift to human review of Error items.","procurementImplications":"Procure diagnostic agents on evidence-grounding and reproduction-audit capabilities, not raw finding volume; budget L3 compute separately.","operatingModelImplications":"Splits drafting assistance from verification ownership: authors supply artifacts, integrity offices triage, research computing runs approved reproductions.","sourceVerification":{"openedUrl":"https://arxiv.org/html/2609.16995v1","referenceExcerpt":"Across all 1,299 items the acceptance rate was 70.6%","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}