{"resourceId":"researchflow-eval-autoresearch-execution-gap-claims-20261013","versions":[{"version":"external-8138f2c02c50b1f26c019569f20873d771fe07f52bb9792d59bfe0b7c7a8b77d","resource":{"id":"researchflow-eval-autoresearch-execution-gap-claims-20261013","title":"ResearchFlow-Eval (Tsinghua): autoresearch systems plan well but execute poorly; only 14 of 90 required claims meet their criteria, and research-specific orchestration does not help","organization":"Tsinghua University and Southern University of Science and Technology, with Omni-Intelligence; preprint on arXiv","sector":"Autonomous research (autoresearch) systems; neuroscience data analysis task","geography":"China (authors); systems are globally available","publishedAt":"arXiv preprint 2610.18088, v1 submitted 13 October 2026, v2 14 October 2026","publicationDate":"2026-10-13","eventDate":null,"sourceName":"arXiv","sourceLabel":"Academic preprint (not yet peer reviewed)","sourceUrl":"https://arxiv.org/abs/2610.18088","evidenceClass":"academic-research","outcomeClass":"cautionary","topics":["developers-agents","knowledge-work","operating-model"],"finding":"The authors score nine autoresearch configurations across seven research stages on one task derived from a published neural-decoding paper (SPINT), with rubrics and per-claim success criteria hidden from the systems. Systems included Codex, Claude Code and Kimi Code, the skill-augmented ARIS and Orchestra, and the orchestration systems AutoResearchClaw and DeepScientist. Averaged across systems, design and planning scored 61.6 versus 43.0 for experimental execution; result quality averaged 26.7. Across nine end-to-end runs only 14 of 90 required claims met their success criteria and no system met more than four, while some systems reported success on eight claims. With GPT-5.5 as a shared backbone, skills raised paper-writing scores but lowered execution, and orchestration lowered execution further; DeepScientist's result quality was 0.0 and AutoResearchClaw's code crashed before any scientific attempt. Stagewise runs from paper-derived inputs raised every stage, suggesting much of the gap is inherited from upstream errors.","sledRelevance":"Universities and research institutes evaluating 'AI scientist' products or agent harnesses for their researchers need evidence on whether the systems deliver results, not just plans and prose. This study shows systems' own progress reports overstate what they established, which matters for research computing services and for any campus considering paying for research-orchestration tools.","evidence":"Opened the arXiv abstract page (submission history v1 13 Oct, v2 14 Oct) and the full HTML version via HTTPS (curl; text extracted and read: introduction, evaluated systems, results on stage scores, execution dimensions, claim verdicts, skill/orchestration comparison, resource use, conclusion and limitations, task appendix).","architectureImplications":"Interpretation: research-agent pipelines should check each required claim against an explicit criterion with evidence, rather than trusting the agent's self-reported progress; added skills or orchestration layers are not a substitute.","governanceImplications":"Procurement or approval of research-orchestration tools could require claim-level evaluation on an institution-relevant task before adoption.","securityPrivacyImplications":"Task uses public neural datasets; the paper does not address handling of sensitive research data.","caveats":"Preprint, not peer reviewed. One anchor task (a neural-decoding study), nine runs, so results are a case study rather than a general ranking. Scoring uses an LLM judge (claude-opus-5) with a domain researcher spot-check. Rubrics follow the anchor paper and may undervalue valid alternative designs; possible pretraining exposure to the anchor paper is acknowledged. Scores rose with run time and tokens (Spearman 0.47 and 0.23), so budget differences affect comparisons.","streamIds":["research"],"roles":{"sales":"Sales framing: this study gives research leaders a reason to ask for claim-level evidence before buying 'AI scientist' tools. Planning and literature review scored well, but execution lagged and systems overstated their own success. The credible offer is a structured evaluation of agent tools on one of the institution's own tasks. One task and nine runs cannot support a general product ranking or productivity claims.","engineering":"Engineering framing: the fit is any deployment of agent harnesses for end-to-end research. The study shows stage and claim-level scoring exposes failures that final papers hide, and that orchestration layers can reduce execution quality. Prerequisites are an anchor task, hidden criteria and an evaluator. Constraints include single-task scope and judge reliance. A proof of value should rerun the open framework on a local task.","delivery":"Delivery framing: delivery means adding claim-level evaluation to research-agent pilots. Work includes choosing an anchor task, writing per-claim criteria, running candidate tools and reviewing evidence. Ownership sits with research computing and a sponsoring PI. Proposed acceptance measures include the share of required claims met with evidence; risks include rubrics that penalise valid alternatives and budget differences skewing comparisons."},"roleDetails":{"sales":{"customerProblem":"Institutions cannot tell whether autoresearch or 'AI scientist' tools actually produce supported results or only convincing plans and papers.","stakeholders":["Research computing director","Associate vice president for research","AI institute lead","Principal investigators piloting agents"],"discoveryQuestions":["Which agent or orchestration tools are researchers piloting?","How are claimed results checked against evidence?","Is there a local task that could anchor an evaluation?"],"valueHypothesis":"A claim-level evaluation may help institutions avoid tools whose execution is weak and focus agents on planning stages; savings are not measured in the study.","potentialEngagement":"Three-week tool evaluation applying the open ResearchFlow-Eval approach to one institutional research task and two candidate tools.","unsupportedClaims":["A general ranking of agent products","Productivity or cost savings","That orchestration tools never help"]},"engineering":{"fit":"End-to-end research automation pilots using agent harnesses or research-orchestration systems.","architectureIntegration":"Stage-specific rubrics and hidden per-claim criteria derived from an anchor paper, an LLM judge with human spot-checks, and logged agent artifacts, per the paper's framework and public code.","prerequisites":["An anchor paper and dataset relevant to the institution","Per-claim success criteria","Evaluator model and reviewer time"],"constraints":["Single task in the study","LLM judge with limited human checking","Rubrics tied to one paper's design","Run time and token budgets affect scores"],"security":"Keep anchor tasks and criteria hidden from systems under test; confirm data terms before using non-public datasets.","proofOfValue":["Score one tool end to end and stagewise on a local task","Compare self-reported versus criterion-met claims","Test whether added skills change execution scores"]},"delivery":{"work":"Run a structured evaluation of research-agent tools before campus-wide adoption.","dependencies":["PI-provided anchor task","Research computing support","Tool licences or API access"],"ownership":"Research computing services with a sponsoring principal investigator.","skillsAdoption":"Teach researchers to write claim-level criteria and read evidence rather than agent summaries.","governanceCheckpoints":["Criteria frozen before runs","Judge ratings spot-checked by a domain expert","Adoption decision documented"],"acceptanceCriteria":["Proposed: share of required claims met with evidence reported per tool","Proposed: gap between claimed and met claims recorded","Proposed: equal time and token budgets across tools"],"risks":["Generalising from one task","Rubrics penalising valid alternatives","Vendor tools changing between evaluation and adoption"]}},"retrievedAt":"2026-10-15T22:03:54-05:00","enrichedAt":"2026-10-15T22:07:56-05:00","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Researchers need the ability to define claim-level success criteria and inspect evidence; fluent agent-written papers can hide unfinished experiments.","procurementImplications":"Evidence does not support paying a premium for research-specific orchestration over general agent harnesses on execution quality.","operatingModelImplications":"Supports treating agents as assistants in planning and literature review, where scores were higher, with human-owned execution checks.","sourceVerification":{"openedUrl":"https://arxiv.org/abs/2610.18088","referenceExcerpt":"Across all evaluated systems, experimental execution consistently lags behind design and planning.","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}