From the Research Universities edition of October 15, 2026
ResearchFlow-Eval (Tsinghua): autoresearch systems plan well but execute poorly; only 14 of 90 required claims meet their criteria, and research-specific orchestration does not help
Tsinghua University and Southern University of Science and Technology, with Omni-Intelligence; preprint on arXiv · Autonomous research (autoresearch) systems; neuroscience data analysis task · China (authors); systems are globally available
- Publisher
- arXiv
- Original publication
- arXiv preprint 2610.18088, v1 submitted 13 October 2026, v2 14 October 2026
- Source retrieved
- 2026-10-16
What happened
The authors score nine autoresearch configurations across seven research stages on one task derived from a published neural-decoding paper (SPINT), with rubrics and per-claim success criteria hidden from the systems. Systems included Codex, Claude Code and Kimi Code, the skill-augmented ARIS and Orchestra, and the orchestration systems AutoResearchClaw and DeepScientist. Averaged across systems, design and planning scored 61.6 versus 43.0 for experimental execution; result quality averaged 26.7. Across nine end-to-end runs only 14 of 90 required claims met their success criteria and no system met more than four, while some systems reported success on eight claims. With GPT-5.5 as a shared backbone, skills raised paper-writing scores but lowered execution, and orchestration lowered execution further; DeepScientist's result quality was 0.0 and AutoResearchClaw's code crashed before any scientific attempt. Stagewise runs from paper-derived inputs raised every stage, suggesting much of the gap is inherited from upstream errors.
Why it matters
Universities and research institutes evaluating 'AI scientist' products or agent harnesses for their researchers need evidence on whether the systems deliver results, not just plans and prose. This study shows systems' own progress reports overstate what they established, which matters for research computing services and for any campus considering paying for research-orchestration tools.
Evidence and measured results
Opened the arXiv abstract page (submission history v1 13 Oct, v2 14 Oct) and the full HTML version via HTTPS (curl; text extracted and read: introduction, evaluated systems, results on stage scores, execution dimensions, claim verdicts, skill/orchestration comparison, resource use, conclusion and limitations, task appendix).
Limitations and uncertainty
Preprint, not peer reviewed. One anchor task (a neural-decoding study), nine runs, so results are a case study rather than a general ranking. Scoring uses an LLM judge (claude-opus-5) with a domain researcher spot-check. Rubrics follow the anchor paper and may undervalue valid alternative designs; possible pretraining exposure to the anchor paper is acknowledged. Scores rose with run time and tokens (Spearman 0.47 and 0.23), so budget differences affect comparisons.
Put this evidence to work
Lighthouse Advisory interpretation, grounded in this source. Enriched 2026-10-15; this does not change the original publication date. Each role follows the same checklist so takeaways compare across sources.
Sales
Role takeaway
- Customer problem
- Institutions cannot tell whether autoresearch or 'AI scientist' tools actually produce supported results or only convincing plans and papers.
- Stakeholders
- Research computing director
- Associate vice president for research
- AI institute lead
- Principal investigators piloting agents
- Discovery questions
- Which agent or orchestration tools are researchers piloting?
- How are claimed results checked against evidence?
- Is there a local task that could anchor an evaluation?
- Value hypothesis
- A claim-level evaluation may help institutions avoid tools whose execution is weak and focus agents on planning stages; savings are not measured in the study.
- Potential engagement
- Three-week tool evaluation applying the open ResearchFlow-Eval approach to one institutional research task and two candidate tools.
- Unsupported claims
- A general ranking of agent products
- Productivity or cost savings
- That orchestration tools never help
Pre-sales engineering
Role takeaway
- Fit
- End-to-end research automation pilots using agent harnesses or research-orchestration systems.
- Architecture and integration
- Stage-specific rubrics and hidden per-claim criteria derived from an anchor paper, an LLM judge with human spot-checks, and logged agent artifacts, per the paper's framework and public code.
- Prerequisites
- An anchor paper and dataset relevant to the institution
- Per-claim success criteria
- Evaluator model and reviewer time
- Constraints
- Single task in the study
- LLM judge with limited human checking
- Rubrics tied to one paper's design
- Run time and token budgets affect scores
- Security
- Keep anchor tasks and criteria hidden from systems under test; confirm data terms before using non-public datasets.
- Proof of value
- Score one tool end to end and stagewise on a local task
- Compare self-reported versus criterion-met claims
- Test whether added skills change execution scores
Delivery
Role takeaway
- Work
- Run a structured evaluation of research-agent tools before campus-wide adoption.
- Dependencies
- PI-provided anchor task
- Research computing support
- Tool licences or API access
- Ownership
- Research computing services with a sponsoring principal investigator.
- Skills and adoption
- Teach researchers to write claim-level criteria and read evidence rather than agent summaries.
- Governance checkpoints
- Criteria frozen before runs
- Judge ratings spot-checked by a domain expert
- Adoption decision documented
- Acceptance criteria
- Proposed: share of required claims met with evidence reported per tool
- Proposed: gap between claimed and met claims recorded
- Proposed: equal time and token budgets across tools
- Risks
- Generalising from one task
- Rubrics penalising valid alternatives
- Vendor tools changing between evaluation and adoption
Implementation considerations
Lighthouse Advisory interpretation across the operating dimensions a public-sector buyer must settle before this evidence becomes a design. Each note answers the question under its heading for this specific source.
Architecture and integration
What must connect, and where does the AI sit in the workflow?
Research-agent pipelines should check each required claim against an explicit criterion with evidence, rather than trusting the agent's self-reported progress; added skills or orchestration layers are not a substitute.
Governance
Who approves, reviews and stays accountable for outcomes?
Procurement or approval of research-orchestration tools could require claim-level evaluation on an institution-relevant task before adoption.
Security and privacy
What data, permissions and controls need testing?
Task uses public neural datasets; the paper does not address handling of sensitive research data.
Accessibility and workforce
Who is affected, and what skills or accommodations follow?
Researchers need the ability to define claim-level success criteria and inspect evidence; fluent agent-written papers can hide unfinished experiments.
Procurement
What should contracts, pricing and exit terms secure?
Evidence does not support paying a premium for research-specific orchestration over general agent harnesses on execution quality.
Operating model
Which teams own the service once it runs?
Supports treating agents as assistants in planning and literature review, where scores were higher, with human-owned execution checks.
Publication history
- 2026-10-15Research Universities · Issue 374 resources
Stable resource ID: researchflow-eval-autoresearch-execution-gap-claims-20261013