From the Research Universities edition of October 5, 2026
CMU counterfactual test: AI-scientist agents score 95% on single-cell questions but only 41% evidence-grounded accuracy; benchmark scores and LLM judges reward ignoring changed data
Carnegie Mellon University School of Computer Science (Zhang and Li), workshop paper for Agentic AI for Biological Discovery · University research / evaluation of AI scientists for data-driven biology · United States (Pittsburgh); benchmark tasks drawn from published international single-cell studies
- Publisher
- arXiv (cs.AI)
- Original publication
- arXiv:2610.04915v1, submitted 4 October 2026
- Source retrieved
- 2026-10-06
What happened
The authors edited the raw data behind 18 BAISBench single-cell questions and four GeneBench-Pro problems into verified null, withdrawal and flip versions, then ran Claude Code (Opus 5.5, Sonnet 5, Haiku 4.5), Codex CLI (GPT-5.6-Sol, GPT-5.6-Terra) and Biomni on every version. Claude agents were 95% accurate with the data and 83% correct with no data at all; their evidence-grounded accuracy (correct, reacts when evidence is withdrawn, follows a reversal) was 41%. After withdrawal the benchmark still awarded 77% of its credit; only 61% of runs followed reversed evidence. Hiding gene names raised flip-following from 58% to 93% and EGA from 39% to 69%. The official score had a flip AUC of 0.11 and an Opus process judge 0.12, so both rewarded ignoring the evidence; a grounding-focused judge prompt raised flip AUC above 0.7 and an 'insufficient data' option cut withdrawal credit from 77% to 31%, but no repair fixed every failure. An audit of 2,944 runs found no sandbox escapes.
Why it matters
Universities and research-computing groups now cite agent benchmark scores when choosing or promoting AI-scientist tools. This study shows a high score can mostly reflect recall of known biology, so campus evaluations need counterfactual checks before claiming agents analyze researchers' own data.
Evidence and measured results
Opened the arXiv abstract page and full HTML text via HTTPS (curl, text extracted); read abstract, method, results tables 2 and 6–12, limitations and case studies. Findings are the authors' measurements; no independent replication.
Limitations and uncertainty
Small task set (18 BAIS-SD questions, 4 GeneBench-Pro problems), three replicates per model and one for Biomni; counterfactuals hand-built; LLM judges not human-validated; single domain (single-cell biology); workshop preprint; authors note responsiveness to evidence is necessary but not sufficient for scientific quality.
Put this evidence to work
Lighthouse Advisory interpretation, grounded in this source. Enriched 2026-10-05; this does not change the original publication date. Each role follows the same checklist so takeaways compare across sources.
Sales
Role takeaway
- Customer problem
- Research groups cannot tell whether an AI scientist's correct answers came from their data or from memorized prior knowledge, and standard benchmark scores hide the difference.
- Stakeholders
- Bioinformatics or data-science core director
- Principal investigators adopting agents
- Research computing director
- Vice president for research / research integrity office
- Discovery questions
- Which agent benchmark results were used to select or justify the current tool?
- Do your analysis protocols include a no-data baseline or an 'insufficient data' response?
- Who validates LLM judges or automated graders used to score agent analyses?
- Value hypothesis
- Adding counterfactual checks to agent pilots can expose recall-driven answers before they enter papers or grants; the study does not quantify cost or time savings.
- Potential engagement
- Six-week evaluation-design engagement for one core facility: build verified withdrawal and flip versions for 15–20 local tasks and report evidence-grounded accuracy alongside accuracy.
- Unsupported claims
- That any tested agent is reliable on a campus's own datasets
- That a grounding-focused judge prompt fixes evaluation
- Generalization beyond single-cell and statistical-genetics tasks
Pre-sales engineering
Role takeaway
- Fit
- Labs and cores running coding-agent or Biomni-style analyses on structured scientific data (h5ad, tabular) with known reference statistics.
- Architecture and integration
- Per-task data versioning (null, withdrawal, flip) with pre-registered reference statistics; canonical file writer; fresh sandbox per run; answer.json with confidence and data-support label; paired-run scoring and judge stress tests.
- Prerequisites
- Tasks with a computable reference statistic
- Sandboxed agent runtime without web access
- Run logging and transcript audit
- Domain expert to verify each counterfactual
- Constraints
- Hand-built counterfactuals limit scale
- Results from 22 tasks in one domain
- Model versions change faster than evaluations
- Some questions admit alternative valid analyses
- Security
- Isolate runs from answer keys and other runs' workspaces, block network access, strip file metadata that leaks edits, and audit all tool calls before scoring.
- Proof of value
- Compare accuracy with evidence-grounded accuracy on local tasks
- Measure no-data accuracy to size the prior
- Test whether the deployed grader's flip AUC exceeds 0.5
- Check abstention rate on withdrawn-evidence versions
Delivery
Role takeaway
- Work
- Stand up a counterfactual evaluation suite for the campus agent pilot, run it on each model or harness release, and publish evidence-grounded metrics with raw accuracy.
- Dependencies
- Domain-verified reference statistics
- Sandboxed compute allocation
- Agent vendor version pinning
- Research office reporting policy
- Ownership
- Core facility or research computing evaluation lead owns the suite; PIs own domain correctness; research integrity office owns reporting expectations.
- Skills and adoption
- Train analysts to build verified counterfactuals and to read evidence-grounded accuracy, and teach researchers to treat 'data insufficient' as a valid agent output.
- Governance checkpoints
- Counterfactual review before tool approval
- Judge validation on paired runs before automated grading
- Re-evaluation on each model upgrade
- Disclosure of agent use and grounding metrics in publications
- Acceptance criteria
- Proposed: evidence-grounded accuracy reported for every approved agent configuration
- Proposed: grader flip AUC above 0.7 before use in scoring
- Proposed: agents abstain or flag on at least half of withdrawn-evidence tasks
- Proposed: no-data baseline documented per task set
- Risks
- Recall-driven answers accepted as data analysis
- Graders rewarding unchanged answers
- Metadata leaks invalidating counterfactuals
- Evaluation cost limiting coverage
Implementation considerations
Lighthouse Advisory interpretation across the operating dimensions a public-sector buyer must settle before this evidence becomes a design. Each note answers the question under its heading for this specific source.
Architecture and integration
What must connect, and where does the AI sit in the workflow?
Evaluation harnesses for campus agent pilots should run each item on intact, withdrawn and reversed data versions passed through one canonical file writer (the authors found an agent recovering an edit from a gzip header), sandbox each run without web access, and log every tool call for audit.
Governance
Who approves, reviews and stays accountable for outcomes?
Research offices and core facilities should not accept vendor or benchmark accuracy as evidence that an agent derived results from supplied data; require evidence-grounded metrics, a no-data baseline and an explicit 'insufficient data' path in analysis protocols.
Security and privacy
What data, permissions and controls need testing?
Runs were sandboxed with no web or retrieval access and audited (16,091 tool calls) for access to answer material; the same isolation and audit pattern applies when agents analyze controlled campus datasets.
Accessibility and workforce
Who is affected, and what skills or accommodations follow?
Building verified counterfactuals requires domain and data-engineering skill; cores need staff time, and junior researchers need training to question confident agent answers.
Procurement
What should contracts, pricing and exit terms secure?
Ask vendors of AI-scientist tools for evidence-grounded or counterfactual results, not only benchmark accuracy, and for an explicit insufficient-data output.
Operating model
Which teams own the service once it runs?
Agent evaluation becomes a recurring service tied to model releases rather than a one-time selection step.
Publication history
- 2026-10-05Research Universities · Issue 284 resources
Stable resource ID: cmu-ai-scientist-evidence-grounding-counterfactuals-20261004