{"resourceId":"cmu-ai-scientist-evidence-grounding-counterfactuals-20261004","versions":[{"version":"external-852eb9ab880e2da8f7c741f634b6226072813841cea5015e51041a502264e1ae","resource":{"id":"cmu-ai-scientist-evidence-grounding-counterfactuals-20261004","title":"CMU counterfactual test: AI-scientist agents score 95% on single-cell questions but only 41% evidence-grounded accuracy; benchmark scores and LLM judges reward ignoring changed data","organization":"Carnegie Mellon University School of Computer Science (Zhang and Li), workshop paper for Agentic AI for Biological Discovery","sector":"University research / evaluation of AI scientists for data-driven biology","geography":"United States (Pittsburgh); benchmark tasks drawn from published international single-cell studies","publishedAt":"arXiv:2610.04915v1, submitted 4 October 2026","publicationDate":"2026-10-04","eventDate":null,"sourceName":"arXiv (cs.AI)","sourceLabel":"Academic preprint (workshop paper, not peer-reviewed journal)","sourceUrl":"https://arxiv.org/abs/2610.04915","evidenceClass":"academic-research","outcomeClass":"cautionary","topics":["developers-agents","knowledge-work","governance-procurement"],"finding":"The authors edited the raw data behind 18 BAISBench single-cell questions and four GeneBench-Pro problems into verified null, withdrawal and flip versions, then ran Claude Code (Opus 5.5, Sonnet 5, Haiku 4.5), Codex CLI (GPT-5.6-Sol, GPT-5.6-Terra) and Biomni on every version. Claude agents were 95% accurate with the data and 83% correct with no data at all; their evidence-grounded accuracy (correct, reacts when evidence is withdrawn, follows a reversal) was 41%. After withdrawal the benchmark still awarded 77% of its credit; only 61% of runs followed reversed evidence. Hiding gene names raised flip-following from 58% to 93% and EGA from 39% to 69%. The official score had a flip AUC of 0.11 and an Opus process judge 0.12, so both rewarded ignoring the evidence; a grounding-focused judge prompt raised flip AUC above 0.7 and an 'insufficient data' option cut withdrawal credit from 77% to 31%, but no repair fixed every failure. An audit of 2,944 runs found no sandbox escapes.","sledRelevance":"Universities and research-computing groups now cite agent benchmark scores when choosing or promoting AI-scientist tools. This study shows a high score can mostly reflect recall of known biology, so campus evaluations need counterfactual checks before claiming agents analyze researchers' own data.","evidence":"Opened the arXiv abstract page and full HTML text via HTTPS (curl, text extracted); read abstract, method, results tables 2 and 6–12, limitations and case studies. Findings are the authors' measurements; no independent replication.","architectureImplications":"Evaluation harnesses for campus agent pilots should run each item on intact, withdrawn and reversed data versions passed through one canonical file writer (the authors found an agent recovering an edit from a gzip header), sandbox each run without web access, and log every tool call for audit.","governanceImplications":"Interpretation: research offices and core facilities should not accept vendor or benchmark accuracy as evidence that an agent derived results from supplied data; require evidence-grounded metrics, a no-data baseline and an explicit 'insufficient data' path in analysis protocols.","securityPrivacyImplications":"Runs were sandboxed with no web or retrieval access and audited (16,091 tool calls) for access to answer material; the same isolation and audit pattern applies when agents analyze controlled campus datasets.","caveats":"Small task set (18 BAIS-SD questions, 4 GeneBench-Pro problems), three replicates per model and one for Biomni; counterfactuals hand-built; LLM judges not human-validated; single domain (single-cell biology); workshop preprint; authors note responsiveness to evidence is necessary but not sufficient for scientific quality.","streamIds":["research"],"roles":{"sales":"Sales framing: campuses buying AI-scientist tools need a way to tell analysis from recall. This CMU study shows agents scoring 95% on single-cell questions while only 41% of the Claude agents' answers met its evidence-grounding test, and standard scores rewarded ignoring changed evidence. A bounded offer is an evaluation-design engagement that adds withdrawn and reversed data versions to a department's agent pilot. Do not claim any agent is reliable on local data; the study covers a small biology task set only.","engineering":"Engineering framing: build counterfactual evaluation into agent pilots. For each task create intact, withdrawn and flipped data versions verified by a reference statistic, write every file through one canonical writer to strip metadata leaks, sandbox runs without web access, and audit tool calls. Score evidence-grounded accuracy and a no-data baseline beside raw accuracy, and test any LLM judge on the same pairs, since judges here rewarded ignoring flips. Expect manual effort per item.","delivery":"Delivery framing: make evidence grounding an acceptance gate before an AI-scientist tool supports production research. A core facility or research computing team owns the counterfactual suite, PIs own domain reference statistics, and the research office owns reporting rules. Track evidence-grounded accuracy, abstention on insufficient data, and judge validity across releases. Interpretation: thresholds below are proposals; the paper reports observed results only for its own small benchmark."},"roleDetails":{"sales":{"customerProblem":"Research groups cannot tell whether an AI scientist's correct answers came from their data or from memorized prior knowledge, and standard benchmark scores hide the difference.","stakeholders":["Bioinformatics or data-science core director","Principal investigators adopting agents","Research computing director","Vice president for research / research integrity office"],"discoveryQuestions":["Which agent benchmark results were used to select or justify the current tool?","Do your analysis protocols include a no-data baseline or an 'insufficient data' response?","Who validates LLM judges or automated graders used to score agent analyses?"],"valueHypothesis":"Adding counterfactual checks to agent pilots can expose recall-driven answers before they enter papers or grants; the study does not quantify cost or time savings.","potentialEngagement":"Six-week evaluation-design engagement for one core facility: build verified withdrawal and flip versions for 15–20 local tasks and report evidence-grounded accuracy alongside accuracy.","unsupportedClaims":["That any tested agent is reliable on a campus's own datasets","That a grounding-focused judge prompt fixes evaluation","Generalization beyond single-cell and statistical-genetics tasks"]},"engineering":{"fit":"Labs and cores running coding-agent or Biomni-style analyses on structured scientific data (h5ad, tabular) with known reference statistics.","architectureIntegration":"Per-task data versioning (null, withdrawal, flip) with pre-registered reference statistics; canonical file writer; fresh sandbox per run; answer.json with confidence and data-support label; paired-run scoring and judge stress tests.","prerequisites":["Tasks with a computable reference statistic","Sandboxed agent runtime without web access","Run logging and transcript audit","Domain expert to verify each counterfactual"],"constraints":["Hand-built counterfactuals limit scale","Results from 22 tasks in one domain","Model versions change faster than evaluations","Some questions admit alternative valid analyses"],"security":"Isolate runs from answer keys and other runs' workspaces, block network access, strip file metadata that leaks edits, and audit all tool calls before scoring.","proofOfValue":["Compare accuracy with evidence-grounded accuracy on local tasks","Measure no-data accuracy to size the prior","Test whether the deployed grader's flip AUC exceeds 0.5","Check abstention rate on withdrawn-evidence versions"]},"delivery":{"work":"Stand up a counterfactual evaluation suite for the campus agent pilot, run it on each model or harness release, and publish evidence-grounded metrics with raw accuracy.","dependencies":["Domain-verified reference statistics","Sandboxed compute allocation","Agent vendor version pinning","Research office reporting policy"],"ownership":"Core facility or research computing evaluation lead owns the suite; PIs own domain correctness; research integrity office owns reporting expectations.","skillsAdoption":"Train analysts to build verified counterfactuals and to read evidence-grounded accuracy, and teach researchers to treat 'data insufficient' as a valid agent output.","governanceCheckpoints":["Counterfactual review before tool approval","Judge validation on paired runs before automated grading","Re-evaluation on each model upgrade","Disclosure of agent use and grounding metrics in publications"],"acceptanceCriteria":["Proposed: evidence-grounded accuracy reported for every approved agent configuration","Proposed: grader flip AUC above 0.7 before use in scoring","Proposed: agents abstain or flag on at least half of withdrawn-evidence tasks","Proposed: no-data baseline documented per task set"],"risks":["Recall-driven answers accepted as data analysis","Graders rewarding unchanged answers","Metadata leaks invalidating counterfactuals","Evaluation cost limiting coverage"]}},"retrievedAt":"2026-10-05T22:07:51-05:00","enrichedAt":"2026-10-05T22:12:00-05:00","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Building verified counterfactuals requires domain and data-engineering skill; cores need staff time, and junior researchers need training to question confident agent answers.","procurementImplications":"Ask vendors of AI-scientist tools for evidence-grounded or counterfactual results, not only benchmark accuracy, and for an explicit insufficient-data output.","operatingModelImplications":"Agent evaluation becomes a recurring service tied to model releases rather than a one-time selection step.","sourceVerification":{"openedUrl":"https://arxiv.org/abs/2610.04915","referenceExcerpt":"On the single-cell questions, the Claude agents are 95% accurate and answer 83% correctly without any data, but their evidence-grounded accuracy is only 41%.","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}