{"resourceId":"cess-deep-research-evidence-selection-bias-20260930","versions":[{"version":"external-f2871f38079ad41923fe89efc7847ffff203a3a0a401d6f5bb1e2d8214a6ff38","resource":{"id":"cess-deep-research-evidence-selection-bias-20260930","title":"CESS: Deep Research citation-correct reports can mislead; selection correction cuts pool MAE 9.2% (MS2) / 60.1% (public agent)","organization":"Beijing Normal University and Ke Holdings — arXiv preprint (cs.AI)","sector":"Higher education research synthesis / AI Deep Research evaluation and research administration","geography":"China (authors); methods transferable to any institution deploying Deep Research agents over literature corpora","publishedAt":"September 30, 2026 (arXiv 2609.39026v1)","publicationDate":"2026-09-30","eventDate":null,"sourceName":"arXiv 2609.39026","sourceLabel":"Academic preprint (arXiv)","sourceUrl":"https://arxiv.org/abs/2609.39026","evidenceClass":"academic-research","outcomeClass":"cautionary","topics":["knowledge-work","developers-agents","governance-procurement","data-security"],"finding":"Search Shapes Conclusions (arXiv 2609.39026, 30 Sep 2026) shows Deep Research agents can produce well-cited yet misleading reports because adaptive search selectively samples the candidate document pool. Citation correctness does not measure whether unread available documents would change the conclusion. Causal Evidence Selection Correction (CESS) predicts each candidate’s evidence direction and corrects the pool average using logged selection and round-reach probabilities, with shrinkage for short searches. On MS2 systematic-review questions, CESS reduces MAE against the candidate-pool average by 9.2% and ranking-change sensitivity by 39.4% versus averaging evidence scores of documents read; on trajectories from a public Open Deep Research agent the reductions reach 60.1% and 87.2%. A further 4,800 paired-intervention trajectories confirm pool correction and search-policy effect estimation are different tasks.","sledRelevance":"Universities deploying Deep Research for literature synthesis and research administration need selection-bias audits—citation checkers alone understate one-sided evidence exposure.","evidence":"arXiv abs HTML and PDF via HTTPS curl; abstract quantitative claims confirmed in PDF text; check_sources not archived.","architectureImplications":"Deep Research stacks need candidate-pool logging and selection-probability telemetry, not only final citations and claim-support checks.","governanceImplications":"Research offices should require evidence-balance audits before using agent reports in funding scans, policy briefs, or systematic reviews.","securityPrivacyImplications":"Selection logs expose researcher queries and corpus access patterns; apply research-data retention and access controls.","caveats":"Preprint; depends on defined candidate pools and evidence-direction predictors; MS2/Open Deep Research settings; does not certify scientific truth of conclusions.","streamIds":["research"],"roles":{"sales":"Sales framing: citation-correct Deep Research reports can still mislead if adaptive search samples a one-sided evidence pool. CESS (Beijing Normal University / Ke Holdings; arXiv 2609.39026, 30 Sep 2026) treats search as adaptive evidence sampling and corrects candidate-pool averages using logged selection probabilities. On MS2 systematic-review questions it cuts MAE vs the pool average by 9.2% and ranking sensitivity by 39.4% versus averaging only documents read; on a public Open Deep Research agent the reductions reach 60.1% and 87.2%. Sell evidence-selection audits beside citation checkers for research synthesis tools.","engineering":"Engineering framing: log per-document selection probabilities and search-round reachability, predict evidence direction for unread candidates, and apply shrinkage/intervals when sampling is incomplete. Separate pool-estimate correction from intervention analysis of search-policy effects (authors prove these are different tasks; 4,800 paired-intervention trajectories). Integrate as an audit sidecar to campus Deep Research agents—do not confuse local citation support with representative coverage of the available literature set.","delivery":"Delivery framing: instrument Deep Research pilots to retain candidate pools and selection logs, then run CESS-style pool audits on a fixed question set (e.g., systematic-review style). Dependencies include retrieval logs, evidence-direction models, and research-library stakeholders. Acceptance: lower MAE vs pool average and reduced sensitivity to adversarial document orderings versus read-only averages. Risks: incomplete candidate pools, mis-specified evidence-direction predictors, and treating audit scores as proof of scientific truth."},"roleDetails":{"sales":{"customerProblem":"Well-cited AI research reports can still mislead when adaptive search never exposes contrary documents in the candidate pool.","stakeholders":["University libraries / knowledge services","Research integrity","VPR analytics","Deep Research platform owners"],"discoveryQuestions":["Do your agents log documents not selected?","How do you audit evidence balance beyond citation support?","What question sets define the candidate pool?"],"valueHypothesis":"Selection-corrected pool audits can cut error versus read-only evidence averages and expose one-sided Deep Research trajectories.","potentialEngagement":"Pilot CESS-style audits on 30–40 systematic-review-style campus questions with logged Deep Research trajectories.","unsupportedClaims":["Guaranteed truthful conclusions","Replacement for peer review","Zero need for human literature expertise"]},"engineering":{"fit":"Causal Evidence Selection Correction sidecar over logged Deep Research trajectories.","architectureIntegration":"Capture candidate pools, selection probabilities, and round reachability; estimate corrected pool evidence direction with shrinkage/intervals.","prerequisites":["Deep Research agent with logging","Candidate corpus definition","Evidence-direction scorer","Audit dashboard"],"constraints":["Preprint","MS2 and Open Deep Research settings","Depends on logged probabilities","Not end-to-end science discovery"],"security":"Logs may contain query content and document metadata—apply retention and access controls appropriate to research data policy.","proofOfValue":["MAE reduction vs pool average","Ranking-sensitivity reduction","Direction-error rate","Intervention vs correction separation check"]},"delivery":{"work":"Enable selection logging, define candidate pools per question class, run weekly pool audits, escalate one-sided reports before downstream use.","dependencies":["Agent platform logging","Library/research computing","Question packs","Privacy review"],"ownership":"Knowledge services + research computing co-own the audit; investigators own scientific interpretation.","skillsAdoption":"Train analysts to distinguish citation correctness from pool representativeness.","governanceCheckpoints":["Logging privacy review","First audit calibration","Policy on blocking one-sided reports"],"acceptanceCriteria":["Documented MAE/sensitivity gains vs baseline","Operational runbook for failed audits","Retention policy approved"],"risks":["Wrong candidate pool definition","Overtrust in automated evidence direction","Performance cost of logging"]}},"retrievedAt":"2026-10-01T22:08:28-05:00","enrichedAt":"2026-10-01T22:14:00-05:00","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Pool audits help less-resourced research staff detect one-sided syntheses without manually reading every candidate document.","procurementImplications":"Require Deep Research vendors to export selection logs and support third-party pool audits—not only citation-support scores.","operatingModelImplications":"Add evidence-selection review to research intelligence workflows alongside plagiarism and citation checks.","sourceVerification":{"openedUrl":"https://arxiv.org/abs/2609.39026","referenceExcerpt":"a well-cited report can still reach a misleading conclusion","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}