From the Research Universities edition of October 1, 2026
CESS: Deep Research citation-correct reports can mislead; selection correction cuts pool MAE 9.2% (MS2) / 60.1% (public agent)
Beijing Normal University and Ke Holdings — arXiv preprint (cs.AI) · Higher education research synthesis / AI Deep Research evaluation and research administration · China (authors); methods transferable to any institution deploying Deep Research agents over literature corpora
- Publisher
- arXiv 2609.39026
- Original publication
- September 30, 2026 (arXiv 2609.39026v1)
- Source retrieved
- 2026-10-02
What happened
Search Shapes Conclusions (arXiv 2609.39026, 30 Sep 2026) shows Deep Research agents can produce well-cited yet misleading reports because adaptive search selectively samples the candidate document pool. Citation correctness does not measure whether unread available documents would change the conclusion. Causal Evidence Selection Correction (CESS) predicts each candidate’s evidence direction and corrects the pool average using logged selection and round-reach probabilities, with shrinkage for short searches. On MS2 systematic-review questions, CESS reduces MAE against the candidate-pool average by 9.2% and ranking-change sensitivity by 39.4% versus averaging evidence scores of documents read; on trajectories from a public Open Deep Research agent the reductions reach 60.1% and 87.2%. A further 4,800 paired-intervention trajectories confirm pool correction and search-policy effect estimation are different tasks.
Why it matters
Universities deploying Deep Research for literature synthesis and research administration need selection-bias audits—citation checkers alone understate one-sided evidence exposure.
Evidence and measured results
arXiv abs HTML and PDF via HTTPS curl; abstract quantitative claims confirmed in PDF text; check_sources not archived.
Limitations and uncertainty
Preprint; depends on defined candidate pools and evidence-direction predictors; MS2/Open Deep Research settings; does not certify scientific truth of conclusions.
Put this evidence to work
Lighthouse Advisory interpretation, grounded in this source. Enriched 2026-10-01; this does not change the original publication date. Each role follows the same checklist so takeaways compare across sources.
Sales
Role takeaway
- Customer problem
- Well-cited AI research reports can still mislead when adaptive search never exposes contrary documents in the candidate pool.
- Stakeholders
- University libraries / knowledge services
- Research integrity
- VPR analytics
- Deep Research platform owners
- Discovery questions
- Do your agents log documents not selected?
- How do you audit evidence balance beyond citation support?
- What question sets define the candidate pool?
- Value hypothesis
- Selection-corrected pool audits can cut error versus read-only evidence averages and expose one-sided Deep Research trajectories.
- Potential engagement
- Pilot CESS-style audits on 30–40 systematic-review-style campus questions with logged Deep Research trajectories.
- Unsupported claims
- Guaranteed truthful conclusions
- Replacement for peer review
- Zero need for human literature expertise
Pre-sales engineering
Role takeaway
- Fit
- Causal Evidence Selection Correction sidecar over logged Deep Research trajectories.
- Architecture and integration
- Capture candidate pools, selection probabilities, and round reachability; estimate corrected pool evidence direction with shrinkage/intervals.
- Prerequisites
- Deep Research agent with logging
- Candidate corpus definition
- Evidence-direction scorer
- Audit dashboard
- Constraints
- Preprint
- MS2 and Open Deep Research settings
- Depends on logged probabilities
- Not end-to-end science discovery
- Security
- Logs may contain query content and document metadata—apply retention and access controls appropriate to research data policy.
- Proof of value
- MAE reduction vs pool average
- Ranking-sensitivity reduction
- Direction-error rate
- Intervention vs correction separation check
Delivery
Role takeaway
- Work
- Enable selection logging, define candidate pools per question class, run weekly pool audits, escalate one-sided reports before downstream use.
- Dependencies
- Agent platform logging
- Library/research computing
- Question packs
- Privacy review
- Ownership
- Knowledge services + research computing co-own the audit; investigators own scientific interpretation.
- Skills and adoption
- Train analysts to distinguish citation correctness from pool representativeness.
- Governance checkpoints
- Logging privacy review
- First audit calibration
- Policy on blocking one-sided reports
- Acceptance criteria
- Documented MAE/sensitivity gains vs baseline
- Operational runbook for failed audits
- Retention policy approved
- Risks
- Wrong candidate pool definition
- Overtrust in automated evidence direction
- Performance cost of logging
Implementation considerations
Lighthouse Advisory interpretation across the operating dimensions a public-sector buyer must settle before this evidence becomes a design. Each note answers the question under its heading for this specific source.
Architecture and integration
What must connect, and where does the AI sit in the workflow?
Deep Research stacks need candidate-pool logging and selection-probability telemetry, not only final citations and claim-support checks.
Governance
Who approves, reviews and stays accountable for outcomes?
Research offices should require evidence-balance audits before using agent reports in funding scans, policy briefs, or systematic reviews.
Security and privacy
What data, permissions and controls need testing?
Selection logs expose researcher queries and corpus access patterns; apply research-data retention and access controls.
Accessibility and workforce
Who is affected, and what skills or accommodations follow?
Pool audits help less-resourced research staff detect one-sided syntheses without manually reading every candidate document.
Procurement
What should contracts, pricing and exit terms secure?
Require Deep Research vendors to export selection logs and support third-party pool audits—not only citation-support scores.
Operating model
Which teams own the service once it runs?
Add evidence-selection review to research intelligence workflows alongside plagiarism and citation checks.
Publication history
- 2026-10-01Research Universities · Issue 244 resources
Stable resource ID: cess-deep-research-evidence-selection-bias-20260930