{"resourceId":"discern-scientific-agent-judgment-benchmark-20260927","versions":[{"version":"external-f2871f38079ad41923fe89efc7847ffff203a3a0a401d6f5bb1e2d8214a6ff38","resource":{"id":"discern-scientific-agent-judgment-benchmark-20260927","title":"DISCERN: agents perfect only 60.8%/34.2%/0.6% of L1/L2/L3 scientific-judgment tasks across 203 life-science evaluations","organization":"University of California San Diego — arXiv preprint (cs.AI)","sector":"Higher education life-science AI research agents / reproducibility and scientific judgment evaluation","geography":"United States (UC San Diego); transferable evaluation methods for campus scientific-agent pilots","publishedAt":"September 27, 2026 (arXiv 2609.33357v1); why-now: not archived and newly pairs with 1 Oct YouRA/RAC agent-architecture claims as independent scientific-judgment scrutiny","publicationDate":"2026-09-27","eventDate":null,"sourceName":"arXiv 2609.33357","sourceLabel":"Academic preprint (arXiv)","sourceUrl":"https://arxiv.org/abs/2609.33357","evidenceClass":"academic-research","outcomeClass":"cautionary","topics":["developers-agents","knowledge-work","governance-procurement","operating-model"],"finding":"DISCERN (arXiv 2609.33357, submitted 27 Sep 2026) evaluates whether AI agents can vet data, verify analyses, and generate evidence-grounded hypotheses on real public life-science datasets. Across 203 tasks, eight tracks, and eight models, agents earn perfect scores in only 60.8% of Level 1 (data integrity), 34.2% of Level 2 (analysis verification), and 0.6% of Level 3 (hypothesis generation/revision) evaluations. Failures include rejecting sound clean-control data, recognizing confounds but not carrying them into conclusions, and wide variation in hypothesis commitment/novelty under adversarial review. Token/code-use rankings are much more stable across tracks than evidence-judgment rankings. Authors conclude current agents may support supervised assistance but do not yet demonstrate reliable autonomous analysis or discovery. Why-now: complements same-day YouRA/RAC architecture papers and prior RECLAIM claim-reproduction evidence with staged scientific-judgment measurement not previously archived.","sledRelevance":"Research leadership must distinguish benchmark task completion from scientific judgment; DISCERN’s near-zero L3 perfect rate argues for supervised assistance policies over autonomous discovery claims.","evidence":"arXiv abs HTML and full PDF via HTTPS curl; perfect-score rates confirmed in abstract and body; check_sources not archived.","architectureImplications":"Scientific-agent platforms need stage-specific eval harnesses (integrity → analysis → hypothesis) with clean controls and sealed sandboxes—not a single Overall score.","governanceImplications":"Do not authorize unattended hypothesis generation or analysis publication when L3 reliability is near zero; require faculty adjudication.","securityPrivacyImplications":"Sealed sandboxes and no network access during eval reduce leakage of grading keys and sensitive derived data; production use still needs data-use agreements.","caveats":"Preprint; life-science core with separate physics extension; LLM graders with human adjudication; bounded tasks not end-to-end discovery; public datasets may be familiar to models.","streamIds":["research"],"roles":{"sales":"Sales framing: benchmark “AI scientist” assistants on scientific judgment under confounds—not only code execution. DISCERN (UC San Diego; arXiv 2609.33357, 27 Sep 2026) evaluates 203 tasks across eight life-science tracks and three levels—data integrity, analysis verification, and hypothesis generation under adversarial review. Across eight models, perfect-score rates are only 60.8% (L1), 34.2% (L2), and 0.6% (L3). Aggregate leaderboards mask track-specific collapse and clean-control false rejects. Position supervised scientific assistance, not autonomous discovery, as the credible near-term offer.","engineering":"Engineering framing: evaluate agents in sealed sandboxes with matched clean controls, injected defects, tool traps, and prior-conflict hypothesis dossiers. Score data diagnosis/action, analysis recognition/calibration, and hypothesis evidence fidelity/novelty separately; use geometric means so one strong stage cannot hide another. Expect resource-use rankings to be more stable across tracks than evidence-judgment rankings. Do not deploy unattended analysis pipelines when L3 perfect rates are near zero.","delivery":"Delivery framing: adopt DISCERN-like staged acceptance tests before granting research agents write-access to analysis pipelines or hypothesis-driven follow-ups. Dependencies include domain datasets, sandbox tooling, and faculty adjudicators for hypothesis novelty. Acceptance criteria should include clean-control specificity, calibration after issue recognition, and documented human oversight for hypothesis stages. Risks include overfitting to public datasets, LLM graders, and mistaking high L1 scores for discovery readiness."},"roleDetails":{"sales":{"customerProblem":"Campus AI research assistants may look strong on aggregate scores while failing data integrity, analysis calibration, or hypothesis novelty.","stakeholders":["Life-science PIs","Research computing","Research integrity","Biomedical informatics leads"],"discoveryQuestions":["What share of clean controls does your agent wrongly reject?","Do recognized confounds change final claims?","Who adjudicates agent hypotheses?"],"valueHypothesis":"Staged DISCERN-like evaluation can bound where agents help under supervision versus where autonomy is unsafe.","potentialEngagement":"Eight-week evaluation on a campus life-science subset mirroring L1–L3 tasks with human adjudication on hypotheses.","unsupportedClaims":["Reliable autonomous discovery today","One Overall score as go-live criterion","Zero need for domain faculty review"]},"engineering":{"fit":"Three-level scientific-agent benchmark harness on real public life-science datasets with sealed sandboxes.","architectureIntegration":"Implement L1 integrity, L2 analysis-verification, and L3 adversarial hypothesis review as separately scored stages with clean controls.","prerequisites":["Sandbox without net exfiltration","Track datasets/tools","Grading rubrics","Human adjudication capacity for L3"],"constraints":["Preprint","Life-science focus (physics extension separate)","LLM verifiers with human adjudication","Not end-to-end wet-lab autonomy"],"security":"Keep gold answers sealed; prevent agents from reaching grading keys or sibling task worlds.","proofOfValue":["Perfect-score rates by level","Clean-control acceptance rate","Recognition-to-calibration gap","Cross-track rank stability of judgment vs tokens"]},"delivery":{"work":"Stand up staged eval, freeze task packs, run model bake-off, publish oversight policy for each level before production use.","dependencies":["Domain faculty","Research computing sandboxes","Data use agreements","Eval engineers"],"ownership":"Biomedical informatics / research computing own the harness; PIs own scientific go-live decisions.","skillsAdoption":"Train analysts to read level-specific scorecards and to escalate L3 outputs to faculty review.","governanceCheckpoints":["Dataset ethics review","L1/L2 automation approval","L3 human-adjudication mandate"],"acceptanceCriteria":["Documented L1/L2 floors met","L3 outputs never auto-publish","Clean-control false-reject rate tracked"],"risks":["Dataset leakage","Leaderboard chasing","Autonomy creep past supervised assistance"]}},"retrievedAt":"2026-10-01T22:08:28-05:00","enrichedAt":"2026-10-01T22:14:00-05:00","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Staged scorecards help train research staff on where agents help versus where human scientific judgment remains mandatory.","procurementImplications":"Require vendors to report level-specific scientific-judgment metrics with clean-control rates—not only coding-agent leaderboards.","operatingModelImplications":"Operate agents as supervised scientific assistants with hard stops before autonomous hypothesis-driven experimental changes.","sourceVerification":{"openedUrl":"https://arxiv.org/abs/2609.33357","referenceExcerpt":"Agents earn perfect scores in only 60.8% of Level 1, 34.2% of Level 2, and 0.6% of Level 3 evaluations","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}