{"resourceId":"dcp-discovery-certification-protocol-20260907","versions":[{"version":"external-dbef9f2527b82962097de2411f6c863323b90ff125e4d9b73f601cbffd33e60e","resource":{"id":"dcp-discovery-certification-protocol-20260907","title":"Discovery Certification Protocol: scores alone do not prove discovery; 0/96 recoveries in controlled audits","organization":"Ning, Zhong, Li, Zeng (arXiv preprint)","sector":"AI research-agent evaluation / scientific discovery auditing","geography":"International academic authors; transferable protocol with site limits","publishedAt":"September 7, 2026 (arXiv v1)","publicationDate":"2026-09-07","eventDate":null,"sourceName":"arXiv","sourceLabel":"Academic preprint — Discovery Certification Protocol (DCP)","sourceUrl":"https://arxiv.org/html/2609.09219v1","evidenceClass":"academic-research","outcomeClass":"cautionary","topics":["developers-agents","governance-procurement","operating-model","knowledge-work"],"finding":"DCP turns AI research-agent claims into executable sealed-eval, matched-agent recovery, and optional feedback-effect tests. Two controlled audits (SQLite optimization and virtual catalyst control) each produced zero recoveries in 96 episodes with an upper bound of 0.0468, while paired studies yielded 30 truthful vs 0 neutral recoveries with passing null calibrations; a deterministic LLM-free verifier reproduces decisions from frozen evidence.","sledRelevance":"Why now: Newly relevant scrutiny against the same-week Virtual Biotech multi-agent score narrative and prior editions' agent harness wins—DCP supplies an executable standard that treats numerical improvement without recovery controls as insufficient for discovery claims.","evidence":"Gate definitions; 0/96 recoveries and 0.0468 bound; 30/0 truthful vs neutral recoveries; deterministic verifier claim; DeepSeek audit case metrics including 88.55% SQLite work reduction under sealed scoring.","architectureImplications":"Campus research-agent stacks need sealed evaluation packs and matched challenger harnesses, not leaderboard adapters alone.","governanceImplications":"Require precommitted targets and recovery bounds before accepting discovery language in reports or procurement.","securityPrivacyImplications":"Sealed labels and registered starting packs must be integrity-protected against challenger leakage.","caveats":"Controlled domains; finite-sample bounds; preprint; does not evaluate Virtual Biotech specifically; not a government audit.","streamIds":["research"],"roles":{"sales":"Sales framing: Campuses hearing strong AI-scientist scores need a way to ask whether a claimed discovery is recoverable by matched agents without the privileged research history. DCP evidence supports a bounded engagement around sealed utility checks, recovery audits, and optional feedback-effect gates—reporting zero recoveries in 96 episodes with upper bound 0.0468 in two controlled audits—not a productized campus service. Discovery should map which numerical outcomes, registered starting information, and web-visible bytes would define a fair challenger. Unsupported: that every research agent claim will pass Core, or that DCP replaces peer review.","engineering":"Engineering framing: Fit is an executable certification protocol with Gate 1 sealed improvement, Gate 2 matched-agent recovery under registered challenger information, and optional Gate 3 truthful-vs-neutral feedback effects, verified by a deterministic LLM-free checker. Integration points are frozen evidence packs, sealed evaluation workloads, and registered episode logs—not leaderboard APIs alone. Prerequisites: precommitted targets, sealed eval, matched agent harness, and finite-sample recovery bounds. Constraints: controlled SQLite and virtual-catalyst audits; not multi-campus production. Security: protect sealed labels and registered starting packs from leakage into challenger contexts. Proof of value is reproducible Core/Evidence decisions from frozen records.","delivery":"Delivery framing: Pilot DCP-style certification on one campus research-agent claim with sealed eval, recovery episodes, and deterministic verifier replay. Dependencies: research computing, evaluation owners, and PI claim registration. Ownership: evaluation lead owns certification decision; project team owns agent under test. Skills adoption teaches staff to register starting information and withhold privileged history from challengers. Governance checkpoints: precommitment review, recovery-bound sign-off, incomplete-audit handling. Acceptance: frozen evidence pack that a verifier reproduces; risks include incomplete audits and treating vendor scores as Core passes."},"roleDetails":{"sales":{"customerProblem":"Research AI claims are often sold on scores without tests that matched agents cannot recover the same result from registered public information.","stakeholders":["VPR / research integrity","Research computing","Evaluation leads","Faculty deploying research agents"],"discoveryQuestions":["What numerical outcome is claimed?","What starting information is registered?","Who runs matched challengers?","What recovery bound is acceptable?"],"valueHypothesis":"Executable recovery and sealed-eval gates can separate useful outcomes from unrecovered discoveries when Core requires zero recoveries and a finite-sample bound—as in the authors' 0/96 episodes with 0.0468 upper bound.","potentialEngagement":"Single-claim DCP Core pilot with sealed eval and matched recovery episodes plus deterministic verifier.","unsupportedClaims":["All agent research will pass Core","DCP replaces peer review","Guaranteed feedback-effect Evidence passes"]},"engineering":{"fit":"Protocolized sealed-eval, recovery, and optional feedback-effect certification with LLM-free verification.","architectureIntegration":"Store frozen evidence; isolate sealed labels; run matched agents on registered public bytes; emit Core/Evidence decisions.","prerequisites":["Precommitted targets","Sealed evaluation workload","Matched agent harness","Deterministic verifier"],"constraints":["Controlled audit domains (SQLite/catalyst)","Finite-sample bounds not absolute impossibility","Not a production SaaS study"],"security":"Prevent sealed-label and privileged-history leakage into challenger contexts; integrity-protect frozen evidence.","proofOfValue":["Zero recoveries with stated upper bound","Verifier reproduces decisions","Positive-control recall above registered minimum"]},"delivery":{"work":"Register one claim, run sealed Gate 1, execute recovery episodes, freeze evidence, and reproduce decisions with the verifier.","dependencies":["Evaluation owners","Compute for matched agents","PI claim registration"],"ownership":"Evaluation lead owns pass/fail; project owns agent under test.","skillsAdoption":"Train staff to withhold privileged research history from challengers and document incomplete audits.","governanceCheckpoints":["Precommitment review","Recovery-bound approval","Incomplete-audit escalation"],"acceptanceCriteria":["Frozen pack verifier-reproducible","Documented recovery count and bound","Written scope limits"],"risks":["Incomplete audits treated as passes","Score shopping around Core","Leakage of sealed labels"]}},"retrievedAt":"2026-09-20T03:13:20Z","enrichedAt":"2026-09-20T03:13:20Z","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Adds evaluation specialist work to interpret Core/Evidence decisions; benefits research integrity staff more than end users alone.","procurementImplications":"Write recovery-audit and sealed-eval clauses into research-agent RFPs; reject score-only acceptance criteria.","operatingModelImplications":"Splits agent builders from independent certifiers; incomplete audits must escalate rather than soft-pass.","sourceVerification":{"openedUrl":"https://arxiv.org/html/2609.09219v1","referenceExcerpt":"Each produced zero recoveries in 96 episodes, with an upper bound of 0.0468","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}