From the Research Universities edition of September 19, 2026
Discovery Certification Protocol: scores alone do not prove discovery; 0/96 recoveries in controlled audits
Ning, Zhong, Li, Zeng (arXiv preprint) · AI research-agent evaluation / scientific discovery auditing · International academic authors; transferable protocol with site limits
- Publisher
- arXiv
- Original publication
- September 7, 2026 (arXiv v1)
- Source retrieved
- 2026-09-20
What happened
DCP turns AI research-agent claims into executable sealed-eval, matched-agent recovery, and optional feedback-effect tests. Two controlled audits (SQLite optimization and virtual catalyst control) each produced zero recoveries in 96 episodes with an upper bound of 0.0468, while paired studies yielded 30 truthful vs 0 neutral recoveries with passing null calibrations; a deterministic LLM-free verifier reproduces decisions from frozen evidence.
Why it matters
Why now: Newly relevant scrutiny against the same-week Virtual Biotech multi-agent score narrative and prior editions' agent harness wins—DCP supplies an executable standard that treats numerical improvement without recovery controls as insufficient for discovery claims.
Evidence and measured results
Gate definitions; 0/96 recoveries and 0.0468 bound; 30/0 truthful vs neutral recoveries; deterministic verifier claim; DeepSeek audit case metrics including 88.55% SQLite work reduction under sealed scoring.
Limitations and uncertainty
Controlled domains; finite-sample bounds; preprint; does not evaluate Virtual Biotech specifically; not a government audit.
Put this evidence to work
Lighthouse Advisory interpretation, grounded in this source. Enriched 2026-09-20; this does not change the original publication date. Each role follows the same checklist so takeaways compare across sources.
Sales
Role takeaway
- Customer problem
- Research AI claims are often sold on scores without tests that matched agents cannot recover the same result from registered public information.
- Stakeholders
- VPR / research integrity
- Research computing
- Evaluation leads
- Faculty deploying research agents
- Discovery questions
- What numerical outcome is claimed?
- What starting information is registered?
- Who runs matched challengers?
- What recovery bound is acceptable?
- Value hypothesis
- Executable recovery and sealed-eval gates can separate useful outcomes from unrecovered discoveries when Core requires zero recoveries and a finite-sample bound—as in the authors' 0/96 episodes with 0.0468 upper bound.
- Potential engagement
- Single-claim DCP Core pilot with sealed eval and matched recovery episodes plus deterministic verifier.
- Unsupported claims
- All agent research will pass Core
- DCP replaces peer review
- Guaranteed feedback-effect Evidence passes
Pre-sales engineering
Role takeaway
- Fit
- Protocolized sealed-eval, recovery, and optional feedback-effect certification with LLM-free verification.
- Architecture and integration
- Store frozen evidence; isolate sealed labels; run matched agents on registered public bytes; emit Core/Evidence decisions.
- Prerequisites
- Precommitted targets
- Sealed evaluation workload
- Matched agent harness
- Deterministic verifier
- Constraints
- Controlled audit domains (SQLite/catalyst)
- Finite-sample bounds not absolute impossibility
- Not a production SaaS study
- Security
- Prevent sealed-label and privileged-history leakage into challenger contexts; integrity-protect frozen evidence.
- Proof of value
- Zero recoveries with stated upper bound
- Verifier reproduces decisions
- Positive-control recall above registered minimum
Delivery
Role takeaway
- Work
- Register one claim, run sealed Gate 1, execute recovery episodes, freeze evidence, and reproduce decisions with the verifier.
- Dependencies
- Evaluation owners
- Compute for matched agents
- PI claim registration
- Ownership
- Evaluation lead owns pass/fail; project owns agent under test.
- Skills and adoption
- Train staff to withhold privileged research history from challengers and document incomplete audits.
- Governance checkpoints
- Precommitment review
- Recovery-bound approval
- Incomplete-audit escalation
- Acceptance criteria
- Frozen pack verifier-reproducible
- Documented recovery count and bound
- Written scope limits
- Risks
- Incomplete audits treated as passes
- Score shopping around Core
- Leakage of sealed labels
Implementation considerations
Lighthouse Advisory interpretation across the operating dimensions a public-sector buyer must settle before this evidence becomes a design. Each note answers the question under its heading for this specific source.
Architecture and integration
What must connect, and where does the AI sit in the workflow?
Campus research-agent stacks need sealed evaluation packs and matched challenger harnesses, not leaderboard adapters alone.
Governance
Who approves, reviews and stays accountable for outcomes?
Require precommitted targets and recovery bounds before accepting discovery language in reports or procurement.
Security and privacy
What data, permissions and controls need testing?
Sealed labels and registered starting packs must be integrity-protected against challenger leakage.
Accessibility and workforce
Who is affected, and what skills or accommodations follow?
Adds evaluation specialist work to interpret Core/Evidence decisions; benefits research integrity staff more than end users alone.
Procurement
What should contracts, pricing and exit terms secure?
Write recovery-audit and sealed-eval clauses into research-agent RFPs; reject score-only acceptance criteria.
Operating model
Which teams own the service once it runs?
Splits agent builders from independent certifiers; incomplete audits must escalate rather than soft-pass.
Publication history
- 2026-09-19Research Universities · Issue 144 resources
Stable resource ID: dcp-discovery-certification-protocol-20260907