From the Research Universities edition of September 19, 2026
OpenDiscoveryTrace: matched 84–89% success rates hide ~30× process-error gaps across frontier models
OpenDiscoveryTrace authors (arXiv preprint) · AI scientist evaluation / process-level auditing · International academic dataset; transferable methods with judge limits
- Publisher
- arXiv
- Original publication
- September 5, 2026 (arXiv v1)
- Source retrieved
- 2026-09-20
What happened
OpenDiscoveryTrace releases 558 full AI scientific-agent trajectories with 9-field-per-step process logs across 124 tasks. Pilot analysis on 363 LLM-judged trajectories finds frontier models achieve comparable success rates (84–89%) while Claude Opus 4.6 produces ~30× more errors per trajectory than GPT-5.4 (2.5 vs 0.08; p<0.0001), with different mixes (66.7% tool misuse vs 83.6% reasoning errors). Hallucinations were rare in their taxonomy (1/688 errors).
Why it matters
Why now: Newly relevant against same-week multi-agent deployment and score-centric narratives—shows endpoint success is insufficient process evidence for campus research-agent operations.
Evidence and measured results
558 trajectories; success-rate band; 2.5 vs 0.08 errors; error-type percentages; failure taxonomy including hallucination rarity.
Limitations and uncertainty
LLM-judged pilot subset; author harness; hallucination rarity may not generalize; not institutional deployment study.
Put this evidence to work
Lighthouse Advisory interpretation, grounded in this source. Enriched 2026-09-20; this does not change the original publication date. Each role follows the same checklist so takeaways compare across sources.
Sales
Role takeaway
- Customer problem
- Vendors present similar agent success rates while hiding large differences in process errors that affect trust and ops cost.
- Stakeholders
- Research computing
- AI evaluation leads
- Faculty adopting science agents
- Risk/compliance
- Discovery questions
- Do you log tool calls and revisions?
- How do you compare models beyond accuracy?
- Who reviews error taxonomies?
- What judge model is used?
- Value hypothesis
- Process traces can expose large error-rate gaps (e.g., ~30×) among models with similar 84–89% success, improving procurement and ops decisions without claiming scientific correctness.
- Potential engagement
- Trace-instrumented dual-model pilot on a fixed scientific task panel with success-plus-error reporting.
- Unsupported claims
- Traces prove scientific validity
- Hallucination rarity generalizes
- Judge scores are unbiased
Pre-sales engineering
Role takeaway
- Fit
- Agent-agnostic trajectory logging with structured per-step fields for process evaluation.
- Architecture and integration
- Attach logger to agent tool loop; store traces; run process benchmarks and failure attribution offline.
- Prerequisites
- Trace schema adoption
- Secure trace storage
- Fixed task panel
- Disclosed judge/eval method
- Constraints
- LLM-judged pilot subset
- Author-controlled harness
- Not production multi-site
- Security
- Restrict access to traces containing prompts, tool outputs, and possibly sensitive research content.
- Proof of value
- Success rates with error rates per model
- Error-type mix (tool vs reasoning)
- Statistical significance of differentials
Delivery
Role takeaway
- Work
- Enable trace logging, run dual-model panel, analyze error profiles, and publish ops-facing metrics beside success rates.
- Dependencies
- Logging platform
- Model access
- Evaluation staff
- Ownership
- Evaluation lead owns metrics; research groups own tasks.
- Skills and adoption
- Train operators to act on tool-misuse vs reasoning-error patterns differently.
- Governance checkpoints
- Judge disclosure
- Sample-size adequacy
- Sensitive-trace access review
- Acceptance criteria
- Success+error report delivered
- At least two models compared
- Documented limitations
- Risks
- Judge-model bias
- Trace storage sprawl
- Overinterpreting rare hallucinations
Implementation considerations
Lighthouse Advisory interpretation across the operating dimensions a public-sector buyer must settle before this evidence becomes a design. Each note answers the question under its heading for this specific source.
Architecture and integration
What must connect, and where does the AI sit in the workflow?
Instrument agents to emit structured process traces; evaluate models on error profiles as well as endpoints.
Governance
Who approves, reviews and stays accountable for outcomes?
Disclose judge models and require process metrics in research-agent acceptance reviews.
Security and privacy
What data, permissions and controls need testing?
Trace stores may contain sensitive prompts and tool outputs—apply access control and retention limits.
Accessibility and workforce
Who is affected, and what skills or accommodations follow?
Creates demand for operators who can read process-error mixes; may reduce blind reliance on endpoint dashboards.
Procurement
What should contracts, pricing and exit terms secure?
Require trace export and dual-metric (success+error) reporting in research-agent purchases.
Operating model
Which teams own the service once it runs?
Moves evaluation from scoreboards to continuous process review with model-specific remediation paths.
Publication history
- 2026-09-19Research Universities · Issue 144 resources
Stable resource ID: opendiscoverytrace-process-traces-20260905