{"resourceId":"opendiscoverytrace-process-traces-20260905","versions":[{"version":"external-dbef9f2527b82962097de2411f6c863323b90ff125e4d9b73f601cbffd33e60e","resource":{"id":"opendiscoverytrace-process-traces-20260905","title":"OpenDiscoveryTrace: matched 84–89% success rates hide ~30× process-error gaps across frontier models","organization":"OpenDiscoveryTrace authors (arXiv preprint)","sector":"AI scientist evaluation / process-level auditing","geography":"International academic dataset; transferable methods with judge limits","publishedAt":"September 5, 2026 (arXiv v1)","publicationDate":"2026-09-05","eventDate":null,"sourceName":"arXiv","sourceLabel":"Academic preprint — OpenDiscoveryTrace process trajectories","sourceUrl":"https://arxiv.org/html/2609.09203","evidenceClass":"academic-research","outcomeClass":"cautionary","topics":["developers-agents","operating-model","governance-procurement","knowledge-work"],"finding":"OpenDiscoveryTrace releases 558 full AI scientific-agent trajectories with 9-field-per-step process logs across 124 tasks. Pilot analysis on 363 LLM-judged trajectories finds frontier models achieve comparable success rates (84–89%) while Claude Opus 4.6 produces ~30× more errors per trajectory than GPT-5.4 (2.5 vs 0.08; p<0.0001), with different mixes (66.7% tool misuse vs 83.6% reasoning errors). Hallucinations were rare in their taxonomy (1/688 errors).","sledRelevance":"Why now: Newly relevant against same-week multi-agent deployment and score-centric narratives—shows endpoint success is insufficient process evidence for campus research-agent operations.","evidence":"558 trajectories; success-rate band; 2.5 vs 0.08 errors; error-type percentages; failure taxonomy including hallucination rarity.","architectureImplications":"Instrument agents to emit structured process traces; evaluate models on error profiles as well as endpoints.","governanceImplications":"Disclose judge models and require process metrics in research-agent acceptance reviews.","securityPrivacyImplications":"Trace stores may contain sensitive prompts and tool outputs—apply access control and retention limits.","caveats":"LLM-judged pilot subset; author harness; hallucination rarity may not generalize; not institutional deployment study.","streamIds":["research"],"roles":{"sales":"Sales framing: Buyers comparing AI scientist tools often see similar success rates and assume equivalent process quality. OpenDiscoveryTrace evidence supports a bounded pitch for process-trace auditing—frontier models hit comparable 84–89% success yet Claude Opus 4.6 produced ~30× more errors per trajectory than GPT-5.4 (2.5 vs 0.08)—so campus programs should buy evaluation that keeps thoughts, tool calls, and error taxonomies, not endpoint scores alone. Unsupported: that low hallucination counts (1/688 errors) generalize to all deployments, or that traces replace domain peer review.","engineering":"Engineering framing: Fit is a trajectory-logging harness with a 9-field-per-step schema (thoughts, tools, observations, errors, revisions, confidence) across scientific tasks, enabling process-level benchmarks beyond outputs. Integration attaches to existing agents without replacing them. Prerequisites: standardized trace schema, storage for hundreds of trajectories, and judge/process-eval pipelines with disclosed limitations. Constraints: pilot LLM-judged subset (363/558); author harness; not a multi-institution production study. Security: traces may contain prompts and tool outputs needing access control. Proof of value is statistically significant process differentials at matched success rates.","delivery":"Delivery framing: Instrument one research-agent pilot to emit OpenDiscoveryTrace-like fields, compare at least two models on a fixed task panel, and report error profiles alongside success rates. Dependencies: research computing logging, evaluation owners, and model API access. Ownership: evaluation lead owns process metrics; labs own task definitions. Skills adoption teaches staff to inspect tool-misuse vs reasoning-error mixes. Governance checkpoints: judge-model disclosure and sample-size review. Acceptance: published success-plus-error profile; risks include judge bias and treating rare hallucinations as absence of risk."},"roleDetails":{"sales":{"customerProblem":"Vendors present similar agent success rates while hiding large differences in process errors that affect trust and ops cost.","stakeholders":["Research computing","AI evaluation leads","Faculty adopting science agents","Risk/compliance"],"discoveryQuestions":["Do you log tool calls and revisions?","How do you compare models beyond accuracy?","Who reviews error taxonomies?","What judge model is used?"],"valueHypothesis":"Process traces can expose large error-rate gaps (e.g., ~30×) among models with similar 84–89% success, improving procurement and ops decisions without claiming scientific correctness.","potentialEngagement":"Trace-instrumented dual-model pilot on a fixed scientific task panel with success-plus-error reporting.","unsupportedClaims":["Traces prove scientific validity","Hallucination rarity generalizes","Judge scores are unbiased"]},"engineering":{"fit":"Agent-agnostic trajectory logging with structured per-step fields for process evaluation.","architectureIntegration":"Attach logger to agent tool loop; store traces; run process benchmarks and failure attribution offline.","prerequisites":["Trace schema adoption","Secure trace storage","Fixed task panel","Disclosed judge/eval method"],"constraints":["LLM-judged pilot subset","Author-controlled harness","Not production multi-site"],"security":"Restrict access to traces containing prompts, tool outputs, and possibly sensitive research content.","proofOfValue":["Success rates with error rates per model","Error-type mix (tool vs reasoning)","Statistical significance of differentials"]},"delivery":{"work":"Enable trace logging, run dual-model panel, analyze error profiles, and publish ops-facing metrics beside success rates.","dependencies":["Logging platform","Model access","Evaluation staff"],"ownership":"Evaluation lead owns metrics; research groups own tasks.","skillsAdoption":"Train operators to act on tool-misuse vs reasoning-error patterns differently.","governanceCheckpoints":["Judge disclosure","Sample-size adequacy","Sensitive-trace access review"],"acceptanceCriteria":["Success+error report delivered","At least two models compared","Documented limitations"],"risks":["Judge-model bias","Trace storage sprawl","Overinterpreting rare hallucinations"]}},"retrievedAt":"2026-09-20T03:13:20Z","enrichedAt":"2026-09-20T03:13:20Z","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Creates demand for operators who can read process-error mixes; may reduce blind reliance on endpoint dashboards.","procurementImplications":"Require trace export and dual-metric (success+error) reporting in research-agent purchases.","operatingModelImplications":"Moves evaluation from scoreboards to continuous process review with model-specific remediation paths.","sourceVerification":{"openedUrl":"https://arxiv.org/html/2609.09203","referenceExcerpt":"all three frontier models achieve comparable success rates (84–89%)","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}