{"resourceId":"ntu-omniq-ai-scientists-quantum-reproduction-20261102","versions":[{"version":"external-3e0b6b977c0fe359bf762e5839c5567e473d67691a8eba9fa8ec953de4b882b8","resource":{"id":"ntu-omniq-ai-scientists-quantum-reproduction-20261102","title":"Reimagining computational quantum research with AI scientists (OmniQ and Residual Agents)","organization":"Nanyang Technological University, College of Computing and Data Science","sector":"University research (AI-agent reproduction of published computational science)","geography":"Singapore (international; transferability to U.S. research universities is partial)","publishedAt":"2 November 2026 (arXiv submission)","publicationDate":"2026-11-02","eventDate":null,"sourceName":"arXiv","sourceLabel":"arXiv 2611.02833, Du, Zhao et al., \"Reimagining computational quantum research with AI scientists,\" submitted 2 Nov 2026","sourceUrl":"https://arxiv.org/abs/2611.02833","evidenceClass":"academic-research","outcomeClass":"mixed","topics":["developers-agents","knowledge-work","operating-model"],"finding":"NTU researchers built OmniQ, an expert-audited environment turning 60 published computational quantum studies into reproduction tasks for AI agents. Expert review found reproducibility issues in 18 of the 60 studies, including one unresolvable inconsistency between paper, code and figure. On the cleaned 59-paper set, success ranged from about 19% (a third-party harness) to about 93% (best first-party agent). A self-improving agent (RA) trained on expert-verified repairs raised a low-cost model from 68% to about 85%.","sledRelevance":"Relevant to research-integrity and research-computing leaders evaluating agentic reproduction: it shows agent failures are partly failures of the published record, and that human expert review remains load-bearing. Limited direct SLED relevance beyond research universities with computational science groups.","evidence":"Authors evaluated five LLM agent systems and six third-party harnesses with a strict reproduction criterion, cross-checking with source code and domain experts. RA used five hardest tasks for development and 54 unseen tasks for evaluation; a human expert approves each memory update. Costs per task reported (e.g., $5.6 for RA). Preprint, not peer reviewed; agents and evaluators chosen by the authors; judge-agent comparison of figures is part of the pipeline.","architectureImplications":"Reproduction ran as plan/execute/judge agents with MCP-based access to verified workflows. Institutions would need sandboxed compute with budgets comparable to the original work and storage for reproduction artifacts.","governanceImplications":"Human expert sign-off on memory updates and on classifying source-side inconsistencies is part of the method; institutions should keep that role explicit if adopting agent reproduction.","securityPrivacyImplications":"Agents executed code from published repositories; sandboxing and limits on network/file access are implied needs but not described in detail.","caveats":"Single domain (computational quantum science); model names and versions are 2026 frontier systems that will change; harness comparisons are sensitive to configuration; failure analysis noted agents declaring success without validation or silently shrinking problems.","streamIds":["research"],"roles":{"sales":"Research-integrity framing: agent reproduction is a way to surface problems in the published record, not just to automate work. NTU found issues in 18 of 60 studies and wide agent variance (about 19% to 93%). A bounded conversation with a research computing or research integrity office could scope a reproduction pilot in one computational field, with expert review built in. Interpretation: this is not evidence that agents can replace reviewers or that savings are predictable.","engineering":"Fit is strongest where published work ships code and runs on modest compute. The architecture is plan/execute/judge agents plus an MCP platform of verified workflows. Prerequisites are sandboxed execution, compute budgets matched to the original study, and artifact storage. Constraints: single domain, rapidly changing model versions, and harness behavior that does not transfer. A proof of value should log cases where agents claim success without validating against the paper, and compare against expert reruns.","delivery":"Delivery work is a supervised reproduction service: intake of target papers, agent runs, expert triage of failures into agent error versus record inconsistency, and reporting back to authors or editors. Ownership sits with research computing plus a faculty reviewer pool. Governance checkpoints include expert approval of any agent memory update and policy on contacting authors. Acceptance criteria should be proposed as measurable reproduction rates with documented expert time; risks include over-trusting judge agents."},"roleDetails":{"sales":{"customerProblem":"Research offices cannot cheaply check whether published computational results reproduce, and agent tools vary widely in reliability.","stakeholders":["Research computing director","Research integrity officer","Computational science faculty","Library/data services"],"discoveryQuestions":["Which computational fields at your institution release code with papers?","Who would triage agent failures into agent error versus paper inconsistency?","What compute budget per reproduction is acceptable?","How would findings be communicated to authors?"],"valueHypothesis":"In fields with released code, supervised agent reproduction may flag record inconsistencies earlier than manual checks, at reported per-task inference costs of a few to tens of dollars.","potentialEngagement":"A one-field pilot reproducing a small set of institution-authored papers with expert triage and a written findings report.","unsupportedClaims":["Agents can reliably reproduce science without experts","Transfer of the 85-93% rates to other fields","Predictable labor savings"]},"engineering":{"fit":"Good for computational fields with public code and modest compute; weak for wet-lab or proprietary-data work.","architectureIntegration":"Plan/execute/judge agent loop, sandboxed runners, MCP access to verified workflows, artifact store.","prerequisites":["Sandboxed compute with per-task budgets","Curated target-paper set with code","Expert reviewers available","Logging of agent actions"],"constraints":["Single-domain evidence","Model versions change quickly","Harness results do not transfer across domains","Running third-party code carries risk"],"security":"Run agents in isolated containers without credentials; restrict network egress; treat repository code as untrusted.","proofOfValue":["Compare agent outcomes with expert reruns on the same papers","Count unvalidated success claims","Record expert hours per resolved case"]},"delivery":{"work":"Set up supervised agent reproduction runs and expert triage workflow; Report reproduction outcomes and record inconsistencies","dependencies":["Research computing sandbox","Faculty reviewer availability","Agent/model access agreements"],"ownership":"Research computing, with a faculty research-integrity reviewer as accountable approver.","skillsAdoption":"Reviewers need training to classify Level I-III issues and to read agent logs.","governanceCheckpoints":["Expert approval of agent memory updates","Policy before contacting authors about inconsistencies","Data-use review for any non-public data"],"acceptanceCriteria":["Proposed: reproduction outcome documented for every target paper","Proposed: expert time per paper recorded","Proposed: zero agent success claims accepted without expert check"],"risks":["Judge-agent errors","Reputational issues when flagging colleagues work","Cost overruns on long computations"]}},"retrievedAt":"2026-11-03T22:06:02-06:00","enrichedAt":"2026-11-03T22:07:06-06:00","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Expert reviewers remain essential; workforce implication is shifting expert time to triage rather than manual reruns.","procurementImplications":"Procurement of agent tooling should allow switching models and harnesses because performance varied markedly across systems and cost did not track success.","operatingModelImplications":"Agent reproduction fits as a research-computing or integrity-office service with faculty review, not as a self-service tool without oversight.","peerReview":"preprint","sourceVerification":{"openedUrl":"https://arxiv.org/abs/2611.02833","referenceExcerpt":"current AI scientists often fail to reproduce published results, exposing a fundamental mismatch between what scientific papers communicate and what autonomous agents require.","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}