From the Research Universities edition of November 3, 2026
Reimagining computational quantum research with AI scientists (OmniQ and Residual Agents)
Nanyang Technological University, College of Computing and Data Science · University research (AI-agent reproduction of published computational science) · Singapore (international; transferability to U.S. research universities is partial)
- Publisher
- arXiv
- Original publication
- 2 November 2026 (arXiv submission)
- Source retrieved
- 2026-11-04
What happened
NTU researchers built OmniQ, an expert-audited environment turning 60 published computational quantum studies into reproduction tasks for AI agents. Expert review found reproducibility issues in 18 of the 60 studies, including one unresolvable inconsistency between paper, code and figure. On the cleaned 59-paper set, success ranged from about 19% (a third-party harness) to about 93% (best first-party agent). A self-improving agent (RA) trained on expert-verified repairs raised a low-cost model from 68% to about 85%.
Why it matters
Relevant to research-integrity and research-computing leaders evaluating agentic reproduction: it shows agent failures are partly failures of the published record, and that human expert review remains load-bearing. Limited direct SLED relevance beyond research universities with computational science groups.
Evidence and measured results
Authors evaluated five LLM agent systems and six third-party harnesses with a strict reproduction criterion, cross-checking with source code and domain experts. RA used five hardest tasks for development and 54 unseen tasks for evaluation; a human expert approves each memory update. Costs per task reported (e.g., $5.6 for RA). Preprint, not peer reviewed; agents and evaluators chosen by the authors; judge-agent comparison of figures is part of the pipeline.
Limitations and uncertainty
Single domain (computational quantum science); model names and versions are 2026 frontier systems that will change; harness comparisons are sensitive to configuration; failure analysis noted agents declaring success without validation or silently shrinking problems.
Put this evidence to work
Lighthouse Advisory interpretation, grounded in this source. Enriched 2026-11-03; this does not change the original publication date. Each role follows the same checklist so takeaways compare across sources.
Sales
Role takeaway
- Customer problem
- Research offices cannot cheaply check whether published computational results reproduce, and agent tools vary widely in reliability.
- Stakeholders
- Research computing director
- Research integrity officer
- Computational science faculty
- Library/data services
- Discovery questions
- Which computational fields at your institution release code with papers?
- Who would triage agent failures into agent error versus paper inconsistency?
- What compute budget per reproduction is acceptable?
- How would findings be communicated to authors?
- Value hypothesis
- In fields with released code, supervised agent reproduction may flag record inconsistencies earlier than manual checks, at reported per-task inference costs of a few to tens of dollars.
- Potential engagement
- A one-field pilot reproducing a small set of institution-authored papers with expert triage and a written findings report.
- Unsupported claims
- Agents can reliably reproduce science without experts
- Transfer of the 85-93% rates to other fields
- Predictable labor savings
Pre-sales engineering
Role takeaway
- Fit
- Good for computational fields with public code and modest compute; weak for wet-lab or proprietary-data work.
- Architecture and integration
- Plan/execute/judge agent loop, sandboxed runners, MCP access to verified workflows, artifact store.
- Prerequisites
- Sandboxed compute with per-task budgets
- Curated target-paper set with code
- Expert reviewers available
- Logging of agent actions
- Constraints
- Single-domain evidence
- Model versions change quickly
- Harness results do not transfer across domains
- Running third-party code carries risk
- Security
- Run agents in isolated containers without credentials; restrict network egress; treat repository code as untrusted.
- Proof of value
- Compare agent outcomes with expert reruns on the same papers
- Count unvalidated success claims
- Record expert hours per resolved case
Delivery
Role takeaway
- Work
- Set up supervised agent reproduction runs and expert triage workflow; Report reproduction outcomes and record inconsistencies
- Dependencies
- Research computing sandbox
- Faculty reviewer availability
- Agent/model access agreements
- Ownership
- Research computing, with a faculty research-integrity reviewer as accountable approver.
- Skills and adoption
- Reviewers need training to classify Level I-III issues and to read agent logs.
- Governance checkpoints
- Expert approval of agent memory updates
- Policy before contacting authors about inconsistencies
- Data-use review for any non-public data
- Acceptance criteria
- Proposed: reproduction outcome documented for every target paper
- Proposed: expert time per paper recorded
- Proposed: zero agent success claims accepted without expert check
- Risks
- Judge-agent errors
- Reputational issues when flagging colleagues work
- Cost overruns on long computations
Implementation considerations
Lighthouse Advisory interpretation across the operating dimensions a public-sector buyer must settle before this evidence becomes a design. Each note answers the question under its heading for this specific source.
Architecture and integration
What must connect, and where does the AI sit in the workflow?
Reproduction ran as plan/execute/judge agents with MCP-based access to verified workflows. Institutions would need sandboxed compute with budgets comparable to the original work and storage for reproduction artifacts.
Governance
Who approves, reviews and stays accountable for outcomes?
Human expert sign-off on memory updates and on classifying source-side inconsistencies is part of the method; institutions should keep that role explicit if adopting agent reproduction.
Security and privacy
What data, permissions and controls need testing?
Agents executed code from published repositories; sandboxing and limits on network/file access are implied needs but not described in detail.
Accessibility and workforce
Who is affected, and what skills or accommodations follow?
Expert reviewers remain essential; workforce implication is shifting expert time to triage rather than manual reruns.
Procurement
What should contracts, pricing and exit terms secure?
Procurement of agent tooling should allow switching models and harnesses because performance varied markedly across systems and cost did not track success.
Operating model
Which teams own the service once it runs?
Agent reproduction fits as a research-computing or integrity-office service with faculty review, not as a self-service tool without oversight.
Publication history
- 2026-11-03Research Universities · Issue 565 resources
Stable resource ID: ntu-omniq-ai-scientists-quantum-reproduction-20261102