From the Research Universities edition of November 7, 2026
TCSExam: Evaluating Frontier LLMs on the Full Range of TCS Research
Tsinghua University; Moonshot AI; Shanghai Qizhi Institute · University / industry research (theoretical computer science evaluation of LLM agents) · China-led (Tsinghua, Shanghai Qizhi Institute, Moonshot AI); transferable as an evaluation method for CS theory groups elsewhere, not as a claim about campus IT deployments
- Publisher
- arXiv
- Original publication
- 5 November 2026 (arXiv 2611.07962v1)
- Source retrieved
- 2026-11-08
What happened
Tsinghua, Moonshot AI, and Shanghai Qizhi Institute authors (including Andrew Chi-Chih Yao) introduce TCSExam, a benchmark meant to cover the full range of theoretical computer science research: theorem proving, solver engineering, constructions, and direct attempts to resolve open conjectures. TCSExam-Full has 100 expert-reviewed tasks (20 Lean 4 research theorems, 30 solver-engineering problems, 20 constructions, and 30 open conjectures in prove-or-refute form); TCSExam-Lite selects 20 tasks, five per track, cutting task attempts by 80 percent at the same per-attempt budget. Every submission is checked by the Lean kernel or an executable task checker. Eight agents are evaluated on Full and nine on Lite. The combined TCSExam Index is led by GPT-6 Astra on Full (71.9) and Opus 5.5 on Lite (72.3); Full and Lite rankings align strongly across eight shared agents (Spearman rho = 0.93). Evaluations produced improved solver outputs and mathematical constructions, plus independently verified special cases and bounds from open-conjecture attempts. The paper reports that all five Lite open conjectures remain unresolved. Docker build recipes and evaluation tools are released for reproducible use.
Why it matters
Helps research universities distinguish benchmark progress on machine-checkable TCS tasks from resolution of open scientific questions. Useful for CS theory groups and research computing units asked to host research-agent evaluations with executable graders rather than LLM-as-judge scores alone.
Evidence and measured results
arXiv preprint describing an expert-reviewed 100-task suite, Lean or executable verification, a multi-agent leaderboard, rank-correlation between Full and Lite, and an explicit statement that Lite open conjectures remain unresolved. Industry co-authors (Moonshot AI). Not peer-reviewed.
Limitations and uncertainty
Preprint; TCS-specific and partly industry-affiliated (Moonshot AI). The Index is a combined score, not a percentage of conjectures solved. Open-conjecture success should not be inferred from the 71.9/72.3 leads. Transfer to wet-lab or observational science is limited because those domains often lack Lean-like checkers.
Put this evidence to work
Lighthouse Advisory interpretation, grounded in this source. Enriched 2026-11-07; this does not change the original publication date. Each role follows the same checklist so takeaways compare across sources.
Sales
Role takeaway
- Customer problem
- Campuses lack a way to tell whether research agents are making checkable TCS progress or merely producing plausible writeups of unsolved problems.
- Stakeholders
- CS theory faculty
- Research computing
- Vice President for Research / research integrity
- Graduate program leadership in CS
- Discovery questions
- Do current agent evaluations use Lean or executable checkers, or only model judges?
- Are open problems scored separately from theorem, solver, and construction tracks?
- Can another group rerun your evaluation from Docker recipes?
- Have any claimed conjecture resolutions been independently verified?
- Value hypothesis
- A Lean or executable TCS exam can rank agents on checkable research outputs (paper: Astra 71.9 Full; Opus 5.5 72.3 Lite) while keeping unresolved conjectures visible.
- Potential engagement
- A Lite-track campus rerun with sandboxed checkers and a public unresolved-conjecture appendix.
- Unsupported claims
- That leading Index scores mean open TCS conjectures have been resolved (all five Lite conjectures remain unresolved)
- Transfer of Index values to non-TCS scientific domains
- Vendor superiority beyond the eight or nine agents and harness reported
Pre-sales engineering
Role takeaway
- Fit
- Strong for theory and formal-methods groups with Lean or executable graders; weak where scientific success cannot be machine-checked.
- Architecture and integration
- Task package plus agent attempt plus Lean kernel or executable checker plus per-track scores rolled into an Index; Lite as a cheaper proxy (rho = 0.93 with Full in this study).
- Prerequisites
- Sandboxed Docker or equivalent runners
- Lean 4 toolchain for proof tasks
- Expert-reviewed task statements
- Model or agent API access
- Constraints
- TCS-only task distribution
- Industry co-authorship (Moonshot AI)
- Open conjectures remain unresolved on Lite
- Preprint leaderboard may shift with new models
- Security
- Execute untrusted generated proofs and solver code in isolation; avoid sending private student or unpublished problems to external APIs without review.
- Proof of value
- Reproducible Lite or Full leaderboard using the released recipes
- Separate reporting of unresolved conjectures vs verified special cases
- Rank stability check between Lite and Full if both are run
Delivery
Role takeaway
- Work
- Deploy the Docker evaluation tools; select Lite or Full; run listed agents; publish track scores and unresolved items.
- Dependencies
- Theory-faculty task review
- Compute for agent attempts and checkers
- License or access to evaluated models
- Ownership
- CS theory faculty own scientific interpretation; research computing owns the harness.
- Skills and adoption
- Staff learn to read Lean and checker logs instead of grading prose writeups.
- Governance checkpoints
- No public claim of a solved open conjecture without checker-plus-expert confirmation
- Track scores reported separately from the combined Index
- Harness version pinned with the leaderboard
- Acceptance criteria
- Proposed: all submissions pass through Lean or the executable checker
- Proposed: Lite or Full documentation includes unresolved conjectures
- Proposed: Docker rerun instructions work on campus hardware
- Risks
- Index scores being advertised as conjecture resolution
- Overfitting agents to the public 100-task set
- Unsafe execution of generated solver code
Implementation considerations
Lighthouse Advisory interpretation across the operating dimensions a public-sector buyer must settle before this evidence becomes a design. Each note answers the question under its heading for this specific source.
Architecture and integration
What must connect, and where does the AI sit in the workflow?
Prefer evaluation harnesses that score research-agent output with Lean kernels or executable checkers, and publish Docker recipes so other campuses can rerun the same tasks. Keep open-conjecture attempts in a separate track so index scores cannot be mistaken for solved science.
Governance
Who approves, reviews and stays accountable for outcomes?
Research leadership should not treat a roughly 72 Index lead as evidence that agents can settle open TCS conjectures; the paper itself reports those Lite conjectures unresolved and counts verified special cases separately.
Security and privacy
What data, permissions and controls need testing?
Running agent evaluations with Dockerized checkers executes model-generated proofs and solver code. Campus deployments need sandboxed runners and careful handling of any non-public task variants. The preprint does not describe production security controls.
Accessibility and workforce
Who is affected, and what skills or accommodations follow?
Executable checkers can lower subjective grading burden, but preparing Lean tasks and interpreting special-case proofs still requires expert TCS labor.
Procurement
What should contracts, pricing and exit terms secure?
When procuring research-agent evaluation services, require machine-checkable graders, published Docker recipes, and separate open-problem reporting—not a single headline score.
Operating model
Which teams own the service once it runs?
Suggests a faculty-owned exam with research-computing-operated sandboxes, analogous to a shared benchmark service rather than a discovery product.
Publication history
- 2026-11-07Research Universities · Issue 604 resources
Stable resource ID: tcsexam-frontier-llm-tcs-research-benchmark-20261105