{"resourceId":"tcsexam-frontier-llm-tcs-research-benchmark-20261105","versions":[{"version":"external-2c4ce84ba1df08c7421f0e5e403639f09bec2da1e6740df819715f867e0f42d1","resource":{"id":"tcsexam-frontier-llm-tcs-research-benchmark-20261105","title":"TCSExam: Evaluating Frontier LLMs on the Full Range of TCS Research","organization":"Tsinghua University; Moonshot AI; Shanghai Qizhi Institute","sector":"University / industry research (theoretical computer science evaluation of LLM agents)","geography":"China-led (Tsinghua, Shanghai Qizhi Institute, Moonshot AI); transferable as an evaluation method for CS theory groups elsewhere, not as a claim about campus IT deployments","publishedAt":"5 November 2026 (arXiv 2611.07962v1)","publicationDate":"2026-11-05","eventDate":null,"sourceName":"arXiv","sourceLabel":"Luo, Y., Liu, Z., Yuan, Y., Yang, Z. & Yao, A. C.-C., \"TCSExam: Evaluating Frontier LLMs on the Full Range of TCS Research,\" arXiv:2611.07962, 5 Nov 2026","sourceUrl":"https://arxiv.org/abs/2611.07962","evidenceClass":"academic-research","outcomeClass":"mixed","topics":["developers-agents","knowledge-work"],"finding":"Tsinghua, Moonshot AI, and Shanghai Qizhi Institute authors (including Andrew Chi-Chih Yao) introduce TCSExam, a benchmark meant to cover the full range of theoretical computer science research: theorem proving, solver engineering, constructions, and direct attempts to resolve open conjectures. TCSExam-Full has 100 expert-reviewed tasks (20 Lean 4 research theorems, 30 solver-engineering problems, 20 constructions, and 30 open conjectures in prove-or-refute form); TCSExam-Lite selects 20 tasks, five per track, cutting task attempts by 80 percent at the same per-attempt budget. Every submission is checked by the Lean kernel or an executable task checker. Eight agents are evaluated on Full and nine on Lite. The combined TCSExam Index is led by GPT-6 Astra on Full (71.9) and Opus 5.5 on Lite (72.3); Full and Lite rankings align strongly across eight shared agents (Spearman rho = 0.93). Evaluations produced improved solver outputs and mathematical constructions, plus independently verified special cases and bounds from open-conjecture attempts. The paper reports that all five Lite open conjectures remain unresolved. Docker build recipes and evaluation tools are released for reproducible use.","sledRelevance":"Helps research universities distinguish benchmark progress on machine-checkable TCS tasks from resolution of open scientific questions. Useful for CS theory groups and research computing units asked to host research-agent evaluations with executable graders rather than LLM-as-judge scores alone.","evidence":"arXiv preprint describing an expert-reviewed 100-task suite, Lean or executable verification, a multi-agent leaderboard, rank-correlation between Full and Lite, and an explicit statement that Lite open conjectures remain unresolved. Industry co-authors (Moonshot AI). Not peer-reviewed.","architectureImplications":"Prefer evaluation harnesses that score research-agent output with Lean kernels or executable checkers, and publish Docker recipes so other campuses can rerun the same tasks. Keep open-conjecture attempts in a separate track so index scores cannot be mistaken for solved science.","governanceImplications":"Research leadership should not treat a roughly 72 Index lead as evidence that agents can settle open TCS conjectures; the paper itself reports those Lite conjectures unresolved and counts verified special cases separately.","securityPrivacyImplications":"Running agent evaluations with Dockerized checkers executes model-generated proofs and solver code. Campus deployments need sandboxed runners and careful handling of any non-public task variants. The preprint does not describe production security controls.","caveats":"Preprint; TCS-specific and partly industry-affiliated (Moonshot AI). The Index is a combined score, not a percentage of conjectures solved. Open-conjecture success should not be inferred from the 71.9/72.3 leads. Transfer to wet-lab or observational science is limited because those domains often lack Lean-like checkers.","streamIds":["research"],"roles":{"sales":"Evaluation framing: TCSExam shows frontier LLM agents can be scored on a 100-task, expert-reviewed suite spanning theorem proving, solver engineering, constructions, and open conjectures, with every submission checked by the Lean kernel or an executable task checker. GPT-6 Astra leads the Full TCSExam Index at 71.9 and Opus 5.5 leads Lite at 72.3, and Full/Lite rankings align (Spearman rho = 0.93). Use this with CS theory faculty and research computing when the question is how to evaluate research agents—not as proof that agents resolve long-standing conjectures. The paper reports all five Lite open conjectures remain unresolved, while recording verified special cases and improved solvers separately.","engineering":"Architecture framing: build research-agent evaluations around machine-checkable artifacts (Lean 4 proofs, executable solvers, constructions scored against documented records) rather than LLM-as-judge rubrics. Separate tracks keep concrete research outputs distinct from prove-or-refute open problems. Docker build recipes and task sources in the paper are the integration surface for a campus rerun. Do not collapse the combined Index into a claim that open science problems are being solved automatically.","delivery":"Delivery framing: if a campus wants a local research-agent exam, start with TCSExam-Lite (20 tasks, five per track) to cut attempts by 80 percent at the same per-attempt budget, then expand to Full if rankings need confirmation. Theory faculty should review task significance; research computing hosts sandboxed Lean and checker runs. Acceptance is a reproducible leaderboard plus an explicit unresolved-conjecture report, not a press claim that agents advanced a named open problem."},"roleDetails":{"sales":{"customerProblem":"Campuses lack a way to tell whether research agents are making checkable TCS progress or merely producing plausible writeups of unsolved problems.","stakeholders":["CS theory faculty","Research computing","Vice President for Research / research integrity","Graduate program leadership in CS"],"discoveryQuestions":["Do current agent evaluations use Lean or executable checkers, or only model judges?","Are open problems scored separately from theorem, solver, and construction tracks?","Can another group rerun your evaluation from Docker recipes?","Have any claimed conjecture resolutions been independently verified?"],"valueHypothesis":"A Lean or executable TCS exam can rank agents on checkable research outputs (paper: Astra 71.9 Full; Opus 5.5 72.3 Lite) while keeping unresolved conjectures visible.","potentialEngagement":"A Lite-track campus rerun with sandboxed checkers and a public unresolved-conjecture appendix.","unsupportedClaims":["That leading Index scores mean open TCS conjectures have been resolved (all five Lite conjectures remain unresolved)","Transfer of Index values to non-TCS scientific domains","Vendor superiority beyond the eight or nine agents and harness reported"]},"engineering":{"fit":"Strong for theory and formal-methods groups with Lean or executable graders; weak where scientific success cannot be machine-checked.","architectureIntegration":"Task package plus agent attempt plus Lean kernel or executable checker plus per-track scores rolled into an Index; Lite as a cheaper proxy (rho = 0.93 with Full in this study).","prerequisites":["Sandboxed Docker or equivalent runners","Lean 4 toolchain for proof tasks","Expert-reviewed task statements","Model or agent API access"],"constraints":["TCS-only task distribution","Industry co-authorship (Moonshot AI)","Open conjectures remain unresolved on Lite","Preprint leaderboard may shift with new models"],"security":"Execute untrusted generated proofs and solver code in isolation; avoid sending private student or unpublished problems to external APIs without review.","proofOfValue":["Reproducible Lite or Full leaderboard using the released recipes","Separate reporting of unresolved conjectures vs verified special cases","Rank stability check between Lite and Full if both are run"]},"delivery":{"work":"Deploy the Docker evaluation tools; select Lite or Full; run listed agents; publish track scores and unresolved items.","dependencies":["Theory-faculty task review","Compute for agent attempts and checkers","License or access to evaluated models"],"ownership":"CS theory faculty own scientific interpretation; research computing owns the harness.","skillsAdoption":"Staff learn to read Lean and checker logs instead of grading prose writeups.","governanceCheckpoints":["No public claim of a solved open conjecture without checker-plus-expert confirmation","Track scores reported separately from the combined Index","Harness version pinned with the leaderboard"],"acceptanceCriteria":["Proposed: all submissions pass through Lean or the executable checker","Proposed: Lite or Full documentation includes unresolved conjectures","Proposed: Docker rerun instructions work on campus hardware"],"risks":["Index scores being advertised as conjecture resolution","Overfitting agents to the public 100-task set","Unsafe execution of generated solver code"]}},"retrievedAt":"2026-11-07T22:08:55-06:00","enrichedAt":"2026-11-07T22:14:26-06:00","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Executable checkers can lower subjective grading burden, but preparing Lean tasks and interpreting special-case proofs still requires expert TCS labor.","procurementImplications":"When procuring research-agent evaluation services, require machine-checkable graders, published Docker recipes, and separate open-problem reporting—not a single headline score.","operatingModelImplications":"Suggests a faculty-owned exam with research-computing-operated sandboxes, analogous to a shared benchmark service rather than a discovery product.","peerReview":"preprint","sourceVerification":{"openedUrl":"https://arxiv.org/abs/2611.07962","referenceExcerpt":"The TCSExam Index combines all four tracks: GPT-6 Astra leads Full (71.9), and Opus 5.5 leads Lite (72.3).","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}