{"resourceId":"sa-bench-semantic-alignment-20260825","versions":[{"version":"external-1e77418a136a9c3a425ded913bfc79ec2940206c5d6ef2054e3c0bbab44c140b","resource":{"id":"sa-bench-semantic-alignment-20260825","title":"SA-Bench: best LLM paper-reproduction config averages only 0.301 SAU; overall mean 0.221","organization":"Beihang University / Shanghai Jiao Tong University / Peking University et al. (Hu, Pan, Su, Liu, Zhang)","sector":"Academic AI evaluation / scientific code reproduction","geography":"International academic authors; ML conference paper corpus (ICLR/ICML/NeurIPS 2025)","publishedAt":"August 25, 2026 (arXiv 2608.24252v1)","publicationDate":"2026-08-25","eventDate":null,"sourceName":"arXiv","sourceLabel":"Academic preprint — SemanticAlign-Bench (SA-Bench); Findings of EMNLP 2026 acceptance noted by authors","sourceUrl":"https://arxiv.org/html/2608.24252","evidenceClass":"academic-research","outcomeClass":"cautionary","topics":["developers-agents","knowledge-work","governance-procurement","operating-model"],"finding":"SA-Bench decomposes 30 ML conference papers into 1,491 Semantic Alignment Units across numerical, methodological, protocol, and ordering dimensions and evaluates 12 generator configurations over 360 runs. Even the strongest configuration (Claude + PaperCoder) achieves a mean SAU score of only 0.301 out of 1.0, with an overall mean of 0.221. A failure taxonomy finds agents often attempt requirements but implement them incorrectly (implementation mismatch and stubs), arguing that scaffolds optimized for executability provide limited leverage for scientifically faithful reproduction.","sledRelevance":"Why now: Newly relevant scrutiny after the 2026-09-19 DCP/OpenDiscoveryTrace/ASI-Bench edition—adds a semantic-alignment dimension showing that runnable reproduction code can still silently diverge from paper specifications, complementary to recovery and process-trace gates.","evidence":"1,491 SAUs; 30 papers; 12 configs; 360 evaluations; best mean 0.301; overall mean 0.221; mismatch/stub failure modes; authors' executability-vs-semantics discussion.","architectureImplications":"Research-agent platforms need specification checkers and SAU-style claim inventories—not only sandboxes that reward green builds.","governanceImplications":"Treat 'code runs' as insufficient acceptance for AI-assisted reproduction claims in theses, grants, or vendor demos.","securityPrivacyImplications":"Limited direct privacy impact; integrity risk is silent methodological drift in reproduced experiments used for decisions.","caveats":"Preprint; ML papers only; judge-model scoring; not a multi-campus field deployment; EMNLP Findings status author-claimed on abstract page metadata pathways.","streamIds":["research"],"roles":{"sales":"Sales framing: Research integrity and computing leaders hearing that agents can 'reproduce papers' need evidence that implementations match specifications—not just that jobs exit zero. SA-Bench supports a bounded pitch around semantic-alignment evaluation (best mean SAU 0.301; overall 0.221 across 360 evaluations)—not a productized campus service that guarantees faithful reproduction. Discovery should map which papers, claim inventories, and human review steps define acceptance. Unsupported: that any scaffold will clear 0.5+ mean SAU, or that executability benchmarks equal scientific fidelity.","engineering":"Engineering framing: Fit is a diagnostic benchmark of atomic Semantic Alignment Units spanning numerical, method, protocol, and ordering drift for paper-to-code agents. Integration points are claim inventories, repository artifacts, and judge/scoring pipelines alongside existing CI—not replacing domain review. Prerequisites: paper claim extraction, controlled generator configs, and human-audited SAU samples. Constraints: ML corpus; LLM judges; preprint. Security focuses on integrity of evaluation packs against gaming. Proof of value is SAU scorecards with failure taxonomy—not build-success rates alone.","delivery":"Delivery framing: Pilot SAU-style semantic checks on one high-stakes paper reproduction before accepting agent-generated code into a research workflow. Dependencies: ML/research software engineering, evaluation owners, and PI claim owners. Ownership: evaluation lead owns fidelity gates; project owns generator under test. Skills adoption teaches staff to spot stubs and protocol drift. Governance checkpoints: SAU inventory sign-off, judge calibration, and incomplete-score handling. Acceptance: documented SAU scores with human spot-checks; risks include treating 0.3 means as success and overgeneralizing beyond ML."},"roleDetails":{"sales":{"customerProblem":"AI paper-reproduction demos often equate executable repositories with scientifically faithful implementations.","stakeholders":["Research integrity","Research computing","Faculty using coding agents","Evaluation leads"],"discoveryQuestions":["Which claims must match the paper?","Who judges semantic alignment?","What SAU threshold is acceptable?","How are stubs handled?"],"valueHypothesis":"Atomic semantic-alignment scoring can expose silent drift when even strong configs average only ~0.30 SAU—bounding claims that agents reproduce papers faithfully.","potentialEngagement":"Single-paper SAU pilot with human-audited claim inventory and scorecard.","unsupportedClaims":["Guaranteed high SAU with any scaffold","Executability equals scientific fidelity","Results transfer unchanged to non-ML domains"]},"engineering":{"fit":"Diagnostic SAU benchmark for numerical, methodological, protocol, and ordering alignment of generated reproduction code.","architectureIntegration":"Add claim-inventory and scoring stages beside CI sandboxes; retain human review for contested SAUs.","prerequisites":["Paper claim extraction","Repository under test","Scoring/judge pipeline","Calibration set"],"constraints":["ML conference corpus","LLM-as-judge variance","Preprint status"],"security":"Protect evaluation packs from leakage into generators; integrity-check scored artifacts.","proofOfValue":["Mean SAU with confidence bounds","Failure taxonomy breakdown","Human agreement on sampled SAUs"]},"delivery":{"work":"Build SAU inventory for one paper, run generator configs, score, and human-spot-check failures.","dependencies":["RSE/evaluation owners","PI claim owner","Compute for generators"],"ownership":"Evaluation lead owns pass criteria; project owns agent/scaffold.","skillsAdoption":"Train staff to detect stubs, protocol skips, and numerical precision drift.","governanceCheckpoints":["Inventory approval","Judge calibration","Incomplete-score policy"],"acceptanceCriteria":["Published SAU scorecard","Documented failure modes","Human spot-check log"],"risks":["Overclaiming from low means","Judge bias","Scope creep beyond ML papers"]}},"retrievedAt":"2026-09-20T22:11:20-05:00","enrichedAt":"2026-09-20T22:11:20-05:00","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Raises the bar for who can certify agent reproductions—needs evaluation-literate RSEs, not only developer agents.","procurementImplications":"Require semantic-alignment or equivalent fidelity gates in research-agent procurements; reject executability-only acceptance tests.","operatingModelImplications":"Adds an evaluation function between coding agents and scientific acceptance; fund that function explicitly.","sourceVerification":{"openedUrl":"https://arxiv.org/html/2608.24252","referenceExcerpt":"achieves a mean SAU score of only 0.301 out of 1.0","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}