{"resourceId":"rules-to-tools-scientific-computing-checks-20260929","versions":[{"version":"external-d75f29e31e507350f0a263bae857d39e0fc4320558141c5b596528935418393d","resource":{"id":"rules-to-tools-scientific-computing-checks-20260929","title":"Rules to Tools: prepared executable checks lift SciCode repair 26/30→29/30 and cut PDE model output 31.2%","organization":"Carnegie Mellon University School of Computer Science and DP Technology — arXiv preprint (cs.AI / scientific computing agents)","sector":"Higher education / scientific software engineering and research computing agent tooling","geography":"United States / China (authors); transferable to campus SciCode-style scientific coding agent platforms","publishedAt":"September 29, 2026 (arXiv 2610.00313v1)","publicationDate":"2026-09-29","eventDate":null,"sourceName":"arXiv 2610.00313","sourceLabel":"Academic preprint (arXiv)","sourceUrl":"https://arxiv.org/abs/2610.00313","evidenceClass":"academic-research","outcomeClass":"mixed","topics":["developers-agents","infrastructure","operating-model","governance-procurement"],"finding":"Rules to Tools (R2T; arXiv 2610.00313, submitted 29 Sep 2026) packages public scientific requirements (boundaries, equations, output contracts) as callable checks during agent code repair. Matched SciCode groups share written checks, starts, model, and budgets; the tool group alone gets a prepared implementation. Across two task-ID cohorts, complete repair is 26/30 with text and 29/30 with checks (eight-ID cohort 13/16→15/16; bootstrap 95% CI [-12.5, 43.75] pp). Three task IDs favor tools, one favors text, eleven tie. Shared-definition cohort ties 13/24; development-exposed alternate starts score 3/10 vs 7/10. Matched detailed-text PDE comparison: 23/24 vs 24/24 with 31.2% lower reported model output for checks, while public CPU rises in task-ID cohorts. Gains are task-dependent and can appear even when initial checks report no violation.","sledRelevance":"Campus research computing deploying coding agents on scientific numerical work should invest in prepared executable scientific checks—not only written specs—and meter both token savings and CPU cost.","evidence":"arXiv abs HTML and PDF via HTTPS curl; abstract repair fractions and PDE 31.2% output claim confirmed in pdftotext; check_sources not archived.","architectureImplications":"Expose public scientific relations as callable diagnostics on the current artifact alongside ordinary Python; keep independent hidden graders for final scores.","governanceImplications":"Treat repair lifts as task-dependent; require multi-task bake-offs and cost reports before mandating checker middleware campus-wide.","securityPrivacyImplications":"Checker containers should see only public probes and candidate code; keep hidden tests and credentials outside the agent workspace.","caveats":"Preprint; DeepSeek Flash controlled runs; wide bootstrap CI on eight-ID difference; checker authoring time unrecorded; not a multi-site HPC ops study.","streamIds":["research"],"roles":{"sales":"Sales framing: scientific coding agents fail quietly when they only have written equations. R2T (CMU/DP Technology; arXiv 2610.00313, 29 Sep 2026) supplies prepared executable checks of public requirements during repair. Matched SciCode cohorts rise from 26/30 to 29/30 complete repairs with tools; PDE checks match or beat detailed text (24/24 vs 23/24) with 31.2% lower reported model output. Pitch a checker middleware pilot with token and CPU meters—not a blanket claim of uniform gains.","engineering":"Engineering framing: author public relations as Describe/Implement pairs, freeze checks before paired repair, and expose a dedicated check command (or checker source via ordinary Python). Retain independent graders. Expect task-level ties and occasional text wins; measure output tokens and public CPU separately because tools can save agent tokens while raising CPU.","delivery":"Delivery framing: pick failed SciCode/PDE-like campus modules, freeze written+tool checks, run matched text vs tool trajectories, and review task signs plus cost. Dependencies include checker authors, sandbox CPU quotas, and hidden graders. Acceptance needs net repair gain without unacceptable CPU blowups. Risks include task-dependent effects, authoring cost, and over-generalizing from one favorable task ID."},"roleDetails":{"sales":{"customerProblem":"Scientific coding agents revise programs against prose requirements and miss measurable physical/numerical violations.","stakeholders":["Research computing","Research software engineers","Computational science PIs","FinOps"],"discoveryQuestions":["Which public equations/boundaries already gate acceptance?","What is the CPU vs token cost ceiling?","Which task families failed recent agent repair pilots?"],"valueHypothesis":"Prepared executable checks can raise complete-repair rates and cut agent output tokens versus text-only scientific specs on matched starts.","potentialEngagement":"Four-week matched text-vs-tool bake-off on 8–15 campus scientific coding failures with cost dashboards.","unsupportedClaims":["Uniform gains on every task","Zero authoring cost","CPU-neutral deployment"]},"engineering":{"fit":"Callable scientific requirement checks layered on SciCode/PDE-style agent repair loops.","architectureIntegration":"Freeze public relation/probe/tolerance specs; deliver written cards to both arms and prepared implementations to the tool arm; grade with independent hidden tests.","prerequisites":["Failed starting modules","Public requirement catalog","Sandbox with CPU/memory caps","Independent grader"],"constraints":["Task-dependent effects","Bootstrap CI may include zero","Authoring time unmeasured","Preprint"],"security":"Public-only probes in tool containers; hide reference solutions and credentials from the agent.","proofOfValue":["Complete-repair delta","Task sign tally","Output-token delta","Public CPU delta"]},"delivery":{"work":"Author/freeze checks, run matched trajectories, audit traces where tools win without initial violations, decide middleware rollout.","dependencies":["Checker authors","Agent harness","CPU quotas","Grader environment"],"ownership":"RSE/research computing own checker middleware; PIs own scientific acceptance criteria.","skillsAdoption":"Train RSEs to write public checks and operators to read diagnostic traces without treating them as final grades.","governanceCheckpoints":["Check freeze before repair","Mid-pilot cost review","Go/no-go by task family"],"acceptanceCriteria":["Net repair improvement on held-out IDs","Token/CPU within band","No leakage of hidden tests"],"risks":["Authoring bottleneck","CPU cost shock","Overfit to development-exposed tasks"]}},"retrievedAt":"2026-10-02T22:05:20-05:00","enrichedAt":"2026-10-02T22:12:00-05:00","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Executable checks shift labor toward RSEs who can encode public scientific relations; lowers some agent token load but raises need for checker authors.","procurementImplications":"When buying scientific coding agents, require callable public-check support and published repair/cost deltas—not prose-only validation claims.","operatingModelImplications":"Operate checkers as shared research-computing middleware with frozen specs, independent grading, and separate token/CPU budgets.","sourceVerification":{"openedUrl":"https://arxiv.org/abs/2610.00313","referenceExcerpt":"complete repair is 26/30 with text and 29/30 with the prepared checks","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}