{"resourceId":"nvidia-swe-serve-20260923","versions":[{"version":"external-775c940a26ece947e676b4f1f6ffeb058427c499c3bcc07db14cc37b7a404e79","resource":{"id":"nvidia-swe-serve-20260923","title":"SWE-Serve: live-serving checks cut agent pass rates from 69.4% to 45.9% on SGLang tasks","organization":"NVIDIA","sector":"AI developer tooling / inference engineering evaluation","geography":"Global (vendor benchmark; transferable to SGLang-like serving stacks)","publishedAt":"2026-09-23T16:00:00Z","publicationDate":"2026-09-23","eventDate":"2026-09-23","sourceName":"NVIDIA Developer Blog","sourceLabel":"Vendor technical blog","sourceUrl":"https://developer.nvidia.com/blog/how-swe-serve-exposes-the-gap-between-local-tests-and-live-serving/","evidenceClass":"vendor-claim","outcomeClass":"cautionary","topics":["developers-agents","infrastructure","operating-model"],"finding":"On Sep 23, 2026 NVIDIA published SWE-Serve, a benchmark of 53 inference-engineering tasks derived from 83 merged SGLang pull requests across six families. Nineteen tasks start a live server. The blog reports patches pass 45.9% of the time with complete verification but 69.4% when live-serving checks are excluded—about one in three patches that pass other tests fail live-serving validation. This is a measured agent-evaluation gap, not a campus product launch.","sledRelevance":"Platform teams using coding agents on inference stacks should treat local unit/green CI as insufficient acceptance for production serving changes.","evidence":"Vendor blog dated Sep 23, 2026; states 53 tasks / 83 PRs; quotes 45.9% vs 69.4% pass rates with/without live-serving checks.","architectureImplications":"CI for inference platforms needs live-server end-to-end stages before merging agent-proposed serving patches.","governanceImplications":"Who may promote agent-generated serving changes without live-serving gates, and what rollback authority exists?","securityPrivacyImplications":"Failed live-serving patches can create availability and config regressions on shared inference endpoints.","caveats":"Vendor-authored SGLang-derived benchmark; results may not transfer to other engines; leaderboards vary by configuration.","streamIds":["nvidia"],"roles":{"sales":"Frame SWE-Serve as a caution for agent-assisted platform engineering—not a reason to buy a new chatbot. Open with green CI that still breaks production serving, then cite NVIDIA’s Sep 23 45.9% versus 69.4% live-serving gap on 53 SGLang-derived tasks. Bound the engagement to adding live-serving acceptance gates for agent-proposed inference changes, without promising any specific coding model will hit a target pass rate on campus stacks.","engineering":"Fit is strong wherever coding agents already open PRs against serving stacks; weak if the org has no inference CI. Integrate live-server smoke tests into merge gates for scheduling, caching, and API changes. Prerequisites: reproducible serving harness, test runners, artifact pinning. Constraints: vendor benchmark is SGLang-centric; nineteen tasks need live servers. Proof: measure agent patch pass rate with and without live-serving checks on your fork.","delivery":"Work: add live-serving CI jobs, define flaky-test budgets, train on-call to reject agent PRs lacking E2E evidence. Dependencies: CI runners, model/runtime fixtures, rollback playbooks. Ownership: inference platform engineering. Skills: serving observability and agent-PR review. Governance: mandatory live-serving checks for production paths. Acceptance: documented gap metric on a fixed task sample. Risks: flaky E2E blocks velocity; over-trusting vendor leaderboards. Keep agent-PR exceptions time-boxed and logged."},"roleDetails":{"sales":{"customerProblem":"Agent-written inference patches pass local tests yet fail when a real server starts.","stakeholders":["inference platform leads","SRE","developer experience","security"],"discoveryQuestions":["Do merge gates start a live server?","Which agent tools open serving PRs today?","What is rollback time for bad serving configs?"],"valueHypothesis":"Live-serving verification can catch roughly a third of otherwise-green agent patches if SWE-Serve-like gaps hold.","potentialEngagement":"Assessment: instrument live-serving gates on one inference repo and report before/after agent pass rates.","unsupportedClaims":["Any coding agent will reach 75% pass@1","SWE-Serve proves production readiness of a campus stack","Unit tests alone are sufficient"]},"engineering":{"fit":"Strong for SGLang-like serving platforms using coding agents; adapt carefully for other engines.","architectureIntegration":"Add live-server E2E stage in CI before merge; keep offline unit tests as a fast filter only.","prerequisites":["CI with containerized server","Pinned runtime images","Task fixtures"],"constraints":["SGLang-derived benchmark scope","Live tasks need more compute","Flaky E2E risk"],"security":"Isolate CI serving endpoints; avoid production credentials in agent sandboxes.","proofOfValue":["Pass-rate delta with/without live checks","Mean time to detect bad serving PR","Rollback success on injected failure"]},"delivery":{"work":"Implement live-serving CI, document agent PR policy, train reviewers.","dependencies":["CI capacity","Serving fixtures","On-call runbooks"],"ownership":"Inference platform engineering manager.","skillsAdoption":"Reviewer checklist for agent-generated serving diffs.","governanceCheckpoints":["Mandatory live-serving gate for prod paths","Exception log for emergency hotfixes"],"acceptanceCriteria":["Live-serving job required on protected branch","Measured pass-rate report for one sprint"],"risks":["CI flakiness","False confidence from vendor leaderboard"]}},"retrievedAt":"2026-09-27T02:58:40Z","enrichedAt":"2026-09-27T03:05:00Z","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Stronger E2E gates raise the skill bar for agent-assisted contributors; pair with clearer failure diagnostics.","procurementImplications":"Budget CI GPU/CPU runners and evaluation harnesses before expanding coding-agent licenses for platform work.","operatingModelImplications":"Treat agent-generated serving changes as high-risk production engineering, not autocomplete.","sourceVerification":{"openedUrl":"https://developer.nvidia.com/blog/how-swe-serve-exposes-the-gap-between-local-tests-and-live-serving/","referenceExcerpt":"patches pass 45.9% of the time with complete verification but 69.4% when live-serving checks are excluded","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}