{"resourceId":"agent-skill-hpc-code-translation-cuda-openmp-sycl-llm-verifier-117-false-positives-20261106","versions":[{"version":"external-945144ec1081dc20f1d56175ae72ceb810fad500579396678ca43feaff1c7b79","resource":{"id":"agent-skill-hpc-code-translation-cuda-openmp-sycl-llm-verifier-117-false-positives-20261106","title":"More with Less: Efficient Parallel Scientific Code Translation via an Agent-Skill Workflow","organization":"South China University of Technology and Michigan Technological University","sector":"Research computing and HPC software (scientific code porting)","geography":"China and United States (authors' institutions); benchmark kernels, not site-specific deployments","publishedAt":"6 November 2026 (arXiv 2611.08947v1, cs.DC; listed 9 November)","publicationDate":"2026-11-06","eventDate":null,"sourceName":"arXiv","sourceLabel":"Shen, J., Xu, P., and Zhang, B., \"More with Less: Efficient Parallel Scientific Code Translation via an Agent-Skill Workflow,\" arXiv:2611.08947, 6 Nov 2026","sourceUrl":"https://arxiv.org/abs/2611.08947","evidenceClass":"academic-research","outcomeClass":"mixed","topics":["developers-agents","infrastructure"],"finding":"The authors translate 30 HeCBench GPU kernels among CUDA, OpenMP and SYCL (six directions, three open-source LLMs) with three agents coordinated by a verdict-driven orchestrator that routes each failure to a targeted repair skill, and compare it with the LASSI baseline run on the same models. Their framework averages 88.7% verified correctness against 65.5% for LASSI using DeepSeek V4 Flash, higher in every one of 18 model and direction combinations, with 26.2% fewer tokens on average (1,071k vs 1,451k per direction) and up to 52.4% fewer overall. Results vary: SYCL as target is harder (74.6% vs 40.0% for LASSI), OpenMP to SYCL scores 79.2% vs 33.3%, and per-model averages span 78.6% to 94.6%. A stricter offline LLM verification lowers the framework's pass rates by 8.3 to 17.3 percentage points and identified 117 false positives that passed compiler and runtime checks, led by placeholder implementations (31), numerical errors (28) and skipped verification (25). Included now because it was submitted on 6 Nov, appeared in the 9 Nov arXiv listings and was not archived; it is research-computing evidence that automated pass signals overstate correctness when agents port scientific code.","sledRelevance":"Relevant to university research-computing and HPC centers facing CUDA-to-portable-model migrations: it shows what an agent translation loop can reach on kernels, how much a second verification stage changes the answer, and that scientific correctness needs checks beyond compilation success.","evidence":"arXiv preprint with a controlled comparison on 30 HeCBench kernels, three open-source models, six translation directions, a re-run LASSI baseline on identical models and environment, repair-round curves, a pass-rate comparison under two verification stages and a taxonomy of 117 false positives. Artifact-evaluation appendix is provided. Not peer reviewed.","architectureImplications":"Separate translation, compile diagnosis and execution validation, route failures by verdict to targeted repair skills with a repair cap (9 iterations here), and add an offline verifier that audits artifacts for stubs, skipped checks and numerical error. Plan for a Slurm GPU partition and the CUDA, OpenMP and SYCL toolchains used in the study.","governanceImplications":"Treat agent-ported code as unverified until domain owners check outputs; use the verifier's confidence scores only to prioritize human inspection, not to approve. Record which models and compilers produced each port and keep the original code as reference.","securityPrivacyImplications":"The study uses public benchmark kernels and OpenAI-compatible model endpoints. Campus codes may be unpublished or export-controlled, so model hosting, data handling and licensing of sources would need review that the preprint does not address.","caveats":"Preprint; kernel-level benchmarks only (30 kernels, six lack OpenMP versions so 24 per OpenMP-source direction), not multi-file applications; baseline is one prior system (LASSI) re-run with different models than its original study; the complementary verifier is itself an LLM and can err; correctness is numerical, with no performance results; low-resource SYCL targets remain weak; no cost or staffing data.","streamIds":["research"],"roles":{"sales":"Framing point: porting scientific codes between GPU programming models is costly, and this preprint gives bounded evidence on how far agents can help. On 30 kernels the authors' agent workflow averaged 88.7% verified correctness against 65.5% for the baseline while using about a quarter fewer tokens, yet a stricter verifier found 117 false passes. Use it with research-computing directors and HPC users weighing portability projects. Position a small pilot on a few kernels from your own codes, not an application-scale porting service, and avoid promising time or cost savings the study did not measure.","engineering":"Framing point: the useful design is a verdict-driven loop, not a longer prompt. Translation, compile diagnosis and execution validation run as separate agents, each failure goes to a targeted repair skill, and an offline verifier audits finished artifacts. Needed: a Slurm GPU partition, CUDA 12.8, a SYCL compiler, OpenMP offload, and an endpoint for open-source models. The 117 false positives (stubs, skipped checks, numerical errors) show output checks must compare against reference results, not 'PASS' strings. Kernel results may not transfer to whole applications.","delivery":"Framing point: this is a porting workflow that needs scientist sign-off, not an unattended conversion. Start with a handful of representative kernels, set the repair cap and test inputs, and have the code owner approve outputs against reference runs. Plan time for reviewing low-confidence verifier judgments and for SYCL targets, where results were weakest. Measure accepted ports, review hours and defects found after acceptance before extending to multi-file codes. The study provides no staffing or schedule data."},"roleDetails":{"sales":{"customerProblem":"Research groups must move GPU codes between CUDA, OpenMP and SYCL as hardware changes, and manual porting is slow and error-prone.","stakeholders":["Research computing director","HPC applications specialists","Principal investigators with legacy GPU codes","Research software engineers"],"discoveryQuestions":["Which codes must be ported in the next 12 months and between which models?","How is correctness of a ported code checked today?","Who owns sign-off on numerical equivalence?","Which target hardware and compilers are in use?"],"valueHypothesis":"An agent translation loop with an independent verification stage may reduce first-draft porting effort on kernel-sized code while exposing silent failures, if reviewed by code owners.","potentialEngagement":"A bounded pilot porting and verifying a small set of kernels from two or three local codes, with a measured comparison of accepted ports and review effort.","unsupportedClaims":["That agents can port full applications","Guaranteed time or cost savings","That the LLM verifier removes the need for human review","Performance parity of translated code"]},"engineering":{"fit":"Good for kernel-sized CUDA, OpenMP and SYCL translations with reference outputs; weak evidence for multi-file applications or other models such as HIP and Kokkos.","architectureIntegration":"Orchestrator with translation, compile-diagnosis and execution agents, a repair-skill router, a Slurm GPU partition for execution, and an offline verifier that scores artifacts with a confidence value.","prerequisites":["Reference outputs and tests per kernel","Slurm GPU partition and compilers","Open-source model endpoint","Repair cap and token budget"],"constraints":["Kernel-level benchmark only","SYCL targets markedly weaker","No performance evaluation","Verifier is an LLM and can be wrong"],"security":"Unpublished or export-controlled codes need controlled model hosting and licensing review; the study used public kernels.","proofOfValue":["Port 5 to 10 local kernels and compare verified correctness with and without the offline verifier","Count false passes found by reference-output comparison","Record tokens and repair rounds per accepted port"]},"delivery":{"work":"Select pilot kernels, build reference tests, configure the verdict-driven loop and verifier, run ports on the GPU partition, and have code owners review results.","dependencies":["Code owner time for review","GPU partition and compiler access","Reference test data"],"ownership":"The code's principal investigator owns acceptance; research computing owns the pipeline and environment.","skillsAdoption":"Research software engineers move from writing ports to specifying tests and reviewing agent output and verifier confidence.","governanceCheckpoints":["Reference tests approved before runs","Human review of low-confidence verifier judgments","Record of models and compiler versions per port"],"acceptanceCriteria":["Proposed: accepted ports match reference outputs within agreed tolerance","Proposed: zero stub or skipped-verification artifacts after review","Proposed: review hours per accepted port recorded"],"risks":["Silent false passes such as placeholder code","Weak SYCL translations","Overreliance on verifier confidence","Kernel results not carrying over to applications"]}},"retrievedAt":"2026-11-09T21:56:10-06:00","enrichedAt":"2026-11-09T21:56:10-06:00","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Shifts effort from manual porting toward writing tests and reviewing outputs; HPC specialists remain needed to judge numerical and performance behavior.","procurementImplications":"When buying or building agentic porting tools, ask for independent verification of outputs, per-direction results rather than one average, and model-agnostic operation, since open-source models are a documented option.","operatingModelImplications":"Supports a research-computing service that hosts the pipeline and verifier while code owners remain accountable for acceptance of each port.","peerReview":"preprint","sourceVerification":{"openedUrl":"https://arxiv.org/abs/2611.08947","referenceExcerpt":"attains the higher verified correctness rate in all 18 model and direction combinations over a state-of-the-art baseline","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}