{"resourceId":"nvidia-aicr-issue-2217-validation-fixture-false-pass-v1-boundary-20260915","versions":[{"version":"external-f0dc9e170748189aed0383944336a91c975e6ec8f50eff00e442686709dae1f8","resource":{"id":"nvidia-aicr-issue-2217-validation-fixture-false-pass-v1-boundary-20260915","title":"AICR issue #2217: performance validation could pass by measuring a temporary fixture, not the shipped runtime; v1 fix limited to the GKE TCPXO path","organization":"NVIDIA (AICR open-source project)","sector":"Technology vendor (open-source infrastructure project)","geography":"Global","publishedAt":"2026-08-17T17:53:52Z (opened); closed 2026-09-15T17:30:43Z","publicationDate":"2026-08-17","eventDate":"2026-09-15","sourceName":"GitHub — NVIDIA/aicr issues","sourceLabel":"Project issue tracker (self-disclosed limitation)","sourceUrl":"https://github.com/NVIDIA/aicr/issues/2217","evidenceClass":"vendor-claim","outcomeClass":"cautionary","topics":["infrastructure","governance-procurement"],"finding":"An AICR epic opened Aug 17, 2026 states that the performance validator could build and measure a temporary fabric-wired runtime while the runtime the recipe delivered remained unwired, permitting a false PASS: published bandwidth evidence could describe a validator fixture rather than what users receive. The issue was closed Sep 15 with acceptance criteria met for the supported GKE TCPXO recipe: the recipe now ships the wired runtime, the required network name fails closed if absent, and evidence records the runtime source and provenance. Its stated v1 boundary does not require broad EFA, RDMA, NVLS or multi-cloud runtime expansion, every machine shape, or full component upgrade-lifecycle qualification.","sledRelevance":"Why now: AICR v1.0 launched Oct 5-6 promoting signed validation evidence. This project record shows the evidence model had a real false-pass gap and that the fix is scoped to one supported path, which bounds how much weight campus or state teams should give evidence for other fabrics.","evidence":"Opened issue #2217 via the GitHub API and read the full body (problem, v1 scope, acceptance criteria, v1 boundary). Self-disclosed by maintainers; not an external audit.","architectureImplications":"For RDMA, EFA or NVLink-switch fabrics outside the GKE TCPXO path, operators should confirm that performance evidence was produced against the delivered runtime, not a test-built one.","governanceImplications":"Evidence review should check runtimeSource and runtimeProvenance fields in signed bundles before accepting a performance PASS.","securityPrivacyImplications":"No direct security finding; the issue concerns integrity of validation evidence, which supports supply-chain and acceptance assurance.","caveats":"Older than tonight's release (Aug-Sep 2026); included because it is newly relevant to v1.0's evidence claims. Whether similar gaps exist on other fabrics is not established either way.","streamIds":["nvidia"],"roles":{"sales":"This issue is a credibility check worth raising with research computing buyers: AICR's own maintainers found that a performance PASS could describe a test fixture rather than the delivered setup, and fixed it for one supported path. Being open about that builds trust. We can offer to review any AICR evidence a customer relies on for acceptance. We should not tell customers that all AICR performance evidence is now trustworthy; the fix is limited to GKE TCPXO by the project's own description.","engineering":"The engineering takeaway is to verify what a performance result actually measured. For GKE TCPXO, v1.0 derives benchmarks from the shipped runtime and records runtimeSource and provenance in the signed bundle. For EFA, RDMA or NVLS fabrics, the project did not commit to the same guarantee in v1. Prerequisite: access to the evidence bundle. Proof of value: confirm runtime provenance fields match the deployed runtime, and rerun a bandwidth test using your own workload runtime.","delivery":"Delivery teams should add an evidence-integrity step to cluster acceptance: read the runtime provenance in any AICR performance evidence and rerun one benchmark with the production workload runtime. The platform lead owns this; procurement should require it when integrators deliver clusters. Checkpoints are acceptance testing and each recipe upgrade. Proposed acceptance criterion: performance evidence names the delivered runtime. Risk: a pass recorded on another fabric is accepted without confirming what it measured."},"roleDetails":{"sales":{"customerProblem":"Buyers may accept signed performance evidence that does not describe the configuration they received.","stakeholders":["Research computing director","Procurement officer","Platform engineering lead"],"discoveryQuestions":["Which fabric does your cluster use (TCPXO, EFA, InfiniBand/RDMA, NVLS)?","Does your acceptance test rerun benchmarks on the delivered runtime?"],"valueHypothesis":"An evidence review step can prevent accepting clusters on misleading benchmark passes (interpretation).","potentialEngagement":"A one-off review of AICR or integrator evidence used for cluster acceptance.","unsupportedClaims":["All AICR performance evidence is now reliable","Similar gaps exist on other fabrics (not shown)","Any measured incidence of false passes"]},"engineering":{"fit":"Applies to any team using AICR performance evidence for acceptance.","architectureIntegration":"Evidence bundles include runtimeSource and runtimeProvenance for the GKE TCPXO path after the fix.","prerequisites":["Access to signed evidence bundles","Ability to run NCCL or similar benchmarks"],"constraints":["Fix scoped to GKE TCPXO","No v1 commitment for EFA, RDMA, NVLS or multi-cloud runtimes"],"security":"Evidence integrity, not confidentiality; verify signatures and provenance offline.","proofOfValue":["Check runtime provenance matches the deployed runtime","Rerun one bandwidth benchmark with the production runtime"]},"delivery":{"work":"Add an evidence-integrity step to GPU cluster acceptance and upgrades.","dependencies":["Evidence bundles from AICR or integrator","Benchmark tooling","Procurement acceptance terms"],"ownership":"Platform lead owns acceptance; procurement enforces evidence requirements in contracts.","skillsAdoption":"Staff learn to read AICR evidence fields and rerun fabric benchmarks.","governanceCheckpoints":["Cluster acceptance","Recipe upgrade"],"acceptanceCriteria":["Performance evidence names the delivered runtime (proposed)","One benchmark rerun on the production runtime before acceptance (proposed)"],"risks":["Accepting fixture-based passes on non-TCPXO fabrics","Evidence produced by pre-fix AICR versions"]}},"retrievedAt":"2026-10-07T03:02:13Z","enrichedAt":"2026-10-07T03:09:23Z","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Requires staff who can interpret benchmark provenance; smaller teams may need integrator help.","procurementImplications":"Contracts for delivered GPU clusters should require that performance evidence be produced on the delivered runtime and fabric.","operatingModelImplications":"Acceptance evidence becomes something to audit rather than simply file.","sourceVerification":{"openedUrl":"https://github.com/NVIDIA/aicr/issues/2217","referenceExcerpt":"That permits a false PASS: published bandwidth evidence can describe a validator fixture rather than the runtime users receive.","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}