{"resourceId":"models-at-work-nemotron-transcribe-vendor-run-benchmarks-offline-capacity-20261028","versions":[{"version":"external-3b71819e9fb9b5041a701366fc1201336b5bfef9e610f75dfb3c6a61aa6d6215","resource":{"id":"models-at-work-nemotron-transcribe-vendor-run-benchmarks-offline-capacity-20261028","title":"Models at Work: most Nemotron Transcribe figures are NVIDIA's own runs, the open release is slower than headline speeds, and only Artificial Analysis is third-party","organization":"Models at Work","sector":"AI trade newsletter (independent reporting, AI-written)","geography":"United States (Nashville-based publication); global subject","publishedAt":"2026-10-28","publicationDate":"2026-10-28","eventDate":null,"sourceName":"Models at Work","sourceLabel":"Nvidia releases open Nemotron 3.5 Transcribe for 40 languages","sourceUrl":"https://modelsatwork.news/article/nvidia-nemotron-3-5-transcribe-open-weight-40-languages-speaker-labels-offline-only-model-card","evidenceClass":"independent-reporting","outcomeClass":"mixed","topics":["knowledge-work","infrastructure","governance-procurement"],"finding":"The newsletter, drawing on the model card, The Decoder and TechCrunch, says most accuracy figures are NVIDIA's own runs on public benchmarks and that Artificial Analysis (2.81% error, close to Mistral Voxtral Small at 2.77%) is the only third-party figure, showing the model near the top rather than ahead. It flags the card's statement that published speeds come from an optimized NIM container and the open version is expected to be roughly two times slower, notes a parameter-count discrepancy between press and the card (34B total, 4.3B active per the card), and says none of the sources reports results on a buyer's own audio.","sledRelevance":"A useful buyer's lens for SLED procurement teams: separate vendor-run benchmarks from third-party results, plan capacity on the open release and insist on local testing.","evidence":"Secondary analysis of vendor documentation and press; the publication discloses that an AI editor writes it and that it did not test the model.","architectureImplications":"Reinforces planning on open-release throughput and on a single H100 80 GB quick-start configuration.","governanceImplications":"Supports procurement language distinguishing vendor-reported from independently measured accuracy.","securityPrivacyImplications":"Not addressed beyond noting self-hosting.","caveats":"AI-written newsletter relying on secondary sources; no original testing; figures should be checked against the model card.","streamIds":["nvidia"],"roles":{"sales":"This analysis helps sales avoid overclaiming. Only one cited figure comes from a party other than NVIDIA, and it shows the model near the top rather than clearly ahead. In SLED conversations, present Nemotron Transcribe as a strong open option worth testing, not a proven winner. Offer to run the customer's own recordings through it and the incumbent side by side. That framing respects procurement rules on fair comparison and avoids disputes later if local results disappoint.","engineering":"Engineering should note two planning points from this piece. First, capacity should be sized on the open release, which the card expects to be about two times slower than NIM figures. Second, the quick-start uses one H100 80 GB GPU, so smaller agency hardware needs testing. Treat leaderboard results as screening only. Build a side-by-side harness that scores the incumbent and the open model on identical local audio with the same scoring rules.","delivery":"Delivery should use a side-by-side evaluation as the gate for any vendor change, with results documented for procurement files. Record which figures came from the vendor, third parties and local tests. Keep the incumbent service running until local results meet agreed thresholds. Plan hardware based on measured throughput from the pilot, not published numbers, and revisit the decision if NVIDIA upstreams its NIM optimizations."},"roleDetails":{"sales":{"customerProblem":"Buyers can be swayed by vendor leaderboard claims that may not reflect their audio.","stakeholders":["Procurement officer","IT director","Clerk or records lead"],"discoveryQuestions":["Which evaluations will your procurement accept as evidence?","Can you run a side-by-side test with your incumbent?"],"valueHypothesis":"Side-by-side local testing gives a defensible basis for any switch.","potentialEngagement":"Comparative pilot of incumbent versus open model on local recordings.","unsupportedClaims":["That the model is clearly best overall","NIM speeds on the open release"]},"engineering":{"fit":"Agencies comparing transcription options.","architectureIntegration":"Side-by-side scoring harness on identical local audio.","prerequisites":["Local test audio with references","H100-class GPU or equivalent for quick-start"],"constraints":["Most figures vendor-run","Open release slower than headline"],"security":"Keep test audio under existing access controls.","proofOfValue":["Side-by-side WER and speaker error","Measured throughput on available hardware"]},"delivery":{"work":"Run a documented comparison and gate any switch on local results.","dependencies":["Incumbent access for comparison","Test audio"],"ownership":"IT with procurement oversight.","skillsAdoption":"Teach staff to read benchmark sources critically.","governanceCheckpoints":["Procurement review of evaluation evidence"],"acceptanceCriteria":["Proposed: open model meets or beats incumbent on local audio"],"risks":["Decision based on vendor claims","Hardware undersized"]}},"retrievedAt":"2026-10-29T02:59:40Z","enrichedAt":"2026-10-29T03:06:28Z","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Not addressed directly.","procurementImplications":"Buyers should weight third-party evaluations and local tests above vendor leaderboard claims.","operatingModelImplications":"Encourages a test-before-replace approach to switching transcription vendors.","sourceVerification":{"openedUrl":"https://modelsatwork.news/article/nvidia-nemotron-3-5-transcribe-open-weight-40-languages-speaker-labels-offline-only-model-card","referenceExcerpt":"Most of these are Nvidia's own runs on public benchmarks.","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}