Lighthouse AdvisorySLED AI Adoption Intelligence
← Back to results

From the NVIDIA edition of September 15, 2026

Standards or public-body guidanceCautionaryUndated source

NIM Operator guidance requires inference metrics for HPA and long model-load startup probes

NVIDIA · AI platform vendor documentation · Global

NVIDIA
Publisher
NVIDIA NIM Operator Deployment documentation
Original publication
NIM LLM 2.0.12 operator deployment docs (retrieved 2026-09-15)
Source retrieved
2026-09-15
Read original source

What happened

Official NIM Operator deployment guidance for 2.0.12 centers on NIMCache + NIMService, requires GPU Operator, PVC caching, and NGC secrets, and states CPU/memory metrics are not useful for HPA—use backend-native inference metrics (e.g., vllm:num_requests_waiting) via Prometheus Adapter. Startup probes allow up to ~20 minutes by default for model load; multi-node spec.args are not propagated on the LeaderWorkerSet path.

Why it matters

Campus and agency Kubernetes platforms cannot reuse generic HPA patterns for NIM; observability and probe budgets become acceptance criteria for production AI services.

Evidence and measured results

Opened docs detail prerequisites (K8s 1.26+, GPU Operator, storage, NGC key, Operator 3.0.2+), autoscaling example, health endpoint semantics (live vs ready), and troubleshooting for cache jobs and probe failures.

Limitations and uncertainty

Documentation guidance, not a measured multi-tenant SLED deployment. Metric names differ between vLLM and SGLang backends.

Cross-source patterns this evidence supports

Put this evidence to work

Lighthouse Advisory interpretation, grounded in this source. Enriched 2026-09-15; this does not change the original publication date. Each role follows the same checklist so takeaways compare across sources.

Sales

Role takeaway
Customer problem
Agencies expect autoscaling chat/agents like web apps using CPU HPA.
Stakeholders
  • platform SRE
  • CISO
  • app owner
Discovery questions
  • Is Prometheus Adapter available?
  • What storage class supports model caches?
  • How long may models take to become ready?
Value hypothesis
Inference-aware platform packaging prevents failed rollouts and noisy-neighbor outages.
Potential engagement
Platform readiness assessment against Operator prerequisites.
Unsupported claims
  • CPU HPA is sufficient for NIM
  • Models are ready in seconds regardless of size

Pre-sales engineering

Role takeaway
Fit
Fit for Kubernetes NIM when GPU Operator, PVCs, and custom metrics exist.
Architecture and integration
Deploy Operator 3.0.2+ with NIMCache then NIMService; wire ServiceMonitor and HPA to waiting-request metrics.
Prerequisites
  • K8s 1.26+
  • GPU Operator
  • storage provisioner
  • NGC secrets
  • Prometheus Adapter
Constraints
  • CPU/mem HPA ineffective
  • Multi-node args not passed via LeaderWorkerSet
  • Long model loads need probe budget
Security
Protect NGC credentials; isolate nim-service namespaces.
Proof of value
  • Cache Ready before service
  • HPA scales on inference metric
  • Startup probe survives model load

Delivery

Role takeaway
Work
Cluster prep, Operator install, sample-to-prod promotion, and observability validation.
Dependencies
  • platform team
  • secrets management
  • app team
Ownership
Platform owns Operator; apps own NIMService specs.
Skills and adoption
SREs need NIM metrics literacy.
Governance checkpoints
  • Prereq checklist
  • Cache job success
  • HPA load test
Acceptance criteria
  • Service ready only after cache Ready
  • Autoscaling observed under load
  • Secrets rotated
Risks
  • Missing custom metrics
  • Undersized PVC
  • Probe kills during load

Implementation considerations

Lighthouse Advisory interpretation across the operating dimensions a public-sector buyer must settle before this evidence becomes a design. Each note answers the question under its heading for this specific source.

Architecture and integration

What must connect, and where does the AI sit in the workflow?

Install Prometheus Adapter and scrape /v1/metrics; size PVCs for model caches; tune startupProbe failureThreshold for large models; avoid multi-node args assumptions.

Governance

Who approves, reviews and stays accountable for outcomes?

Platform standards should mandate inference-metric HPA and explicit model-cache readiness before exposing student/citizen-facing endpoints.

Security and privacy

What data, permissions and controls need testing?

NGC API keys and pull secrets are cluster secrets; namespace isolation and secret rotation are mandatory.

Accessibility and workforce

Who is affected, and what skills or accommodations follow?

Stable scaling protects shared campus assistants used by large student/staff populations.

Procurement

What should contracts, pricing and exit terms secure?

Include platform observability and storage prerequisites in NIM/AI Enterprise SOWs.

Operating model

Which teams own the service once it runs?

Define who owns cache PVCs, HPA policies, and NGC secret rotation.

Publication history

  1. 2026-09-15NVIDIA · Issue 1010 resources
Read preserved resource versions (JSON)

Stable resource ID: nvidia-nim-operator-hpa-inference-metrics-2012