---
name: kserve-vllm-llm-serving
description: "Production LLM inference on Kubernetes with vLLM + KServe — InferenceService, GPU scheduling, scale-to-zero, canary. Deploys LLM inference on Kubernetes with the vLLM serving runtime under KServe: InferenceService manifests with vLLM ServingRuntime, GPU scheduling via nvidia.com/gpu limits and node selectors, KEDA/Knative scale-to-zero, canary rollouts with canaryTrafficPercent, and OpenAI-compatible /v1/completions endpoints. Covers tensor-parallel and PagedAttention tuning, model storage from S3/HF Hub, and KV-cache-aware routing through the Gateway API Inference Extension (InferencePool/EndpointPicker)."
license: MIT
---

# KServe vLLM Serving

Deploy and operate large language models on Kubernetes using vLLM as the serving engine behind KServe's `InferenceService` abstraction. This skill scaffolds the manifests, GPU scheduling, autoscaling, and traffic-splitting configuration needed to run an OpenAI-compatible inference endpoint in a real cluster — for platform and ML engineers who already have a GPU node pool and want production-grade serving rather than a notebook demo.

## When to use it

- You have a Hugging Face model (or a fine-tuned checkpoint in S3/GCS/PVC) and need it reachable as an HTTP API inside a cluster.
- You want requests to speak the OpenAI wire format (`/v1/completions`, `/v1/chat/completions`) so existing clients and SDKs work unchanged.
- You need GPUs scheduled correctly with `nvidia.com/gpu` limits, node selectors, and tolerations.
- You want the deployment to scale to zero when idle and roll out new model versions with a controlled canary percentage.

## How it works

KServe ships a vLLM `ServingRuntime` (the `kserve-huggingfaceserver` / vLLM backend) that wraps vLLM's `OpenAI`-compatible server. You declare an `InferenceService` whose `predictor` references the model via `storageUri` (e.g. `hf://meta-llama/Llama-3.1-8B-Instruct`) and passes engine args like `--max-model-len`, `--dtype`, and `--tensor-parallel-size` through the container `args`.

Autoscaling runs through the Knative Serving (Serverless) install path, where `minReplicas: 0` enables scale-to-zero via the activator; the Raw Deployment path uses HPA/KEDA instead. Canary rollouts use `canaryTrafficPercent` on the `InferenceService` spec — KServe keeps the previous revision live and shifts the named percentage of traffic to the new one.

## Usage

```yaml
apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: llama3-8b
spec:
  predictor:
    minReplicas: 0
    canaryTrafficPercent: 20
    model:
      modelFormat: { name: huggingface }
      runtime: kserve-huggingfaceserver
      storageUri: hf://meta-llama/Llama-3.1-8B-Instruct
      args: ["--tensor-parallel-size", "1", "--max-model-len", "8192"]
      resources:
        limits: { nvidia.com/gpu: "1" }
```

```bash
kubectl apply -f llama3-8b.yaml
kubectl get isvc llama3-8b   # wait for READY=True, note the URL
curl $URL/v1/chat/completions -d '{"model":"llama3-8b","messages":[{"role":"user","content":"hi"}]}'
```

## Tips & gotchas

- Scale-to-zero requires the Knative/Serverless deployment mode. On Raw Deployment, `minReplicas: 0` is not honored — use KEDA.
- `--tensor-parallel-size` must equal the GPU count in `resources.limits`, or vLLM fails on startup.
- Cold starts are slow: weights download plus CUDA graph capture can take minutes. Pre-warm with a PVC-cached model or set a non-zero `minReplicas` for latency-sensitive paths.
- Gated HF models need a token mounted as a secret referenced by the storage initializer.

## FAQ

**Does the model name in requests have to match the InferenceService?** The `model` field in the request must match what vLLM was launched with; KServe routing is by URL/host, not the body's model field.

**Can I serve quantized models?** Yes — pass `--quantization awq` (or the relevant method) in `args`, assuming the checkpoint is in that format.

**How do I roll back a bad canary?** Set `canaryTrafficPercent: 0` or remove it; KServe routes all traffic back to the last good revision.