KServe vLLM Serving

Deploy and operate large language models on Kubernetes using vLLM as the serving engine behind KServe's InferenceService abstraction. This skill scaffolds the manifests, GPU scheduling, autoscaling, and traffic-splitting configuration needed to run an OpenAI-compatible inference endpoint in a real cluster — for platform and ML engineers who already have a GPU node pool and want production-grade serving rather than a notebook demo.

When to use it

• You have a Hugging Face model (or a fine-tuned checkpoint in S3/GCS/PVC) and need it reachable as an HTTP API inside a cluster.
• You want requests to speak the OpenAI wire format (/v1/completions, /v1/chat/completions) so existing clients and SDKs work unchanged.
• You need GPUs scheduled correctly with nvidia.com/gpu limits, node selectors, and tolerations.
• You want the deployment to scale to zero when idle and roll out new model versions with a controlled canary percentage.

How it works

KServe ships a vLLM ServingRuntime (the kserve-huggingfaceserver / vLLM backend) that wraps vLLM's OpenAI-compatible server. You declare an InferenceService whose predictor references the model via storageUri (e.g. hf://meta-llama/Llama-3.1-8B-Instruct) and passes engine args like --max-model-len, --dtype, and --tensor-parallel-size through the container args.

Autoscaling runs through the Knative Serving (Serverless) install path, where minReplicas: 0 enables scale-to-zero via the activator; the Raw Deployment path uses HPA/KEDA instead. Canary rollouts use canaryTrafficPercent on the InferenceService spec — KServe keeps the previous revision live and shifts the named percentage of traffic to the new one.

Usage

curl -fsSL --create-dirs -o ~/.claude/skills/kserve-vllm-llm-serving/SKILL.md https://claudeskil.com/skills/kserve-vllm-llm-serving/SKILL.md

apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: llama3-8b
spec:
  predictor:
    minReplicas: 0
    canaryTrafficPercent: 20
    model:
      modelFormat: { name: huggingface }
      runtime: kserve-huggingfaceserver
      storageUri: hf://meta-llama/Llama-3.1-8B-Instruct
      args: ["--tensor-parallel-size", "1", "--max-model-len", "8192"]
      resources:
        limits: { nvidia.com/gpu: "1" }

kubectl apply -f llama3-8b.yaml
kubectl get isvc llama3-8b   # wait for READY=True, note the URL
curl $URL/v1/chat/completions -d '{"model":"llama3-8b","messages":[{"role":"user","content":"hi"}]}'

Tips & gotchas

• Scale-to-zero requires the Knative/Serverless deployment mode. On Raw Deployment, minReplicas: 0 is not honored — use KEDA.
• --tensor-parallel-size must equal the GPU count in resources.limits, or vLLM fails on startup.
• Cold starts are slow: weights download plus CUDA graph capture can take minutes. Pre-warm with a PVC-cached model or set a non-zero minReplicas for latency-sensitive paths.
• Gated HF models need a token mounted as a secret referenced by the storage initializer.

FAQ

Does the model name in requests have to match the InferenceService? The model field in the request must match what vLLM was launched with; KServe routing is by URL/host, not the body's model field.

Can I serve quantized models? Yes — pass --quantization awq (or the relevant method) in args, assuming the checkpoint is in that format.

How do I roll back a bad canary? Set canaryTrafficPercent: 0 or remove it; KServe routes all traffic back to the last good revision.