SGLang Inference Server

This skill helps you stand up and tune an SGLang inference server for production LLM serving. It covers launching the server with the right parallelism and memory flags, exploiting RadixAttention prefix caching, enforcing structured output, and using both the OpenAI-compatible and native endpoints. It is aimed at engineers deploying open-weight models (Llama, Qwen, DeepSeek, Mixtral, etc.) on one or more GPUs who care about throughput and tail latency.

When to use it

• You are serving a model behind an API and want continuous batching plus prefix-cache reuse for shared system prompts or few-shot examples.
• You need guaranteed-valid JSON, regex-constrained, or grammar-constrained output.
• A model does not fit on one GPU and you need tensor or data parallelism.
• You are migrating from vLLM and want to compare throughput/TTFT on the same hardware.

How it works

The entry point is python -m sglang.launch_server. Key flags this skill helps you set:

• --tp-size (tensor parallelism across GPUs) and --dp-size (data-parallel replicas).
• --mem-fraction-static to size the KV cache pool; lower it if you hit CUDA OOM.
• --chunked-prefill-size to cap tokens per prefill step and protect decode latency.
• RadixAttention is on by default; disable with --disable-radix-cache when benchmarking cold paths.

Structured generation is handled by the --grammar-backend (xgrammar or outlines). Requests pass regex, json_schema, or ebnf constraints. The server exposes an OpenAI-compatible surface at /v1/chat/completions and /v1/completions, plus the native /generate endpoint.

Example

curl -fsSL --create-dirs -o ~/.claude/skills/sglang-inference-server/SKILL.md https://claudeskil.com/skills/sglang-inference-server/SKILL.md

# Launch a 7B model across 2 GPUs
python -m sglang.launch_server \
  --model-path meta-llama/Llama-3.1-8B-Instruct \
  --tp-size 2 --mem-fraction-static 0.85 \
  --chunked-prefill-size 4096 --port 30000

# JSON-constrained request via the native API
curl http://localhost:30000/generate -d '{
  "text": "List two primes as JSON: ",
  "sampling_params": {"max_new_tokens": 64,
    "json_schema": "{\"type\":\"array\",\"items\":{\"type\":\"integer\"}}"}
}'

Tips & gotchas

• RadixAttention only helps when requests share prefixes. Put static system prompts and few-shot blocks first so they hit the cache; volatile content goes last.
• OOM at startup is almost always --mem-fraction-static set too high alongside a large --context-length. Drop the fraction before lowering batch size.
• xgrammar is faster than outlines for JSON schema; switch to outlines if you hit an unsupported grammar feature.
• Use --dp-size for replicas of a model that fits on one GPU, and --tp-size only when a single copy does not fit.

FAQ

Is it a drop-in vLLM replacement? The /v1 endpoints match the OpenAI spec, so most clients work unchanged. Native features (/generate, RadixAttention, structured outputs) need SGLang-specific request fields.

How do I benchmark it? Use python -m sglang.bench_serving against a running server, or sglang.bench_one_batch for offline throughput; compare TTFT and output token throughput at matched concurrency.

Can I constrain output to a grammar? Yes — pass ebnf, regex, or json_schema in sampling_params, backed by --grammar-backend.