---
name: sglang-inference-server
description: "Launch and tune SGLang serving — RadixAttention prefix caching, structured generation, tensor parallelism, vLLM benchmarks. Production SGLang serving: launch_server with --tp-size/--dp-size, RadixAttention prefix caching and --chunked-prefill-size tuning, structured generation via xgrammar/outlines (regex, JSON schema, EBNF), the OpenAI-compatible /v1 endpoint and native /generate API, continuous batching with --mem-fraction-static, CUDA graph and torch.compile flags, speculative decoding (EAGLE), and bench_serving against vLLM on H100 prefix-heavy RAG and multi-turn workloads."
license: MIT
---

# SGLang Inference Server

This skill helps you stand up and tune an [SGLang](https://github.com/sgl-project/sglang) inference server for production LLM serving. It covers launching the server with the right parallelism and memory flags, exploiting RadixAttention prefix caching, enforcing structured output, and using both the OpenAI-compatible and native endpoints. It is aimed at engineers deploying open-weight models (Llama, Qwen, DeepSeek, Mixtral, etc.) on one or more GPUs who care about throughput and tail latency.

## When to use it
- You are serving a model behind an API and want continuous batching plus prefix-cache reuse for shared system prompts or few-shot examples.
- You need guaranteed-valid JSON, regex-constrained, or grammar-constrained output.
- A model does not fit on one GPU and you need tensor or data parallelism.
- You are migrating from vLLM and want to compare throughput/TTFT on the same hardware.

## How it works
The entry point is `python -m sglang.launch_server`. Key flags this skill helps you set:
- `--tp-size` (tensor parallelism across GPUs) and `--dp-size` (data-parallel replicas).
- `--mem-fraction-static` to size the KV cache pool; lower it if you hit CUDA OOM.
- `--chunked-prefill-size` to cap tokens per prefill step and protect decode latency.
- RadixAttention is on by default; disable with `--disable-radix-cache` when benchmarking cold paths.

Structured generation is handled by the `--grammar-backend` (`xgrammar` or `outlines`). Requests pass `regex`, `json_schema`, or `ebnf` constraints. The server exposes an OpenAI-compatible surface at `/v1/chat/completions` and `/v1/completions`, plus the native `/generate` endpoint.

## Example
```bash
curl -fsSL --create-dirs -o ~/.claude/skills/sglang-inference-server/SKILL.md https://claudeskil.com/skills/sglang-inference-server/SKILL.md

# Launch a 7B model across 2 GPUs
python -m sglang.launch_server \
  --model-path meta-llama/Llama-3.1-8B-Instruct \
  --tp-size 2 --mem-fraction-static 0.85 \
  --chunked-prefill-size 4096 --port 30000

# JSON-constrained request via the native API
curl http://localhost:30000/generate -d '{
  "text": "List two primes as JSON: ",
  "sampling_params": {"max_new_tokens": 64,
    "json_schema": "{\"type\":\"array\",\"items\":{\"type\":\"integer\"}}"}
}'
```

## Tips & gotchas
- RadixAttention only helps when requests share *prefixes*. Put static system prompts and few-shot blocks first so they hit the cache; volatile content goes last.
- OOM at startup is almost always `--mem-fraction-static` set too high alongside a large `--context-length`. Drop the fraction before lowering batch size.
- `xgrammar` is faster than `outlines` for JSON schema; switch to `outlines` if you hit an unsupported grammar feature.
- Use `--dp-size` for replicas of a model that fits on one GPU, and `--tp-size` only when a single copy does not fit.

## FAQ
**Is it a drop-in vLLM replacement?** The `/v1` endpoints match the OpenAI spec, so most clients work unchanged. Native features (`/generate`, RadixAttention, structured outputs) need SGLang-specific request fields.

**How do I benchmark it?** Use `python -m sglang.bench_serving` against a running server, or `sglang.bench_one_batch` for offline throughput; compare TTFT and output token throughput at matched concurrency.

**Can I constrain output to a grammar?** Yes — pass `ebnf`, `regex`, or `json_schema` in `sampling_params`, backed by `--grammar-backend`.