SGLang: Fast LLM Serving and a Structured Generation Programming Model

SGLang is two things in one package: a high-throughput serving runtime and a frontend language for writing structured, multi-step LLM programs. The runtime is interesting because of RadixAttention, which automatically reuses the key-value cache across requests that share a prefix, and the frontend lets you express forks, constrained outputs, and control flow without manually stitching prompts together. This tutorial walks through both pillars with a coherent extraction-and-reasoning example you can adapt to real workloads.

What SGLang Is

It helps to separate the two layers from the start, because most confusion about SGLang comes from conflating them.

The runtime is a server you launch once and query over HTTP. It handles batching, scheduling, and KV-cache management on the GPU. You can use it through an OpenAI-compatible API and never touch the frontend language at all.

The frontend language is a Python DSL (embedded as decorators and helper functions) for describing generation programs. It compiles your program into a sequence of calls against a backend, which can be the local SGLang runtime, or an external provider such as OpenAI. The frontend is where features like parallel forks and constrained decoding become ergonomic.

The two pillars

How this differs from vLLM

If you have used vLLM, the runtime will feel familiar: both do continuous batching, tensor parallelism, and expose an OpenAI-compatible API. The practical differences are that SGLang's RadixAttention makes shared-prefix workloads (few-shot prompting, agents that reuse a long system prompt, tree-of-thought style branching) noticeably cheaper, and that SGLang ships a frontend language for orchestrating multi-step programs. vLLM focuses on being a serving engine; SGLang is a serving engine plus a programming model. We will not re-explain PagedAttention here.

Installation and Hardware Notes

The simplest install pulls the runtime, the frontend, and the common dependencies.

# Recommended: install everything (runtime + frontend + extras)
pip install "sglang[all]"

Frontend only, if you just want the DSL against a remote backend
pip install "sglang[openai]"

Verify
python -c "import sglang as sgl; print(sgl.version)"

Hardware notes worth keeping in mind:

• The runtime targets NVIDIA GPUs with recent CUDA. Ampere (A100) and newer (L4, L40S, H100) are the smooth path; some kernels assume sm80 or higher.
• Memory governs how much KV cache you can hold and therefore how much RadixAttention can reuse. More free VRAM means a larger radix tree and higher cache-hit rates.
• AMD ROCm support exists but is less battle-tested; check the current support matrix before committing.
• For multi-GPU, tensor parallelism (--tp) shards the model weights across GPUs on one node.

# A quick environment sanity check
python -c "import torch; print('cuda', torch.cuda.isavailable(), 'gpus', torch.cuda.devicecount())"

nvidia-smi --query-gpu=name,memory.total,memory.used --format=csv

python -m sglang.launchserver \
  --model-path meta-llama/Llama-3.1-8B-Instruct \
  --port 30000 \
  --host 0.0.0.0

Common flags you will reach for:

# Multi-GPU with tensor parallelism (4 GPUs on one node)
python -m sglang.launchserver \

  --model-path meta-llama/Llama-3.1-70B-Instruct \
  --tp 4 \
  --port 30000

Quantization to fit a larger model on smaller GPUs
python -m sglang.launchserver \
  --model-path meta-llama/Llama-3.1-8B-Instruct \
  --quantization fp8 \
  --port 30000

Control context length and KV-cache memory budget
python -m sglang.launchserver \

  --model-path meta-llama/Llama-3.1-8B-Instruct \
  --context-length 8192 \
  --mem-fraction-static 0.85 \
  --port 30000

• --tp N requires N visible GPUs; combine with CUDAVISIBLEDEVICES to pin specific devices.
• --quantization accepts values such as fp8, awq, and gptq. Quantized weights reduce memory and can raise throughput at a usually small quality cost.
• --context-length sets the maximum sequence length; raising it increases per-request KV-cache footprint.
• --mem-fraction-static reserves a fraction of VRAM for the static KV pool. Lower it if you hit out-of-memory at startup; raise it to give RadixAttention more room to cache.

curl http://localhost:30000/health

curl http://localhost:30000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Llama-3.1-8B-Instruct",
    "messages": [
      {"role": "system", "content": "You are a concise assistant."},
      {"role": "user", "content": "List three causes of GPU OOM during serving."}
    ],
    "temperature": 0.2,
    "maxtokens": 256
  }'

The same endpoint from the OpenAI Python client:

from openai import OpenAI

client = OpenAI(baseurl="http://localhost:30000/v1", apikey="EMPTY")

resp = client.chat.completions.create(
    model="meta-llama/Llama-3.1-8B-Instruct",
    messages=[
        {"role": "system", "content": "You are a concise assistant."},
        {"role": "user", "content": "Summarize RadixAttention in one sentence."},
    ],
    temperature=0.2,
    maxtokens=128,

)
print(resp.choices[0].message.content)

stream = client.chat.completions.create(
    model="meta-llama/Llama-3.1-8B-Instruct",
    messages=[{"role": "user", "content": "Explain continuous batching briefly."}],
    stream=True,
)
for chunk in stream:
    delta = chunk.choices[0].delta.content
    if delta:
        print(delta, end="", flush=True)

curl http://localhost:30000/generate \
  -H "Content-Type: application/json" \
  -d '{
    "text": "The three pillars of observability are",
    "samplingparams": {"temperature": 0.0, "maxnewtokens": 64}
  }'

To use the frontend language against this runtime, point the SGLang client at the server endpoint.

import sglang as sgl

Bind the frontend to the local runtime
backend = sgl.RuntimeEndpoint("http://localhost:30000")
sgl.setdefaultbackend(backend)

From here on, the frontend functions described below will execute against that runtime.

The Frontend Language in Depth

The frontend turns a generation program into an ordinary Python function. The key building blocks:

• @sgl.function decorates a function whose first argument is a program state s.
• s += ... appends text or a message role to the state.
• sgl.user(...), sgl.assistant(...), sgl.system(...) mark chat roles.
• sgl.gen("name", ...) fills a named slot by generating tokens; the result is read back via s["name"].
• s.fork(n) creates parallel copies of the state for independent branches.

A minimal example:

import sglang as sgl

@sgl.function
def explain(s, topic):
    s += sgl.system("You are a precise technical writer.")
    s += sgl.user(f"Explain {topic} in two sentences.")
    s += sgl.assistant(sgl.gen("answer", maxtokens=120, temperature=0.2))


state = explain.run(topic="RadixAttention")
print(state["answer"])

@sgl.function
def classify(s, ticket):
    s += sgl.system("Classify the support ticket.")
    s += sgl.user(ticket)
    # The model must pick exactly one of these labels
    s += sgl.assistant(sgl.gen("label", choices=["billing", "bug", "feature", "other"]))

state = classify.run(ticket="The app crashes when I upload a PDF.")
print(state["label"])  # one of the four choices, guaranteed

@sgl.function
def extractphone(s, text):
    s += sgl.user(f"Extract the phone number from: {text}")
    s += sgl.assistant(
        sgl.gen("phone", regex=r"\+?\d{1,3}[- ]?\d{3,4}[- ]?\d{3,4}")
    )

For structured records, constrain to a JSON schema so the output parses on the first try:

import json

productschema = json.dumps({

    "type": "object",
    "properties": {
        "name": {"type": "string"},
        "price": {"type": "number"},
        "instock": {"type": "boolean"},
    },
    "required": ["name", "price", "instock"],

})

@sgl.function
def torecord(s, description):
    s += sgl.user(f"Convert to JSON: {description}")
    s += sgl.assistant(sgl.gen("record", maxtokens=128, jsonschema=productschema))


state = torecord.run(description="The Helix mug costs 12.50 and is available.")
record = json.loads(state["record"])
print(record["name"], record["price"], record["instock"])

@sgl.function
def multiperspective(s, question):
    s += sgl.system("Answer carefully.")
    s += sgl.user(question)

    forks = s.fork(3)
    for f in forks:
        f += sgl.assistant(sgl.gen("view", maxtokens=100, temperature=0.8))


    # Merge the three independent drafts back into the parent
    drafts = [f["view"] for f in forks]
    s += sgl.user("Here are three drafts:\n" + "\n---\n".join(drafts))
    s += sgl.user("Write one consolidated answer.")
    s += sgl.assistant(sgl.gen("final", maxtokens=160))

state = multiperspective.run(question="How should we size KV cache for a chat service?")

print(state["final"])

import json
import sglang as sgl

sgl.setdefaultbackend(sgl.RuntimeEndpoint("http://localhost:30000"))


incidentschema = json.dumps({
    "type": "object",
    "properties": {
        "service": {"type": "string"},
        "severity": {"type": "string", "enum": ["low", "medium", "high"]},
        "rootcause": {"type": "string"},

    },
    "required": ["service", "severity", "rootcause"],
})

@sgl.function
def triageincident(s, report):

    # Shared prefix: system instructions reused across every request.
    # RadixAttention caches this once.
    s += sgl.system(
        "You are an SRE assistant. Be precise and avoid speculation."
    )
    s += sgl.user(f"Incident report:\n{report}")

    # Step 1: a constrained routing decision
    s += sgl.assistant(
        "Escalation needed: "
sgl.gen("escalate", choices=["yes", "no"])
    )


    # Step 2: extract structured fields with a JSON schema
    s += sgl.user("Extract the structured fields as JSON.")
    s += sgl.assistant(sgl.gen("fields", maxtokens=200, jsonschema=incidentschema))

    # Step 3: branch only when escalation is needed
    if s["escalate"] == "yes":
        s += sgl.user("Draft a short remediation note for an on-call engineer.")
        s += sgl.assistant(sgl.gen("note", maxtokens=180))


reporttext = (
    "At 02:14 the checkout-api returned 500s for ~8 minutes. "
    "Error logs show a connection pool exhaustion against the orders DB."
)

state = triageincident.run(report=reporttext)
print("escalate:", state["escalate"])
print("fields:", json.loads(state["fields"]))
if state["escalate"] == "yes":
    print("note:", state["note"])

What makes this efficient: the system prompt and the report are a shared prefix. If you run this program over a batch of incidents that share the same instructions, the instruction tokens are computed once and reused. The constrained escalate slot guarantees a parseable yes/no, and the JSON-schema slot guarantees the fields parse without defensive string cleanup.

Why RadixAttention Helps Shared-Prefix Workloads

The cost of LLM inference is dominated by attention over the KV cache. When many requests share leading tokens — a long system prompt, a few-shot block, a retrieved document — recomputing those tokens for each request is wasted work. RadixAttention stores KV cache keyed by token sequence in a radix tree, so a new request that matches an existing prefix reuses the cached KV and only computes the novel suffix.

This is why few-shot prompting and agent loops benefit most: the few-shot examples or the agent's standing instructions form a stable, long prefix that gets reused on every call. The frontend's fork makes this explicit — branches share a prefix by construction — but the benefit also applies to plain API requests that happen to share leading text.

Batching Multiple Requests with runbatch

When you have many inputs for the same program, runbatch submits them together so the runtime can schedule and cache across them.

reports = [
    {"report": "DB latency spiked to 4s during the nightly backup window."},
    {"report": "Frontend 404s after the last deploy; assets path changed."},
    {"report": "Auth service CPU at 95% under normal traffic."},
]

states = triageincident.runbatch(
    reports,
    progressbar=True,

)

for st in states:
    print(st["escalate"], json.loads(st["fields"])["severity"])

@sgl.function
def describe(s, imagepath):
    s += sgl.user(sgl.image(imagepath) + "Describe this image in one sentence.")

    s += sgl.assistant(sgl.gen("caption", maxtokens=64))

python -m sglang.launchserver \
  --model-path BAAI/bge-large-en-v1.5 \
  --is-embedding --port 30000

python -m sglang.benchserving \
  --backend sglang \
  --host 127.0.0.1 --port 30000 \
  --dataset-name random \
  --num-prompts 1000 \
  --random-input-len 1024 \
  --random-output-len 256

Read the output as a distribution, not a single number: look at request throughput, output-token throughput, and the time-to-first-token and per-token latency percentiles (p50/p99).

A balanced take on SGLang vs vLLM:

• For workloads with heavy prefix sharing — few-shot prompting, agents with long standing instructions, tree-style branching — SGLang's RadixAttention often wins on throughput and latency because it skips recomputation.
• For workloads where each request is essentially unique with little shared prefix, the two runtimes are closer; both rely on continuous batching and efficient attention kernels, and the winner depends on model, hardware, and version.
• If you need a programming model for multi-step, constrained, or parallel generation, SGLang's frontend is a genuine differentiator and reduces application glue code.
• Benchmark on your own traffic. Synthetic numbers rarely transfer cleanly to production prompt shapes.

Best Practices

• Put the stable, reusable text (system prompt, few-shot examples, shared documents) at the front of every prompt so RadixAttention can cache it. Variable content goes last.
• Use constrained decoding (choices, regex, jsonschema) whenever you intend to parse the output. It removes a class of brittle string-cleanup bugs.
• Prefer runbatch for offline or bulk workloads; it gives the scheduler and the cache the most to work with.
• Tune --mem-fraction-static deliberately. Too low wastes cache capacity; too high risks OOM under long contexts. Watch nvidia-smi while load testing.
• Keep the frontend program deterministic in structure. Branch on constrained outputs (like the yes/no above) rather than free text, so control flow stays predictable.
• Separate concerns: use the OpenAI-compatible API for simple chat integrations, and reach for the frontend language only when you actually need forks, constraints, or multi-step orchestration.
• Pin the SGLang version in production and re-run benchserving after upgrades; kernel and scheduler changes can shift performance.

• Two pillars: a continuous-batching runtime with RadixAttention prefix caching, and a frontend DSL for structured generation.
• Install with pip install "sglang[all]"; launch with python -m sglang.launchserver; query through the OpenAI-compatible API or the native /generate endpoint.
• The frontend's @sgl.function, sgl.gen, role helpers, s.fork, and constrained decoding make multi-step programs concise and reliable.
• RadixAttention pays off most when requests share long prefixes; put stable text first to maximize cache hits.
• Use runbatch for bulk jobs and sglang.benchserving to validate performance on your own traffic before drawing conclusions about SGLang versus vLLM.