Serving LLMs in Production with Text Generation Inference (TGI)

Text Generation Inference (TGI) is Hugging Face's purpose-built toolkit for serving large language models behind a production HTTP API. It pairs a Rust web layer with a Python inference backend, ships as a ready-to-run Docker image, and exposes an OpenAI-compatible Messages API so existing clients work with minimal changes. This tutorial walks through deploying TGI, tuning its launcher, and querying it from curl, the OpenAI client, and the Hugging Face Hub client.

What TGI Is

TGI is an inference server maintained by Hugging Face and used to power their hosted Inference Endpoints. Architecturally it splits responsibilities:

• A Rust router/webserver handles HTTP, request validation, batching decisions, and token streaming. Rust keeps the request-handling path fast and memory-safe under concurrency.
• A Python model server (one process per shard) runs the actual model forward passes on the GPU, using optimized attention and batching kernels.

This separation lets the busy networking layer stay lean while the heavy numerical work sits in well-tested Python/CUDA code. Because TGI is built by the same team behind the Transformers and Hub libraries, model loading, tokenizer handling, and gated-model authentication line up naturally with the rest of the Hugging Face ecosystem.

Key Features

• Continuous batching. Incoming requests join an in-flight batch as soon as slots free up, rather than waiting for a fixed batch window. This keeps the GPU busy and improves throughput under mixed load.
• Tensor parallelism (sharding). A model can be split across multiple GPUs with --num-shard, enabling models that do not fit on a single card.
• Optimized attention. TGI uses Flash Attention and Paged Attention kernels to reduce memory use and speed up both prefill and decode.
• Token streaming. Server-Sent Events (SSE) stream tokens as they are produced, which lowers perceived latency for chat UIs.
• Quantization. Built-in support for bitsandbytes, GPTQ, AWQ, and other schemes to fit larger models into less VRAM.
• OpenAI-compatible Messages API. A /v1/chat/completions endpoint that mirrors the OpenAI request/response shape.

If you have read our earlier vLLM tutorial: both servers do continuous batching and paged attention, and both expose an OpenAI-style API. The practical difference is packaging and ecosystem. TGI is Docker-first, tightly integrated with the Hub and Inference Endpoints, and driven by a single launcher binary with explicit flags. We compare them more concretely at the end.

Prerequisites

• A machine with one or more NVIDIA GPUs and recent drivers.
• Docker with the NVIDIA Container Toolkit installed, so --gpus all works.
• A Hugging Face account and an access token if you plan to serve gated models such as the Llama family. Create one under Settings → Access Tokens.

Verify GPU access inside Docker before going further:

docker run --rm --gpus all nvidia/cuda:12.4.0-base-ubuntu22.04 nvidia-smi

If the GPU table prints, the container runtime can see your hardware.

The Fastest Path: Running TGI with Docker

TGI is distributed as a container image at ghcr.io/huggingface/text-generation-inference. The minimum you need is a model ID and GPU access. We will use a Mistral instruct model, which is open and small enough to run on a single mid-range GPU.

# Reuse a host directory for the model cache so re-runs do not re-download
mkdir -p $HOME/.cache/huggingface

docker run --gpus all --shm-size 1g -p 8080:80 \
  -v $HOME/.cache/huggingface:/data \
  ghcr.io/huggingface/text-generation-inference:latest \
  --model-id mistralai/Mistral-7B-Instruct-v0.3

A few details matter here:

• --gpus all exposes the GPUs to the container.
• -p 8080:80 maps the container's port 80 to host port 8080. TGI listens on port 80 inside the container by default.
• -v $HOME/.cache/huggingface:/data mounts a persistent cache. Weights are downloaded once into /data and reused across container restarts, which saves time and bandwidth.
• --shm-size 1g raises shared memory; the model server uses it for inter-process communication, and the default can be too small for sharded setups.
• Everything after the image name is passed to the TGI launcher, not to Docker. --model-id is the launcher's required argument.

Serving Gated Models

Models like meta-llama/Llama-3.1-8B-Instruct require accepting a license and authenticating. Pass your token through the HFTOKEN environment variable:

docker run --gpus all --shm-size 1g -p 8080:80 \
  -v $HOME/.cache/huggingface:/data \
  -e HFTOKEN=$HFTOKEN \

  ghcr.io/huggingface/text-generation-inference:latest \
  --model-id meta-llama/Llama-3.1-8B-Instruct

Set the token in your shell first with export HFTOKEN=hfxxx. Never bake the token into an image or commit it to source control.

curl http://localhost:8080/health

docker run --gpus all --shm-size 1g -p 8080:80 \
  -v $HOME/.cache/huggingface:/data \
  ghcr.io/huggingface/text-generation-inference:latest \
  --model-id mistralai/Mistral-7B-Instruct-v0.3 \
  --max-input-tokens 4096 \
  --max-total-tokens 8192 \
  --max-batch-prefill-tokens 8192

docker run --gpus all --shm-size 1g -p 8080:80 \
  -v $HOME/.cache/huggingface:/data \
  ghcr.io/huggingface/text-generation-inference:latest \
  --model-id meta-llama/Llama-3.1-70B-Instruct \
  --num-shard 4

docker run --gpus all --shm-size 1g -p 8080:80 \
  -v $HOME/.cache/huggingface:/data \
  ghcr.io/huggingface/text-generation-inference:latest \
  --model-id mistralai/Mistral-7B-Instruct-v0.3 \
  --quantize bitsandbytes \
  --dtype float16

Native /generate and /generatestream

The native endpoint takes raw input text and generation parameters. It does not apply a chat template, so you control the exact prompt.

curl http://localhost:8080/generate \
  -H 'Content-Type: application/json' \
  -d '{
    "inputs": "[INST] Explain continuous batching in one paragraph. [/INST]",
    "parameters": {
      "maxnewtokens": 200,
      "temperature": 0.7,
      "topp": 0.95

    }
  }'

The response is a JSON object with a generatedtext field. For streaming, use /generatestream, which returns Server-Sent Events:

curl http://localhost:8080/generatestream \
  -H 'Content-Type: application/json' \
  -d '{
    "inputs": "[INST] List three uses for token streaming. [/INST]",
    "parameters": { "maxnewtokens": 150 }
  }'

Each event carries a token object as it is generated, and the final event includes generatedtext and timing details.

curl http://localhost:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "tgi",
    "messages": [
      {"role": "system", "content": "You are a concise assistant."},
      {"role": "user", "content": "What is tensor parallelism?"}
    ],
    "maxtokens": 200,
    "temperature": 0.7
  }'

The model field is accepted but not used for routing; TGI serves whatever model it was launched with. Set it to any placeholder such as "tgi".

The Messages API with the OpenAI Python Client

Because the endpoint is OpenAI-compatible, the official openai library works by pointing baseurl at your TGI server. This is the simplest migration path for code already written against OpenAI.

from openai import OpenAI

client = OpenAI(
    baseurl="http://localhost:8080/v1",
    apikey="not-needed",  # TGI ignores the key unless you front it with auth

)

resp = client.chat.completions.create(
    model="tgi",
    messages=[
        {"role": "system", "content": "You are a concise assistant."},
        {"role": "user", "content": "Summarize what TGI does in two sentences."},
    ],
    maxtokens=200,
    temperature=0.7,
)

print(resp.choices[0].message.content)

Streaming uses the same stream=True pattern as OpenAI:

stream = client.chat.completions.create(
    model="tgi",
    messages=[{"role": "user", "content": "Count from 1 to 5 slowly."}],
    maxtokens=100,

    stream=True,
)

for chunk in stream:
    delta = chunk.choices[0].delta.content
    if delta:
        print(delta, end="", flush=True)
print()

Using huggingfacehub.InferenceClient

The huggingfacehub library ships an InferenceClient that talks to TGI directly. It offers a low-level textgeneration method matching the native API and a chatcompletion method matching the Messages API.

from huggingfacehub import InferenceClient

client = InferenceClient(baseurl="http://localhost:8080")


Native text generation
out = client.textgeneration(
    "[INST] Give one benefit of paged attention. [/INST]",
    maxnewtokens=120,
    temperature=0.7,
)
print(out)

Chat completion with the same client:

completion = client.chatcompletion(
    messages=[
        {"role": "user", "content": "Explain quantization in two sentences."},
    ],
    maxtokens=150,
)
print(completion.choices[0].message.content)

Both methods support streaming. For text generation, pass stream=True and iterate over tokens:

for token in client.textgeneration(
    "[INST] Name two TGI launcher flags. [/INST]",
    maxnewtokens=80,

    stream=True,
):
    print(token, end="", flush=True)
print()

For chat, chatcompletion(..., stream=True) yields chunks whose choices[0].delta.content holds each incremental piece, mirroring the OpenAI shape.

Structured Output with JSON Grammar

TGI can constrain generation so the model emits text that conforms to a JSON schema. With the Messages API you use responseformat, which maps to a grammar internally and guarantees the output parses.

from openai import OpenAI

client = OpenAI(baseurl="http://localhost:8080/v1", apikey="not-needed")


schema = {
    "type": "object",
    "properties": {
        "city": {"type": "string"},
        "population": {"type": "integer"},
    },
    "required": ["city", "population"],
}

resp = client.chat.completions.create(
    model="tgi",
    messages=[{"role": "user", "content": "Give the city and population of Jakarta."}],
    responseformat={"type": "jsonobject", "value": schema},

    maxtokens=100,
)
print(resp.choices[0].message.content)

On the native /generate endpoint the equivalent is a grammar field inside parameters with {"type": "json", "value": }. Constrained decoding is useful for extraction pipelines where downstream code must parse the result without defensive cleanup.

Deploying on Kubernetes

TGI runs well as a standard Kubernetes Deployment plus Service. The container needs a GPU resource request, the cache mounted as a volume, and the launcher flags passed as args. The example below assumes the NVIDIA device plugin is installed so nvidia.com/gpu is schedulable.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: tgi-mistral
  labels:
    app: tgi-mistral
spec:
  replicas: 1
  selector:
    matchLabels:
      app: tgi-mistral
  template:
    metadata:
      labels:
        app: tgi-mistral
    spec:
      containers:
name: tgi
          image: ghcr.io/huggingface/text-generation-inference:latest

          args:
"--model-id"
"mistralai/Mistral-7B-Instruct-v0.3"
"--max-input-tokens"
"4096"
"--max-total-tokens"
"8192"
          ports:

containerPort: 80
          env:

name: HFTOKEN

              valueFrom:

                secretKeyRef:
                  name: hf-secret
                  key: token
          resources:
            limits:
              nvidia.com/gpu: 1
          volumeMounts:
name: model-cache
              mountPath: /data

name: dshm
              mountPath: /dev/shm

      volumes:
name: model-cache
          persistentVolumeClaim:

            claimName: tgi-cache-pvc
name: dshm
          emptyDir:

            medium: Memory
            sizeLimit: 1Gi

apiVersion: v1
kind: Service
metadata:
  name: tgi-mistral
spec:
  selector:
    app: tgi-mistral
  ports:
port: 8080
      targetPort: 80

curl http://localhost:8080/metrics

scrapeconfigs:
jobname: tgi

    staticconfigs:

targets: ["tgi-mistral:8080"]

Watch queue duration and batch fill: rising queue time under steady load signals you are saturated and should scale out or raise the token budgets.

Benchmarking with text-generation-benchmark

The image ships a benchmarking tool that drives load against a running server and reports latency and throughput. Run it inside the same container:

docker exec -it  \
  text-generation-benchmark \
  --tokenizer-name mistralai/Mistral-7B-Instruct-v0.3 \
  --sequence-length 512 \
  --decode-length 128 \
  --batch-size 1 --batch-size 8 --batch-size 32

It sweeps the listed batch sizes and prints prefill and decode timings per batch, which helps you pick token budgets and understand how throughput scales with concurrency before you commit to production settings.

Choosing TGI vs vLLM

Both are strong, actively maintained inference servers with overlapping core techniques. A balanced view:

• Choose TGI when you are already in the Hugging Face ecosystem, want a single supported Docker image, value tight Hub and Inference Endpoints integration, or prefer a launcher with explicit, well-documented flags. The Rust router is robust under concurrency, and the managed-endpoint path is a clean escape hatch.
• Choose vLLM when you want the broadest and often earliest support for new models and quantization formats, need its specific scheduling features, or are building on a Python-native stack where vLLM's library API fits better.

In practice the two converge on throughput for common models, so the decision usually comes down to ecosystem fit and operational preference rather than raw speed. Benchmark both on your own model and traffic shape before deciding.

Best Practices

• Mount the cache. Always bind a host or PVC volume to /data so you download weights once.
• Set token budgets deliberately. Tune --max-input-tokens, --max-total-tokens, and --max-batch-prefill-tokens to your real prompt lengths rather than leaving defaults; this reclaims VRAM for concurrency.
• Keep secrets out of images. Inject HFTOKEN via environment variables or Kubernetes Secrets, never in the image or repo.
• Prefer pre-quantized checkpoints. AWQ or GPTQ weights generally outperform on-the-fly bitsandbytes quantization for serving.
• Right-size /dev/shm. Sharded deployments need adequate shared memory; set --shm-size in Docker or a memory emptyDir in Kubernetes.
• Stream for chat. Use /generatestream or stream=True to cut perceived latency in interactive interfaces.
• Measure before scaling. Use text-generation-benchmark and /metrics to find the real bottleneck instead of guessing.
• Pin the image tag. Use a specific version tag rather than latest in production so deployments are reproducible.

Conclusion and Key Takeaways

TGI gives you a production-grade LLM server with very little setup: one Docker command, a model ID, and a mounted cache get you a running endpoint with continuous batching and optimized attention. Its OpenAI-compatible Messages API means existing OpenAI client code works by changing only the baseurl, while the native /generate endpoints and the huggingfacehub.InferenceClient give you finer control and direct Hub integration.

Key takeaways:

• TGI is a Rust router plus Python model server, distributed as a Docker image and tightly integrated with the Hugging Face Hub.
• The launcher flags for token budgets, sharding, and quantization are your primary levers for fitting a model and tuning throughput.
• You can call it three ways: native /generate, the OpenAI-compatible /v1/chat/completions, and the InferenceClient — all with streaming and structured-output support.
• It deploys cleanly on Kubernetes and is available fully managed via Inference Endpoints.
• Against vLLM, the choice usually hinges on ecosystem fit; benchmark both on your workload before committing.

With the commands and configuration in this guide you can stand up a Mistral or Llama instruct model, query it from multiple clients, and move it to Kubernetes when you are ready for production traffic.