TensorRT-LLM: Squeezing Maximum Throughput Out of NVIDIA GPUs for LLM Inference

Hey everyone, today we look at one of the most serious weapons available if you run LLMs in production on NVIDIA GPUs: TensorRT-LLM. Where vLLM is known for being easy and already fast, TensorRT-LLM is a level beyond that: it compiles your model into a binary engine optimized specifically for the GPU architecture you own. In many workloads the result is lower latency and higher throughput than any other runtime.

The trade-off is complexity. There is a build step that can take tens of minutes, the resulting engine is tied to your GPU and library versions, and there are a lot of knobs. So I am writing this tutorial honestly: I will show you how to use it, and also when you should not.

We will cover the architecture and core concepts, installation through Docker, checkpoint conversion and engine building, inference with the Python API, FP8 and INT4 quantization, deployment with Triton, and how to choose between TensorRT-LLM and simpler alternatives.

Introduction

What Is TensorRT-LLM

TensorRT-LLM is an open source library from NVIDIA that provides optimized LLM model definitions plus a high performance inference runtime. It is built on TensorRT, NVIDIA's deep learning compiler that has long been used for vision models.

Its flow differs from what you do in PyTorch. In PyTorch a model executes layer by layer dynamically. In TensorRT-LLM the model is compiled ahead of time into an engine, a binary file containing an optimized execution plan: kernels chosen specifically for your GPU, fused operations, planned memory layouts, and fixed precision.

Its main optimizations include heavily tuned attention kernels, in-flight batching (often called continuous batching) that merges requests arriving at different times without making them wait for each other, a paged KV cache for efficient memory use, FP8 and INT4 quantization with dedicated kernels, and tensor plus pipeline parallelism for models that do not fit on one GPU.

When to Use It, and When Not To

Use TensorRT-LLM when you serve LLM traffic that is high and steady on NVIDIA GPUs, when GPU cost per token genuinely matters, when the model you serve rarely changes, and when you have time for build and tuning work.

Do not use TensorRT-LLM when you are still experimenting and switching models often, when you need support for a very new or unusual architecture, when your team is small and cannot own a build pipeline, or when you need portability to non-NVIDIA hardware. In those cases vLLM or SGLang give you 80 to 90 percent of the performance at a tenth of the complexity.

My usual advice: start with vLLM, measure, and move to TensorRT-LLM only when your cost numbers demand it.

Installation

The sane path is the official NVIDIA container, because TensorRT-LLM is very sensitive to CUDA, driver, and TensorRT versions.

Prerequisites: an NVIDIA GPU ideally with compute capability 8.0 or higher (Ampere, Ada, Hopper), a current driver, and the NVIDIA Container Toolkit installed.

docker run --rm -it --gpus all \
  --shm-size=8g \
  -v $(pwd):/workspace \
  -w /workspace \
  nvcr.io/nvidia/tritonserver:24.10-trtllm-python-py3 bash

Inside the container, verify:

python3 -c "import tensorrtllm; print(tensorrtllm.version)"

If you insist on installing directly on the host:

pip install tensorrtllm --extra-index-url https://pypi.nvidia.com

from tensorrtllm import LLM, SamplingParams

Builds the engine behind the scenes, then runs it
llm = LLM(model="TinyLlama/TinyLlama-1.1B-Chat-v1.0")

sampling = SamplingParams(temperature=0.7, topp=0.9, maxtokens=256)

prompts = [
    "Explain what a vector database is in two sentences.",
    "Write a Python function that reverses a string.",
]

for out in llm.generate(prompts, sampling):
    print("PROMPT:", out.prompt)
    print("OUTPUT:", out.outputs[0].text)
    print("-" * 60)

The first call is slow because the engine is being built. Later calls reuse the cached engine and start much faster. Keep that cache directory if you want to move the engine to another machine with an identical GPU.

For larger models across several GPUs:

llm = LLM(
    model="meta-llama/Llama-3.1-8B-Instruct",
    tensorparallelsize=2,          # shard the model across 2 GPUs
    dtype="bfloat16",
    maxbatchsize=64,
    maxinputlen=4096,
    maxseqlen=8192,
)

The Manual Flow: Convert and Build

When you need full control, for custom quantization or Triton deployment, use the two stage flow: convert the checkpoint, then build the engine.

Stage one converts HuggingFace weights into the TensorRT-LLM checkpoint format.

python3 convertcheckpoint.py \
  --modeldir ./Llama-3.1-8B-Instruct \
  --outputdir ./ckpt/llama-8b-bf16 \

  --dtype bfloat16 \
  --tpsize 1

Stage two compiles that checkpoint into an engine.

trtllm-build \
  --checkpointdir ./ckpt/llama-8b-bf16 \

  --outputdir ./engines/llama-8b-bf16 \
  --gemmplugin auto \

  --maxbatchsize 64 \

  --maxinputlen 4096 \

  --maxseqlen 8192 \

  --usepagedcontextfmha enable

Note that the built engine is tied to three things: the GPU architecture, the TensorRT-LLM version, and the build parameters above. An engine built for an A100 will not run on an H100. So do not ship engines as cross-machine release artifacts, ship the checkpoint and the build script.

Running a built engine:

from tensorrtllm import LLM, SamplingParams

llm = LLM(model="./engines/llama-8b-bf16",
          tokenizer="meta-llama/Llama-3.1-8B-Instruct")

out = llm.generate(["What are the benefits of a paged KV cache?"],
                   SamplingParams(maxtokens=200))
print(out[0].outputs[0].text)

Quantization

This is where TensorRT-LLM really shines, because its quantized kernels are deeply optimized for NVIDIA hardware.

FP8 is available on Hopper (H100) and Ada (L40S, RTX 4090) GPUs. Quality is usually nearly indistinguishable from bf16, while throughput rises noticeably and KV cache memory halves.

python3 quantize.py \
  --modeldir ./Llama-3.1-8B-Instruct \

  --outputdir ./ckpt/llama-8b-fp8 \
  --dtype bfloat16 \
  --qformat fp8 \
  --kvcachedtype fp8 \
  --calibsize 512

python3 quantize.py \
  --modeldir ./Llama-3.1-8B-Instruct \
  --outputdir ./ckpt/llama-8b-int4awq \

  --qformat int4awq \
  --awqblocksize 128 \
  --calibsize 512

After quantization, run trtllm-build as before with --checkpointdir pointing at the quantized output.

One frequently forgotten point: quantization needs calibration data. If your domain is specific, say Indonesian legal or medical text, calibrate on samples from that domain rather than a generic English default. The quality difference is noticeable.

And always re-evaluate the model after quantization. Do not assume quality is preserved just because a public benchmark says so.

In-Flight Batching

This is the feature that gives the largest throughput jump on real workloads, and the concept is worth understanding.

In traditional static batching, the server waits for a batch to fill, processes it together, and only accepts the next batch once every request in it finishes. The problem is that output lengths differ. A request that finishes in 20 tokens waits for one that needs 800. The GPU idles a lot.

With in-flight batching, as soon as one request finishes its slot is immediately filled by a queued request, without waiting for the rest of the batch. GPU utilization goes up substantially and queueing latency drops.

This is enabled by default in the modern runtime, but its effectiveness depends on build parameters such as maxbatchsize and maxnumtokens. Too small and you cap parallelism. Too large and the KV cache runs out, causing rejections or preemption.

Deploying with Triton Inference Server

For production, the standard combination is TensorRT-LLM as a backend inside Triton Inference Server. Triton handles HTTP and gRPC, Prometheus metrics, the model repository, and health checks.

The model repository usually looks like this:

modelrepo/
  preprocessing/        # tokenization
  tensorrtllm/         # the built engine
  postprocessing/       # detokenization
  ensemble/             # wires the three together

Start the server:

tritonserver --model-repository=/workspace/modelrepo

curl -X POST localhost:8000/v2/models/ensemble/generate \
  -H "Content-Type: application/json" \
  -d '{"textinput": "What is in-flight batching?", "maxtokens": 128, "temperature": 0.7}'

trtllm-serve ./engines/llama-8b-bf16 \
  --tokenizer meta-llama/Llama-3.1-8B-Instruct \
  --port 8000

from openai import OpenAI

client = OpenAI(baseurl="http://localhost:8000/v1", apikey="empty")

resp = client.chat.completions.create(
    model="llama",
    messages=[{"role": "user", "content": "Hello, who are you?"}],
)
print(resp.choices[0].message.content)

trtllm-bench --model meta-llama/Llama-3.1-8B-Instruct \
  throughput \
  --dataset dataset.json \
  --enginedir ./engines/llama-8b-bf16

Four metrics matter. Time to first token, how fast the first token appears, which drives chat experience. Inter-token latency, the gap between tokens, which determines how smoothly text streams. Total throughput in tokens per second, which sets your cost per token. And the maximum concurrency that still meets your latency target.

Measure with a traffic pattern resembling production, not one long prompt repeated. Results differ enormously.

Tips and Best Practices

Always use the official NVIDIA container and record the tag in your team documentation. This stack has a strict compatibility matrix.

Ship checkpoints and build scripts, not engine files, as release artifacts. Engines must be rebuilt on the target machine.

Do not set maxinputlen and maxseqlen far above real needs. Every increase eats KV cache memory that could otherwise serve more concurrent requests.

Start at bf16, measure, then try FP8, then INT4 only if needed. Climb the quantization ladder gradually while evaluating quality on your own data.

Monitor GPU utilization and KV cache usage in production. Low utilization with a long queue usually means maxbatchsize is too small.

Budget build time in your CI pipeline. Building an engine for an 8B model can take 10 to 30 minutes, and that should be planned rather than discovered at deploy time.

Conclusion

TensorRT-LLM is the right choice once you are seriously serving LLMs on NVIDIA GPUs and cost per token matters. The summary:

Models are compiled ahead of time into engines optimized for your specific GPU, and those engines are not portable across architectures.

The high level LLM API is the easiest entry point, while the convert_checkpoint plus trtllm-build flow gives full control.

In-flight batching and the paged KV cache are the main sources of real world throughput gains.

FP8 quantization on Hopper and Ada delivers large gains with usually minimal quality loss, while INT4 AWQ pushes memory further with more quality risk.

For production, pair it with Triton Inference Server, or use trtllm-serve if an OpenAI compatible endpoint is enough.

Most importantly, be honest about your needs. If your traffic is still modest, vLLM gets you nearly the same result with far less complexity. Move to TensorRT-LLM when your cost numbers tell you to, not because it is the most advanced option available. Happy serving.