LMDeploy Tutorial: A Complete Toolkit for Deploying and Serving Large Language Models
Large Language Models (LLMs) are increasingly used in various applications, from chatbots to code assistants. However, the main challenges in using LLMs are the high GPU resource requirements and inference latency. LMDeploy provides a comprehensive solution for compressing, deploying, and serving LLMs efficiently.
LMDeploy is developed by the InternLM team at Shanghai AI Laboratory. The toolkit supports various popular models such as LLaMA, Qwen, Mistral, InternLM, and many others. With its optimized TurboMind engine and W4A16/W8A8 quantization support, LMDeploy allows you to run large models with fewer resources without significantly sacrificing output quality.
In this tutorial, we will cover installation, basic usage, model quantization, API serving, and best practices for production deployment.
Installation
Prerequisites
Before installing LMDeploy, make sure you have:
- Python 3.8 or later
- CUDA 11.8 or 12.x (for GPU acceleration)
- NVIDIA GPU with at least 8GB VRAM (for quantized 7B models)
Installation via pip
The easiest way to install LMDeploy is using pip:
pip install lmdeploy
For installation with specific CUDA support:
# For CUDA 12.x
pip install lmdeploy
For CUDA 11.8
pip install lmdeploy --extra-index-url https://download.pytorch.org/whl/cu118
Verify Installation
After installation is complete, verify by running:
import lmdeploy
print(lmdeploy.version)
Or via the command line:
lmdeploy version
Installation from Source (Optional)
If you need the latest features from the development branch:
git clone https://github.com/InternLM/lmdeploy.git
cd lmdeploy
pip install -e .
Basic Usage
Interactive Chat via Command Line
The quickest way to try LMDeploy is using the interactive chat mode:
lmdeploy chat internlm/internlm25-7b-chat
This command will download the model from Hugging Face (if not already present) and start an interactive chat session in the terminal. You can directly type questions and get responses from the model.
To use a model from a local path:
lmdeploy chat /path/to/your/model
Using the Pipeline API in Python
For integration into Python applications, LMDeploy provides a simple Pipeline API:
from lmdeploy import pipeline
Initialize the pipeline
pipe = pipeline("internlm/internlm25-7b-chat")
Single prompt inference
response = pipe(["Explain what machine learning is in 3 sentences"])
print(response[0].text)
Batch Inference
One of LMDeploy's strengths is efficient batch processing:
from lmdeploy import pipeline
pipe = pipeline("internlm/internlm25-7b-chat")
prompts = [
"What is a neural network?",
"Explain the difference between supervised and unsupervised learning",
"How do transformers work?",
"What is the purpose of the attention mechanism?"
]
responses = pipe(prompts)
for prompt, resp in zip(prompts, responses):
print(f"Q: {prompt}")
print(f"A: {resp.text}\n")
Generation Configuration
You can control text generation parameters using GenerationConfig:
from lmdeploy import pipeline, GenerationConfig
pipe = pipeline("internlm/internlm25-7b-chat")
genconfig = GenerationConfig(
maxnewtokens=512,
temperature=0.7,
topp=0.9,
topk=40,
repetitionpenalty=1.1
)
response = pipe(
["Write a short poem about artificial intelligence"],
genconfig=genconfig
)
print(response[0].text)
Multi-turn Conversation
LMDeploy supports multi-turn conversations while maintaining context:
from lmdeploy import pipeline, GenerationConfig, ChatTemplateConfig
pipe = pipeline(
"internlm/internlm25-7b-chat",
chattemplateconfig=ChatTemplateConfig(
modelname="internlm2",
system="You are a helpful AI assistant."
)
)
Simulate multi-turn conversation
messages = [
{"role": "user", "content": "Who invented electricity?"},
]
response = pipe([messages])
print(f"Assistant: {response[0].text}")
Continue the conversation
messages.append({"role": "assistant", "content": response[0].text})
messages.append({"role": "user", "content": "What was their greatest contribution?"})
response = pipe([messages])
print(f"Assistant: {response[0].text}")
Model Quantization
Quantization is a technique to reduce model size and speed up inference by lowering the precision of model weights. LMDeploy supports several quantization methods.
W4A16 Quantization (AWQ)
W4A16 means weights are quantized to 4-bit while activations remain at 16-bit. This is the most commonly used method:
lmdeploy lite autoawq \
internlm/internlm2
5-7b-chat \
--calib-dataset ptb \
--calib-samples 128 \
--calib-seqlen 2048 \
--w-bits 4 \
--w-group-size 128 \
--work-dir ./internlm25-7b-chat-4bit
Parameter explanation:
--calib-dataset: Calibration dataset for determining quantization ranges--calib-samples: Number of calibration samples--calib-seqlen: Calibration sequence length--w-bits: Number of bits for weight quantization (4-bit)--w-group-size: Group size for group-wise quantization--work-dir: Output directory for the quantized model
W8A8 Quantization
For more precise quantization with slightly higher overhead:
lmdeploy lite smoothquant \
internlm/internlm25-7b-chat \
--calib-dataset ptb \
--calib-samples 128 \
--calib-seqlen 2048 \
--work-dir ./internlm25-7b-chat-w8a8
Using Quantized Models
After quantization is complete, use the model as usual:
from lmdeploy import pipeline
Use the AWQ 4-bit model
pipe = pipeline("./internlm25-7b-chat-4bit")
response = pipe(["Explain quantum computing simply"])
print(response[0].text)
Memory Usage Comparison
Here is the approximate VRAM usage for a 7B model:
| Precision | VRAM (approx) | Relative Speed |
|-----------|---------------|----------------|
| FP16 | ~14 GB | 1x (baseline) |
| W8A8 | ~8 GB | 1.2-1.5x |
| W4A16 | ~5 GB | 1.5-2x |
KV Cache Quantization
Besides model quantization, LMDeploy also supports KV cache quantization to reduce memory usage during inference:
from lmdeploy import pipeline, TurbomindEngineConfig
engineconfig = TurbomindEngineConfig(
quantpolicy=8 # Quantize KV cache to 8-bit
)
pipe = pipeline(
"internlm/internlm25-7b-chat",
backendconfig=engineconfig
)
response = pipe(["What is a large language model?"])
print(response[0].text)
Advanced Usage
Serving via API Server
LMDeploy provides an API server compatible with the OpenAI API format, making it easy to integrate with applications already using the OpenAI SDK:
lmdeploy serve apiserver \
internlm/internlm2
5-7b-chat \
--server-name 0.0.0.0 \
--server-port 23333 \
--tp 1
Key parameters:
--server-name: Bind address (0.0.0.0 for external access)--server-port: Server port--tp: Tensor parallelism (number of GPUs to use)
Accessing the API Server
Once the server is running, you can access it using the OpenAI SDK:
from openai import OpenAI
client = OpenAI(
apikey="not-needed",
baseurl="http://localhost:23333/v1"
)
Chat completion
response = client.chat.completions.create(
model="internlm/internlm25-7b-chat",
messages=[
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "What are the advantages of using LMDeploy?"}
],
temperature=0.7,
maxtokens=512
)
print(response.choices[0].message.content)
Or using curl:
curl http://localhost:23333/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "internlm/internlm25-7b-chat",
"messages": [
{"role": "user", "content": "Hello, how are you?"}
],
"temperature": 0.7
}'
Streaming Response
To get streaming responses:
from openai import OpenAI
client = OpenAI(
apikey="not-needed",
baseurl="http://localhost:23333/v1"
)
stream = client.chat.completions.create(
model="internlm/internlm25-7b-chat",
messages=[
{"role": "user", "content": "Tell me a brief history of AI"}
],
stream=True
)
for chunk in stream:
if chunk.choices[0].delta.content:
print(chunk.choices[0].delta.content, end="", flush=True)
print()
Tensor Parallelism (Multi-GPU)
For larger models, you can distribute the model across multiple GPUs:
# Using 2 GPUs
lmdeploy serve apiserver \
meta-llama/Meta-Llama-3-70B-Instruct \
--tp 2 \
--server-port 23333
In Python:
from lmdeploy import pipeline, TurbomindEngineConfig
engineconfig = TurbomindEngineConfig(tp=2)
pipe = pipeline(
"meta-llama/Meta-Llama-3-70B-Instruct",
backendconfig=engineconfig
)
Vision Language Model (VLM) Support
LMDeploy also supports multimodal models that can process images:
from lmdeploy import pipeline
from lmdeploy.vl import loadimage
pipe = pipeline("OpenGVLab/InternVL2-8B")
image = loadimage("https://example.com/image.jpg")
response = pipe([("Describe this image", image)])
print(response[0].text)
For multiple images:
from lmdeploy import pipeline
from lmdeploy.vl import loadimage
pipe = pipeline("OpenGVLab/InternVL2-8B")
image1 = loadimage("image1.jpg")
image2 = loadimage("image2.jpg")
response = pipe([("Compare these two images", [image1, image2])])
print(response[0].text)
TurboMind Engine Configuration
TurboMind is LMDeploy's core inference engine. Here is the full configuration:
from lmdeploy import pipeline, TurbomindEngineConfig
engineconfig = TurbomindEngineConfig(
tp=1, # Tensor parallelism
sessionlen=4096, # Maximum context length
maxbatchsize=128, # Maximum batch size
cachemaxentrycount=0.8, # GPU memory proportion for KV cache
quantpolicy=0, # 0: no KV quant, 4: 4-bit, 8: 8-bit
ropescalingfactor=1.0, # RoPE scaling factor
)
pipe = pipeline(
"internlm/internlm25-7b-chat",
backendconfig=engineconfig
)
PyTorch Backend
Besides TurboMind, LMDeploy also supports a PyTorch backend for broader compatibility:
from lmdeploy import pipeline, PytorchEngineConfig
engineconfig = PytorchEngineConfig(
tp=1,
sessionlen=4096,
maxbatchsize=64,
cachemaxentrycount=0.8
)
pipe = pipeline(
"internlm/internlm25-7b-chat",
backendconfig=engineconfig
)
The PyTorch backend is useful when:
- The model is not yet supported by TurboMind
- You need specific PyTorch features
- Debugging and development
Docker Deployment
Running with Docker
LMDeploy provides an official Docker image:
docker run --gpus all \
-v ~/.cache/huggingface:/root/.cache/huggingface \
-p 23333:23333 \
--ipc=host \
openmmlab/lmdeploy:latest \
lmdeploy serve apiserver \
internlm/internlm25-7b-chat \
--server-port 23333
Docker Compose
For a more structured setup:
version: '3.8'
services:
lmdeploy:
image: openmmlab/lmdeploy:latest
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: 1
capabilities: [gpu]
ports:
- "23333:23333"
volumes:
- huggingface-cache:/root/.cache/huggingface
ipc: host
command: >
lmdeploy serve apiserver
internlm/internlm25-7b-chat
--server-port 23333
--server-name 0.0.0.0
volumes:
huggingface-cache:
Best Practices
1. Choose the Right Quantization Method
- W4A16 (AWQ): Best choice for most use cases. Reduces memory by up to 70% with minimal quality degradation.
- W8A8: Use when you need higher precision or the model is sensitive to quantization.
- FP16: Use when you have sufficient GPU resources and need maximum quality.
2. Optimize Memory Configuration
engineconfig = TurbomindEngineConfig(
cache
maxentrycount=0.8, # Use 80% of free memory for KV cache
quantpolicy=8, # 8-bit KV cache for efficiency
)
Do not set cachemaxentrycount too high as it can cause out-of-memory errors with large batches.
3. Use Tensor Parallelism for Large Models
For 70B+ models, distribute across multiple GPUs:
engineconfig = TurbomindEngineConfig(
tp=4, # 4 GPUs for 70B models
)
4. Monitor Performance
LMDeploy provides a metrics endpoint that can be integrated with Prometheus:
# Metrics available at
curl http://localhost:23333/metrics
5. Set Session Length Wisely
Excessively long session lengths consume more memory:
engineconfig = TurbomindEngineConfig(
sessionlen=2048, # Adjust to actual needs
)
6. Use Continuous Batching
LMDeploy uses continuous batching by default, enabling higher throughput compared to static batching. Make sure maxbatchsize is set according to GPU capacity:
engineconfig = TurbomindEngineConfig(
maxbatchsize=64, # Adjust to GPU capacity
)
7. Benchmark Before Production
Always benchmark before deployment:
lmdeploy serve apiserver internlm/internlm25-7b-chat --server-port 23333 &
Benchmark using built-in tools
pip install openai
python -c "
from openai import OpenAI
import time
client = OpenAI(apikey='none', baseurl='http://localhost:23333/v1')
start = time.time()
for i in range(100):
client.chat.completions.create(
model='internlm/internlm25-7b-chat',
messages=[{'role': 'user', 'content': 'Hi'}],
maxtokens=50
)
elapsed = time.time() - start
print(f'100 requests in {elapsed:.2f}s ({100/elapsed:.1f} req/s)')
"
Supported Models
LMDeploy supports various popular models:
| Model Family | Examples |
|-------------|----------|
| LLaMA | LLaMA-2, LLaMA-3, LLaMA-3.1 |
| Qwen | Qwen-7B, Qwen-14B, Qwen-72B, Qwen2 |
| InternLM | InternLM-7B, InternLM2, InternLM2.5 |
| Mistral | Mistral-7B, Mixtral-8x7B |
| Gemma | Gemma-2B, Gemma-7B |
| ChatGLM | ChatGLM3, GLM-4 |
| Baichuan | Baichuan-7B, Baichuan-13B |
| Yi | Yi-6B, Yi-34B |
| DeepSeek | DeepSeek-V2 |
| Phi | Phi-2, Phi-3 |
For the complete list, visit the official LMDeploy documentation.
Conclusion
LMDeploy is an extremely useful toolkit for anyone who needs to deploy Large Language Models efficiently. With key features such as:
- TurboMind Engine: Fast inference with continuous batching and CUDA kernel optimizations
- AWQ/W8A8 Quantization: Significantly reduces memory usage
- OpenAI-Compatible API Server: Easy integration with existing ecosystems
- Multi-GPU Support: Tensor parallelism for large models
- VLM Support: Supports multimodal models
LMDeploy allows you to run large models on more affordable hardware while maintaining good output quality. The toolkit is suitable for both prototyping and production deployment.
Next steps you can explore: