LMDeploy Tutorial: A Complete Toolkit for Deploying and Serving Large Language Models

# Tutorial LMDeploy: Toolkit Lengkap untuk Deploy dan Serving Large Language Model Large Language Model (LLM) semakin banyak digunakan dalam berbagai aplikasi, mulai dari chatbot hingga asisten kode....

By Ruby Abdullah · · tutorial
LMDeployLLM DeploymentModel QuantizationTurboMindMLOps

LMDeploy Tutorial: A Complete Toolkit for Deploying and Serving Large Language Models

Large Language Models (LLMs) are increasingly used in various applications, from chatbots to code assistants. However, the main challenges in using LLMs are the high GPU resource requirements and inference latency. LMDeploy provides a comprehensive solution for compressing, deploying, and serving LLMs efficiently.

LMDeploy is developed by the InternLM team at Shanghai AI Laboratory. The toolkit supports various popular models such as LLaMA, Qwen, Mistral, InternLM, and many others. With its optimized TurboMind engine and W4A16/W8A8 quantization support, LMDeploy allows you to run large models with fewer resources without significantly sacrificing output quality.

In this tutorial, we will cover installation, basic usage, model quantization, API serving, and best practices for production deployment.

Installation

Prerequisites

Before installing LMDeploy, make sure you have:

  • Python 3.8 or later
  • CUDA 11.8 or 12.x (for GPU acceleration)
  • NVIDIA GPU with at least 8GB VRAM (for quantized 7B models)

Installation via pip

The easiest way to install LMDeploy is using pip:

pip install lmdeploy

For installation with specific CUDA support:

# For CUDA 12.x

pip install lmdeploy

For CUDA 11.8

pip install lmdeploy --extra-index-url https://download.pytorch.org/whl/cu118

Verify Installation

After installation is complete, verify by running:

import lmdeploy

print(lmdeploy.version)

Or via the command line:

lmdeploy version

Installation from Source (Optional)

If you need the latest features from the development branch:

git clone https://github.com/InternLM/lmdeploy.git

cd lmdeploy

pip install -e .

Basic Usage

Interactive Chat via Command Line

The quickest way to try LMDeploy is using the interactive chat mode:

lmdeploy chat internlm/internlm25-7b-chat

This command will download the model from Hugging Face (if not already present) and start an interactive chat session in the terminal. You can directly type questions and get responses from the model.

To use a model from a local path:

lmdeploy chat /path/to/your/model

Using the Pipeline API in Python

For integration into Python applications, LMDeploy provides a simple Pipeline API:

from lmdeploy import pipeline

Initialize the pipeline

pipe = pipeline("internlm/internlm25-7b-chat")

Single prompt inference

response = pipe(["Explain what machine learning is in 3 sentences"])

print(response[0].text)

Batch Inference

One of LMDeploy's strengths is efficient batch processing:

from lmdeploy import pipeline

pipe = pipeline("internlm/internlm25-7b-chat")

prompts = [

"What is a neural network?",

"Explain the difference between supervised and unsupervised learning",

"How do transformers work?",

"What is the purpose of the attention mechanism?"

]

responses = pipe(prompts)

for prompt, resp in zip(prompts, responses):

print(f"Q: {prompt}")

print(f"A: {resp.text}\n")

Generation Configuration

You can control text generation parameters using GenerationConfig:

from lmdeploy import pipeline, GenerationConfig

pipe = pipeline("internlm/internlm25-7b-chat")

genconfig = GenerationConfig(

maxnewtokens=512,

temperature=0.7,

topp=0.9,

topk=40,

repetitionpenalty=1.1

)

response = pipe(

["Write a short poem about artificial intelligence"],

genconfig=genconfig

)

print(response[0].text)

Multi-turn Conversation

LMDeploy supports multi-turn conversations while maintaining context:

from lmdeploy import pipeline, GenerationConfig, ChatTemplateConfig

pipe = pipeline(

"internlm/internlm25-7b-chat",

chattemplateconfig=ChatTemplateConfig(

modelname="internlm2",

system="You are a helpful AI assistant."

)

)

Simulate multi-turn conversation

messages = [

{"role": "user", "content": "Who invented electricity?"},

]

response = pipe([messages])

print(f"Assistant: {response[0].text}")

Continue the conversation

messages.append({"role": "assistant", "content": response[0].text})

messages.append({"role": "user", "content": "What was their greatest contribution?"})

response = pipe([messages])

print(f"Assistant: {response[0].text}")

Model Quantization

Quantization is a technique to reduce model size and speed up inference by lowering the precision of model weights. LMDeploy supports several quantization methods.

W4A16 Quantization (AWQ)

W4A16 means weights are quantized to 4-bit while activations remain at 16-bit. This is the most commonly used method:

lmdeploy lite autoawq \

internlm/internlm25-7b-chat \

--calib-dataset ptb \

--calib-samples 128 \

--calib-seqlen 2048 \

--w-bits 4 \

--w-group-size 128 \

--work-dir ./internlm25-7b-chat-4bit

Parameter explanation:

  • --calib-dataset: Calibration dataset for determining quantization ranges
  • --calib-samples: Number of calibration samples
  • --calib-seqlen: Calibration sequence length
  • --w-bits: Number of bits for weight quantization (4-bit)
  • --w-group-size: Group size for group-wise quantization
  • --work-dir: Output directory for the quantized model

W8A8 Quantization

For more precise quantization with slightly higher overhead:

lmdeploy lite smoothquant \

internlm/internlm25-7b-chat \

--calib-dataset ptb \

--calib-samples 128 \

--calib-seqlen 2048 \

--work-dir ./internlm25-7b-chat-w8a8

Using Quantized Models

After quantization is complete, use the model as usual:

from lmdeploy import pipeline

Use the AWQ 4-bit model

pipe = pipeline("./internlm25-7b-chat-4bit")

response = pipe(["Explain quantum computing simply"])

print(response[0].text)

Memory Usage Comparison

Here is the approximate VRAM usage for a 7B model:

| Precision | VRAM (approx) | Relative Speed |

|-----------|---------------|----------------|

| FP16 | ~14 GB | 1x (baseline) |

| W8A8 | ~8 GB | 1.2-1.5x |

| W4A16 | ~5 GB | 1.5-2x |

KV Cache Quantization

Besides model quantization, LMDeploy also supports KV cache quantization to reduce memory usage during inference:

from lmdeploy import pipeline, TurbomindEngineConfig

engineconfig = TurbomindEngineConfig(

quantpolicy=8 # Quantize KV cache to 8-bit

)

pipe = pipeline(

"internlm/internlm25-7b-chat",

backendconfig=engineconfig

)

response = pipe(["What is a large language model?"])

print(response[0].text)

Advanced Usage

Serving via API Server

LMDeploy provides an API server compatible with the OpenAI API format, making it easy to integrate with applications already using the OpenAI SDK:

lmdeploy serve apiserver \

internlm/internlm25-7b-chat \

--server-name 0.0.0.0 \

--server-port 23333 \

--tp 1

Key parameters:

  • --server-name: Bind address (0.0.0.0 for external access)
  • --server-port: Server port
  • --tp: Tensor parallelism (number of GPUs to use)

Accessing the API Server

Once the server is running, you can access it using the OpenAI SDK:

from openai import OpenAI

client = OpenAI(

apikey="not-needed",

baseurl="http://localhost:23333/v1"

)

Chat completion

response = client.chat.completions.create(

model="internlm/internlm25-7b-chat",

messages=[

{"role": "system", "content": "You are a helpful assistant."},

{"role": "user", "content": "What are the advantages of using LMDeploy?"}

],

temperature=0.7,

maxtokens=512

)

print(response.choices[0].message.content)

Or using curl:

curl http://localhost:23333/v1/chat/completions \

-H "Content-Type: application/json" \

-d '{

"model": "internlm/internlm25-7b-chat",

"messages": [

{"role": "user", "content": "Hello, how are you?"}

],

"temperature": 0.7

}'

Streaming Response

To get streaming responses:

from openai import OpenAI

client = OpenAI(

apikey="not-needed",

baseurl="http://localhost:23333/v1"

)

stream = client.chat.completions.create(

model="internlm/internlm25-7b-chat",

messages=[

{"role": "user", "content": "Tell me a brief history of AI"}

],

stream=True

)

for chunk in stream:

if chunk.choices[0].delta.content:

print(chunk.choices[0].delta.content, end="", flush=True)

print()

Tensor Parallelism (Multi-GPU)

For larger models, you can distribute the model across multiple GPUs:

# Using 2 GPUs

lmdeploy serve apiserver \

meta-llama/Meta-Llama-3-70B-Instruct \

--tp 2 \

--server-port 23333

In Python:

from lmdeploy import pipeline, TurbomindEngineConfig

engineconfig = TurbomindEngineConfig(tp=2)

pipe = pipeline(

"meta-llama/Meta-Llama-3-70B-Instruct",

backendconfig=engineconfig

)

Vision Language Model (VLM) Support

LMDeploy also supports multimodal models that can process images:

from lmdeploy import pipeline

from lmdeploy.vl import loadimage

pipe = pipeline("OpenGVLab/InternVL2-8B")

image = loadimage("https://example.com/image.jpg")

response = pipe([("Describe this image", image)])

print(response[0].text)

For multiple images:

from lmdeploy import pipeline

from lmdeploy.vl import loadimage

pipe = pipeline("OpenGVLab/InternVL2-8B")

image1 = loadimage("image1.jpg")

image2 = loadimage("image2.jpg")

response = pipe([("Compare these two images", [image1, image2])])

print(response[0].text)

TurboMind Engine Configuration

TurboMind is LMDeploy's core inference engine. Here is the full configuration:

from lmdeploy import pipeline, TurbomindEngineConfig

engineconfig = TurbomindEngineConfig(

tp=1, # Tensor parallelism

sessionlen=4096, # Maximum context length

maxbatchsize=128, # Maximum batch size

cachemaxentrycount=0.8, # GPU memory proportion for KV cache

quantpolicy=0, # 0: no KV quant, 4: 4-bit, 8: 8-bit

ropescalingfactor=1.0, # RoPE scaling factor

)

pipe = pipeline(

"internlm/internlm25-7b-chat",

backendconfig=engineconfig

)

PyTorch Backend

Besides TurboMind, LMDeploy also supports a PyTorch backend for broader compatibility:

from lmdeploy import pipeline, PytorchEngineConfig

engineconfig = PytorchEngineConfig(

tp=1,

sessionlen=4096,

maxbatchsize=64,

cachemaxentrycount=0.8

)

pipe = pipeline(

"internlm/internlm25-7b-chat",

backendconfig=engineconfig

)

The PyTorch backend is useful when:

  • The model is not yet supported by TurboMind
  • You need specific PyTorch features
  • Debugging and development

Docker Deployment

Running with Docker

LMDeploy provides an official Docker image:

docker run --gpus all \

-v ~/.cache/huggingface:/root/.cache/huggingface \

-p 23333:23333 \

--ipc=host \

openmmlab/lmdeploy:latest \

lmdeploy serve apiserver \

internlm/internlm25-7b-chat \

--server-port 23333

Docker Compose

For a more structured setup:

version: '3.8'

services:

lmdeploy:

image: openmmlab/lmdeploy:latest

deploy:

resources:

reservations:

devices:

  • driver: nvidia
count: 1

capabilities: [gpu]

ports:

  • "23333:23333"
volumes:

  • huggingface-cache:/root/.cache/huggingface
ipc: host

command: >

lmdeploy serve apiserver

internlm/internlm25-7b-chat

--server-port 23333

--server-name 0.0.0.0

volumes:

huggingface-cache:

Best Practices

1. Choose the Right Quantization Method

  • W4A16 (AWQ): Best choice for most use cases. Reduces memory by up to 70% with minimal quality degradation.
  • W8A8: Use when you need higher precision or the model is sensitive to quantization.
  • FP16: Use when you have sufficient GPU resources and need maximum quality.

2. Optimize Memory Configuration

engineconfig = TurbomindEngineConfig(

cachemaxentrycount=0.8, # Use 80% of free memory for KV cache

quantpolicy=8, # 8-bit KV cache for efficiency

)

Do not set cachemaxentrycount too high as it can cause out-of-memory errors with large batches.

3. Use Tensor Parallelism for Large Models

For 70B+ models, distribute across multiple GPUs:

engineconfig = TurbomindEngineConfig(

tp=4, # 4 GPUs for 70B models

)

4. Monitor Performance

LMDeploy provides a metrics endpoint that can be integrated with Prometheus:

# Metrics available at

curl http://localhost:23333/metrics

5. Set Session Length Wisely

Excessively long session lengths consume more memory:

engineconfig = TurbomindEngineConfig(

sessionlen=2048, # Adjust to actual needs

)

6. Use Continuous Batching

LMDeploy uses continuous batching by default, enabling higher throughput compared to static batching. Make sure maxbatchsize is set according to GPU capacity:

engineconfig = TurbomindEngineConfig(

maxbatchsize=64, # Adjust to GPU capacity

)

7. Benchmark Before Production

Always benchmark before deployment:

lmdeploy serve apiserver internlm/internlm25-7b-chat --server-port 23333 &

Benchmark using built-in tools

pip install openai

python -c "

from openai import OpenAI

import time

client = OpenAI(apikey='none', baseurl='http://localhost:23333/v1')

start = time.time()

for i in range(100):

client.chat.completions.create(

model='internlm/internlm25-7b-chat',

messages=[{'role': 'user', 'content': 'Hi'}],

maxtokens=50

)

elapsed = time.time() - start

print(f'100 requests in {elapsed:.2f}s ({100/elapsed:.1f} req/s)')

"

Supported Models

LMDeploy supports various popular models:

| Model Family | Examples |

|-------------|----------|

| LLaMA | LLaMA-2, LLaMA-3, LLaMA-3.1 |

| Qwen | Qwen-7B, Qwen-14B, Qwen-72B, Qwen2 |

| InternLM | InternLM-7B, InternLM2, InternLM2.5 |

| Mistral | Mistral-7B, Mixtral-8x7B |

| Gemma | Gemma-2B, Gemma-7B |

| ChatGLM | ChatGLM3, GLM-4 |

| Baichuan | Baichuan-7B, Baichuan-13B |

| Yi | Yi-6B, Yi-34B |

| DeepSeek | DeepSeek-V2 |

| Phi | Phi-2, Phi-3 |

For the complete list, visit the official LMDeploy documentation.

Conclusion

LMDeploy is an extremely useful toolkit for anyone who needs to deploy Large Language Models efficiently. With key features such as:

  • TurboMind Engine: Fast inference with continuous batching and CUDA kernel optimizations
  • AWQ/W8A8 Quantization: Significantly reduces memory usage
  • OpenAI-Compatible API Server: Easy integration with existing ecosystems
  • Multi-GPU Support: Tensor parallelism for large models
  • VLM Support: Supports multimodal models

LMDeploy allows you to run large models on more affordable hardware while maintaining good output quality. The toolkit is suitable for both prototyping and production deployment.

Next steps you can explore:

  • Try quantizing your favorite model and compare the results
  • Set up an API server for your team
  • Integrate with your applications using the OpenAI SDK
  • Explore VLM support for multimodal use cases
  • Benchmark and optimize for your production workload
  • Related Articles

    TorchServe: Deploying PyTorch Models to Production

    TorchServe: Panduan Lengkap Deployment Model PyTorch ke Production Penulis: Ruby Abdullah Kategori: Tutorial Tags: PyTor...

    MLflow Tutorial: Open-Source Platform for End-to-End MLOps

    Tutorial MLflow: Platform Open-Source untuk MLOps End-to-End MLflow adalah platform open-source yang dikembangkan oleh D...

    Complete Prompt Flow Tutorial: Microsoft Framework for Building LLM Applications

    Tutorial Lengkap Prompt Flow: Framework Microsoft untuk Membangun Aplikasi LLM Prompt Flow adalah framework open-source ...

    Truss: How to Package, Serve, and Deploy ML Models Cleanly

    Truss: Cara Package, Serve, dan Deploy Model ML dengan Rapi Halo temen-temen, ketemu lagi sama aku, Ruby. Kali ini aku p...