LitGPT Tutorial: A Framework for Training and Fine-Tuning Large Language Models

LitGPT is an open-source framework from Lightning AI designed specifically for pretraining, fine-tuning, and deploying Large Language Models (LLMs). Built on top of PyTorch Lightning, LitGPT provides clean, modular reference implementations for various popular LLM architectures such as Llama, Mistral, Phi, Gemma, and more. This framework has become a go-to choice for practitioners who want to train or customize language models without writing boilerplate code from scratch.

In this tutorial, we will walk through how to use LitGPT from installation, model downloading, fine-tuning with custom datasets, to deployment for inference. Every step includes code examples that you can run directly.

Why LitGPT?

Before diving into implementation, it is important to understand the advantages of LitGPT compared to other approaches:

• Multi-architecture: Supports 20+ LLM architectures including Llama 3, Mistral, Phi-3, Gemma 2, Falcon, and others within a single consistent framework
• Built-in optimizations: Flash Attention 2, quantization (QLoRA, GPTQ), gradient checkpointing, and FSDP are available without complex configuration
• Intuitive CLI: Common operations like download, finetune, and chat are available as simple CLI commands
• Modular and hackable: Clean, well-documented source code makes it easy to modify for research or production needs
• Checkpoint conversion: Supports conversion from/to Hugging Face format, making integration with the existing ecosystem straightforward

Installation

Prerequisites

Ensure your system meets the following requirements:

• Python 3.10 or newer
• CUDA 12.1+ (for NVIDIA GPUs) or MPS support (for Apple Silicon)
• Minimum 16GB RAM (32GB+ recommended)
• GPU with at least 8GB VRAM for fine-tuning (24GB+ recommended for 7B models)

Installation via pip

pip install 'litgpt[all]'

The [all] option installs all dependencies including tokenizers, datasets, and additional utilities. If you only need specific features:

# Minimal installation
pip install litgpt

With quantization support
pip install 'litgpt[quantize]'

Installation from Source

For development or access to the latest features:

git clone https://github.com/Lightning-AI/litgpt.git
cd litgpt
pip install -e '.[all]'

Verify Installation

import litgpt
print(litgpt.version)

Check available models
import subprocess
result = subprocess.run(['litgpt', 'download', 'list'], captureoutput=True, text=True)

print(result.stdout[:500])

# Download Llama 3.1 8B
litgpt download meta-llama/Meta-Llama-3.1-8B-Instruct

Download Mistral 7B
litgpt download mistralai/Mistral-7B-Instruct-v0.3

Download a small model for testing
litgpt download microsoft/phi-2

Models are saved to the checkpoints/ directory by default. You can change the location with the --checkpointdir flag:

litgpt download meta-llama/Meta-Llama-3.1-8B-Instruct \
    --checkpointdir ./mymodels/

Interactive Chat

Once the model is downloaded, you can start chatting immediately:

litgpt chat meta-llama/Meta-Llama-3.1-8B-Instruct

Or with additional options:

litgpt chat meta-llama/Meta-Llama-3.1-8B-Instruct \
    --maxnewtokens 512 \
    --temperature 0.7 \
    --topp 0.9 \

    --quantize bnb.nf4  # Quantize to save VRAM

from litgpt import LLM

Load model
llm = LLM.load("meta-llama/Meta-Llama-3.1-8B-Instruct")

Generate text
response = llm.generate("Explain what machine learning is in 3 sentences.")
print(response)

Generate with custom parameters
response = llm.generate(
    prompt="Write a Python function to calculate fibonacci:",
    maxnewtokens=256,

    temperature=0.3,
    topk=50
)
print(response)

For memory-efficient usage with quantization:

from litgpt import LLM

Load with 4-bit quantization
llm = LLM.load(
    "meta-llama/Meta-Llama-3.1-8B-Instruct",
    quantize="bnb.nf4"
)

response = llm.generate("What are the benefits of using Docker?")
print(response)

Fine-Tuning with Custom Datasets

Preparing the Dataset

LitGPT supports several dataset formats. The most common format is JSON with an instruction-response structure:

[
    {
        "instruction": "Explain what a REST API is",
        "input": "",
        "output": "A REST API (Representational State Transfer Application Programming Interface) is a communication architecture that uses the HTTP protocol for data exchange between systems. REST APIs use standard HTTP methods such as GET, POST, PUT, and DELETE for CRUD operations. Data is typically sent in JSON or XML format."
    },
    {
        "instruction": "Convert the following code to Python 3",
        "input": "print 'Hello World'",
        "output": "print('Hello World')"
    }
]

Save this file as mydataset.json. Then create a script to prepare the data:

import json
from pathlib import Path

def preparedataset(inputfile, outputdir):
    """Prepare dataset for LitGPT fine-tuning."""
    outputdir = Path(outputdir)
    outputdir.mkdir(parents=True, existok=True)

    with open(inputfile, 'r', encoding='utf-8') as f:

        data = json.load(f)

    # Split 90% train, 10% validation
    splitidx = int(len(data)  0.9)

    traindata = data[:splitidx]
    valdata = data[splitidx:]

    with open(outputdir / "train.json", 'w', encoding='utf-8') as f:

        json.dump(traindata, f, ensureascii=False, indent=2)


    with open(outputdir / "val.json", 'w', encoding='utf-8') as f:
        json.dump(valdata, f, ensureascii=False, indent=2)

    print(f"Train samples: {len(traindata)}")

    print(f"Validation samples: {len(valdata)}")

preparedataset("mydataset.json", "data/mycustomdata")

litgpt finetune lora \
    --checkpointdir checkpoints/meta-llama/Meta-Llama-3.1-8B-Instruct \

    --data JSON \
    --data.jsonpath data/mycustomdata/ \
    --train.epochs 3 \
    --train.lrwarmupsteps 100 \
    --train.learningrate 3e-4 \

    --train.microbatchsize 4 \

    --train.maxseqlength 1024 \

    --lorar 16 \
    --loraalpha 32 \

    --loradropout 0.05 \
    --outdir out/my-finetuned-model

litgpt finetune lora \
    --checkpointdir checkpoints/meta-llama/Meta-Llama-3.1-8B-Instruct \

    --data JSON \
    --data.jsonpath data/mycustomdata/ \
    --quantize bnb.nf4 \
    --train.epochs 3 \
    --train.learningrate 2e-4 \

    --train.microbatchsize 2 \

    --lorar 8 \
    --loraalpha 16 \

    --precision bf16-true \
    --outdir out/my-qlora-model

litgpt finetune full \
    --checkpointdir checkpoints/microsoft/phi-2 \

    --data JSON \
    --data.jsonpath data/mycustomdata/ \
    --train.epochs 2 \
    --train.learningrate 2e-5 \

    --train.microbatchsize 1 \

    --train.maxseqlength 1024 \

    --outdir out/phi2-full-finetune

# Launch TensorBoard
tensorboard --logdir out/my-finetuned-model/logs

Access in browser: http://localhost:6006

litgpt finetune lora \
    --checkpointdir checkpoints/meta-llama/Meta-Llama-3.1-8B-Instruct \

    --data JSON \
    --data.jsonpath data/mycustomdata/ \
    --devices 4 \
    --train.epochs 3 \
    --train.learningrate 3e-4 \

    --lorar 16 \
    --loraalpha 32 \

    --outdir out/multi-gpu-model

litgpt pretrain \
    --modelname tiny-llama-1.1b \

    --data TextFiles \
    --data.path data/mycorpus/ \
    --train.maxsteps 50000 \

    --train.learningrate 4e-4 \
    --train.microbatchsize 8 \
    --train.maxseqlength 2048 \
    --devices 4 \
    --outdir out/my-pretrained-model

# LitGPT to Hugging Face
from litgpt.scripts.convertlitcheckpoint import convertlitcheckpoint


convertlitcheckpoint(

    checkpointdir="out/my-finetuned-model/final",
    outputdir="out/my-finetuned-model/hfformat"
)

# LitGPT -> Hugging Face format
litgpt convertfromlitgpt \
    --checkpointdir out/my-finetuned-model/final \

    --outputdir out/hfmodel


Hugging Face -> LitGPT format
litgpt converttolitgpt \

    --checkpointdir path/to/hfmodel \

    --outputdir checkpoints/myconvertedmodel

import litserve as ls
from litgpt import LLM

class LitGPTAPI(ls.LitAPI):
    def setup(self, device):
        self.llm = LLM.load(
            "out/my-finetuned-model/final",
            quantize="bnb.nf4"
        )

    def decoderequest(self, request):

        return request["prompt"]

    def predict(self, prompt):
        return self.llm.generate(
            prompt,
            maxnewtokens=512,

            temperature=0.7
        )

    def encoderesponse(self, output):
        return {"response": output}

if name == "main":
    api = LitGPTAPI()
    server = ls.LitServer(api, accelerator="gpu", devices=1)
    server.run(port=8000)

curl -X POST http://localhost:8000/predict \
    -H "Content-Type: application/json" \
    -d '{"prompt": "Explain the concept of microservices:"}'

# Install evaluation harness
pip install lm-eval

Convert to HF format first
litgpt convertfromlitgpt \
    --checkpointdir out/my-finetuned-model/final \

    --outputdir out/hfevalmodel

Run evaluation
lmeval --model hf \

    --modelargs pretrained=out/hfevalmodel \
    --tasks hellaswag,arceasy,mmlu \

    --batchsize 4

# Estimate VRAM requirements
def estimatevram(modelparamsbillion, method="qlora"):

    if method == "full":
        return modelparamsbillion  6  # ~6GB per 1B params (mixed precision)
    elif method == "lora":
        return modelparamsbillion  2  # ~2GB per 1B params

    elif method == "qlora":
        return modelparamsbillion  1.2  # ~1.2GB per 1B params

print(f"Llama 8B LoRA: ~{estimatevram(8, 'lora')}GB VRAM")
print(f"Llama 8B QLoRA: ~{estimatevram(8, 'qlora')}GB VRAM")

print(f"Llama 8B Full: ~{estimatevram(8, 'full')}GB VRAM")

3. Dataset Quality

• Ensure your dataset is clean and consistent in format
• Minimum 100-500 samples for task-specific fine-tuning with LoRA
• Use deduplication to avoid overfitting
• Validate JSON format before training

4. Hyperparameter Tuning

Start with conservative settings and adjust:

# Safe starting configuration
learningrate: 2e-4

epochs: 3
microbatchsize: 4

lorar: 8
loraalpha: 16

warmupsteps: 100
weight_decay: 0.01

5. Checkpoint Management

# LitGPT automatically saves checkpoints
Output structure:
out/my-model/
├── final/           # Final checkpoint
├── step-1000/       # Intermediate checkpoints
├── step-2000/
├── logs/            # TensorBoard logs
└── hyperparameters.yaml

Conclusion

LitGPT simplifies the LLM workflow from download to deployment. Its main strengths lie in extensive multi-architecture support, an intuitive CLI, and integrated implementation of optimization techniques like LoRA/QLoRA. This framework is well-suited for practitioners who want to focus on experimentation and results without getting bogged down in low-level implementation details.

Next steps you can explore:

Complete documentation and additional examples are available in the official LitGPT repository on GitHub.