Complete Tutorial: Fine-tuning LLMs with LoRA and PEFT
Fine-tuning Large Language Models (LLMs) traditionally requires massive GPU memory and long training times. LoRA (Low-Rank Adaptation) and PEFT (Parameter-Efficient Fine-Tuning) enable you to fine-tune with significantly fewer resources while maintaining competitive performance.
What are LoRA and PEFT?
LoRA (Low-Rank Adaptation)
LoRA is a technique that adds trainable low-rank matrices to frozen model layers. Instead of updating all model parameters:
- Freeze all original model weights
- Inject trainable rank decomposition matrices (A and B)
- Only train these new matrices (< 1% of total parameters)
- Memory efficient: ~10x smaller than full fine-tuning
- Faster training: Training is faster
- No inference latency: Weights can be merged
- Task switching: Swap LoRA adapters for different tasks
PEFT (Parameter-Efficient Fine-Tuning)
PEFT is a library from Hugging Face that provides various efficient fine-tuning methods:
- LoRA: Low-rank adaptation
- QLoRA: LoRA with quantization
- Prefix Tuning: Prepend trainable tokens
- Prompt Tuning: Learn soft prompts
- IA3: Infused Adapter by Inhibiting and Amplifying
Installation
# Install packages
pip install transformers datasets accelerate peft bitsandbytes
pip install trl # For training utilities
pip install wandb # Optional: experiment tracking
For QLoRA (4-bit quantization)
pip install bitsandbytes>=0.41.0
Verify GPU
python -c "import torch; print(torch.cuda.isavailable())"
Quick Start: Fine-tune with LoRA
1. Basic LoRA Fine-tuning
from transformers import AutoModelForCausalLM, AutoTokenizer, TrainingArguments
from peft import LoraConfig, getpeftmodel, TaskType
from datasets import loaddataset
from trl import SFTTrainer
Load base model and tokenizer
modelname = "meta-llama/Llama-2-7b-hf" # Or another model
tokenizer = AutoTokenizer.frompretrained(modelname)
tokenizer.padtoken = tokenizer.eostoken
model = AutoModelForCausalLM.frompretrained(
modelname,
torchdtype="auto",
devicemap="auto"
)
LoRA configuration
loraconfig = LoraConfig(
r=8, # Rank of the update matrices
loraalpha=32, # Scaling factor
targetmodules=["qproj", "vproj"], # Modules to apply LoRA
loradropout=0.05,
bias="none",
tasktype=TaskType.CAUSALLM
)
Apply LoRA to model
model = getpeftmodel(model, loraconfig)
model.printtrainableparameters()
Output: trainable params: 4,194,304 || all params: 6,742,609,920 || trainable%: 0.06%
Load dataset
dataset = loaddataset("timdettmers/openassistant-guanaco", split="train")
Training arguments
trainingargs = TrainingArguments(
outputdir="./lora-output",
numtrainepochs=3,
perdevicetrainbatchsize=4,
gradientaccumulationsteps=4,
learningrate=2e-4,
fp16=True,
loggingsteps=10,
savestrategy="epoch",
warmupratio=0.03,
)
Trainer
trainer = SFTTrainer(
model=model,
args=trainingargs,
traindataset=dataset,
tokenizer=tokenizer,
datasettextfield="text",
maxseqlength=512,
)
Train
trainer.train()
Save LoRA weights
model.savepretrained("./lora-adapter")
2. LoRA Config Parameters
from peft import LoraConfig
config = LoraConfig(
# Core parameters
r=8, # Rank: dimension of low-rank matrices
# Higher = more expressive but more params
# Typical: 4, 8, 16, 32, 64
loraalpha=32, # Scaling factor
# Rule: alpha = 2 r for optimal results
# Scaling = alpha / r
# Target modules (varies by model architecture)
targetmodules=[
"qproj", # Query projection
"kproj", # Key projection
"vproj", # Value projection
"oproj", # Output projection
"gateproj", # MLP gate
"upproj", # MLP up
"downproj", # MLP down
],
loradropout=0.05, # Dropout for regularization
bias="none", # "none", "all", "loraonly"
tasktype=TaskType.CAUSALLM, # Or SEQCLS, SEQ2SEQLM, TOKENCLS
# Advanced options
modulestosave=["embedtokens", "lmhead"], # Modules to train fully
layerstotransform=None, # Specific layers to apply LoRA
layerspattern=None, # Pattern for layer selection
)
QLoRA: 4-bit Quantization + LoRA
QLoRA enables fine-tuning large models with minimal memory by using 4-bit quantization.
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import LoraConfig, getpeftmodel, preparemodelforkbittraining
import torch
Quantization config
bnbconfig = BitsAndBytesConfig(
loadin4bit=True,
bnb4bitquanttype="nf4", # Normal Float 4-bit
bnb4bitcomputedtype=torch.bfloat16,
bnb4bitusedoublequant=True, # Nested quantization
)
Load model with 4-bit quantization
modelname = "meta-llama/Llama-2-7b-hf"
model = AutoModelForCausalLM.frompretrained(
modelname,
quantizationconfig=bnbconfig,
devicemap="auto",
trustremotecode=True,
)
tokenizer = AutoTokenizer.frompretrained(modelname)
tokenizer.padtoken = tokenizer.eostoken
Prepare model for k-bit training
model = preparemodelforkbittraining(model)
LoRA config
loraconfig = LoraConfig(
r=16,
loraalpha=32,
targetmodules=[
"qproj", "kproj", "vproj", "oproj",
"gateproj", "upproj", "downproj"
],
loradropout=0.05,
bias="none",
tasktype="CAUSALLM"
)
Apply LoRA
model = getpeftmodel(model, loraconfig)
model.printtrainableparameters()
Dataset Preparation
1. Instruction Dataset Format
from datasets import Dataset
Format: instruction-input-output
data = [
{
"instruction": "Translate to Spanish",
"input": "Hello, how are you?",
"output": "Hola, como estas?"
},
{
"instruction": "Explain the following concept",
"input": "Machine Learning",
"output": "Machine Learning is a branch of AI..."
}
]
Create formatting function
def formatinstruction(sample):
return f"""### Instruction:
{sample['instruction']}
Input:
{sample['input']}
Response:
{sample['output']}"""
dataset = Dataset.fromlist(data)
dataset = dataset.map(lambda x: {"text": formatinstruction(x)})
2. Chat Format (ChatML)
def formatchat(sample):
messages = sample.get("messages", [])
text = ""
for msg in messages:
role = msg["role"]
content = msg["content"]
if role == "system":
text += f"<|system|>\n{content}\n"
elif role == "user":
text += f"<|user|>\n{content}\n"
elif role == "assistant":
text += f"<|assistant|>\n{content}\n"
return {"text": text}
Data format
data = [
{
"messages": [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "What is Python?"},
{"role": "assistant", "content": "Python is a programming language..."}
]
}
]
3. Custom Dataset from File
import json
from datasets import Dataset
Load from JSON
with open("trainingdata.json", "r") as f:
data = json.load(f)
Or from JSONL
data = []
with open("trainingdata.jsonl", "r") as f:
for line in f:
data.append(json.loads(line))
Create dataset
dataset = Dataset.fromlist(data)
Split train/val
dataset = dataset.traintestsplit(testsize=0.1)
traindataset = dataset["train"]
evaldataset = dataset["test"]
Training with SFTTrainer
from transformers import TrainingArguments
from trl import SFTTrainer, DataCollatorForCompletionOnlyLM
Data collator for completion-only training
responsetemplate = "### Response:"
collator = DataCollatorForCompletionOnlyLM(
responsetemplate=responsetemplate,
tokenizer=tokenizer
)
Training arguments
trainingargs = TrainingArguments(
outputdir="./output",
# Training hyperparameters
numtrainepochs=3,
perdevicetrainbatchsize=4,
perdeviceevalbatchsize=4,
gradientaccumulationsteps=4,
gradientcheckpointing=True, # Memory saving
# Optimizer
learningrate=2e-4,
weightdecay=0.001,
warmupratio=0.03,
lrschedulertype="cosine",
# Precision
fp16=True, # Or bf16=True for Ampere+
# Logging
loggingsteps=10,
loggingdir="./logs",
# Saving
savestrategy="steps",
savesteps=100,
savetotallimit=3,
# Evaluation
evalstrategy="steps",
evalsteps=100,
# Misc
reportto="wandb", # Or "tensorboard"
runname="lora-experiment",
)
Trainer
trainer = SFTTrainer(
model=model,
args=trainingargs,
traindataset=traindataset,
evaldataset=evaldataset,
tokenizer=tokenizer,
datacollator=collator,
maxseqlength=1024,
packing=False, # True for packing multiple samples
)
Train
trainer.train()
Save
trainer.savemodel("./final-model")
model.savepretrained("./lora-adapter")
tokenizer.savepretrained("./lora-adapter")
Inference with LoRA
1. Load and Merge Adapter
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
Load base model
basemodel = AutoModelForCausalLM.frompretrained(
"meta-llama/Llama-2-7b-hf",
torchdtype="auto",
devicemap="auto"
)
Load LoRA adapter
model = PeftModel.frompretrained(basemodel, "./lora-adapter")
Merge weights (optional, for faster inference)
model = model.mergeandunload()
Save merged model
model.savepretrained("./merged-model")
2. Inference
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
Load model (merged or with adapter)
model = AutoModelForCausalLM.frompretrained(
"./merged-model",
torchdtype=torch.float16,
devicemap="auto"
)
tokenizer = AutoTokenizer.frompretrained("./merged-model")
def generateresponse(prompt, maxlength=512):
inputs = tokenizer(prompt, returntensors="pt").to(model.device)
with torch.nograd():
outputs = model.generate(
inputs,
maxnewtokens=maxlength,
temperature=0.7,
topp=0.9,
dosample=True,
padtokenid=tokenizer.eostokenid
)
response = tokenizer.decode(outputs[0], skipspecialtokens=True)
return response[len(prompt):]
Test
prompt = """### Instruction:
Explain what machine learning is
Response:
"""
response = generateresponse(prompt)
print(response)
3. Multiple LoRA Adapters
from peft import PeftModel
Load base model
basemodel = AutoModelForCausalLM.frompretrained("base-model")
Load adapter 1
model = PeftModel.frompretrained(basemodel, "./adapter-task1", adaptername="task1")
Load adapter 2
model.loadadapter("./adapter-task2", adaptername="task2")
Switch between adapters
model.setadapter("task1")
output1 = model.generate(...)
model.setadapter("task2")
output2 = model.generate(...)
Disable adapter (use base model)
with model.disableadapter():
baseoutput = model.generate(...)
Fine-tuning for Specific Tasks
1. Text Classification
from transformers import AutoModelForSequenceClassification
from peft import LoraConfig, getpeftmodel, TaskType
model = AutoModelForSequenceClassification.frompretrained(
"bert-base-uncased",
numlabels=3
)
loraconfig = LoraConfig(
r=8,
loraalpha=16,
targetmodules=["query", "value"],
loradropout=0.1,
bias="none",
tasktype=TaskType.SEQCLS
)
model = getpeftmodel(model, loraconfig)
2. Question Answering
from transformers import AutoModelForQuestionAnswering
from peft import LoraConfig, getpeftmodel, TaskType
model = AutoModelForQuestionAnswering.frompretrained("bert-base-uncased")
loraconfig = LoraConfig(
r=16,
loraalpha=32,
targetmodules=["query", "key", "value"],
tasktype=TaskType.QUESTIONANS
)
model = getpeftmodel(model, loraconfig)
3. Summarization
from transformers import AutoModelForSeq2SeqLM
from peft import LoraConfig, getpeftmodel, TaskType
model = AutoModelForSeq2SeqLM.frompretrained("t5-base")
loraconfig = LoraConfig(
r=8,
loraalpha=16,
targetmodules=["q", "v"],
tasktype=TaskType.SEQ2SEQLM
)
model = getpeftmodel(model, loraconfig)
Tips and Best Practices
1. Hyperparameter Tuning
# Start with this config
loraconfig = LoraConfig(
r=8, # Start small, increase if underfitting
loraalpha=16, # 2 r
targetmodules=["qproj", "vproj"], # Minimal
loradropout=0.05,
)
For complex tasks
loraconfig = LoraConfig(
r=32,
loraalpha=64,
targetmodules=[
"qproj", "kproj", "vproj", "oproj",
"gateproj", "upproj", "downproj"
],
loradropout=0.1,
)
2. Memory Optimization
# Gradient checkpointing
trainingargs = TrainingArguments(
gradientcheckpointing=True,
...
)
Mixed precision
trainingargs = TrainingArguments(
fp16=True, # Or bf16=True
...
)
Gradient accumulation
trainingargs = TrainingArguments(
perdevicetrainbatchsize=1,
gradientaccumulationsteps=16, # Effective batch = 16
...
)
3. Debugging
# Check trainable parameters
model.printtrainableparameters()
Check LoRA modules
for name, param in model.namedparameters():
if param.requiresgrad:
print(f"{name}: {param.shape}")
Verify adapter loaded
print(model.peftconfig)
print(model.activeadapter)
4. Common Issues
# Issue: OOM (Out of Memory)
Solution: Reduce batch size, use gradient checkpointing
Issue: Loss not decreasing
Solution: Check learning rate, increase r, add more target modules
Issue: Overfitting
Solution: Increase dropout, reduce epochs, use more data
Issue: Catastrophic forgetting
Solution: Lower learning rate, use fewer epochs
Conclusion
LoRA and PEFT enable LLM fine-tuning with minimal resources:
| Method | Memory | Speed | Quality |
|--------|--------|-------|---------|
| Full Fine-tune | 100% | Slow | Best |
| LoRA | ~10% | Fast | Very Good |
| QLoRA | ~5% | Medium | Good |
Key takeaways: