Tutorial Lengkap Fine-tuning LLM dengan LoRA dan PEFT
Fine-tuning Large Language Models (LLM) secara tradisional membutuhkan GPU memory yang sangat besar dan waktu yang lama. LoRA (Low-Rank Adaptation) dan PEFT (Parameter-Efficient Fine-Tuning) memungkinkan Anda melakukan fine-tuning dengan resource yang jauh lebih kecil sambil mempertahankan performa yang kompetitif.
Apa itu LoRA dan PEFT?
LoRA (Low-Rank Adaptation)
LoRA adalah teknik yang menambahkan trainable low-rank matrices ke layer-layer model yang di-freeze. Daripada mengupdate semua parameter model:
- Freeze semua weights original model
- Inject trainable rank decomposition matrices (A dan B)
- Hanya train matrices baru ini (< 1% dari total parameters)
- Memory efficient: ~10x lebih kecil dari full fine-tuning
- Faster training: Training lebih cepat
- No inference latency: Weights bisa di-merge
- Task switching: Swap LoRA adapters untuk tasks berbeda
PEFT (Parameter-Efficient Fine-Tuning)
PEFT adalah library dari Hugging Face yang menyediakan berbagai metode fine-tuning efisien:
- LoRA: Low-rank adaptation
- QLoRA: LoRA dengan quantization
- Prefix Tuning: Prepend trainable tokens
- Prompt Tuning: Learn soft prompts
- IA3: Infused Adapter by Inhibiting and Amplifying
Instalasi
# Install packages
pip install transformers datasets accelerate peft bitsandbytes
pip install trl # Untuk training utilities
pip install wandb # Optional: experiment tracking
Untuk QLoRA (4-bit quantization)
pip install bitsandbytes>=0.41.0
Verify GPU
python -c "import torch; print(torch.cuda.isavailable())"
Quick Start: Fine-tune dengan LoRA
1. Basic LoRA Fine-tuning
from transformers import AutoModelForCausalLM, AutoTokenizer, TrainingArguments
from peft import LoraConfig, getpeftmodel, TaskType
from datasets import loaddataset
from trl import SFTTrainer
Load base model dan tokenizer
modelname = "meta-llama/Llama-2-7b-hf" # Atau model lain
tokenizer = AutoTokenizer.frompretrained(modelname)
tokenizer.padtoken = tokenizer.eostoken
model = AutoModelForCausalLM.frompretrained(
modelname,
torchdtype="auto",
devicemap="auto"
)
Konfigurasi LoRA
loraconfig = LoraConfig(
r=8, # Rank of the update matrices
loraalpha=32, # Scaling factor
targetmodules=["qproj", "vproj"], # Modules to apply LoRA
loradropout=0.05,
bias="none",
tasktype=TaskType.CAUSALLM
)
Apply LoRA ke model
model = getpeftmodel(model, loraconfig)
model.printtrainableparameters()
Output: trainable params: 4,194,304 || all params: 6,742,609,920 || trainable%: 0.06%
Load dataset
dataset = loaddataset("timdettmers/openassistant-guanaco", split="train")
Training arguments
trainingargs = TrainingArguments(
outputdir="./lora-output",
numtrainepochs=3,
perdevicetrainbatchsize=4,
gradientaccumulationsteps=4,
learningrate=2e-4,
fp16=True,
loggingsteps=10,
savestrategy="epoch",
warmupratio=0.03,
)
Trainer
trainer = SFTTrainer(
model=model,
args=trainingargs,
traindataset=dataset,
tokenizer=tokenizer,
datasettextfield="text",
maxseqlength=512,
)
Train
trainer.train()
Save LoRA weights
model.savepretrained("./lora-adapter")
2. LoRA Config Parameters
from peft import LoraConfig
config = LoraConfig(
# Core parameters
r=8, # Rank: dimensi dari low-rank matrices
# Lebih tinggi = lebih ekspresif tapi lebih banyak params
# Typical: 4, 8, 16, 32, 64
loraalpha=32, # Scaling factor
# Rule: alpha = 2 r untuk hasil optimal
# Scaling = alpha / r
# Target modules (varies by model architecture)
targetmodules=[
"qproj", # Query projection
"kproj", # Key projection
"vproj", # Value projection
"oproj", # Output projection
"gateproj", # MLP gate
"upproj", # MLP up
"downproj", # MLP down
],
loradropout=0.05, # Dropout untuk regularization
bias="none", # "none", "all", "loraonly"
tasktype=TaskType.CAUSALLM, # Atau SEQCLS, SEQ2SEQLM, TOKENCLS
# Advanced options
modulestosave=["embedtokens", "lmhead"], # Modules to train fully
layerstotransform=None, # Specific layers to apply LoRA
layerspattern=None, # Pattern untuk layer selection
)
QLoRA: 4-bit Quantization + LoRA
QLoRA memungkinkan fine-tuning model besar dengan memory sangat kecil dengan menggunakan 4-bit quantization.
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import LoraConfig, getpeftmodel, preparemodelforkbittraining
import torch
Quantization config
bnbconfig = BitsAndBytesConfig(
loadin4bit=True,
bnb4bitquanttype="nf4", # Normal Float 4-bit
bnb4bitcomputedtype=torch.bfloat16,
bnb4bitusedoublequant=True, # Nested quantization
)
Load model dengan 4-bit quantization
modelname = "meta-llama/Llama-2-7b-hf"
model = AutoModelForCausalLM.frompretrained(
modelname,
quantizationconfig=bnbconfig,
devicemap="auto",
trustremotecode=True,
)
tokenizer = AutoTokenizer.frompretrained(modelname)
tokenizer.padtoken = tokenizer.eostoken
Prepare model untuk k-bit training
model = preparemodelforkbittraining(model)
LoRA config
loraconfig = LoraConfig(
r=16,
loraalpha=32,
targetmodules=[
"qproj", "kproj", "vproj", "oproj",
"gateproj", "upproj", "downproj"
],
loradropout=0.05,
bias="none",
tasktype="CAUSALLM"
)
Apply LoRA
model = getpeftmodel(model, loraconfig)
model.printtrainableparameters()
Dataset Preparation
1. Instruction Dataset Format
from datasets import Dataset
Format: instruction-input-output
data = [
{
"instruction": "Terjemahkan ke bahasa Indonesia",
"input": "Hello, how are you?",
"output": "Halo, apa kabar?"
},
{
"instruction": "Jelaskan konsep berikut",
"input": "Machine Learning",
"output": "Machine Learning adalah cabang dari AI..."
}
]
Buat formatting function
def formatinstruction(sample):
return f"""### Instruction:
{sample['instruction']}
Input:
{sample['input']}
Response:
{sample['output']}"""
dataset = Dataset.fromlist(data)
dataset = dataset.map(lambda x: {"text": formatinstruction(x)})
2. Chat Format (ChatML)
def formatchat(sample):
messages = sample.get("messages", [])
text = ""
for msg in messages:
role = msg["role"]
content = msg["content"]
if role == "system":
text += f"<|system|>\n{content}\n"
elif role == "user":
text += f"<|user|>\n{content}\n"
elif role == "assistant":
text += f"<|assistant|>\n{content}\n"
return {"text": text}
Data format
data = [
{
"messages": [
{"role": "system", "content": "Kamu adalah asisten yang helpful."},
{"role": "user", "content": "Apa itu Python?"},
{"role": "assistant", "content": "Python adalah bahasa pemrograman..."}
]
}
]
3. Custom Dataset dari File
import json
from datasets import Dataset
Load dari JSON
with open("trainingdata.json", "r") as f:
data = json.load(f)
Atau dari JSONL
data = []
with open("trainingdata.jsonl", "r") as f:
for line in f:
data.append(json.loads(line))
Buat dataset
dataset = Dataset.fromlist(data)
Split train/val
dataset = dataset.traintestsplit(testsize=0.1)
traindataset = dataset["train"]
evaldataset = dataset["test"]
Training dengan SFTTrainer
from transformers import TrainingArguments
from trl import SFTTrainer, DataCollatorForCompletionOnlyLM
Data collator untuk completion-only training
responsetemplate = "### Response:"
collator = DataCollatorForCompletionOnlyLM(
responsetemplate=responsetemplate,
tokenizer=tokenizer
)
Training arguments
trainingargs = TrainingArguments(
outputdir="./output",
# Training hyperparameters
numtrainepochs=3,
perdevicetrainbatchsize=4,
perdeviceevalbatchsize=4,
gradientaccumulationsteps=4,
gradientcheckpointing=True, # Memory saving
# Optimizer
learningrate=2e-4,
weightdecay=0.001,
warmupratio=0.03,
lrschedulertype="cosine",
# Precision
fp16=True, # Atau bf16=True untuk Ampere+
# Logging
loggingsteps=10,
loggingdir="./logs",
# Saving
savestrategy="steps",
savesteps=100,
savetotallimit=3,
# Evaluation
evalstrategy="steps",
evalsteps=100,
# Misc
reportto="wandb", # Atau "tensorboard"
runname="lora-experiment",
)
Trainer
trainer = SFTTrainer(
model=model,
args=trainingargs,
traindataset=traindataset,
evaldataset=evaldataset,
tokenizer=tokenizer,
datacollator=collator,
maxseqlength=1024,
packing=False, # True untuk packing multiple samples
)
Train
trainer.train()
Save
trainer.savemodel("./final-model")
model.savepretrained("./lora-adapter")
tokenizer.savepretrained("./lora-adapter")
Inference dengan LoRA
1. Load dan Merge Adapter
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
Load base model
basemodel = AutoModelForCausalLM.frompretrained(
"meta-llama/Llama-2-7b-hf",
torchdtype="auto",
devicemap="auto"
)
Load LoRA adapter
model = PeftModel.frompretrained(basemodel, "./lora-adapter")
Merge weights (optional, untuk inference lebih cepat)
model = model.mergeandunload()
Save merged model
model.savepretrained("./merged-model")
2. Inference
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
Load model (merged atau dengan adapter)
model = AutoModelForCausalLM.frompretrained(
"./merged-model",
torchdtype=torch.float16,
devicemap="auto"
)
tokenizer = AutoTokenizer.frompretrained("./merged-model")
def generateresponse(prompt, maxlength=512):
inputs = tokenizer(prompt, returntensors="pt").to(model.device)
with torch.nograd():
outputs = model.generate(
inputs,
maxnewtokens=maxlength,
temperature=0.7,
topp=0.9,
dosample=True,
padtokenid=tokenizer.eostokenid
)
response = tokenizer.decode(outputs[0], skipspecialtokens=True)
return response[len(prompt):]
Test
prompt = """### Instruction:
Jelaskan apa itu machine learning
Response:
"""
response = generateresponse(prompt)
print(response)
3. Multiple LoRA Adapters
from peft import PeftModel
Load base model
basemodel = AutoModelForCausalLM.frompretrained("base-model")
Load adapter 1
model = PeftModel.frompretrained(basemodel, "./adapter-task1", adaptername="task1")
Load adapter 2
model.loadadapter("./adapter-task2", adaptername="task2")
Switch between adapters
model.setadapter("task1")
output1 = model.generate(...)
model.setadapter("task2")
output2 = model.generate(...)
Disable adapter (use base model)
with model.disableadapter():
baseoutput = model.generate(...)
Fine-tuning untuk Task Spesifik
1. Text Classification
from transformers import AutoModelForSequenceClassification
from peft import LoraConfig, getpeftmodel, TaskType
model = AutoModelForSequenceClassification.frompretrained(
"bert-base-uncased",
numlabels=3
)
loraconfig = LoraConfig(
r=8,
loraalpha=16,
targetmodules=["query", "value"],
loradropout=0.1,
bias="none",
tasktype=TaskType.SEQCLS
)
model = getpeftmodel(model, loraconfig)
2. Question Answering
from transformers import AutoModelForQuestionAnswering
from peft import LoraConfig, getpeftmodel, TaskType
model = AutoModelForQuestionAnswering.frompretrained("bert-base-uncased")
loraconfig = LoraConfig(
r=16,
loraalpha=32,
targetmodules=["query", "key", "value"],
tasktype=TaskType.QUESTIONANS
)
model = getpeftmodel(model, loraconfig)
3. Summarization
from transformers import AutoModelForSeq2SeqLM
from peft import LoraConfig, getpeftmodel, TaskType
model = AutoModelForSeq2SeqLM.frompretrained("t5-base")
loraconfig = LoraConfig(
r=8,
loraalpha=16,
targetmodules=["q", "v"],
tasktype=TaskType.SEQ2SEQLM
)
model = getpeftmodel(model, loraconfig)
Tips dan Best Practices
1. Hyperparameter Tuning
# Start dengan config ini
loraconfig = LoraConfig(
r=8, # Mulai kecil, naikkan jika underfitting
loraalpha=16, # 2 r
targetmodules=["qproj", "vproj"], # Minimal
loradropout=0.05,
)
Untuk task kompleks
loraconfig = LoraConfig(
r=32,
loraalpha=64,
targetmodules=[
"qproj", "kproj", "vproj", "oproj",
"gateproj", "upproj", "downproj"
],
loradropout=0.1,
)
2. Memory Optimization
# Gradient checkpointing
trainingargs = TrainingArguments(
gradientcheckpointing=True,
...
)
Mixed precision
trainingargs = TrainingArguments(
fp16=True, # Atau bf16=True
...
)
Gradient accumulation
trainingargs = TrainingArguments(
perdevicetrainbatchsize=1,
gradientaccumulationsteps=16, # Effective batch = 16
...
)
3. Debugging
# Check trainable parameters
model.printtrainableparameters()
Check LoRA modules
for name, param in model.namedparameters():
if param.requiresgrad:
print(f"{name}: {param.shape}")
Verify adapter loaded
print(model.peftconfig)
print(model.activeadapter)
4. Common Issues
# Issue: OOM (Out of Memory)
Solution: Reduce batch size, use gradient checkpointing
Issue: Loss not decreasing
Solution: Check learning rate, increase r, add more target modules
Issue: Overfitting
Solution: Increase dropout, reduce epochs, use more data
Issue: Catastrophic forgetting
Solution: Lower learning rate, use less epochs
Kesimpulan
LoRA dan PEFT memungkinkan fine-tuning LLM dengan resource minimal:
| Metode | Memory | Speed | Quality |
|--------|--------|-------|---------|
| Full Fine-tune | 100% | Slow | Best |
| LoRA | ~10% | Fast | Very Good |
| QLoRA | ~5% | Medium | Good |
Key takeaways: