LLM Post-Training with TRL: SFT, Reward Modeling, and DPO

After a base language model is pretrained, it still needs to be shaped into something useful and aligned with human expectations. TRL (Transformer Reinforcement Learning) is Hugging Face's library for that entire post-training stage, from supervised fine-tuning through preference optimization. This tutorial walks through the modern post-training pipeline and shows how to use TRL's trainers in practice, with Direct Preference Optimization (DPO) as the centerpiece.

What TRL Is and Where It Sits

TRL is a library focused on the steps that come after pretraining. Unlike a wrapper that mainly simplifies configuration, TRL provides concrete trainer classes for the full alignment stack: supervised fine-tuning, reward modeling, and several preference-optimization and reinforcement-learning algorithms.

It is built directly on the Hugging Face ecosystem:

• Transformers for model and tokenizer loading.
• PEFT for parameter-efficient adapters such as LoRA, so you can train large models on modest hardware.
• Accelerate for distributed and mixed-precision training.
• Datasets for loading and preprocessing data.

Because TRL reuses these components, anything you already know about AutoModelForCausalLM, tokenizers, or LoRA configs carries over directly.

The Modern LLM Post-Training Pipeline

A typical instruction-following or chat model goes through three stages:

TRL covers stages 2 and 3. Most production alignment work today combines an SFT pass followed by a preference-optimization pass.

Installation

# Core stack
pip install trl peft datasets accelerate

Recommended extras
pip install transformers bitsandbytes
pip install wandb  # optional experiment tracking

Verify the install and check that a GPU is visible:

import torch
import trl

print("TRL version:", trl.version)
print("CUDA available:", torch.cuda.isavailable())

example = {
    "messages": [
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Explain what a vector database is."},
        {"role": "assistant", "content": "A vector database stores embeddings..."},
    ]
}

example = {
    "prompt": "Explain what a vector database is.",
    "chosen": "A vector database stores embeddings and supports similarity search...",
    "rejected": "It's just a normal database.",
}

from datasets import loaddataset

raw = loaddataset("json", datafiles="mypreferences.jsonl", split="train")


def topreference(example):
    return {
        "prompt": example["question"],
        "chosen": example["goodanswer"],

        "rejected": example["badanswer"],
    }

dataset = raw.map(topreference, removecolumns=raw.columnnames)


Reserve a held-out split for honest evaluation.
splits = dataset.traintestsplit(testsize=0.05, seed=42)
traindataset, evaldataset = splits["train"], splits["test"]

A few habits that pay off:

• Always keep a held-out split. Preference metrics on the training set are misleading; you want eval numbers the model has not seen.
• Inspect a handful of rows by hand. Print five examples and read them. Label noise is far easier to catch by eye than by aggregate statistics.
• Set a seed. Reproducible splits make experiments comparable across runs.

Supervised Fine-Tuning with SFTTrainer

SFT is the foundation: it teaches the base model the response style and instruction-following behavior you want before any preference tuning. In TRL this is SFTTrainer configured with SFTConfig.

from datasets import loaddataset
from trl import SFTConfig, SFTTrainer

dataset = loaddataset("trl-lib/Capybara", split="train")

config = SFTConfig(
    outputdir="sft-model",

    numtrainepochs=1,

    perdevicetrainbatchsize=2,

    gradientaccumulationsteps=8,

    learningrate=2e-5,
    loggingsteps=10,

    packing=True,            # pack short samples to use the context window efficiently
    maxlength=1024,
    bf16=True,
)

trainer = SFTTrainer(
    model="Qwen/Qwen2.5-0.5B",
    args=config,
    traindataset=dataset,

)

trainer.train()
trainer.savemodel("sft-model")

A few points worth noting:

• Chat templates. When the dataset has a messages column, SFTTrainer applies the tokenizer's chat template, so special tokens and role markers are added correctly.
• Packing. Setting packing=True concatenates multiple short examples into one sequence up to maxlength. This improves throughput and reduces wasted padding, at the cost of slightly mixing examples within a sequence.
• Passing a model name. You can pass a model ID string and TRL loads it for you, or pass an already-loaded AutoModelForCausalLM instance.

For larger models you usually do not want full fine-tuning. TRL accepts a peftconfig, and the trainer wraps the model with PEFT internally. LoRA and PEFT are covered in depth in a separate tutorial, so this is intentionally brief.

from peft import LoraConfig
from trl import SFTConfig, SFTTrainer

peftconfig = LoraConfig(

    r=16,
    loraalpha=32,
    loradropout=0.05,

    targetmodules="all-linear",
    tasktype="CAUSALLM",
)

trainer = SFTTrainer(
    model="Qwen/Qwen2.5-0.5B",
    args=SFTConfig(outputdir="sft-lora", bf16=True),

    traindataset=dataset,
    peftconfig=peftconfig,
)

trainer.train()

The same peftconfig argument works with the other TRL trainers, including DPOTrainer.

from datasets import loaddataset
from transformers import AutoModelForSequenceClassification, AutoTokenizer
from trl import RewardConfig, RewardTrainer

modelid = "Qwen/Qwen2.5-0.5B"

tokenizer = AutoTokenizer.frompretrained(modelid)

model = AutoModelForSequenceClassification.frompretrained(modelid, numlabels=1)

dataset = loaddataset("trl-lib/ultrafeedbackbinarized", split="train")

config = RewardConfig(
    outputdir="reward-model",

    perdevicetrainbatchsize=4,

    numtrainepochs=1,

    learningrate=1e-5,
    loggingsteps=10,

    maxlength=1024,
    bf16=True,
)

trainer = RewardTrainer(
    model=model,
    args=config,
    traindataset=dataset,

    processingclass=tokenizer,
)

trainer.train()

The reward model is loaded as a sequence-classification head with a single output (numlabels=1). The trainer optimizes a pairwise loss so that the score for chosen exceeds the score for rejected. This reward model is what PPOTrainer later uses to provide a training signal.

from datasets import loaddataset

dataset = loaddataset("trl-lib/ultrafeedbackbinarized", split="train")
print(dataset[0].keys())  # dictkeys(['prompt', 'chosen', 'rejected', ...])

from datasets import loaddataset
from transformers import AutoModelForCausalLM, AutoTokenizer
from trl import DPOConfig, DPOTrainer

modelid = "Qwen/Qwen2.5-0.5B-Instruct"


model = AutoModelForCausalLM.frompretrained(modelid)

tokenizer = AutoTokenizer.frompretrained(modelid)


traindataset = loaddataset(

    "trl-lib/ultrafeedbackbinarized", split="train"
).select(range(2000))  # small slice for a quick run

config = DPOConfig(
    outputdir="dpo-model",

    perdevicetrainbatchsize=2,

    gradientaccumulationsteps=8,

    numtrainepochs=1,

    learningrate=5e-6,
    beta=0.1,                 # KL strength: how close to stay to the reference
    losstype="sigmoid",      # the standard DPO loss

    loggingsteps=10,
    maxlength=1024,

    maxpromptlength=512,

    bf16=True,
)

trainer = DPOTrainer(
    model=model,
    refmodel=None,           # TRL creates a frozen copy of model as reference
    args=config,
    traindataset=traindataset,
    processingclass=tokenizer,

)

trainer.train()
trainer.savemodel("dpo-model")

Key DPOConfig knobs:

• beta controls how strongly the policy is held to the reference model. Lower values (around 0.05) allow bigger changes; higher values (0.3+) keep the model conservative. A value near 0.1 is a common starting point.
• losstype selects the preference loss. "sigmoid" is standard DPO; alternatives such as "ipo" and "hinge" change how the objective treats the preference margin and can reduce overfitting on some datasets.
• refmodel=None tells TRL to make a frozen copy of the policy as the reference. If you train with LoRA, TRL can use the base model with adapters disabled as the reference, avoiding a second model copy in memory.

To run DPO with LoRA instead of full fine-tuning, pass a peftconfig exactly as in the SFT example. This is the common setup for larger models.

from datasets import loaddataset
from trl import GRPOConfig, GRPOTrainer

dataset = loaddataset("trl-lib/tldr", split="train")


A reward function receives the generated completions and returns a score per item.
def lengthreward(completions, kwargs):

    # Toy example: prefer concise answers under 200 characters.
    return [1.0 if len(c) < 200 else 0.0 for c in completions]

trainer = GRPOTrainer(
    model="Qwen/Qwen2.5-0.5B-Instruct",
    rewardfuncs=lengthreward,
    args=GRPOConfig(outputdir="grpo-model", bf16=True, loggingsteps=10),
    traindataset=dataset,

)

trainer.train()

def correctnessreward(completions, groundtruth, kwargs):
    scores = []
    for completion, answer in zip(completions, groundtruth):
        scores.append(1.0 if answer.strip() in completion else 0.0)
    return scores

Extra dataset columns (like groundtruth) are forwarded to the reward function as keyword arguments, so you can implement any checkable objective.

config = DPOConfig(
    outputdir="dpo-model",
    evalstrategy="steps",

    evalsteps=100,
    loggingsteps=10,

    reportto="wandb",       # or "tensorboard"
    pushtohub=True,
    hubmodelid="your-username/dpo-model",
)

With an evaluation split passed as evaldataset, DPO logs useful metrics such as reward accuracy (how often chosen scores above rejected) and the reward margin. After training you can push the model:

trainer.pushtohub()

accelerate config              # one-time interactive setup
accelerate launch traindpo.py

For large models that do not fit on a single GPU, DeepSpeed ZeRO shards optimizer state, gradients, and parameters across devices:

accelerate launch --configfile deepspeedzero3.yaml traindpo.py

• Post-training has three stages: pretrain, SFT, then preference optimization. TRL covers the last two.
• SFTTrainer handles supervised fine-tuning, applies chat templates, supports packing, and combines cleanly with LoRA via peftconfig.
• RewardTrainer trains a scalar reward model on chosen/rejected pairs for classic RLHF.
• DPOTrainer is the practical default for preference alignment: it optimizes directly on preference pairs, avoids a separate reward model and RL loop, and is tuned mainly through beta and loss_type.
• PPOTrainer, GRPOTrainer, ORPO, and KTO cover the rest of the spectrum, from full RLHF to verifiable-reward RL to binary-feedback methods.
• Success depends less on the algorithm and more on doing SFT first, curating clean preference data, tuning beta, and guarding against overfitting.

Start with an SFT pass, move to DPO on a clean preference dataset, evaluate honestly, and only graduate to PPO or GRPO when your task genuinely needs online reinforcement learning.