Complete LlamaFactory Tutorial: All-in-One LLM Fine-Tuning Toolkit

# Tutorial Lengkap LlamaFactory: Toolkit Fine-Tuning LLM All-in-One Fine-tuning Large Language Model (LLM) sering kali menjadi proses yang kompleks dan membutuhkan banyak konfigurasi manual. LlamaFac...

By Ruby Abdullah · · tutorial
LlamaFactoryLLM Fine-TuningLoRAQLoRAMachine Learning

Complete LlamaFactory Tutorial: All-in-One LLM Fine-Tuning Toolkit

Fine-tuning Large Language Models (LLMs) often involves complex configurations and significant engineering effort. LlamaFactory simplifies this entire process by providing an intuitive interface and support for numerous models and training methods. In this tutorial, we will learn how to use LlamaFactory from installation to deploying your fine-tuned model.

What is LlamaFactory?

LlamaFactory (or LLaMA-Factory) is an open-source toolkit that provides a unified framework for fine-tuning over 100 language models. It supports various training methods including full fine-tuning, LoRA, QLoRA, and other Parameter-Efficient Fine-Tuning (PEFT) approaches. One of LlamaFactory's standout features is LLaMA Board, a Gradio-based web interface that enables fine-tuning without writing any code.

Key Features

  • Multi-Model Support: Supports LLaMA, Mistral, Qwen, Phi, Gemma, Yi, DeepSeek, and many more
  • Diverse Training Methods: Full fine-tuning, LoRA, QLoRA, DoRA, and GaLore
  • LLaMA Board: Web UI for no-code fine-tuning
  • Integrated Datasets: Dozens of built-in datasets ready to use
  • Automatic Evaluation: Automated benchmarking with MMLU, CMMLU, and C-Eval
  • Model Export: Export to GGUF, vLLM, or other formats

Installation

System Requirements

Before installing LlamaFactory, ensure your system meets the following requirements:

  • Python 3.8 or newer
  • CUDA 11.8 or 12.1 (for NVIDIA GPUs)
  • Minimum 16GB RAM (32GB recommended)
  • GPU with minimum 8GB VRAM for QLoRA, 16GB for LoRA, 40GB+ for full fine-tuning

Installation from Source

git clone --depth 1 https://github.com/hiyouga/LLaMA-Factory.git

cd LLaMA-Factory


pip install -e ".[torch,metrics]"

Installation with Docker

docker pull hiyouga/llamafactory:latest


docker run --gpus all \


  -v ./hfcache:/root/.cache/huggingface \


  -v ./data:/app/data \


  -v ./output:/app/output \


  -p 7860:7860 \


  -e GRADIOSHARE=1 \


  hiyouga/llamafactory:latest

Verify Installation

llamafactory-cli version

Expected output:

----------------------------------------------------------

| Welcome to LLaMA Factory, version x.x.x               |


| Project page: https://github.com/hiyouga/LLaMA-Factory |

Optional Dependencies

# For GPTQ quantization

pip install auto-gptq



For AWQ quantization


pip install autoawq



For Flash Attention 2


pip install flash-attn --no-build-isolation



For vLLM inference


pip install vllm

Basic Usage: Fine-Tuning with CLI

Preparing Your Dataset

LlamaFactory uses a specific dataset format. Create a JSON file with the following structure:

[

  {


    "instruction": "Explain what machine learning is in simple terms.",


    "input": "",


    "output": "Machine learning is a branch of artificial intelligence where computers learn from data to make predictions or decisions without being explicitly programmed. Think of it like teaching a child to recognize animals - the more examples they see, the better they become at identifying new animals."


  },


  {


    "instruction": "Write Python code to calculate factorial.",


    "input": "5",


    "output": "def factorial(n):\n    if n <= 1:\n        return 1\n    return n * factorial(n - 1)\n\nresult = factorial(5)\nprint(f'Factorial of 5 is {result}')  # Output: 120"


  }


]

Save this file as data/mydataset.json. Then register it in data/datasetinfo.json:

{

  "mycustomdataset": {


    "filename": "mydataset.json",


    "columns": {


      "prompt": "instruction",


      "query": "input",


      "response": "output"


    }


  }


}

Chat/Conversation Dataset Format

For multi-turn conversation datasets, use this format:

[

  {


    "conversations": [


      {"from": "human", "value": "What is Python?"},


      {"from": "gpt", "value": "Python is a high-level programming language known for its clean and readable syntax."},


      {"from": "human", "value": "What are its advantages over Java?"},


      {"from": "gpt", "value": "Python has several advantages over Java: more concise syntax, a gentler learning curve, an incredibly rich data science library ecosystem, and faster development cycles."}


    ]


  }


]

Running Fine-Tuning with CLI

Here is an example of fine-tuning Qwen2.5-7B using LoRA:

llamafactory-cli train \

  --stage sft \


  --modelnameorpath Qwen/Qwen2.5-7B \


  --dataset mycustomdataset \


  --template qwen \


  --finetuningtype lora \


  --lorarank 16 \


  --loraalpha 32 \


  --loradropout 0.05 \


  --outputdir output/qwen25-7b-lora \


  --perdevicetrainbatchsize 4 \


  --gradientaccumulationsteps 4 \


  --learningrate 5e-5 \


  --numtrainepochs 3 \


  --loggingsteps 10 \


  --savesteps 100 \


  --fp16 true \


  --maxlength 2048

Using YAML Configuration Files

For more complex projects, use YAML configuration files:

# config/trainqwenlora.yaml

Model


modelnameorpath: Qwen/Qwen2.5-7B


trustremotecode: true



Method


stage: sft


dotrain: true


finetuningtype: lora


lorarank: 16


loraalpha: 32


loradropout: 0.05


loratarget: all



Dataset


dataset: mycustomdataset


template: qwen


cutofflen: 2048


overwritecache: true


preprocessingnumworkers: 16



Output


outputdir: output/qwen25-7b-lora


loggingsteps: 10


savesteps: 100


overwriteoutputdir: true



Train


perdevicetrainbatchsize: 4


gradientaccumulationsteps: 4


learningrate: 5.0e-5


numtrainepochs: 3.0


lrschedulertype: cosine


warmupratio: 0.1


fp16: true

Run training with:

llamafactory-cli train config/trainqwenlora.yaml

Using LLaMA Board (Web UI)

LLaMA Board is LlamaFactory's flagship feature that enables fine-tuning through a web interface.

Launching LLaMA Board

llamafactory-cli webui

Your browser will automatically open at http://localhost:7860. The interface provides:

  • Train Tab: Configure and run fine-tuning
  • Evaluate Tab: Evaluate models with benchmarks
  • Chat Tab: Test models interactively
  • Export Tab: Export models to other formats

    • Fine-Tuning Steps via Web UI

  • Select a model from the dropdown (e.g., Qwen2.5-7B)
  • Choose the fine-tuning method (LoRA/QLoRA/Full)
  • Upload or select a dataset
  • Set hyperparameters (learning rate, epochs, batch size)
  • Click "Start" to begin training
  • Monitor progress through real-time loss graphs

    • Advanced Usage

    QLoRA: Fine-Tuning with Limited Resources

    QLoRA enables fine-tuning large models on GPUs with limited VRAM through 4-bit quantization:

    # config/trainqlora.yaml

    modelnameorpath: meta-llama/Llama-3.1-8B-Instruct
    

    quantizationbit: 4
    

    quantizationmethod: bitsandbytes
    


    stage: sft
    

    dotrain: true
    

    finetuningtype: lora
    

    lorarank: 64
    

    loraalpha: 128
    

    loradropout: 0.1
    

    loratarget: all
    


    dataset: mycustomdataset
    

    template: llama3
    

    cutofflen: 4096
    


    outputdir: output/llama31-8b-qlora
    

    perdevicetrainbatchsize: 2
    

    gradientaccumulationsteps: 8
    

    learningrate: 2.0e-4
    

    numtrainepochs: 3.0
    

    lrschedulertype: cosine
    

    warmupratio: 0.1
    

    bf16: true

    With 4-bit QLoRA, an 8B parameter model can be fine-tuned on a GPU with just 8GB VRAM.

    DPO (Direct Preference Optimization)

    DPO is used for model alignment based on human preferences:

    [

      {
    

        "conversations": [
    

          {"from": "human", "value": "How do I write a professional email?"}
    

        ],
    

        "chosen": {
    

          "from": "gpt",
    

          "value": "Here are the steps to write a professional email:\n1. Use a clear and specific subject line\n2. Address the recipient politely\n3. State your purpose in the first paragraph\n4. Provide necessary details\n5. Close with a clear call to action\n6. Use a professional closing"
    

        },
    

        "rejected": {
    

          "from": "gpt",
    

          "value": "just write whatever lol dont overthink it"
    

        }
    

      }
    

    ]

    DPO training configuration:

    stage: dpo

    dotrain: true
    

    modelnameorpath: Qwen/Qwen2.5-7B-Instruct
    

    finetuningtype: lora
    

    lorarank: 16
    


    dataset: mypreferencedataset
    

    template: qwen
    


    outputdir: output/qwen25-dpo
    

    perdevicetrainbatchsize: 2
    

    gradientaccumulationsteps: 4
    

    learningrate: 5.0e-6
    

    numtrainepochs: 1.0
    

    dpobeta: 0.1

    Multi-GPU Training with DeepSpeed

    For training on multiple GPUs, use DeepSpeed:

    # config/traindeepspeed.yaml

    modelnameorpath: Qwen/Qwen2.5-72B
    

    finetuningtype: lora
    

    lorarank: 32
    

    deepspeed: examples/deepspeed/dsz3config.json
    


    dataset: mycustomdataset
    

    template: qwen
    

    cutofflen: 4096
    


    outputdir: output/qwen25-72b-lora
    

    perdevicetrainbatchsize: 1
    

    gradientaccumulationsteps: 16
    

    learningrate: 1.0e-4
    

    numtrainepochs: 2.0
    

    bf16: true

    Run with:

    deepspeed --numgpus 4 src/train.py config/traindeepspeed.yaml

    Merging LoRA Adapter with Base Model

    After training completes, merge the LoRA adapter into the base model:

    llamafactory-cli export \

      --modelnameorpath Qwen/Qwen2.5-7B \
    

      --adapternameorpath output/qwen25-7b-lora \
    

      --template qwen \
    

      --finetuningtype lora \
    

      --exportdir output/qwen25-7b-merged \
    

      --exportsize 4 \
    

      --exportlegacyformat false

    Export to GGUF (for llama.cpp)

    llamafactory-cli export \

      --modelnameorpath output/qwen25-7b-merged \
    

      --template qwen \
    

      --exportdir output/qwen25-7b-gguf \
    

      --exportquantizationbit 4 \
    

      --exportquantizationdataset identity

    Model Evaluation

    Evaluate your fine-tuned model:

    llamafactory-cli eval \

      --modelnameorpath Qwen/Qwen2.5-7B \
    

      --adapternameorpath output/qwen25-7b-lora \
    

      --template qwen \
    

      --finetuningtype lora \
    

      --task mmlu \
    

      --lang en \
    

      --nshot 5 \
    

      --batchsize 4

    Inference and Chat

    Test your model interactively:

    llamafactory-cli chat \

      --modelnameorpath Qwen/Qwen2.5-7B \
    

      --adapternameorpath output/qwen25-7b-lora \
    

      --template qwen \
    

      --finetuningtype lora

    Or deploy as an OpenAI-compatible API server:

    llamafactory-cli api \

      --modelnameorpath Qwen/Qwen2.5-7B \
    

      --adapternameorpath output/qwen25-7b-lora \
    

      --template qwen \
    

      --finetuningtype lora \
    

      --host 0.0.0.0 \
    

      --port 8000

    Then access it like the OpenAI API:

    from openai import OpenAI


    client = OpenAI(
    

        baseurl="http://localhost:8000/v1",
    

        apikey="not-needed"
    

    )
    


    response = client.chat.completions.create(
    

        model="qwen25-7b-lora",
    

        messages=[
    

            {"role": "user", "content": "Explain the concept of transfer learning."}
    

        ],
    

        temperature=0.7,
    

        maxtokens=512
    

    )
    


    print(response.choices[0].message.content)

    Best Practices

    1. Choosing the Right Fine-Tuning Method

    | Method | VRAM | Quality | Speed | When to Use |

    |--------|------|---------|-------|-------------|

    | Full | 40GB+ | Best | Slow | Large datasets, significant changes |

    | LoRA | 16GB | Very Good | Medium | Most use cases |

    | QLoRA | 8GB | Good | Medium | Limited resources |

    | DoRA | 16GB | Very Good | Medium | More stable LoRA alternative |

    2. Hyperparameter Optimization

    • Learning Rate: Start with 2e-5 for full fine-tuning, 1e-4 to 5e-4 for LoRA/QLoRA
    • LoRA Rank: 8 for simple tasks, 16-32 for complex tasks, 64+ for large datasets
    • Batch Size: Use gradient accumulation if VRAM is limited
    • Epochs: 1-3 epochs is usually sufficient; more than 5 risks overfitting

    3. Dataset Quality

    • Ensure data is clean and consistent
    • Minimum 500-1000 samples for specific tasks
    • Use the appropriate format (instruction/conversation)
    • Validate data before starting training
    • Avoid excessive duplication

    4. Training Monitoring

    • Watch training loss, ensure it decreases consistently
    • Use a validation set to detect overfitting
    • Save checkpoints regularly
    • Use TensorBoard or Weights & Biases for visualization

    # Launch TensorBoard

    tensorboard --logdir output/qwen25-7b-lora/runs

    5. Resource-Saving Tips

    • Use gradientcheckpointing: true to save VRAM
    • Enable flashattn: fa2 if your GPU supports it
    • Use packing: true to maximize sequence length utilization
    • Consider the unsloth backend for 2x speedup

    # Resource-efficient configuration

    gradientcheckpointing: true
    

    flashattn: fa2
    

    packing: true
    

    useunsloth: true

    Troubleshooting

    Error: CUDA Out of Memory

    Solutions:

  • Reduce perdevicetrainbatchsize
  • Enable gradientcheckpointing: true
  • Use QLoRA (4-bit quantization)
  • Reduce cutofflen

    • Error: Template Not Found

    Make sure to use the correct template for your model:

    • LLaMA 3/3.1: llama3
    • Qwen2/2.5: qwen
    • Mistral/Mixtral: mistral
    • Phi-3/3.5: phi
    • Gemma 2: gemma
    • DeepSeek V2/V3: deepseek

    Training Loss Not Decreasing

    • Check dataset quality and format
    • Try a higher learning rate
    • Ensure the template matches the model
    • Verify the dataset is registered in dataset_info.json

    Conclusion

    LlamaFactory significantly simplifies the LLM fine-tuning process. With support for over 100 models, various training methods, and an intuitive web interface, this toolkit is suitable for both beginners and experienced practitioners.

    Key takeaways:

  • Choose the fine-tuning method based on your resources and needs (QLoRA for limited resources, LoRA for the best balance)
  • Dataset quality matters more than quantity
  • Use LLaMA Board for quick experiments before running large-scale training
  • Always evaluate your model after fine-tuning and compare with the baseline
  • Leverage the export feature for deployment across various platforms

    • By mastering LlamaFactory, you can customize LLMs for various specific needs such as domain-specific chatbots, coding assistants, document analysis, and more, without having to master complex technical details.

    Related Articles

    Torchtune Tutorial: Fine-Tuning LLMs with PyTorch Native

    Tutorial Torchtune: Fine-Tuning LLM dengan PyTorch Native Pendahuluan Torchtune adalah library resmi dari PyTorch untuk ...

    Unsloth Tutorial: Fast and Memory-Efficient LLM Fine-Tuning

    Fine-Tuning LLM Secara Efisien dengan Unsloth Dahulu, melakukan fine-tuning model bahasa besar membutuhkan server multi-...

    ZenML: Build Portable, Production-Ready ML and LLM Pipelines

    ZenML: Bikin Pipeline ML dan LLM yang Portable dan Siap Produksi Halo temen-temen, ketemu lagi sama aku, Ruby Abdullah. ...

    Complete Replicate Tutorial: Run and Deploy ML Models via API

    Tutorial Lengkap Replicate: Menjalankan dan Deploy Model ML via API Replicate adalah platform cloud yang memungkinkan An...