Complete Guide to Hugging Face AutoTrain: Fine-Tune AI Models Without the Hassle

# Tutorial Lengkap Hugging Face AutoTrain: Fine-Tuning Model AI Tanpa Ribet Hugging Face AutoTrain adalah platform dan library open-source yang memungkinkan kamu melakukan fine-tuning model machine l...

By Ruby Abdullah · · tutorial
Hugging FaceAutoTrainFine-TuningLLMMachine Learning

Complete Guide to Hugging Face AutoTrain: Fine-Tune AI Models Without the Hassle

Hugging Face AutoTrain is an open-source platform and library that enables you to fine-tune machine learning models automatically without writing training code from scratch. With AutoTrain, a process that typically requires hundreds of lines of PyTorch configuration code can be simplified to a few CLI commands or a few lines of Python code.

AutoTrain supports a wide range of tasks including Large Language Model (LLM) fine-tuning, text classification, image classification, object detection, and tabular data. This platform is ideal for data scientists and ML engineers who want to run quick experiments without dealing with boilerplate training loops.

In this tutorial, we will cover how to use AutoTrain from installation, CLI usage, Python API usage, to advanced configuration for various fine-tuning scenarios.

Installation

System Prerequisites

Before installing AutoTrain, make sure you have:

  • Python 3.8 or later
  • pip package manager
  • GPU with CUDA support (recommended for training, minimum NVIDIA GPU with 8GB VRAM)
  • Git (for cloning repositories if needed)

Installation via pip

The easiest way to install AutoTrain is through pip:

pip install autotrain-advanced

For installation with full support including all optional dependencies:

pip install autotrain-advanced[api]

Verify Installation

After installation is complete, verify by running:

autotrain --version

You can also check if the GPU is properly detected:

import torch

print(f"CUDA available: {torch.cuda.isavailable()}")

print(f"GPU: {torch.cuda.getdevicename(0)}")

Setup Hugging Face Token

AutoTrain requires a Hugging Face token to access models and datasets. Create a token at https://huggingface.co/settings/tokens then set it as an environment variable:

export HFTOKEN=hfxxxxxxxxxxxxxxxxxxxxx

Or login via CLI:

huggingface-cli login

Basic Usage via CLI

The AutoTrain CLI is the fastest way to start fine-tuning. You just need to prepare your dataset and run a single command.

Fine-Tuning LLM with CLI

Example of fine-tuning a language model using a custom dataset:

autotrain llm \

--train \

--model meta-llama/Llama-3.2-1B \

--data-path ./my-dataset \

--text-column text \

--lr 2e-4 \

--batch-size 4 \

--epochs 3 \

--trainer sft \

--peft \

--quantization int4 \

--project-name my-llm-finetune

Key parameters:

  • --model: Base model from Hugging Face Hub
  • --data-path: Path to dataset folder (CSV, JSON, or Parquet format)
  • --text-column: Name of the column containing training text
  • --trainer: Trainer type (sft for Supervised Fine-Tuning, reward for Reward Model, dpo for DPO)
  • --peft: Enable LoRA/PEFT for efficient training
  • --quantization: Model quantization (int4 or int8) to save VRAM

Dataset Format for LLM

Datasets must be in CSV, JSON, or Parquet format. For SFT (Supervised Fine-Tuning), the simplest format:

text

"<|system|>You are a helpful assistant.<|end|><|user|>What is Python?<|end|><|assistant|>Python is a high-level programming language.<|end|>"

"<|system|>You are a helpful assistant.<|end|><|user|>Explain machine learning.<|end|><|assistant|>Machine learning is a branch of AI that enables computers to learn from data.<|end|>"

For a more structured chat format, use JSON format:

[

{

"text": [

{"role": "system", "content": "You are a helpful assistant."},

{"role": "user", "content": "What is deep learning?"},

{"role": "assistant", "content": "Deep learning is a subset of machine learning that uses multi-layered neural networks."}

]

}

]

Text Classification with CLI

autotrain text-classification \

--train \

--model bert-base-uncased \

--data-path ./sentiment-dataset \

--text-column review \

--target-column sentiment \

--lr 5e-5 \

--batch-size 16 \

--epochs 5 \

--project-name sentiment-classifier

Image Classification with CLI

autotrain image-classification \

--train \

--model google/vit-base-patch16-224 \

--data-path ./image-dataset \

--image-column image \

--target-column label \

--lr 3e-5 \

--batch-size 8 \

--epochs 10 \

--project-name image-classifier

Dataset folder structure for image classification:

image-dataset/

├── train/

│ ├── metadata.csv

│ └── images/

│ ├── img001.jpg

│ ├── img002.jpg

│ └── ...

└── valid/

├── metadata.csv

└── images/

├── img101.jpg

└── ...

Python API Usage

Besides CLI, AutoTrain also provides a more flexible Python API for integration into existing pipelines.

Setup Project and LLM Training

from autotrain.params import LLMTrainingParams

from autotrain.project import AutoTrainProject

params = LLMTrainingParams(

model="meta-llama/Llama-3.2-1B",

datapath="./my-dataset",

textcolumn="text",

trainsplit="train",

validsplit="validation",

trainer="sft",

# Hyperparameters

learningrate=2e-4,

numepochs=3,

batchsize=4,

gradientaccumulation=4,

maxseqlength=2048,

# Memory efficiency

peft=True,

quantization="int4",

lorar=16,

loraalpha=32,

loradropout=0.05,

# Output

projectname="my-llm-project",

pushtohub=True,

repoid="username/my-finetuned-llm",

)

project = AutoTrainProject(params=params, backend="local")

project.create()

Text Classification with Python API

from autotrain.params import TextClassificationParams

from autotrain.project import AutoTrainProject

params = TextClassificationParams(

model="bert-base-uncased",

datapath="./sentiment-data",

textcolumn="text",

targetcolumn="label",

trainsplit="train",

validsplit="test",

learningrate=5e-5,

numepochs=5,

batchsize=16,

maxseqlength=512,

projectname="sentiment-analysis",

pushtohub=False,

)

project = AutoTrainProject(params=params, backend="local")

project.create()

Token Classification (NER)

from autotrain.params import TokenClassificationParams

from autotrain.project import AutoTrainProject

params = TokenClassificationParams(

model="bert-base-multilingual-cased",

datapath="./ner-dataset",

textcolumn="tokens",

targetcolumn="nertags",

trainsplit="train",

validsplit="validation",

learningrate=3e-5,

numepochs=10,

batchsize=16,

projectname="ner-model",

)

project = AutoTrainProject(params=params, backend="local")

project.create()

Tabular Classification

AutoTrain also supports tabular data for classification and regression tasks:

from autotrain.params import TabularParams

from autotrain.project import AutoTrainProject

params = TabularParams(

datapath="./tabular-data",

targetcolumn="target",

trainsplit="train",

validsplit="test",

task="classification",

# AutoTrain will automatically try various models

# including XGBoost, LightGBM, Random Forest, etc.

numtrials=50, # Number of trials for hyperparameter search

timelimit=3600, # Time limit in seconds

projectname="tabular-classifier",

)

project = AutoTrainProject(params=params, backend="local")

project.create()

Advanced LLM Fine-Tuning

DPO (Direct Preference Optimization)

DPO is an alignment technique that allows models to learn from human preferences without needing a separate reward model:

from autotrain.params import LLMTrainingParams

from autotrain.project import AutoTrainProject

params = LLMTrainingParams(

model="meta-llama/Llama-3.2-1B",

datapath="./preference-dataset",

trainer="dpo",

# DPO dataset columns

textcolumn="prompt",

rejectedtextcolumn="rejected",

# DPO specific params

dpobeta=0.1,

# Hyperparameters

learningrate=5e-6,

numepochs=1,

batchsize=2,

gradientaccumulation=8,

maxseqlength=1024,

# PEFT

peft=True,

quantization="int4",

lorar=16,

loraalpha=32,

projectname="dpo-alignment",

)

project = AutoTrainProject(params=params, backend="local")

project.create()

DPO dataset format:

[

{

"prompt": "Explain renewable energy.",

"chosen": "Renewable energy refers to energy sources that are naturally replenished on a human timescale, such as sunlight, wind, water, and geothermal heat.",

"rejected": "Renewable energy is like the green stuff from the sun or whatever."

}

]

ORPO (Odds Ratio Preference Optimization)

ORPO is a newer and more efficient alignment technique:

autotrain llm \

--train \

--model meta-llama/Llama-3.2-1B \

--data-path ./preference-data \

--trainer orpo \

--text-column prompt \

--rejected-text-column rejected \

--lr 5e-6 \

--batch-size 2 \

--epochs 1 \

--peft \

--quantization int4 \

--project-name orpo-training

Optimal LoRA Configuration

The choice of LoRA parameters significantly affects fine-tuning quality:

params = LLMTrainingParams(

model="meta-llama/Llama-3.2-3B",

datapath="./dataset",

trainer="sft",

textcolumn="text",

peft=True,

quantization="int4",

# LoRA config

lorar=64, # Rank - higher means more parameters are tuned

loraalpha=128, # Alpha - typically 2x of r

loradropout=0.05, # Dropout for regularization

targetmodules="qproj,vproj,kproj,oproj,gateproj,upproj,downproj",

# Training config

learningrate=2e-4,

numepochs=3,

batchsize=4,

gradientaccumulation=4,

warmupratio=0.03,

weightdecay=0.01,

maxgradnorm=0.3,

maxseqlength=2048,

# Optimizer

optimizer="pagedadamw8bit",

scheduler="cosine",

projectname="optimized-lora",

)

LoRA rank selection guide:

  • r=8: Minimal, suitable for simple tasks or small datasets
  • r=16: Good default for most cases
  • r=32-64: For complex tasks or when quality is paramount
  • r=128+: Approaching full fine-tuning, requires more VRAM

Preparing Good Datasets

Creating a Dataset from Scratch

import pandas as pd

import json

Dataset for SFT in chat format

conversations = []

Example training data

qapairs = [

{

"system": "You are a friendly and helpful customer service assistant.",

"user": "How do I return an item?",

"assistant": "To return an item, please follow these steps:\n1. Go to your orders page\n2. Select the order you want to return\n3. Click the 'Request Return' button\n4. Fill in the return reason form\n5. Wait for confirmation from our team within 1-2 business days"

},

{

"system": "You are a friendly and helpful customer service assistant.",

"user": "How long does shipping take?",

"assistant": "Shipping times depend on the service you choose:\n- Standard: 3-5 business days\n- Express: 1-2 business days\n- Same Day: Available for select areas"

}

]

for pair in qapairs:

text = f"<|system|>{pair['system']}<|end|>"

text += f"<|user|>{pair['user']}<|end|>"

text += f"<|assistant|>{pair['assistant']}<|end|>"

conversations.append({"text": text})

df = pd.DataFrame(conversations)

df.tocsv("./my-dataset/train.csv", index=False)

print(f"Dataset created with {len(df)} examples")

Validating Your Dataset

import pandas as pd

df = pd.readcsv("./my-dataset/train.csv")

print(f"Total samples: {len(df)}")

print(f"Columns: {df.columns.tolist()}")

print(f"Null values: {df.isnull().sum().todict()}")

Check text lengths

df['textlength'] = df['text'].str.len()

print(f"Avg text length: {df['textlength'].mean():.0f} chars")

print(f"Max text length: {df['textlength'].max()} chars")

print(f"Min text length: {df['textlength'].min()} chars")

Sample data

print("\nSample:")

print(df['text'].iloc[0][:200])

Splitting Your Dataset

from sklearn.modelselection import traintestsplit

df = pd.readcsv("./my-dataset/data.csv")

traindf, validdf = traintestsplit(df, testsize=0.1, randomstate=42)

traindf.tocsv("./my-dataset/train.csv", index=False)

validdf.tocsv("./my-dataset/valid.csv", index=False)

print(f"Train: {len(traindf)}, Validation: {len(validdf)}")

Using Your Fine-Tuned Model

Loading the Trained Model

from transformers import AutoModelForCausalLM, AutoTokenizer

import torch

modelpath = "./my-llm-project" # Or "username/my-finetuned-llm" from Hub

tokenizer = AutoTokenizer.frompretrained(modelpath)

model = AutoModelForCausalLM.frompretrained(

modelpath,

torchdtype=torch.float16,

devicemap="auto",

)

Generate text

prompt = "<|system|>You are a helpful assistant.<|end|><|user|>What is AutoTrain?<|end|><|assistant|>"

inputs = tokenizer(prompt, returntensors="pt").to(model.device)

with torch.nograd():

outputs = model.generate(

*inputs,

maxnewtokens=256,

temperature=0.7,

dosample=True,

topp=0.9,

)

response = tokenizer.decode(outputs[0], skipspecialtokens=False)

print(response)

Loading a PEFT/LoRA Model

If you used PEFT during training, the model adapter is separate from the base model:

from transformers import AutoModelForCausalLM, AutoTokenizer

from peft import PeftModel

import torch

basemodel = "meta-llama/Llama-3.2-1B"

adapterpath = "./my-llm-project"

tokenizer = AutoTokenizer.frompretrained(adapterpath)

model = AutoModelForCausalLM.frompretrained(

basemodel,

torchdtype=torch.float16,

devicemap="auto",

)

model = PeftModel.frompretrained(model, adapterpath)

Optional: merge adapter into base model for faster inference

model = model.mergeandunload()

Evaluating Your Model

from transformers import pipeline

pipe = pipeline(

"text-generation",

model=model,

tokenizer=tokenizer,

torchdtype=torch.float16,

devicemap="auto",

)

testprompts = [

"Explain machine learning in 2 sentences.",

"What is the difference between Python and JavaScript?",

"How do I create an API with FastAPI?",

]

for prompt in testprompts:

formatted = f"<|user|>{prompt}<|end|><|assistant|>"

result = pipe(

formatted,

maxnewtokens=200,

temperature=0.7,

dosample=True,

)

print(f"Q: {prompt}")

print(f"A: {result[0]['generatedtext']}")

print("-" 50)

Monitoring Training

Integration with Weights & Biases

pip install wandb

autotrain llm \

--train \

--model meta-llama/Llama-3.2-1B \

--data-path ./dataset \

--text-column text \

--trainer sft \

--peft \

--quantization int4 \

--lr 2e-4 \

--batch-size 4 \

--epochs 3 \

--logging-steps 10 \

--project-name my-project \

--log wandb

Integration with TensorBoard

autotrain llm \

--train \

--model meta-llama/Llama-3.2-1B \

--data-path ./dataset \

--text-column text \

--trainer sft \

--peft \

--lr 2e-4 \

--batch-size 4 \

--epochs 3 \

--logging-steps 10 \

--project-name my-project \

--log tensorboard

Then open TensorBoard:

tensorboard --logdir ./my-project/runs

YAML Configuration

For more complex projects, you can use a YAML configuration file:

# config.yaml

task: llm

basemodel: meta-llama/Llama-3.2-3B

projectname: advanced-finetune

dataset:

path: ./training-data

textcolumn: text

trainsplit: train

validsplit: validation

training:

trainer: sft

learningrate: 2e-4

numepochs: 3

batchsize: 4

gradientaccumulation: 4

maxseqlength: 2048

warmupratio: 0.03

weightdecay: 0.01

loggingsteps: 10

savesteps: 500

evalsteps: 500

peft:

enabled: true

quantization: int4

lorar: 32

loraalpha: 64

loradropout: 0.05

targetmodules: "qproj,vproj,kproj,oproj"

output:

pushtohub: true

repoid: username/my-model

Run with:

autotrain --config config.yaml

Best Practices

1. Start with a Small Dataset

Before training with your entire dataset, experiment with a small subset (100-500 samples) to ensure the pipeline works correctly and identify issues early.

head -n 100 train.csv > trainsmall.csv

2. Use Quantization to Save VRAM

For GPUs with limited VRAM, always use quantization:

  • int4: Saves approximately 75% VRAM, suitable for 8-16GB GPUs
  • int8: Saves approximately 50% VRAM, slightly better quality than int4

3. Adjust Batch Size and Gradient Accumulation

If your GPU cannot handle a large batch size, use gradient accumulation:

Effective batch size = batchsize x gradientaccumulation

Example: batchsize=2 with gradientaccumulation=8 yields an effective batch size of 16.

4. Pay Attention to Learning Rate

  • For LLM fine-tuning with LoRA: 1e-4 to 3e-4
  • For full fine-tuning: 1e-5 to 5e-5
  • For text/image classification: 2e-5 to 5e-5

5. Validate Dataset Quality

Poor data produces poor models, regardless of how sophisticated the training technique. Make sure:

  • No excessive duplicate data
  • Consistent format throughout the dataset
  • Accurate labels for classification tasks
  • Text is not truncated or corrupted

6. Push to Hugging Face Hub

Always save models to the Hub for versioning and sharing:

autotrain llm \

--train \

--model meta-llama/Llama-3.2-1B \

--data-path ./dataset \

--text-column text \

--trainer sft \

--peft \

--push-to-hub \

--repo-id username/model-name \

--project-name my-project

7. Use an Evaluation Split

Always set aside a portion of data for evaluation to monitor overfitting:

autotrain llm \

--train \

--model meta-llama/Llama-3.2-1B \

--data-path ./dataset \

--text-column text \

--train-split train \

--valid-split validation \

--trainer sft \

--eval-steps 100 \

--project-name my-project

Troubleshooting

Out of Memory (OOM)

If you get a CUDA OOM error:

  • Reduce batchsize (try 1 or 2)
  • Reduce maxseqlength
  • Enable quantization: int4
  • Increase gradientaccumulation to compensate for smaller batch size
  • Use gradient_checkpointing: true
  • Slow Training

  • Make sure GPU is detected: nvidia-smi
  • Use bf16 or fp16 mixed precision
  • Ensure data loading is not a bottleneck (use SSD)
  • Consider using Flash Attention 2:
  • pip install flash-attn --no-build-isolation
    

    Model Not Converging

  • Check dataset for noise or mislabeled data
  • Try a smaller learning rate
  • Increase the number of epochs
  • Make sure the data format matches what the model expects
  • Conclusion

    Hugging Face AutoTrain significantly simplifies the process of fine-tuning AI models. With support for various tasks (LLM, NLP, Computer Vision, Tabular), multiple training techniques (SFT, DPO, ORPO), and tight integration with the Hugging Face ecosystem, AutoTrain is the right tool to accelerate your ML experiments.

    Key takeaways:

    • Start with a small dataset and iterate incrementally
    • Use PEFT/LoRA and quantization for efficiency
    • Validate dataset quality before starting training
    • Monitor training with WandB or TensorBoard
    • Save models to Hugging Face Hub for versioning

    With the fundamentals covered in this tutorial, you can immediately start fine-tuning models for your specific project needs.

    Related Articles

    Complete Google Gemma Tutorial: Open-Source Models from Google DeepMind

    Tutorial Lengkap Google Gemma: Model Open-Source dari Google DeepMind Google Gemma adalah keluarga model bahasa open-sou...

    Complete Tutorial: Fine-tuning LLMs with LoRA and PEFT

    Tutorial Lengkap Fine-tuning LLM dengan LoRA dan PEFT Fine-tuning Large Language Models (LLM) secara tradisional membutu...

    MergeKit Tutorial: Merging LLM Models to Create Custom AI Models

    Tutorial MergeKit: Menggabungkan Model LLM untuk Membuat Model AI Kustom Model bahasa besar (LLM) open-source terus berk...

    LitGPT Tutorial: A Framework for Training and Fine-Tuning Large Language Models

    Tutorial LitGPT: Framework untuk Melatih dan Fine-Tuning Large Language Models LitGPT adalah framework open-source dari ...