Complete Guide to Hugging Face AutoTrain: Fine-Tune AI Models Without the Hassle
Hugging Face AutoTrain is an open-source platform and library that enables you to fine-tune machine learning models automatically without writing training code from scratch. With AutoTrain, a process that typically requires hundreds of lines of PyTorch configuration code can be simplified to a few CLI commands or a few lines of Python code.
AutoTrain supports a wide range of tasks including Large Language Model (LLM) fine-tuning, text classification, image classification, object detection, and tabular data. This platform is ideal for data scientists and ML engineers who want to run quick experiments without dealing with boilerplate training loops.
In this tutorial, we will cover how to use AutoTrain from installation, CLI usage, Python API usage, to advanced configuration for various fine-tuning scenarios.
Installation
System Prerequisites
Before installing AutoTrain, make sure you have:
- Python 3.8 or later
- pip package manager
- GPU with CUDA support (recommended for training, minimum NVIDIA GPU with 8GB VRAM)
- Git (for cloning repositories if needed)
Installation via pip
The easiest way to install AutoTrain is through pip:
pip install autotrain-advanced
For installation with full support including all optional dependencies:
pip install autotrain-advanced[api]
Verify Installation
After installation is complete, verify by running:
autotrain --version
You can also check if the GPU is properly detected:
import torch
print(f"CUDA available: {torch.cuda.isavailable()}")
print(f"GPU: {torch.cuda.getdevicename(0)}")
Setup Hugging Face Token
AutoTrain requires a Hugging Face token to access models and datasets. Create a token at https://huggingface.co/settings/tokens then set it as an environment variable:
export HFTOKEN=hfxxxxxxxxxxxxxxxxxxxxx
Or login via CLI:
huggingface-cli login
Basic Usage via CLI
The AutoTrain CLI is the fastest way to start fine-tuning. You just need to prepare your dataset and run a single command.
Fine-Tuning LLM with CLI
Example of fine-tuning a language model using a custom dataset:
autotrain llm \
--train \
--model meta-llama/Llama-3.2-1B \
--data-path ./my-dataset \
--text-column text \
--lr 2e-4 \
--batch-size 4 \
--epochs 3 \
--trainer sft \
--peft \
--quantization int4 \
--project-name my-llm-finetune
Key parameters:
--model: Base model from Hugging Face Hub--data-path: Path to dataset folder (CSV, JSON, or Parquet format)--text-column: Name of the column containing training text--trainer: Trainer type (sftfor Supervised Fine-Tuning,rewardfor Reward Model,dpofor DPO)--peft: Enable LoRA/PEFT for efficient training--quantization: Model quantization (int4orint8) to save VRAM
Dataset Format for LLM
Datasets must be in CSV, JSON, or Parquet format. For SFT (Supervised Fine-Tuning), the simplest format:
text
"<|system|>You are a helpful assistant.<|end|><|user|>What is Python?<|end|><|assistant|>Python is a high-level programming language.<|end|>"
"<|system|>You are a helpful assistant.<|end|><|user|>Explain machine learning.<|end|><|assistant|>Machine learning is a branch of AI that enables computers to learn from data.<|end|>"
For a more structured chat format, use JSON format:
[
{
"text": [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "What is deep learning?"},
{"role": "assistant", "content": "Deep learning is a subset of machine learning that uses multi-layered neural networks."}
]
}
]
Text Classification with CLI
autotrain text-classification \
--train \
--model bert-base-uncased \
--data-path ./sentiment-dataset \
--text-column review \
--target-column sentiment \
--lr 5e-5 \
--batch-size 16 \
--epochs 5 \
--project-name sentiment-classifier
Image Classification with CLI
autotrain image-classification \
--train \
--model google/vit-base-patch16-224 \
--data-path ./image-dataset \
--image-column image \
--target-column label \
--lr 3e-5 \
--batch-size 8 \
--epochs 10 \
--project-name image-classifier
Dataset folder structure for image classification:
image-dataset/
├── train/
│ ├── metadata.csv
│ └── images/
│ ├── img001.jpg
│ ├── img002.jpg
│ └── ...
└── valid/
├── metadata.csv
└── images/
├── img101.jpg
└── ...
Python API Usage
Besides CLI, AutoTrain also provides a more flexible Python API for integration into existing pipelines.
Setup Project and LLM Training
from autotrain.params import LLMTrainingParams
from autotrain.project import AutoTrainProject
params = LLMTrainingParams(
model="meta-llama/Llama-3.2-1B",
datapath="./my-dataset",
textcolumn="text",
trainsplit="train",
validsplit="validation",
trainer="sft",
# Hyperparameters
learningrate=2e-4,
numepochs=3,
batchsize=4,
gradientaccumulation=4,
maxseqlength=2048,
# Memory efficiency
peft=True,
quantization="int4",
lorar=16,
loraalpha=32,
loradropout=0.05,
# Output
projectname="my-llm-project",
pushtohub=True,
repoid="username/my-finetuned-llm",
)
project = AutoTrainProject(params=params, backend="local")
project.create()
Text Classification with Python API
from autotrain.params import TextClassificationParams
from autotrain.project import AutoTrainProject
params = TextClassificationParams(
model="bert-base-uncased",
datapath="./sentiment-data",
textcolumn="text",
targetcolumn="label",
trainsplit="train",
validsplit="test",
learningrate=5e-5,
numepochs=5,
batchsize=16,
maxseqlength=512,
projectname="sentiment-analysis",
pushtohub=False,
)
project = AutoTrainProject(params=params, backend="local")
project.create()
Token Classification (NER)
from autotrain.params import TokenClassificationParams
from autotrain.project import AutoTrainProject
params = TokenClassificationParams(
model="bert-base-multilingual-cased",
datapath="./ner-dataset",
textcolumn="tokens",
targetcolumn="nertags",
trainsplit="train",
validsplit="validation",
learningrate=3e-5,
numepochs=10,
batchsize=16,
projectname="ner-model",
)
project = AutoTrainProject(params=params, backend="local")
project.create()
Tabular Classification
AutoTrain also supports tabular data for classification and regression tasks:
from autotrain.params import TabularParams
from autotrain.project import AutoTrainProject
params = TabularParams(
datapath="./tabular-data",
targetcolumn="target",
trainsplit="train",
validsplit="test",
task="classification",
# AutoTrain will automatically try various models
# including XGBoost, LightGBM, Random Forest, etc.
numtrials=50, # Number of trials for hyperparameter search
timelimit=3600, # Time limit in seconds
projectname="tabular-classifier",
)
project = AutoTrainProject(params=params, backend="local")
project.create()
Advanced LLM Fine-Tuning
DPO (Direct Preference Optimization)
DPO is an alignment technique that allows models to learn from human preferences without needing a separate reward model:
from autotrain.params import LLMTrainingParams
from autotrain.project import AutoTrainProject
params = LLMTrainingParams(
model="meta-llama/Llama-3.2-1B",
datapath="./preference-dataset",
trainer="dpo",
# DPO dataset columns
textcolumn="prompt",
rejectedtextcolumn="rejected",
# DPO specific params
dpobeta=0.1,
# Hyperparameters
learningrate=5e-6,
numepochs=1,
batchsize=2,
gradientaccumulation=8,
maxseqlength=1024,
# PEFT
peft=True,
quantization="int4",
lorar=16,
loraalpha=32,
projectname="dpo-alignment",
)
project = AutoTrainProject(params=params, backend="local")
project.create()
DPO dataset format:
[
{
"prompt": "Explain renewable energy.",
"chosen": "Renewable energy refers to energy sources that are naturally replenished on a human timescale, such as sunlight, wind, water, and geothermal heat.",
"rejected": "Renewable energy is like the green stuff from the sun or whatever."
}
]
ORPO (Odds Ratio Preference Optimization)
ORPO is a newer and more efficient alignment technique:
autotrain llm \
--train \
--model meta-llama/Llama-3.2-1B \
--data-path ./preference-data \
--trainer orpo \
--text-column prompt \
--rejected-text-column rejected \
--lr 5e-6 \
--batch-size 2 \
--epochs 1 \
--peft \
--quantization int4 \
--project-name orpo-training
Optimal LoRA Configuration
The choice of LoRA parameters significantly affects fine-tuning quality:
params = LLMTrainingParams(
model="meta-llama/Llama-3.2-3B",
datapath="./dataset",
trainer="sft",
textcolumn="text",
peft=True,
quantization="int4",
# LoRA config
lorar=64, # Rank - higher means more parameters are tuned
loraalpha=128, # Alpha - typically 2x of r
loradropout=0.05, # Dropout for regularization
targetmodules="qproj,vproj,kproj,oproj,gateproj,upproj,downproj",
# Training config
learningrate=2e-4,
numepochs=3,
batchsize=4,
gradientaccumulation=4,
warmupratio=0.03,
weightdecay=0.01,
maxgradnorm=0.3,
maxseqlength=2048,
# Optimizer
optimizer="pagedadamw8bit",
scheduler="cosine",
projectname="optimized-lora",
)
LoRA rank selection guide:
r=8: Minimal, suitable for simple tasks or small datasetsr=16: Good default for most casesr=32-64: For complex tasks or when quality is paramountr=128+: Approaching full fine-tuning, requires more VRAM
Preparing Good Datasets
Creating a Dataset from Scratch
import pandas as pd
import json
Dataset for SFT in chat format
conversations = []
Example training data
qapairs = [
{
"system": "You are a friendly and helpful customer service assistant.",
"user": "How do I return an item?",
"assistant": "To return an item, please follow these steps:\n1. Go to your orders page\n2. Select the order you want to return\n3. Click the 'Request Return' button\n4. Fill in the return reason form\n5. Wait for confirmation from our team within 1-2 business days"
},
{
"system": "You are a friendly and helpful customer service assistant.",
"user": "How long does shipping take?",
"assistant": "Shipping times depend on the service you choose:\n- Standard: 3-5 business days\n- Express: 1-2 business days\n- Same Day: Available for select areas"
}
]
for pair in qapairs:
text = f"<|system|>{pair['system']}<|end|>"
text += f"<|user|>{pair['user']}<|end|>"
text += f"<|assistant|>{pair['assistant']}<|end|>"
conversations.append({"text": text})
df = pd.DataFrame(conversations)
df.tocsv("./my-dataset/train.csv", index=False)
print(f"Dataset created with {len(df)} examples")
Validating Your Dataset
import pandas as pd
df = pd.readcsv("./my-dataset/train.csv")
print(f"Total samples: {len(df)}")
print(f"Columns: {df.columns.tolist()}")
print(f"Null values: {df.isnull().sum().todict()}")
Check text lengths
df['textlength'] = df['text'].str.len()
print(f"Avg text length: {df['textlength'].mean():.0f} chars")
print(f"Max text length: {df['textlength'].max()} chars")
print(f"Min text length: {df['textlength'].min()} chars")
Sample data
print("\nSample:")
print(df['text'].iloc[0][:200])
Splitting Your Dataset
from sklearn.modelselection import traintestsplit
df = pd.readcsv("./my-dataset/data.csv")
traindf, validdf = traintestsplit(df, testsize=0.1, randomstate=42)
traindf.tocsv("./my-dataset/train.csv", index=False)
validdf.tocsv("./my-dataset/valid.csv", index=False)
print(f"Train: {len(traindf)}, Validation: {len(validdf)}")
Using Your Fine-Tuned Model
Loading the Trained Model
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
modelpath = "./my-llm-project" # Or "username/my-finetuned-llm" from Hub
tokenizer = AutoTokenizer.frompretrained(modelpath)
model = AutoModelForCausalLM.frompretrained(
modelpath,
torchdtype=torch.float16,
devicemap="auto",
)
Generate text
prompt = "<|system|>You are a helpful assistant.<|end|><|user|>What is AutoTrain?<|end|><|assistant|>"
inputs = tokenizer(prompt, returntensors="pt").to(model.device)
with torch.nograd():
outputs = model.generate(
*inputs,
maxnewtokens=256,
temperature=0.7,
dosample=True,
topp=0.9,
)
response = tokenizer.decode(outputs[0], skipspecialtokens=False)
print(response)
Loading a PEFT/LoRA Model
If you used PEFT during training, the model adapter is separate from the base model:
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
import torch
basemodel = "meta-llama/Llama-3.2-1B"
adapterpath = "./my-llm-project"
tokenizer = AutoTokenizer.frompretrained(adapterpath)
model = AutoModelForCausalLM.frompretrained(
basemodel,
torchdtype=torch.float16,
devicemap="auto",
)
model = PeftModel.frompretrained(model, adapterpath)
Optional: merge adapter into base model for faster inference
model = model.mergeandunload()
Evaluating Your Model
from transformers import pipeline
pipe = pipeline(
"text-generation",
model=model,
tokenizer=tokenizer,
torchdtype=torch.float16,
devicemap="auto",
)
testprompts = [
"Explain machine learning in 2 sentences.",
"What is the difference between Python and JavaScript?",
"How do I create an API with FastAPI?",
]
for prompt in testprompts:
formatted = f"<|user|>{prompt}<|end|><|assistant|>"
result = pipe(
formatted,
maxnewtokens=200,
temperature=0.7,
dosample=True,
)
print(f"Q: {prompt}")
print(f"A: {result[0]['generatedtext']}")
print("-" 50)
Monitoring Training
Integration with Weights & Biases
pip install wandb
autotrain llm \
--train \
--model meta-llama/Llama-3.2-1B \
--data-path ./dataset \
--text-column text \
--trainer sft \
--peft \
--quantization int4 \
--lr 2e-4 \
--batch-size 4 \
--epochs 3 \
--logging-steps 10 \
--project-name my-project \
--log wandb
Integration with TensorBoard
autotrain llm \
--train \
--model meta-llama/Llama-3.2-1B \
--data-path ./dataset \
--text-column text \
--trainer sft \
--peft \
--lr 2e-4 \
--batch-size 4 \
--epochs 3 \
--logging-steps 10 \
--project-name my-project \
--log tensorboard
Then open TensorBoard:
tensorboard --logdir ./my-project/runs
YAML Configuration
For more complex projects, you can use a YAML configuration file:
# config.yaml
task: llm
basemodel: meta-llama/Llama-3.2-3B
projectname: advanced-finetune
dataset:
path: ./training-data
textcolumn: text
trainsplit: train
validsplit: validation
training:
trainer: sft
learningrate: 2e-4
numepochs: 3
batchsize: 4
gradientaccumulation: 4
maxseqlength: 2048
warmupratio: 0.03
weightdecay: 0.01
loggingsteps: 10
savesteps: 500
evalsteps: 500
peft:
enabled: true
quantization: int4
lorar: 32
loraalpha: 64
loradropout: 0.05
targetmodules: "qproj,vproj,kproj,oproj"
output:
pushtohub: true
repoid: username/my-model
Run with:
autotrain --config config.yaml
Best Practices
1. Start with a Small Dataset
Before training with your entire dataset, experiment with a small subset (100-500 samples) to ensure the pipeline works correctly and identify issues early.
head -n 100 train.csv > trainsmall.csv
2. Use Quantization to Save VRAM
For GPUs with limited VRAM, always use quantization:
int4: Saves approximately 75% VRAM, suitable for 8-16GB GPUsint8: Saves approximately 50% VRAM, slightly better quality than int4
3. Adjust Batch Size and Gradient Accumulation
If your GPU cannot handle a large batch size, use gradient accumulation:
Effective batch size = batchsize x gradientaccumulation
Example: batchsize=2 with gradientaccumulation=8 yields an effective batch size of 16.
4. Pay Attention to Learning Rate
- For LLM fine-tuning with LoRA:
1e-4to3e-4 - For full fine-tuning:
1e-5to5e-5 - For text/image classification:
2e-5to5e-5
5. Validate Dataset Quality
Poor data produces poor models, regardless of how sophisticated the training technique. Make sure:
- No excessive duplicate data
- Consistent format throughout the dataset
- Accurate labels for classification tasks
- Text is not truncated or corrupted
6. Push to Hugging Face Hub
Always save models to the Hub for versioning and sharing:
autotrain llm \
--train \
--model meta-llama/Llama-3.2-1B \
--data-path ./dataset \
--text-column text \
--trainer sft \
--peft \
--push-to-hub \
--repo-id username/model-name \
--project-name my-project
7. Use an Evaluation Split
Always set aside a portion of data for evaluation to monitor overfitting:
autotrain llm \
--train \
--model meta-llama/Llama-3.2-1B \
--data-path ./dataset \
--text-column text \
--train-split train \
--valid-split validation \
--trainer sft \
--eval-steps 100 \
--project-name my-project
Troubleshooting
Out of Memory (OOM)
If you get a CUDA OOM error:
batchsize (try 1 or 2)maxseqlengthquantization: int4gradientaccumulation to compensate for smaller batch sizegradient_checkpointing: trueSlow Training
nvidia-smibf16 or fp16 mixed precisionpip install flash-attn --no-build-isolation
Model Not Converging
Conclusion
Hugging Face AutoTrain significantly simplifies the process of fine-tuning AI models. With support for various tasks (LLM, NLP, Computer Vision, Tabular), multiple training techniques (SFT, DPO, ORPO), and tight integration with the Hugging Face ecosystem, AutoTrain is the right tool to accelerate your ML experiments.
Key takeaways:
- Start with a small dataset and iterate incrementally
- Use PEFT/LoRA and quantization for efficiency
- Validate dataset quality before starting training
- Monitor training with WandB or TensorBoard
- Save models to Hugging Face Hub for versioning
With the fundamentals covered in this tutorial, you can immediately start fine-tuning models for your specific project needs.