HuggingFace Accelerate: One PyTorch Script for CPU, Single GPU, Multi-GPU, and Mixed Precision
Hey everyone, today we are looking at a library that quietly powers a lot of the training code you already use but is rarely studied on its own: Accelerate from HuggingFace. If you have ever written a PyTorch training loop that ran beautifully on your laptop, then were asked to run it on a 4-GPU server and suddenly had to learn DistributedDataParallel, torchrun, ranks, world size, and dist.initprocessgroup, Accelerate is the cure.
Its philosophy is simple and I really like it: you keep writing an ordinary PyTorch training loop, Accelerate adds about five lines, and after that the exact same code runs on CPU, a single GPU, multiple GPUs in one machine, multiple machines, Apple Silicon (MPS), and TPUs. No nested if branches, no two versions of the same script.
In this tutorial we cover installation and configuration, converting a plain training loop, mixed precision, gradient accumulation, correct checkpoint saving and loading in distributed settings, non-duplicated logging, and finally DeepSpeed and FSDP integration for large models. Let us begin.
Introduction
First, let me explain the problem Accelerate actually solves.
Why Multi-GPU Is Annoying
Moving from one GPU to many with raw PyTorch drags in a lot of tedious plumbing.
You have to initialize the process group, choose a communication backend like NCCL, assign a rank to each process, and tear the group down cleanly at the end.
You have to replace the plain DataLoader with one using DistributedSampler, and remember to call setepoch() every epoch so shuffling stays correct across processes.
You have to wrap your model in DistributedDataParallel, then remember that model.module is what you save, not model.
You have to make sure only the main process prints logs, writes files, and reports metrics, otherwise you get eight identical log lines and eight processes fighting over the same file.
And if you want mixed precision, add GradScaler and autocast handling on top.
None of this is your research idea or your product. It is pure plumbing. Accelerate takes it over.
What Accelerate Does
Accelerate gives you one object called Accelerator. It detects the execution environment, prepares your model, optimizer, and dataloaders for that environment, and provides drop-in replacements for the few operations that must be distribution aware, mainly backward(), checkpointing, and logging.
An important framing: Accelerate is not a training framework like PyTorch Lightning. It does not take over your loop. You keep full control of the order of operations. It just deletes the distributed boilerplate.
Installation and Configuration
python -m venv venv
source venv/bin/activate # Linux / Mac
venv\Scripts\activate # Windows
pip install torch torchvision
pip install accelerate
Then run the configuration wizard once per machine:
accelerate config
It asks a handful of questions: one machine or several, how many GPUs, which mixed precision mode (no, fp16, or bf16), whether to use DeepSpeed or FSDP. Your answers are stored in a YAML file at ~/.cache/huggingface/accelerate/defaultconfig.yaml.
If you are in a hurry and just want sensible defaults:
accelerate config default
Inspect what Accelerate detected on your machine:
accelerate env
From a Plain Loop to an Accelerate Loop
This is the core section. Here is the plain PyTorch version first, then the Accelerate version, so you can see exactly what changes.
Plain PyTorch:
import torch
import torch.nn as nn
device = "cuda" if torch.cuda.isavailable() else "cpu"
model = MyModel().to(device)
optimizer = torch.optim.AdamW(model.parameters(), lr=3e-4)
criterion = nn.CrossEntropyLoss()
for epoch in range(EPOCHS):
model.train()
for x, y in trainloader:
x, y = x.to(device), y.to(device)
optimizer.zerograd()
loss = criterion(model(x), y)
loss.backward()
optimizer.step()
With Accelerate:
import torch
import torch.nn as nn
from accelerate import Accelerator
accelerator = Accelerator()
model = MyModel()
optimizer = torch.optim.AdamW(model.parameters(), lr=3e-4)
criterion = nn.CrossEntropyLoss()
model, optimizer, trainloader = accelerator.prepare(model, optimizer, trainloader)
for epoch in range(EPOCHS):
model.train()
for x, y in trainloader:
optimizer.zerograd()
loss = criterion(model(x), y)
accelerator.backward(loss)
optimizer.step()
Only four things differ.
You create an Accelerator().
You delete every .to(device) call. Accelerate moves the model and each batch to the right device.
You call accelerator.prepare() on the model, optimizer, and dataloader. That is where DDP wrapping and sampler replacement happen.
You swap loss.backward() for accelerator.backward(loss), which handles gradient scaling for mixed precision and cross-process synchronization.
That is it. This script now runs on CPU, one GPU, or eight GPUs unchanged.
Launching Training
To run it, do not use python train.py, use the launcher:
accelerate launch train.py
To override the stored configuration from the command line:
# Force 4 GPUs with bf16
accelerate launch --numprocesses=4 --mixedprecision=bf16 train.py
Force CPU for debugging
accelerate launch --cpu train.py
A tip from experience: while debugging, run with a single process first. Logic bugs are far easier to read when logs are not interleaved from eight processes.
Mixed Precision
Mixed precision trains with some operations in 16-bit rather than 32-bit precision. The result is lower memory usage and usually higher throughput on modern GPUs.
In Accelerate you never touch GradScaler or autocast. You just declare it.
accelerator = Accelerator(mixedprecision="bf16")
Choose bf16 if your GPU is Ampere or newer (A100, RTX 30 series, RTX 40 series, H100). Choose fp16 for older cards like V100 or T4. bf16 is more numerically stable because it shares the exponent range of fp32, so overflow problems are rare.
Gradient Accumulation
When your GPU is small but you need a large effective batch size, gradient accumulation is the answer. Accelerate exposes it through a clean context manager.
from accelerate import Accelerator
accelerator = Accelerator(gradientaccumulationsteps=4)
model, optimizer, trainloader = accelerator.prepare(model, optimizer, trainloader)
for x, y in trainloader:
with accelerator.accumulate(model):
loss = criterion(model(x), y)
accelerator.backward(loss)
optimizer.step()
scheduler.step()
optimizer.zerograd()
The nice part is that you still write optimizer.step() inside the loop as usual. Accelerate decides when that step actually executes and when it only accumulates. It also disables cross-GPU gradient synchronization on non-update steps, saving communication bandwidth.
Logging and Printing Without Duplicates
In a multi-process environment, a plain print() runs in every process. With 8 GPUs, one log line becomes eight.
# Printed only by the main process
accelerator.print(f"epoch {epoch} loss {loss.item():.4f}")
A block that only the main process runs
if accelerator.ismainprocess:
sendtelegramnotification("training finished")
Wait until all processes reach this point
accelerator.waitforeveryone()
waitforeveryone() matters before operations such as writing a file or reading a file the main process just wrote. Without it, other processes may read a half-written file.
Aggregating Metrics Across Processes
During evaluation, each process only sees part of the data. Computing local accuracy alone gives a number that does not represent the full dataset. Accelerate provides gatherformetrics, which also removes the duplicates introduced by padding the last batch.
model.eval()
correct = 0
total = 0
for x, y in evalloader:
with torch.nograd():
pred = model(x).argmax(dim=-1)
pred, y = accelerator.gatherformetrics((pred, y))
correct += (pred == y).sum().item()
total += y.numel()
accelerator.print(f"accuracy: {correct/total:.4f}")
This is one of the most valuable features, because metric bugs in distributed settings are silent. The number looks plausible and is simply wrong.
Saving and Loading Models
This is where most people trip. A prepared model is DDP wrapped, so saving it directly produces keys prefixed with module. that are awkward to load later.
The correct way to save final weights:
accelerator.waitforeveryone()
unwrapped = accelerator.unwrapmodel(model)
accelerator.save(unwrapped.statedict(), "modelfinal.pth")
accelerator.save is already smart enough to write only from the main process, so you do not need to wrap it in an if ismainprocess block.
For a full resumable checkpoint (optimizer state, scheduler state, random state included), use the state API:
# Register extra objects you want included
accelerator.registerforcheckpointing(scheduler)
Save
accelerator.savestate("checkpoints/epoch5")
Restore
accelerator.loadstate("checkpoints/epoch5")
This is far more reliable than hand-assembling a checkpoint dictionary, especially if you later move to DeepSpeed or FSDP, which use their own sharded formats.
A Complete Example
Here is a full script you can run as a skeleton.
import torch
import torch.nn as nn
from torch.utils.data import DataLoader, TensorDataset
from accelerate import Accelerator
EPOCHS = 5
def main():
accelerator = Accelerator(mixedprecision="bf16", gradientaccumulationsteps=2)
# Dummy data: 2000 samples, 20 features, 3 classes
x = torch.randn(2000, 20)
y = torch.randint(0, 3, (2000,))
ds = TensorDataset(x, y)
trainloader = DataLoader(ds, batchsize=32, shuffle=True)
model = nn.Sequential(
nn.Linear(20, 128), nn.ReLU(),
nn.Linear(128, 64), nn.ReLU(),
nn.Linear(64, 3),
)
optimizer = torch.optim.AdamW(model.parameters(), lr=1e-3)
scheduler = torch.optim.lrscheduler.CosineAnnealingLR(optimizer, Tmax=EPOCHS)
criterion = nn.CrossEntropyLoss()
model, optimizer, trainloader, scheduler = accelerator.prepare(
model, optimizer, trainloader, scheduler
)
for epoch in range(EPOCHS):
model.train()
for xb, yb in trainloader:
with accelerator.accumulate(model):
loss = criterion(model(xb), yb)
accelerator.backward(loss)
optimizer.step()
optimizer.zerograd()
scheduler.step()
accelerator.print(f"epoch {epoch+1}/{EPOCHS} loss {loss.item():.4f}")
accelerator.waitforeveryone()
unwrapped = accelerator.unwrapmodel(model)
accelerator.save(unwrapped.statedict(), "modelfinal.pth")
accelerator.print("done, model saved")
if name == "main":
main()
Run it with:
accelerate launch example.py
Experiment Tracker Integration
Accelerate wraps several popular trackers so logging automatically happens only on the main process.
accelerator = Accelerator(logwith="wandb")
accelerator.inittrackers("classification-project", config={"lr": 1e-3, "epochs": 5})
accelerator.log({"trainloss": loss.item(), "epoch": epoch}, step=globalstep)
accelerator.endtraining()
Swap "wandb" for "tensorboard", "mlflow", or "cometml" without changing your accelerator.log calls.
Large Models: DeepSpeed and FSDP
If your model does not fit on a single GPU, you need parameter sharding. The good news is that your training code does not change at all, only the configuration does.
Run accelerate config, answer yes when asked about DeepSpeed or FSDP, and pick your ZeRO stage. The same accelerate launch train.py then runs sharded training.
Two important notes here. First, with DeepSpeed ZeRO-3 or FSDP you must checkpoint through accelerator.savestate, since a plain statedict may contain only the local shard. Second, unwrapmodel needs extra arguments in some configurations to gather full parameters, so read the warnings printed in your logs.
Tips and Best Practices
Always wrap your code in a main() function guarded by if name == "main":. In multi-process launches, module-level code can execute repeatedly with side effects you did not intend.
Do not call .to(device) manually after prepare. If you need the current device to create a new tensor, use accelerator.device.
Remember that DataLoader batchsize is per process. With 4 GPUs and batch size 32, your effective batch is 128. Adjust the learning rate accordingly.
If you use a per-step scheduler rather than a per-epoch one, pass the scheduler through prepare so Accelerate adjusts its step count for the number of processes.
Debug on a single process first. Only scale to multi-GPU once the loop is provably correct.
Be careful with operations that rely on a global batch index, such as writing one file per sample. Local indices are not globally unique across processes.
Conclusion
Accelerate hits a sweet spot between raw PyTorch, which is too manual, and a heavy framework that takes over everything. The summary:
You keep writing an ordinary PyTorch loop and add only Accelerator(), prepare(), and accelerator.backward().
The same code runs on CPU, single GPU, multi-GPU, multi-node, MPS, and TPU without branching.
Mixed precision and gradient accumulation become configuration options rather than bug-prone extra code.
Use accelerator.print, ismainprocess, and waitforeveryone so logging and file writes behave in multi-process runs.
Use gatherformetrics for correct evaluation and savestate or unwrap_model for correct checkpointing.
For large models, DeepSpeed and FSDP are enabled through configuration without touching training code.
Take one of your existing training scripts, change those five lines, and launch it with accelerate launch. There is a genuinely satisfying moment when you realize the script that used to run only on your laptop is now server ready with no rewrite. Happy training.