FiftyOne by Voxel51: The Complete Guide to Dataset Curation and Model Evaluation for Computer Vision
Introduction
Building high-quality computer vision models is not just about choosing the right architecture or training for more epochs. In practice, the quality of your dataset determines the ceiling of your model's performance. Noisy labels, duplicate images, class imbalances, and edge cases buried deep in your data can silently degrade your results. Yet most practitioners spend far more time tuning hyperparameters than inspecting their data.
FiftyOne, developed by Voxel51, is an open-source toolkit designed to solve exactly this problem. It provides a powerful set of tools for exploring, visualizing, curating, and evaluating computer vision datasets and model predictions. Whether you are working on image classification, object detection, instance segmentation, or keypoint estimation, FiftyOne gives you a systematic way to understand what is happening in your data and your models.
In this tutorial, we will walk through FiftyOne from installation to advanced usage. You will learn how to load datasets from various sources, launch the interactive App to visually explore your data, evaluate model predictions against ground truth, identify annotation errors, use embeddings for dataset curation, and integrate FiftyOne into your existing PyTorch, TensorFlow, and Hugging Face workflows. Every section includes runnable code examples that you can copy and execute in your own environment.
By the end of this guide, you will have a practical understanding of how FiftyOne fits into a modern computer vision pipeline and how it can help you build better datasets and, consequently, better models.
Installation
FiftyOne runs on Python 3.8 and above. The recommended way to install it is via pip. It is best practice to use a virtual environment to avoid dependency conflicts with other projects.
Basic Installation
# Create and activate a virtual environment
python -m venv fiftyone-env
source fiftyone-env/bin/activate # On Windows: fiftyone-env\Scripts\activate
Install FiftyOne
pip install fiftyone
Installation with Optional Dependencies
FiftyOne has several optional integrations. If you plan to work with specific frameworks, you can install the relevant extras.
# Install with all optional dependencies
pip install fiftyone[all]
Or install specific extras
pip install fiftyone torch torchvision # For PyTorch integration
pip install fiftyone tensorflow # For TensorFlow integration
pip install fiftyone transformers # For Hugging Face integration
Verifying the Installation
After installation, verify that everything works by launching a quick test.
import fiftyone as fo
Print the version
print(fo.version)
Load a small sample dataset to verify
dataset = fo.zoo.loadzoodataset("quickstart", maxsamples=10)
print(dataset)
If the installation is successful, you should see a summary of the dataset printed to your terminal. FiftyOne ships with a built-in dataset zoo that includes popular datasets like COCO, CIFAR-10, ImageNet, and more, making it easy to get started without downloading anything manually.
Launching the FiftyOne App
FiftyOne includes a browser-based application for visual data exploration. You can launch it from Python.
import fiftyone as fo
dataset = fo.zoo.loadzoodataset("quickstart", maxsamples=50)
Launch the App
session = fo.launchapp(dataset)
This opens a web interface at http://localhost:5151 where you can browse images, filter by labels, view model predictions, and interactively explore your dataset. The App is one of FiftyOne's most powerful features, and we will reference it throughout this tutorial.
Basic Usage
Loading Your Own Dataset
While the dataset zoo is convenient for experimentation, you will typically want to work with your own data. FiftyOne supports loading data from a variety of formats.
Loading Images from a Directory
The simplest case is loading a directory of images without any labels.
import fiftyone as fo
dataset = fo.Dataset.fromdir(
datasetdir="/path/to/your/images",
datasettype=fo.types.ImageDirectory,
name="my-image-dataset",
)
print(dataset)
print(dataset.first())
Loading a COCO-Format Dataset
If your dataset uses the COCO format (common for object detection and segmentation), FiftyOne can parse it directly.
import fiftyone as fo
dataset = fo.Dataset.fromdir(
datasettype=fo.types.COCODetectionDataset,
datapath="/path/to/images",
labelspath="/path/to/annotations.json",
name="my-coco-dataset",
)
print(dataset)
Loading a YOLO-Format Dataset
For datasets in YOLO format, FiftyOne handles the conversion seamlessly.
import fiftyone as fo
dataset = fo.Dataset.fromdir(
datasettype=fo.types.YOLOv5Dataset,
datasetdir="/path/to/yolo-dataset",
split="train",
name="my-yolo-dataset",
)
Loading from CSV or Custom Formats
For custom formats, you can build a dataset sample by sample.
import fiftyone as fo
import csv
dataset = fo.Dataset(name="custom-dataset")
with open("annotations.csv", "r") as f:
reader = csv.DictReader(f)
for row in reader:
sample = fo.Sample(filepath=row["imagepath"])
# Add a classification label
sample["groundtruth"] = fo.Classification(label=row["label"])
# Add custom metadata
sample["confidencescore"] = float(row["score"])
dataset.addsample(sample)
print(dataset)
Exploring Data in the App
Once your dataset is loaded, the FiftyOne App becomes your primary tool for visual exploration.
import fiftyone as fo
dataset = fo.zoo.loadzoodataset("quickstart", maxsamples=200)
session = fo.launchapp(dataset)
In the App, you can:
- Browse images in a grid layout and click on any image to see its full details.
- Filter by label to show only images containing a specific class (e.g., "person" or "car").
- Sort by metadata fields such as file size, number of detections, or any custom field you add.
- Select samples to create subsets for further analysis.
Using Views to Filter Data
Views are a core concept in FiftyOne. A view is a filtered, sorted, or transformed perspective on your dataset. Views do not copy data; they are lightweight references.
import fiftyone as fo
from fiftyone import ViewField as F
dataset = fo.zoo.loadzoodataset("quickstart", maxsamples=200)
Filter to samples that contain at least one "person" detection
view = dataset.filterlabels(
"groundtruth", F("label") == "person"
)
print(f"Samples with 'person': {len(view)}")
Filter to high-confidence predictions only
highconfview = dataset.filterlabels(
"predictions", F("confidence") > 0.9
)
Sort by number of detections (descending)
sortedview = dataset.sortby(
F("groundtruth.detections").length(), reverse=True
)
Show the filtered view in the App
session = fo.launchapp(view)
Tagging Samples
Tags allow you to mark samples for later reference, such as flagging problematic images for review.
# Tag specific samples
for sample in dataset.take(10):
sample.tags.append("needs-review")
sample.save()
Retrieve tagged samples
reviewview = dataset.matchtags("needs-review")
print(f"Samples needing review: {len(reviewview)}")
Working with Different Label Types
FiftyOne supports all common computer vision label types.
import fiftyone as fo
sample = fo.Sample(filepath="/path/to/image.jpg")
Classification
sample["classification"] = fo.Classification(
label="cat", confidence=0.95
)
Object detection (bounding box in [x, y, width, height] relative format)
sample["detections"] = fo.Detections(detections=[
fo.Detection(
label="cat",
boundingbox=[0.1, 0.2, 0.3, 0.4], # relative coordinates
confidence=0.92,
),
fo.Detection(
label="dog",
boundingbox=[0.5, 0.1, 0.4, 0.6],
confidence=0.87,
),
])
Semantic segmentation
import numpy as np
mask = np.zeros((480, 640), dtype=np.uint8)
mask[100:300, 200:500] = 1 # Region labeled as class 1
sample["segmentation"] = fo.Segmentation(mask=mask)
Keypoints
sample["keypoints"] = fo.Keypoints(keypoints=[
fo.Keypoint(
label="person",
points=[(0.5, 0.1), (0.45, 0.3), (0.55, 0.3)], # relative coords
),
])
Advanced Usage
Evaluating Model Predictions
One of FiftyOne's most valuable features is its ability to evaluate model predictions against ground truth annotations. This goes beyond simple aggregate metrics by letting you drill into per-sample and per-detection results.
Object Detection Evaluation
import fiftyone as fo
import fiftyone.zoo as foz
Load a dataset with both ground truth and predictions
dataset = foz.loadzoodataset("quickstart", maxsamples=200)
Run evaluation (COCO-style mAP)
results = dataset.evaluatedetections(
"predictions", # field containing model predictions
gtfield="groundtruth", # field containing ground truth
evalkey="eval", # key to store per-sample results
computemAP=True,
)
Print aggregate metrics
print(f"mAP: {results.mAP():.4f}")
results.printreport()
The evaluation results are stored directly on each sample and each detection, allowing you to filter and visualize specific types of errors.
from fiftyone import ViewField as F
Find all false positive detections
fpview = dataset.filterlabels("predictions", F("eval") == "fp")
print(f"Samples with false positives: {len(fpview)}")
Find all false negative (missed) detections
fnview = dataset.filterlabels("groundtruth", F("eval") == "fn")
print(f"Samples with missed detections: {len(fnview)}")
Visualize false positives in the App
session = fo.launchapp(fpview)
Classification Evaluation
import fiftyone as fo
Assuming your dataset has "groundtruth" and "predictions" classification fields
results = dataset.evaluateclassifications(
"predictions",
gtfield="groundtruth",
evalkey="eval",
)
Print a classification report
results.printreport()
Plot a confusion matrix
plot = results.plotconfusionmatrix()
plot.show()
Finding Annotation Errors
Annotation errors are one of the most common sources of model degradation, yet they are notoriously difficult to find manually. FiftyOne provides tools to systematically identify mislabeled samples.
Using Model Predictions to Flag Errors
A practical approach is to use a well-trained model's predictions to flag samples where the model confidently disagrees with the ground truth label.
import fiftyone as fo
import fiftyone.brain as fob
from fiftyone import ViewField as F
dataset = fo.zoo.loadzoodataset("quickstart", maxsamples=500)
Compute mistakenness scores
This analyzes how "mistaken" each label might be based on model predictions
fob.computemistakenness(
dataset,
"predictions", # model predictions field
labelfield="groundtruth", # ground truth field
)
Sort by mistakenness (most likely errors first)
mistakeview = dataset.sortby("mistakenness", reverse=True)
View the top 50 most likely annotation errors
topmistakes = mistakeview.take(50)
session = fo.launchapp(topmistakes)
The mistakenness score ranks samples by how likely it is that the ground truth label is incorrect. Reviewing the top-ranked samples often reveals genuine annotation errors that would be nearly impossible to find by browsing randomly.
Finding Duplicate and Near-Duplicate Images
Duplicate images in your dataset can bias training and inflate evaluation metrics.
import fiftyone as fo
import fiftyone.brain as fob
dataset = fo.zoo.loadzoodataset("quickstart", maxsamples=500)
Compute image similarity using embeddings
fob.computesimilarity(
dataset,
brainkey="imgsim",
)
Find near-duplicate images
dupview = fob.computeexactduplicates(dataset)
print(f"Found {len(dupview)} exact duplicates")
For near-duplicates, use similarity index
results = dataset.sortbysimilarity(
dataset.first().id, # reference image
brainkey="imgsim",
k=10, # top 10 most similar
)
session = fo.launchapp(results)
Dataset Curation with Embeddings
Embeddings provide a numerical representation of your images that captures semantic content. FiftyOne uses embeddings for visualization, similarity search, and dataset curation.
Computing and Visualizing Embeddings
import fiftyone as fo
import fiftyone.brain as fob
import fiftyone.zoo as foz
dataset = foz.loadzoodataset("quickstart", maxsamples=500)
Compute embeddings using a pre-trained model
fob.computevisualization(
dataset,
brainkey="imgviz",
method="umap", # Dimensionality reduction method
)
Launch the App - embeddings plot will be available
session = fo.launchapp(dataset)
In the App, click the "Embeddings" tab to see the 2D visualization
The embeddings visualization reveals the structure of your dataset. Clusters correspond to visually similar images, and outliers may indicate unusual or mislabeled samples.
Finding Representative Subsets
When you have a very large dataset but limited labeling budget, you want to select a diverse, representative subset for annotation.
import fiftyone as fo
import fiftyone.brain as fob
dataset = fo.zoo.loadzoodataset("quickstart", maxsamples=1000)
Compute representativeness scores
fob.computerepresentativeness(dataset)
Select the most representative samples
representativeview = dataset.sortby("representativeness", reverse=True)
toprepresentative = representativeview.take(100)
These 100 samples best represent the diversity of your dataset
session = fo.launchapp(toprepresentative)
Finding Unique Samples
Conversely, you might want to find the most unique or unusual samples in your dataset.
import fiftyone.brain as fob
Compute uniqueness scores
fob.computeuniqueness(dataset)
View the most unique samples
uniqueview = dataset.sortby("uniqueness", reverse=True)
session = fo.launchapp(uniqueview.take(50))
View the least unique (most redundant) samples
redundantview = dataset.sortby("uniqueness")
session.view = redundantview.take(50)
Integration with Popular Frameworks
FiftyOne is designed to fit into existing ML workflows. It integrates seamlessly with PyTorch, TensorFlow, and Hugging Face.
PyTorch Integration
You can convert a FiftyOne dataset into a PyTorch DataLoader for training or use PyTorch models to generate predictions that you then analyze in FiftyOne.
import fiftyone as fo
import fiftyone.utils.torch as fout
import torch
import torchvision
from torchvision import transforms
Load dataset
dataset = fo.zoo.loadzoodataset("cifar10", split="test", maxsamples=500)
Convert to PyTorch DataLoader
transform = transforms.Compose([
transforms.Resize((224, 224)),
transforms.ToTensor(),
transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
])
torchdataset = fout.TorchImageClassificationDataset(
dataset,
labelfield="groundtruth",
transform=transform,
)
dataloader = torch.utils.data.DataLoader(
torchdataset, batchsize=32, shuffle=False
)
Run inference with a pre-trained model
model = torchvision.models.resnet50(pretrained=True)
model.eval()
Add predictions back to FiftyOne
with torch.nograd():
for batchidx, (images, ) in enumerate(dataloader):
outputs = model(images)
probs = torch.nn.functional.softmax(outputs, dim=1)
confs, preds = torch.max(probs, 1)
# Map predictions back to samples
start = batchidx 32
for i, (pred, conf) in enumerate(zip(preds, confs)):
sample = dataset.skip(start + i).first()
sample["torchprediction"] = fo.Classification(
label=str(pred.item()),
confidence=conf.item(),
)
sample.save()
Using FiftyOne's Model Zoo with PyTorch
FiftyOne also provides a model zoo with pre-trained models ready for inference.
import fiftyone as fo
import fiftyone.zoo as foz
dataset = foz.loadzoodataset("quickstart", maxsamples=100)
Load a pre-trained detection model from the zoo
model = foz.loadzoomodel("faster-rcnn-resnet50-fpn-coco-torch")
Apply the model to your dataset
dataset.applymodel(model, labelfield="zoopredictions")
Compare zoo predictions with existing predictions
session = fo.launchapp(dataset)
TensorFlow Integration
FiftyOne works with TensorFlow datasets and models in a similar fashion.
import fiftyone as fo
import fiftyone.utils.tf as fout
Load a FiftyOne dataset
dataset = fo.zoo.loadzoodataset("quickstart", maxsamples=100)
Export to TFRecords for TensorFlow training
dataset.export(
exportdir="/path/to/tfrecords",
datasettype=fo.types.TFObjectDetectionDataset,
labelfield="groundtruth",
)
Or use TensorFlow Hub models for inference
import fiftyone.zoo as foz
model = foz.loadzoomodel("ssd-mobilenet-v2-fpnlite-coco-tf2")
dataset.applymodel(model, labelfield="tfpredictions")
session = fo.launchapp(dataset)
Hugging Face Integration
FiftyOne integrates with the Hugging Face ecosystem, letting you load datasets from the Hugging Face Hub and use Transformers models for inference.
import fiftyone as fo
import fiftyone.utils.huggingface as fouh
Load a dataset from Hugging Face Hub
dataset = fouh.loadfromhub(
"detection-datasets/coco",
name="coco-hub-dataset",
maxsamples=200,
)
print(dataset)
session = fo.launchapp(dataset)
You can also use Hugging Face Transformers models to generate predictions.
import fiftyone as fo
from transformers import pipeline
from PIL import Image
dataset = fo.zoo.loadzoodataset("quickstart", maxsamples=50)
Load a Hugging Face object detection pipeline
detector = pipeline("object-detection", model="facebook/detr-resnet-50")
for sample in dataset.itersamples(progress=True):
img = Image.open(sample.filepath)
results = detector(img)
detections = []
w, h = img.size
for r in results:
box = r["box"]
# Convert from absolute to relative coordinates
x = box["xmin"] / w
y = box["ymin"] / h
bw = (box["xmax"] - box["xmin"]) / w
bh = (box["ymax"] - box["ymin"]) / h
detections.append(fo.Detection(
label=r["label"],
boundingbox=[x, y, bw, bh],
confidence=r["score"],
))
sample["hfpredictions"] = fo.Detections(detections=detections)
sample.save()
session = fo.launchapp(dataset)
Pushing Datasets to Hugging Face Hub
You can also push your curated FiftyOne datasets back to the Hugging Face Hub for sharing and reproducibility.
import fiftyone.utils.huggingface as fouh
Push your curated dataset to the Hub
fouh.pushtohub(
dataset,
"your-username/curated-detection-dataset",
private=True,
)
Comparing Multiple Models
A common workflow is comparing predictions from multiple models to understand their relative strengths and weaknesses.
import fiftyone as fo
import fiftyone.zoo as foz
dataset = foz.loadzoodataset("quickstart", maxsamples=200)
Apply two different models
modela = foz.loadzoomodel("faster-rcnn-resnet50-fpn-coco-torch")
modelb = foz.loadzoomodel("ssd-mobilenet-v2-fpnlite-coco-tf2")
dataset.applymodel(modela, labelfield="modelapreds")
dataset.applymodel(modelb, labelfield="modelbpreds")
Evaluate both models
resultsa = dataset.evaluatedetections(
"modelapreds", gtfield="groundtruth",
evalkey="evala", computemAP=True,
)
resultsb = dataset.evaluatedetections(
"modelbpreds", gtfield="groundtruth",
evalkey="evalb", computemAP=True,
)
print(f"Model A mAP: {resultsa.mAP():.4f}")
print(f"Model B mAP: {resultsb.mAP():.4f}")
Find samples where Model A is correct but Model B fails
from fiftyone import ViewField as F
acorrectbwrong = dataset.match(
(F("modelapreds.detections").length() > 0) &
(F("evalbfp") > F("evalafp"))
)
session = fo.launchapp(acorrectbwrong)
Best Practices
1. Always Persist Your Datasets
By default, FiftyOne datasets are persistent and stored in a local MongoDB database. Give your datasets meaningful names so you can reload them later.
import fiftyone as fo
Create a named, persistent dataset
dataset = fo.Dataset(name="project-v2-train", persistent=True)
Later, reload it
dataset = fo.loaddataset("project-v2-train")
List all available datasets
print(fo.listdatasets())
2. Use Views Instead of Copying Data
Views are lightweight and do not duplicate data. Use them liberally to slice and filter your dataset for analysis.
# Good: use views for filtering
smallobjects = dataset.filterlabels(
"groundtruth",
(F("boundingbox")[2] F("boundingbox")[3]) < 0.01
)
Avoid: creating new datasets for every filter operation
This wastes storage and is harder to manage
3. Track Dataset Versions
When you modify labels or remove samples, keep track of what changed.
# Use tags to track dataset versions
for sample in dataset:
sample.tags.append("v1.0")
sample.save()
After cleaning, tag the new version
cleanedview = dataset.matchtags("v1.0").match(F("quality") == "good")
for sample in cleanedview:
sample.tags.append("v1.1-cleaned")
sample.save()
4. Automate Quality Checks
Build quality check pipelines that run automatically before training.
import fiftyone as fo
import fiftyone.brain as fob
def runqualitychecks(dataset):
"""Run a standard suite of quality checks on a dataset."""
# Check for duplicates
fob.computeexactduplicates(dataset)
# Compute uniqueness
fob.computeuniqueness(dataset)
# Flag low-uniqueness samples
redundant = dataset.match(F("uniqueness") < 0.1)
print(f"Warning: {len(redundant)} potentially redundant samples found")
# Check for missing labels
unlabeled = dataset.match(~F("groundtruth").exists())
print(f"Warning: {len(unlabeled)} samples have no labels")
# Check label distribution
counts = dataset.countvalues("groundtruth.detections.label")
print("Label distribution:")
for label, count in sorted(counts.items(), key=lambda x: -x[1]):
print(f" {label}: {count}")
return dataset
dataset = fo.loaddataset("my-project")
runqualitychecks(dataset)
5. Export Clean Datasets for Training
After curation, export your cleaned dataset in the format your training pipeline expects.
import fiftyone as fo
dataset = fo.loaddataset("my-project")
Create a clean view (exclude flagged samples)
cleanview = dataset.matchtags("approved")
Export to COCO format
cleanview.export(
exportdir="/path/to/clean-dataset",
datasettype=fo.types.COCODetectionDataset,
labelfield="groundtruth",
)
Export to YOLO format
cleanview.export(
exportdir="/path/to/clean-dataset-yolo",
datasettype=fo.types.YOLOv5Dataset,
labelfield="groundtruth",
)
6. Leverage the FiftyOne App During Model Development
Do not treat data exploration as a one-time activity. Launch the App frequently during model development to inspect failure cases, compare model iterations, and track improvements over time. The visual feedback loop is one of the most effective ways to improve model performance.
7. Use Delegated Operations for Large Datasets
When working with datasets containing hundreds of thousands or millions of samples, use FiftyOne's delegated operations to run computations asynchronously.
import fiftyone as fo
import fiftyone.brain as fob
dataset = fo.loaddataset("large-dataset")
For large datasets, delegate heavy computations
fob.computevisualization(
dataset,
brainkey="embeddings",
method="umap",
numworkers=8, # Parallelize computation
)
Conclusion
FiftyOne fills a critical gap in the computer vision workflow. While there are many tools for model training and deployment, few provide robust support for the data-centric tasks that often determine whether a project succeeds or fails. By giving you the ability to visualize, explore, evaluate, and curate your datasets systematically, FiftyOne transforms data quality from a guessing game into an engineering discipline.
In this tutorial, we covered the full spectrum of FiftyOne's capabilities. We started with installation and basic dataset loading, moved through visual exploration with the App and views, and then advanced to model evaluation, annotation error detection, and embedding-based curation. We also demonstrated how FiftyOne integrates with PyTorch, TensorFlow, and Hugging Face, ensuring that it fits into whatever ML stack you are already using.
The key takeaways are:
- Data quality matters more than model complexity. FiftyOne gives you the tools to systematically improve your data.
- Visual exploration is not optional. The FiftyOne App lets you see what aggregate metrics cannot tell you.
- Evaluation should be granular. Per-sample and per-detection evaluation results help you understand exactly where your model fails.
- Annotation errors are common and findable. The mistakenness computation can surface labeling mistakes that would otherwise go unnoticed.
- Embeddings unlock dataset curation at scale. Similarity search, uniqueness scoring, and representativeness help you build better datasets with less labeling effort.
- Integration is seamless. FiftyOne works with the frameworks and formats you already use, so adopting it does not require rearchitecting your pipeline.
To continue learning, visit the official FiftyOne documentation at docs.voxel51.com, explore the FiftyOne Tutorials repository on GitHub, and join the Voxel51 community on Slack. The open-source community around FiftyOne is active and welcoming, and new features and integrations are added regularly.
Start by loading one of your existing datasets into FiftyOne today. Spend thirty minutes exploring it in the App. You will almost certainly discover something about your data that you did not know before, and that discovery is the first step toward building a better model.