Pixeltable: Declarative AI Data Infrastructure for Multimodal Data
If you have ever built an AI pipeline that has to process images, video, audio, or documents, you probably know the same particular kind of fatigue: your data is scattered everywhere. There is a folder with raw files, a script for resizing and converting, another folder for embedding outputs, a separate vector database, one CSV file holding model predictions, and one messy notebook trying to glue all of it together. The moment a single new piece of data arrives, you have to rerun half the pipeline by hand and pray the order is right. Pixeltable is the answer to this chaos.
In this tutorial I want to walk you through Pixeltable from scratch. We start with installation, creating a table, and inserting data, then climb up to the parts that make Pixeltable special: computed columns that update automatically, UDFs, an embedding index for similarity search, and how it handles everything from images to video frames. I am going to keep the tone relaxed so it is easy to digest, but every bit of code here is runnable.
What Pixeltable Is and Why Declarative Matters
Pixeltable is a Python library for AI data infrastructure that is declarative by design. What "declarative" means here is that you describe what you want (for example, "this column contains thumbnails of the images in that column") rather than writing out step by step when and how those thumbnails get made. Pixeltable handles the execution, the caching, and deciding when something needs to be recomputed.
The core idea is simple but powerful: all your data, whether it is text, images, video, audio, or documents, lives inside a table. This table is like a database table, with rows and columns. The difference is that a single column can natively store heavy data types like images or video, not just numbers and strings. On top of that, you can add a computed column, which is a column whose value is derived from other columns through a function. When new data arrives, the computed column fills itself in automatically. When you add a new computed column to a table that already has data, Pixeltable immediately backfills it for every existing row.
Why does this matter? Because AI work is 90 percent repetitive data transformation: resize images, extract video frames, call a model for inference, build embeddings, and store the index. If all of those steps can be expressed as columns, you no longer need to write loops, you no longer need to manage cache files by hand, and you no longer have to fear forgetting to rerun one of the stages. Pixeltable also automatically stores your data along with its versions, so every change is traceable and can be rolled back.
Compare this with the old way. Normally I have one preprocessing script, one inference script, one vector database, and one artifacts directory. Four different systems that I have to keep in sync myself. In Pixeltable, those four things become one table with a few computed columns plus one embedding index. One source of truth, and everything is connected.
Key Concepts to Understand
Before we code, here are a few terms that will keep showing up:
- Table: the main container for data, similar to a SQL table but supporting multimodal types.
- Column: either a regular column (data you insert) or a computed column (derived automatically).
- Insert: how you add new data to a table. Each insert triggers recomputation of the related computed columns.
- UDF (User-Defined Function): a plain Python function you register with Pixeltable to use in computed columns.
- Embedding index: a vector index attached to a column, used for similarity search.
- View and iterator: a way to "explode" one row into many rows, for example turning one video into many frames.
Now let us jump into the practice.
Installation
Pixeltable requires Python 3.9 or higher. I recommend creating a virtual environment first to keep things clean, then installing with pip.
# Create and activate a virtual environment (run in your terminal)
python -m venv venv
source venv/bin/activate # Linux / macOS
venv\Scripts\activate # Windows
Install Pixeltable
pip install pixeltable
Once it is installed, try importing it to make sure everything is fine:
import pixeltable as pxt
print(pxt.version)
The first time you run it, Pixeltable creates a local storage directory (by default at ~/.pixeltable). That is where all your data, metadata, and media files are stored. So the tables you create are persistent, meaning if your script stops and you rerun it tomorrow, the tables are still there. This is different from a Pandas DataFrame, which vanishes the moment the program exits.
For certain features, you may need extra dependencies. For example, for Hugging Face models, install transformers and torch. For video processing, make sure you have the required codecs (usually already bundled). For calling external model APIs like OpenAI, install the SDK and set the API key via an environment variable.
# Example of commonly used extra dependencies
pip install transformers torch sentence-transformers
pip install openai
One thing you should know: Pixeltable organizes tables using directories and namespaces. So we can group tables into logical directories, similar to folders. This becomes very handy once a project grows large.
import pixeltable as pxt
Create a directory for our project (safe if it already exists)
pxt.createdir('demo', ifexists='ignore')
Basic Usage: Tables, Insert, and Query
Now we create our first table. Imagine we are building a simple product catalog that stores product images along with their descriptions.
Create a Table
Creating a table is done with pxt.createtable, where we define a schema of column names and their types.
import pixeltable as pxt
Clean up any old table with the same name first (optional while experimenting)
pxt.droptable('demo.products', ifnotexists='ignore')
Create a table with a column schema
products = pxt.createtable(
'demo.products',
{
'name': pxt.String,
'category': pxt.String,
'price': pxt.Float,
'image': pxt.Image,
'description': pxt.String,
}
)
print(products)
Notice the pxt.Image type. This is not just a path string. Pixeltable understands that the column contains images, so it can display, resize, or pass them to a computer vision model directly. Other multimodal types available include pxt.Video, pxt.Audio, pxt.Document, and pxt.Json. There are also standard types like pxt.String, pxt.Int, pxt.Float, pxt.Bool, pxt.Timestamp, and pxt.Array.
Insert Data
Data is inserted with insert. For an image column, you just provide a local path or a URL. Pixeltable takes care of loading it.
products.insert([
{
'name': 'Lightweight Running Shoes',
'category': 'sports',
'price': 55.0,
'image': 'https://raw.githubusercontent.com/pixeltable/pixeltable/main/docs/resources/images/000000000009.jpg',
'description': 'Running shoes with a cushioned sole and breathable mesh material.',
},
{
'name': 'Canvas Backpack',
'category': 'accessories',
'price': 30.0,
'image': 'https://raw.githubusercontent.com/pixeltable/pixeltable/main/docs/resources/images/000000000025.jpg',
'description': 'A durable canvas backpack with many compartments.',
},
])
print(f'Total rows now: {products.count()}')
We can insert a single dict or a list of dicts. If you have a lot of data, inserts can be done in batches and Pixeltable will process them efficiently.
Query Data
Querying in Pixeltable feels like a mix of Pandas and SQL, but with its own API. We select the columns we want using bracket notation, apply a filter with where, then call collect to fetch the results.
# Get all products in the sports category
result = products.where(products.category == 'sports').select(
products.name, products.price
).collect()
print(result)
Get products priced under 40
cheap = products.where(products.price < 40).select(
products.name, products.price
).collect()
print(cheap)
collect returns a result object that you can convert to a Pandas DataFrame with .topandas() if needed, or iterate directly. To glance at a table's contents, just call products.head() or products.show(3).
# View the first 3 rows
print(products.head(3))
Also select the image column (it will be an image object)
sample = products.select(products.name, products.image).head(2)
print(sample)
At this point we already have a table that stores multimodal data persistently and can be queried. But the real power of Pixeltable only shows up in the next section.
Advanced Usage: Computed Columns, UDFs, and Embeddings
Computed Column with Built-in Functions
Computed columns are the heart of Pixeltable. The basic idea: we add a column whose value is the result of an expression over other columns. Since our data is images, let us create a computed column that produces thumbnails by using the built-in operations on the image type.
# Add a thumbnail computed column (resize the image to 224x224)
products.addcomputedcolumn(
thumbnail=products.image.resize([224, 224])
)
Add a computed column that stores the original image width
products.addcomputedcolumn(
imagewidth=products.image.width
)
print(products.select(products.name, products.imagewidth).collect())
Here is the cool part: because we add these columns after data is already in the table, Pixeltable immediately computes thumbnail and imagewidth for all existing rows. And more importantly, when a new product is inserted later, both columns fill themselves in automatically without us running anything else. This is what I mean by "updates automatically".
Image objects in Pixeltable essentially wrap a PIL Image, so many PIL operations are available as methods, like resize, rotate, crop, convert, and properties like width, height, mode.
Writing Your Own UDF
Built-in functions are limited. If you want custom logic, write a UDF. You just write a plain Python function and add the @pxt.udf decorator. Type annotations on the parameters and return value are important so Pixeltable understands the column types.
import pixeltable as pxt
@pxt.udf
def pricelabel(price: float) -> str:
"""Categorize the price into an easy-to-read label."""
if price < 20:
return 'cheap'
elif price < 50:
return 'mid'
else:
return 'premium'
Use the UDF as a computed column
products.addcomputedcolumn(
pricesegment=pricelabel(products.price)
)
print(products.select(products.name, products.price, products.pricesegment).collect())
UDFs are not limited to simple types. You can write a UDF that takes an image, processes it with OpenCV or NumPy, and returns an image or an array. Here is a UDF that computes the average brightness of an image:
import PIL.Image
import numpy as np
@pxt.udf
def avgbrightness(img: PIL.Image.Image) -> float:
"""Compute the average brightness of an image (0-255)."""
arr = np.asarray(img.convert('L')) # grayscale
return float(arr.mean())
products.addcomputedcolumn(
brightness=avgbrightness(products.image)
)
print(products.select(products.name, products.brightness).collect())
Notice that the UDF parameter is typed as PIL.Image.Image. Pixeltable automatically decodes the image column into a PIL object before handing it to our function. This is what makes writing multimodal logic feel light: we focus on the logic, and the decoding is handled for us.
Model Inference as a Computed Column
One of the most common use cases is calling an AI model in a computed column. Pixeltable has many built-in functions for model integration, for example through the pixeltable.functions module. As an example, we could use an image classification model from Hugging Face to automatically label each product image.
from pixeltable.functions.huggingface import clipimage, cliptext
Conceptual example: zero-shot classification with CLIP
(requires transformers + torch installed)
candidate
categories = ['shoes', 'bag', 'shirt', 'accessory', 'electronics']
Build image and text embeddings via the model, then compare them.
The exact API can vary slightly between versions, so check the pixeltable.functions docs.
The important pattern is not the specific API of any particular model, but the concept: a model call becomes a column. As soon as new data arrives, the inference runs automatically and the result is stored. No more separate batch inference script that you have to run manually, and no more "I forgot to run the model on the data that came in yesterday" problem.
Embedding Index and Similarity Search
Now the part people ask about most: how to build semantic search. In Pixeltable, you add an embedding index to a column, and Pixeltable handles creating the embeddings plus storing the vectors. To find similar items, you just call similarity.
Let us create an embedding index for the text description column using a sentence-transformers model.
from pixeltable.functions.huggingface import sentencetransformer
Register an embedding for the text column 'description'
products.add
embeddingindex(
column='description',
string
embed=sentencetransformer.using(
model
id='sentence-transformers/all-MiniLM-L6-v2'
)
)
After the index is set up, we can find products whose description is most similar to a text query. We use the similarity expression inside the query, then sort.
query = 'gear for morning runs'
result = (
products
.orderby(products.description.similarity(query), asc=False)
.select(products.name, products.description)
.limit(3)
.collect()
)
print(result)
The key thing to underline: this embedding index is also alive. If a new product is inserted later, Pixeltable automatically creates an embedding for that new row and adds it to the index. So the search is always up to date without us rebuilding the index manually. Compared to the usual workflow where you manage a separate vector database and sync embeddings yourself, this is far more concise.
Pixeltable also supports embedding indexes for images. For example, using a CLIP model, we can build an index over the image column, then search for images similar to a text query (text to image search) or to another image (image to image search). The concept is exactly the same, only the embedding model is swapped for one that supports images.
Working with Images and Video Frames
Video is where Pixeltable truly shines, because a single video is thousands of frames and managing that by hand is a headache. Pixeltable uses the concept of a view with an iterator to "explode" one video into many frame rows.
First, we create a table to store videos.
import pixeltable as pxt
pxt.droptable('demo.videos', ifnotexists='ignore')
videos = pxt.createtable(
'demo.videos',
{'source': pxt.Video}
)
videos.insert([
{'source': 'https://raw.githubusercontent.com/pixeltable/pixeltable/main/docs/resources/bangkok.mp4'},
])
Now we create a view that extracts frames from each video using FrameIterator. Each frame becomes one row in the view, complete with a frame column (the image) and a frame position number.
from pixeltable.iterators import FrameIterator
Create a view that grabs 1 frame per second (fps=1)
frames = pxt.createview(
'demo.frames',
videos,
iterator=FrameIterator.create(video=videos.source, fps=1)
)
print(f'Number of frames extracted: {frames.count()}')
print(frames.select(frames.frame, frames.pos).head(3))
What is interesting is that this view is not a copy of the data. It is a derived view computed from the parent video table. When you add a new video to the videos table, the frames view automatically adds frame rows for that video. And since frame is also an image-typed column, we can attach computed columns or an embedding index on top of it, exactly like a regular image table.
# Example: add an object-detection computed column per frame (conceptual)
frames.addcomputedcolumn(detections=yolomodel(frames.frame))
Or build an embedding index to search frames via text
frames.addembeddingindex(column='frame', embedding=clipembed)
With this pattern, building a "find a moment inside a video using a sentence" system becomes just a few lines: create a video table, create a frame view, attach a CLIP embedding on the frame column, then query with similarity. The entire decoding, frame sampling, embedding, and indexing pipeline is handled declaratively.
Incremental Updates and Versioning
One of Pixeltable's promises is incremental processing. This means it only recomputes what needs to change, not everything. If you insert 5 new rows into a table that already has 1000 rows, the computed columns and embeddings are only computed for those 5 rows. The old ones are untouched.
Updating a column value also triggers precisely targeted recomputation. For example, if we change a product's price, the computed columns that depend on price (like pricesegment) get recomputed only for that row.
# Update the price of a specific product
products.where(products.name == 'Canvas Backpack').update(
{'price': 18.0}
)
The pricesegment column automatically changes to 'cheap'
print(products.select(products.name, products.price, products.pricesegment).collect())
Pixeltable also stores versions. Each operation that changes a table (insert, update, add column) bumps the table version. We can view the history and even revert to a previous version with revert. This makes experimentation safe: if an addcomputedcolumn turns out to be wrong, just revert.
# View the current table version via metadata
print(products.getmetadata())
Go back one step to the previous version
products.revert()
This versioning capability also makes data lineage traceable. Because each computed column knows which columns it was computed from and with what function, you can always trace the origin of a value. For projects that need reproducibility, this is a big plus.
Best Practices
After using Pixeltable a few times in real projects, there are some things that in my experience make the experience much smoother.
First, organize tables with directories from the start. Usepxt.createdir to group tables per project or per domain. Clean table names like catalog.products or research.experiment1 will save you from confusion once you have many tables. Remember, tables are persistent, so they will not disappear on their own.
Second, think of computed columns as a pipeline. Instead of making one giant UDF that does five things at once, break it into several chained computed columns. For example a frame column, then detectedframe, then objectcount. This makes each stage debuggable on its own, and Pixeltable can cache the intermediate results. If one stage fails, you do not lose the previous stage.
Third, be careful with expensive UDFs. If a UDF calls a large model or an external API, every insert will trigger that call. For local models, consider batching. For paid APIs, be aware that adding a computed column to a large table will call the API for all existing rows at once, and that can get costly. Test on a small table first.
Fourth, lean on the embedding index instead of managing vectors yourself. The temptation to export embeddings to an external vector database is there, but as long as your needs are met, let Pixeltable handle it. Embeddings that stay in sync with the data automatically eliminate an entire class of sync bugs.
Fifth, use ifexists and ifnotexists when scripting. While experimenting, scripts are often run many times. Parameters like ifexists='ignore' on createdir or ifnotexists='ignore' on droptable make scripts idempotent, so they do not error just because a table already exists or does not exist yet.
Sixth, use correct type annotations in UDFs. Pixeltable uses type annotations to determine the output column type. Wrong annotations can produce unexpected results or errors. If your UDF returns an image, annotate it as -> PIL.Image.Image. If it returns an array, consider whether pxt.Array with a specific element type fits better.
Seventh, take advantage of the incremental nature. Because Pixeltable only computes what changed, design your workflow to feed data in gradually rather than rebuilding everything. The pattern of "insert new data and let the computed columns catch up" is far more efficient and cheaper than dropping the table and rebuilding it from scratch every time.
Conclusion
Pixeltable solves the problem that makes so many AI projects messy: multimodal data scattered across many systems plus a fragile transformation pipeline. By putting everything inside tables and expressing transformations as computed columns, you get a single source of truth that stays consistent automatically. New data comes in, and all derived columns and indexes follow immediately without us running anything.
In this tutorial we went from installation, to creating a multimodal table, inserting and querying data, all the way to the core parts: computed columns with built-in functions and UDFs, an embedding index for similarity search, video frame extraction via views and iterators, and incremental updates plus versioning. What I hope sticks in your mind is not the specific syntax, but the shift in mindset: from "I write a script that executes steps" to "I declare the shape of the data I want, and let the system execute".
Your next step: try replacing the sample dataset above with your own data, whether that is a photo collection, a PDF archive, or a video folder. Add one computed column for your favorite model's inference, attach one embedding index, then feel what an always-in-sync semantic search is like. Once you experience "insert the data and everything just handles itself", it is hard to go back to the old way. Happy tinkering, and may your AI pipelines get a lot tidier.