Daft: The Distributed Multimodal DataFrame for Large Scale ETL and Analytics

Hey everyone, it's me again, Ruby Abdullah. This time I want to introduce you to a tool that I think you'll find yourself reaching for more and more once your work starts touching data that is both large and varied in shape. It's called Daft. If you've been comfortable using pandas or Polars to crunch tables of numbers and text, Daft is like a far more ambitious version: it's designed to handle not just numbers and text, but also images, embeddings, URLs, and even the outputs of machine learning model inference, all inside the same DataFrame.

What really grabbed my attention is the combination in its design. On the surface, you write clean, readable Python code that feels a lot like pandas. But underneath, its execution engine is written in Rust, which makes it fast and memory efficient. And even better, Daft is lazy by nature: it doesn't compute anything the moment you write an operation, but instead waits until you actually ask for the result. Thanks to this laziness, Daft can optimize your entire chain of operations before running, and can even spread the work across many machines using Ray when the data is too big for a single computer.

In this tutorial I'll walk you through everything from scratch. We start with installation, then build our first DataFrame, understand the concepts of lazy execution and .collect(), read data from Parquet, CSV, and S3, play with expressions and UDFs, process image and URL columns (download, decode, resize), run model inference over columns, and finally scale out with Ray. Take it easy, I'll explain slowly with runnable examples you can try right away. Let's get started.

Introduction

Before we dive into code, I want you to understand why Daft exists and what problem it solves. Picture a scenario that's becoming increasingly common: you have millions of rows where each row contains not just numbers, but also paths to images, URLs to documents, chunks of long text, and maybe embedding vectors produced by a model. If you try to process this with pandas, you'll quickly hit two walls. The first wall is memory, because pandas keeps all data in the RAM of a single machine. The second wall is multimodal data types, because pandas has no native concept of an image column or a tensor column, so you end up storing slow, messy Python objects.

Daft was built to answer both of these problems at once. On the scale side, Daft can run on a single laptop for small data, but can also be spread across a Ray cluster for terabyte scale data without you having to rewrite your code. On the multimodal side, Daft has dedicated data types for images, tensors, embeddings, and URLs, plus built in operations for downloading, decoding, and resizing images. So a pipeline that would normally require a mix of pandas, PIL, requests, and manual parallel code can now be written neatly inside a single DataFrame API.

There are three core concepts that will keep showing up throughout this tutorial, so let me introduce them upfront. First, the DataFrame, which is the main table structure where your data lives. Second, the Expression, which is how you describe a computation over columns without immediately running it. Third, lazy execution, the principle that Daft defers all computation until you call an action like .collect() or .show(). These three concepts are what make Daft different from pandas and what allow it to be fast and scalable.

Installation

Alright, let's start with the most basic thing, which is installing Daft on your machine. One thing that sometimes confuses people: the package name on PyPI is getdaft, not daft. But when you import it in Python, you still write import daft. So don't be surprised when the install command looks different from the import name.

To install the base version, just run this command in your terminal.

pip install getdaft

If you already know you'll be working with cloud data or scaling with Ray, I suggest installing the extras at the same time so you don't have to go back and forth. Daft provides several extras you can pick based on your needs.

# Support for AWS S3, Google Cloud Storage, Azure
pip install "getdaft[aws]"

Support for scaling with Ray
pip install "getdaft[ray]"

Install everything at once
pip install "getdaft[all]"

Personally I usually just install getdaft[all] when I'm in a development environment, so I don't run into a "module not found" error halfway through. But if you're building a lean Docker image, it's better to pick only the extras you actually need.

Once installed, it's a good idea to test that everything works. Try running this little script.

import daft

print(daft.version)

df = daft.frompydict({

    "name": ["Ruby", "Sinta", "Bagas"],
    "age": [28, 24, 31],
})
df.show()

If you see the Daft version printed and a small table appears on screen, your installation is done and we're ready to move on. The daft.frompydict function above is the easiest way to build a DataFrame from a Python dictionary, where keys become column names and values become the column contents.

Basic Usage

Building Your First DataFrame

There are many ways to build a DataFrame in Daft, and the one I use most for quick experiments is frompydict. But besides that you can also build from a list of dictionaries using frompylist, or directly from a pandas DataFrame you already have. Take a look at a few of these ways.

import daft

From a dictionary of columns
df1 = daft.frompydict({

    "id": [1, 2, 3],
    "product": ["Coffee", "Tea", "Milk"],
    "price": [25000, 18000, 22000],
})

From a list of rows
df2 = daft.frompylist([
    {"id": 1, "product": "Coffee", "price": 25000},
    {"id": 2, "product": "Tea", "price": 18000},
])

From pandas
import pandas as pd
pdf = pd.DataFrame({"id": [1, 2], "product": ["Coffee", "Tea"]})
df3 = daft.frompandas(pdf)


df1.show()

Now, this is the most important part for you to grasp, so please read slowly everyone. In Daft, when you write operations like select, filter, or withcolumn, Daft doesn't immediately run that computation. All it does is record a plan of what you want to do. The computation only actually runs when you call an action, and the most common action is .collect().

Why is it designed this way? Because by waiting, Daft can see your entire chain of operations at once and optimize it. For example, if you read a huge file but only need two columns and only rows priced above 20000, Daft is smart enough to read only the needed columns and filter rows as early as possible, so the work becomes much lighter.

Take a look at this example to feel the difference.

import daft

df = daft.frompydict({

    "product": ["Coffee", "Tea", "Milk", "Juice"],
    "price": [25000, 18000, 22000, 30000],
})

All of this is still lazy, no computation has run yet
expensive = df.where(df["price"] > 20000).select("product", "price")

Only here does computation run and the result get materialized
result = expensive.collect()
print(result)

In the code above, the line expensive = ... just assembles a plan. Only when expensive.collect() is called does Daft run everything and place the result in memory. Besides .collect(), there are several other actions that also trigger execution, like .show(n) to display the first few rows, .topandas() to convert to pandas, .topydict() to convert to a dictionary, and .writeparquet() to write to disk. As long as you haven't called one of these actions, your DataFrame is still an unexecuted plan.

One tip from me: if you're developing a pipeline and want to peek at intermediate results, use .show(5), not .collect() on everything, especially if the data is large. .show() only takes a few rows so it's much faster and more memory friendly.

Filter, Select, and New Columns

The basic operations for processing tables in Daft are very similar to other DataFrame libraries, but with a distinctive expression style. To pick columns use select, to filter rows use where, and to create new columns use withcolumn. Let's combine them all.

import daft

df = daft.frompydict({
    "product": ["Coffee", "Tea", "Milk", "Juice"],
    "price": [25000, 18000, 22000, 30000],
    "stock": [10, 25, 8, 15],
})

result = (
    df
    .withcolumn("discountedprice", df["price"]  0.9)

    .withcolumn("stockvalue", df["price"]  df["stock"])
    .where(df["stock"] > 9)
    .select("product", "discountedprice", "stockvalue")
)

result.show()

Notice the chained writing style. Each operation returns a new DataFrame, so you can keep attaching the next operation. An expression like df["price"] 0.9 is an example of an Expression, which is a description of a computation over columns that hasn't been executed. I'll cover expressions in more depth in the advanced section.

import daft

df = daft.frompydict({

    "category": ["drink", "food", "drink", "food", "drink"],
    "price": [25000, 40000, 18000, 35000, 30000],
})

summary = df.groupby("category").agg(
    df["price"].mean().alias("avgprice"),
    df["price"].sum().alias("totalprice"),

    df["price"].count().alias("itemcount"),
)

summary.show()

import daft

Read one file or many files at once using a wildcard
dfcsv = daft.readcsv("data/sales.csv")
dfparquet = daft.readparquet("data/.parquet")

Read directly from S3
dfs3 = daft.readparquet("s3://my-bucket/dataset/.parquet")


dfparquet.show()

import daft
from daft.io import IOConfig, S3Config

ioconfig = IOConfig(
    s3=S3Config(
        keyid="AKIA...",

        accesskey="secret...",
        regionname="ap-southeast-1",

    )
)

df = daft.readparquet("s3://my-bucket/data/.parquet", ioconfig=ioconfig)
df.show()

Remember, because Daft is lazy, the readparquet above doesn't immediately suck the entire file into memory. Daft only reads the metadata and schema first. The data is only actually read when you call an action, and only the necessary columns and rows are read thanks to the optimization I told you about.

Expressions are the heart of Daft. Every time you write a computation over columns, you're actually building an expression. The most common way to access a column is df["columnname"], but you can also use daft.col("columnname") which is useful when you don't yet have a reference to the DataFrame. Expressions have many methods grouped into namespaces, for example .str for string operations, .dt for dates, .image for images, and .url for URLs.

import daft

df = daft.frompydict({
    "name": ["ruby abdullah", "sinta dewi", "bagas pratama"],
    "email": ["ruby@mail.com", "sinta@mail.com", "bagas@mail.com"],
})

result = df.select(
    daft.col("name").str.upper().alias("namecaps"),

    daft.col("name").str.length().alias("namelength"),
    daft.col("email").str.split("@").alias("emailparts"),

)

result.show()

import daft

@daft.udf(returndtype=daft.DataType.string())
def pricecategory(price):

    out = []
    for p in price.topylist():
        if p < 20000:
            out.append("cheap")
        elif p < 28000:
            out.append("medium")
        else:
            out.append("expensive")
    return out

df = daft.frompydict({

    "product": ["Coffee", "Tea", "Juice"],
    "price": [25000, 18000, 30000],
})

df = df.withcolumn("segment", pricecategory(df["price"]))

df.show()

Notice two important things. First, you must state returndtype so Daft knows the data type of the result column. Second, the price argument passed to the function is not a single number but a series, which is why I call .topylist() to iterate. This pattern does require a bit of adaptation if you're used to apply in pandas which is per row, but the payoff is much better performance.

import daft

df = daft.frompydict({
    "url": [
        "https://example.com/photo1.jpg",
        "https://example.com/photo2.jpg",
        "https://example.com/photo3.jpg",
    ],
})

df = (
    df
    .withcolumn("bytes", df["url"].url.download(onerror="null"))
    .withcolumn("image", daft.col("bytes").image.decode())

    .withcolumn("small", daft.col("image").image.resize(224, 224))
)

df.select("url", "small").show()

A few things to note. The onerror="null" argument in url.download is a lifesaver, because when you download thousands of URLs some will inevitably fail or be dead, and with this setting the failed row becomes null instead of crashing the entire pipeline. Then, image.resize(224, 224) is an example of standardizing to a 224x224 size, which happens to be a common input size for many vision models. All of these operations stay lazy and will run in parallel when you collect, so you get concurrent downloading and decoding for free without writing your own threading code.

import daft
import numpy as np

@daft.udf(returndtype=daft.DataType.python())
class ImageClassifier:
    def init(self):
        # Load the model once when the worker is initialized
        # Pseudo example: replace with your real model
        self.model = loadyourmodel()

    def call(self, imagecolumn):

        imagebatch = imagecolumn.topylist()
        predictions = []
        for img in imagebatch:

            arr = np.array(img)
            out = self.model.predict(arr[None, ...])
            predictions.append(out.tolist())
        return predictions

df = daft.frompydict({"url": ["https://example.com/a.jpg"]})
df = (
    df
    .withcolumn("image", df["url"].url.download(onerror="null").image.decode())
    .withcolumn("prediction", ImageClassifier(daft.col("image")))

)
df.show()

For models that need a GPU, Daft also gives a way to request resources. You can add an argument like numgpus=1 when declaring the UDF, so the scheduler knows this UDF needs a GPU and arranges its placement on the cluster correctly.

@daft.udf(returndtype=daft.DataType.python(), numgpus=1)
class GPUModel:
    def init(self):
        self.model = loadmodeltogpu()


    def call(self, column):
        return [self.model(x) for x in column.topylist()]

Scaling Out with Ray

Up to this point all the code we've written runs on a single machine using Daft's local runner. But Daft's true power emerges when the data is too big for a single machine. At this point, you can tell Daft to spread the entire workload across a Ray cluster, and what amazed me, you don't need to change your pipeline code at all. Just swap the runner at the start of the script.

import daft

Connect Daft to a Ray cluster
daft.context.setrunnerray(address="ray://cluster-head:10001")

If Ray runs locally, just call without an address
daft.context.setrunnerray()

df = daft.readparquet("s3://my-bucket/big-dataset/*.parquet")

df = df.where(df["price"] > 20000).groupby("category").agg(
    df["price"].mean().alias("avgprice")
)
df.collect()

Notice that the only new line is daft.context.setrunnerray(). All the DataFrame operations below it are exactly the same as what you'd write for a local machine. This is Daft's main promise: you write the logic once, and it handles whether it runs on a single laptop or on a hundred machines. Thanks to that lazy nature, Daft can break your work into many small tasks and spread them across all Ray workers automatically.

This is also why the patterns we discussed earlier, like parallel URL downloading and class UDFs that load the model once per worker, become even more meaningful at cluster scale. Each Ray worker pulls its portion of the data, downloads its images, runs its model, and sends the results back, all concurrently without you writing a single line of distribution code.

Best Practices

After using Daft for several projects, there are some habits that I think make life easier. I'll share them with you so you don't have to go through the same trial and error.

First, take full advantage of the lazy nature. Don't rush to call .collect() in the middle of your pipeline just to peek. Assemble your entire chain of transformations first, then collect at the end. If you just want to check the shape of the data, use .show(5) which only pulls a few rows. This gives Daft the maximum opportunity to optimize the whole plan at once.

Second, always specify the right returndtype in your UDFs. This is not just a formality, it helps Daft manage memory and validate types correctly. If the output is a complex structure with no native Daft type, then use daft.DataType.python(), but if you can use a concrete type like string, integer, or list, prefer that because it's more efficient.

Third, for download and I/O operations that are prone to failure, always set onerror="null". When you process thousands or millions of URLs, some are guaranteed to be dead, time out, or have corrupt formats. You don't want one broken URL to bring down a pipeline that's been running for hours. With onerror="null", the problematic row becomes null and you can filter it out later using where.

Fifth, develop locally, deploy on the cluster. When developing, use a small subset of data and the local runner so your iterations are fast. Once the pipeline is correct, then switch to setrunnerray() and point it at the full data. Because the code is identical, this transition is smooth and you don't need to re-debug business logic on an expensive cluster.

Sixth, pay attention to batch size and resources when working with images and models. Images consume a lot of memory, especially at high resolution. Resize as early as possible in the pipeline so the data flowing to later stages is lighter. And for GPU models, declare numgpus correctly so the scheduler places tasks on machines that have a GPU.

Seventh, understand when to use .topandas(). Daft is very easy to convert to pandas for final steps like plotting or integrating with libraries that don't yet support Daft. But remember, .topandas() pulls all data to a single machine, so don't do this on large data. Do heavy aggregation or filtering in Daft first until the data shrinks, then convert to pandas at the very end.

Conclusion

Alright everyone, we've traveled quite a long way from zero to being able to scale a pipeline to a cluster. I hope you now have a clear picture of why Daft is special and when you'll need it.

The main point I want to emphasize: Daft is the answer to two problems that keep showing up in the modern data era, namely large scale and multimodal data. Thanks to a friendly Python API and a fast Rust engine, plus a lazy design that lets it optimize and distribute the work, you can write pipelines that are pleasant to read yet still capable of processing terabytes of data containing images, text, embeddings, and URLs. And what I love most, you write the logic once, then just swap the runner to jump from your laptop to a Ray cluster without rewriting anything.

My advice, start small. Try installing getdaft, build a DataFrame from a dictionary, play with filters and expressions, then slowly work up to image downloading and UDFs. Once you're comfortable, then touch Ray for scaling. Learning gradually like this makes the concepts of lazy execution and expressions really stick, and that's the foundation that will help you across all of Daft's advanced features.

That's it from me. If you found this tutorial helpful, try it right away in your project and feel the difference yourself of processing multimodal data using a tool that was actually designed for it. See you in the next tutorial, and happy hacking everyone.