Hamilton: Build Clean, Maintainable Dataflows and ML Pipelines in Python
Hey everyone, Ruby here again. This time I want to introduce you to a Python library that I think is seriously underrated but incredibly useful if you deal with data a lot: it's called Hamilton, built by the team at DAGWorks. If you have ever had a data pipeline or a feature engineering script that turned into a mess of hundreds of lines, full of variables like df1, df2, dffinal, dffinalv2, to the point where you yourself forgot the order of the transformations, well, Hamilton is the cure.
The core idea is simple but elegant: every Python function is a node in a dataflow. The function name becomes the name of its output, and the function parameters become its dependencies. So Hamilton reads your functions, automatically figures out who depends on who, and builds a clear DAG (Directed Acyclic Graph) that you can even visualize. You just say "I want this output", and Hamilton handles the execution order. Pretty cool, right?
In this article I will walk you through everything from scratch: how to install it, the basic concept of writing transform functions, building a Driver, requesting outputs, passing inputs and config, visualizing the DAG, and how to use Hamilton for real feature engineering and machine learning pipelines. I will give you plenty of runnable code examples. Let's get started.
Introduction: Why Hamilton?
Before we dive into code, let me talk about a problem that happens all the time in the data world. Imagine you are working on feature engineering. At first it's just two or three transformations, easy. But over time the team adds new columns, new features, and your file grows into 500 lines of procedural code. Code like this suffers from a few classic diseases:
- Hard to test. Because everything is mashed into one long script, you can't test a single transformation without running the whole thing.
- Hard to read. A new person joining the team gets dizzy following a
dfthat gets overwritten again and again. - Hard to trace. If one column has a weird value, you have to manually scroll to find where it came from.
- Hard to reuse. Want to use one transformation in another project? You copy-paste, and the problems start all over again.
Hamilton solves this with one paradigm: you write pure functions that are declarative. Each function is responsible for producing exactly one thing, and it requests its input through parameters. Here's a simple example:
import pandas as pd
def spendpersignup(spend: pd.Series, signups: pd.Series) -> pd.Series:
"""Average marketing spend per signup."""
return spend / signups
Look closely. This function is named spendpersignup. To Hamilton, that means this function produces an output named spendpersignup. Its parameters are spend and signups, so Hamilton understands that spendpersignup depends on two other nodes, namely spend and signups. You don't have to write "call this first then call that". Hamilton infers it all by itself from the function signature.
Because each function is isolated and has a clear name, you get bonuses for free: the code is easy to test (just call the function directly), easy to read (function name equals column name), easy to trace (Hamilton knows the exact lineage of every output), and easy to reuse (just import the module).
One thing I need to underline: Hamilton is framework-agnostic about data. It does not force you to use pandas only. You can use pandas Series, plain Python objects, numpy arrays, and it even integrates with Polars, Dask, Ray, or Spark. Hamilton focuses on orchestrating dependencies between functions, not on the compute engine. So it's very lightweight and does not force you into any particular ecosystem.
Installation
Installation is dead simple. Note that the package name is not hamilton, everyone, but sf-hamilton. This matters because the name hamilton on PyPI is already taken by another package. Just run:
pip install sf-hamilton
If you want the DAG visualization feature (which I think is an absolute must-try), install the extra too:
pip install "sf-hamilton[visualization]"
This pulls in graphviz as a Python dependency. But remember, you also need the Graphviz binary installed on your system. On Ubuntu/Debian just run:
sudo apt-get install graphviz
On Mac with Homebrew:
brew install graphviz
For those who want to try other integrations, Hamilton has many extras. For example, for integration with various execution backends:
pip install "sf-hamilton[ray]" # parallel execution with Ray
pip install "sf-hamilton[dask]" # distributed execution with Dask
To check that the installation succeeded, try running this in Python:
import hamilton
print(hamilton.version)
Notice again, even though the package is sf-hamilton, when importing you still use import hamilton. This often confuses newcomers, so I am emphasizing it once more so you don't get it wrong.
I recommend using a virtual environment to keep things tidy:
python -m venv venv
source venv/bin/activate # on Windows: venv\Scripts\activate
pip install "sf-hamilton[visualization]"
Basic Usage: Nodes, Modules, and the Driver
Alright, now let's get into the heart of Hamilton. There are three main concepts you must understand: the transform function (node), the module, and the Driver.
Writing Transform Functions in a Module
In Hamilton, you write your transform functions in a separate Python file, commonly called a dataflow module. This matters because Hamilton will scan that module to find all the functions and build its DAG. Let's create a file called functions.py:
# functions.py
import pandas as pd
def avg3wkspend(spend: pd.Series) -> pd.Series:
"""3-week rolling average of spend."""
return spend.rolling(3).mean()
def spendpersignup(spend: pd.Series, signups: pd.Series) -> pd.Series:
"""Spend per signup each week."""
return spend / signups
def spendmean(spend: pd.Series) -> float:
"""Overall mean of spend."""
return spend.mean()
def spendzeromean(spend: pd.Series, spendmean: float) -> pd.Series:
"""Spend centered to zero (minus the mean)."""
return spend - spendmean
def spendstddev(spend: pd.Series) -> float:
"""Standard deviation of spend."""
return spend.std()
def spendzeromeanunitvariance(
spendzeromean: pd.Series, spendstddev: float
) -> pd.Series:
"""Normalized (standardized) spend."""
return spendzeromean / spendstddev
Look carefully at the flow. The function spendzeromean needs two inputs: spend and spendmean. Now spendmean is not raw data, it's the output of the spendmean function above it. Hamilton wires this up automatically. Then spendzeromeanunitvariance needs spendzeromean (another function's output) and spendstddev (another function's output too). So you see, from just the parameter names and function names, Hamilton can build a long chain of dependencies.
The raw inputs (the ones you must supply from outside) are spend and signups, because there is no function named spend or signups. Hamilton calls inputs like these nodes that you fill via external data.
Building the Driver and Requesting Outputs
Now let's write the main script to run this dataflow. Call it run.py:
# run.py
import pandas as pd
from hamilton import driver
import functions # the module we just created
1. Build the Driver from the module
dr = (
driver.Builder()
.withmodules(functions)
.build()
)
2. Prepare the raw input data
initialdata = {
"spend": pd.Series([10, 10, 20, 40, 40, 50]),
"signups": pd.Series([1, 10, 50, 100, 200, 400]),
}
3. Decide which outputs we want
outputcolumns = [
"spend",
"signups",
"avg3wkspend",
"spendpersignup",
"spendzeromeanunitvariance",
]
4. Execute
df = dr.execute(outputcolumns, inputs=initialdata)
print(df)
Run python run.py, and Hamilton will:
functions module and gather all functions as nodes.outputcolumns).Point number 4 is really important and often amazes people. If you only request spendpersignup, Hamilton will not bother computing avg3wkspend or spendstddev. It only runs what is relevant to the output you asked for. So it's efficient and doesn't waste computation.
I used driver.Builder() here, which is the modern and recommended approach now. There used to be an old way using driver.Driver(config, module) directly, but the Builder pattern is far more flexible and readable, so I recommend you use this one.
Understanding the Output DataFrame
When you call dr.execute(outputcolumns, ...) and the outputs are a mix of pandas Series, Hamilton by default joins them all into one DataFrame based on the index. The columns match the node names you requested. So the result looks roughly like this:
spend signups avg3wkspend spendpersignup spendzeromeanunitvariance
0 10 1 NaN 10.000000 -1.064581
1 10 10 NaN 1.000000 -1.064581
2 20 50 13.333333 0.400000 -0.483900
...
If the output you request is not a Series (for example a scalar float like spendmean), you can control how Hamilton assembles the output via what's called a resultbuilder, but I'll cover that in the advanced section.
Advanced Usage: Config, Visualization, and Result Builders
Now that you understand the basics, let's level up. This section covers the features that make Hamilton truly powerful for production.
Passing Config to Shape the DAG
There is an important difference between inputs and config in Hamilton:
- inputs: data that changes on every execution (for example a different DataFrame each run).
- config: values that determine the shape of the DAG itself, given when you build the Driver, not at execution time.
Config is very useful when you want different logic branches. For example, you want different transformation logic between daily and weekly data. Hamilton has the @config.when decorator for this:
# functions.py (continued)
from hamilton.functionmodifiers import config
@config.when(datasource="production")
def rawdataprod(dbconnection: object) -> pd.DataFrame:
"""Pull data from the production database."""
return pd.readsql("SELECT FROM events", dbconnection)
@config.when(datasource="testing")
def rawdatatest() -> pd.DataFrame:
"""Dummy data for testing."""
return pd.DataFrame({"value": [1, 2, 3]})
Notice the prod and _test at the end of the function names. That's called name-mangling. Hamilton drops the part after the double-underscore, so both functions produce a node named rawdata. The difference is that only one is active depending on the config you pass:
dr = (
driver.Builder()
.withconfig({"datasource": "testing"})
.withmodules(functions)
.build()
)
With config datasource="testing", only the rawdatatest function enters the DAG. This is a really cool pattern for building one codebase that runs in different environments without if-else scattered everywhere.
Visualizing the DAG
Now this is my favorite feature. Hamilton can draw your DAG as a visual diagram. This is incredibly helpful for documentation, debugging, and onboarding new people. The requirement is that you've installed the [visualization] extra and the graphviz binary mentioned earlier.
# Visualize the whole DAG and save to file
dr.displayallfunctions("mydag.png")
Visualize only the path needed for a specific output
dr.visualizeexecution(
finalvars=["spendzeromeanunitvariance"],
outputfilepath="executionpath.png",
inputs={"spend": pd.Series([1, 2, 3]), "signups": pd.Series([1, 2, 3])},
)
displayallfunctions draws all nodes and their connections. Meanwhile visualizeexecution shows only the nodes that will actually be executed to produce the output you requested. I often use the second one to make sure my pipeline is not running unnecessary computation. You can literally see visually which path is active.
If you work in a Jupyter notebook, these visualization outputs even render inline. So you can iterate quickly while watching the shape of the DAG in real time.
Result Builder: Controlling the Output Shape
By default Hamilton joins outputs into a pandas DataFrame. But sometimes you don't want that. Maybe you want a dictionary, or a numpy array, or still a DataFrame but with specific rules. This is where the resultbuilder comes in:
from hamilton import base, driver
Output as a dictionary
dr = (
driver.Builder()
.withmodules(functions)
.withadapters(base.DictResult())
.build()
)
result = dr.execute(["spendmean", "spendstddev"], inputs=initialdata)
print(result)
{'spendmean': 28.33, 'spendstddev': 17.51}
base.DictResult() makes Hamilton return a dictionary, where the keys are node names and the values are the computed results. This is perfect when your outputs are a mix of types (some scalar, some Series, some model objects). There is also base.PandasDataFrameResult() (the default) and base.NumpyMatrixResult() if you need other formats.
Commonly Used Decorators
Hamilton has many decorators to make your code more concise. Let me cover a few that I use most often.
@tag for adding metadata to a node, useful for filtering and documentation:
from hamilton.functionmodifiers import tag
@tag(owner="data-team", stage="production")
def revenue(price: pd.Series, quantity: pd.Series) -> pd.Series:
return price
quantity
@extractcolumns for splitting one DataFrame into several column nodes at once. This is super useful at the start of a pipeline:
from hamilton.functionmodifiers import extractcolumns
@extractcolumns("spend", "signups", "region")
def loadrawdata(datapath: str) -> pd.DataFrame:
"""Load data and immediately split its columns into separate nodes."""
return pd.readcsv(datapath)
With @extractcolumns, after loadrawdata runs, Hamilton automatically creates nodes spend, signups, and region that each can become a dependency of other functions. So you don't have to write a separate function to extract each column.
@parameterize for creating many similar nodes from one template function, reducing duplication:
from hamilton.functionmodifiers import parameterize, source
@parameterize(
spendrolling3=dict(window=source("window3")),
spendrolling7=dict(window=source("window7")),
)
def spendrolling(spend: pd.Series, window: int) -> pd.Series:
return spend.rolling(window).mean()
This generates two nodes: spendrolling3 and spendrolling7, from one function definition. Imagine if you needed 10 different windows, this decorator saves you from writing 10 nearly identical functions.
Materializers: Automatic Reading and Writing of Data
For production pipelines, you often need to read from a data source and write results somewhere. Hamilton has the concept of materializers via to and from that handle this I/O declaratively:
from hamilton.io.materialization import to
from hamilton import driver
dr = driver.Builder().withmodules(functions).build()
Execute and save the output to CSV at once
materializers = [
to.csv(
path="./output/features.csv",
id="featurestocsv",
dependencies=["spendpersignup", "avg3wkspend"],
)
]
dr.materialize(*materializers, inputs=initialdata)
With materializers, the saving logic is separated from the transformation logic. So your transform functions stay pure and easy to test, while the "where to save" concern is managed at the Driver level.
Feature Engineering and ML Pipelines
Now the most exciting part, which is how we use Hamilton for real cases: feature engineering and machine learning. This is the area where Hamilton truly shines.
Structured Feature Engineering
One of the main reasons Hamilton was built (fun fact: it was originally created at Stitch Fix to manage thousands of time-series features) is feature engineering. Let's build a feature module:
# features.py
import pandas as pd
import numpy as np
from hamilton.functionmodifiers import extractcolumns
@extractcolumns("age", "income", "purchases", "dayssincesignup")
def rawcustomers(datapath: str) -> pd.DataFrame:
return pd.readcsv(datapath)
def incomelog(income: pd.Series) -> pd.Series:
"""Log transform for skewed income."""
return np.log1p(income)
def agebucket(age: pd.Series) -> pd.Series:
"""Group age into categorical buckets."""
return pd.cut(age, bins=[0, 25, 40, 60, 120],
labels=["young", "adult", "mature", "senior"])
def purchasefrequency(purchases: pd.Series, dayssincesignup: pd.Series) -> pd.Series:
"""Purchase frequency per day since signup."""
return purchases / dayssincesignup.clip(lower=1)
def highvalueflag(income: pd.Series, purchases: pd.Series) -> pd.Series:
"""Flag high-value customers."""
return ((income > income.median()) & (purchases > purchases.median())).astype(int)
Look how clean this is. Each feature is a standalone function with a clear name. If later another data scientist wants to add a new feature, they just write a new function without touching the old ones. And most importantly, each feature can be unit tested:
def testpurchasefrequency():
purchases = pd.Series([10, 20])
days = pd.Series([5, 10])
result = purchasefrequency(purchases, days)
assert result.tolist() == [2.0, 2.0]
No need to set up a big pipeline just to test one feature. Call the function directly, pass inputs, check the output. Simple.
End-to-End ML Pipeline
Now let's build a complete machine learning pipeline from raw data to a trained model. Hamilton can hold this entire flow in one coherent DAG:
# mlpipeline.py
import pandas as pd
from sklearn.modelselection import traintestsplit
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracyscore
from hamilton.functionmodifiers import extractcolumns
@extractcolumns("feature1", "feature2", "feature3", "target")
def dataset(datapath: str) -> pd.DataFrame:
return pd.readcsv(datapath)
def featurematrix(
feature1: pd.Series, feature2: pd.Series, feature3: pd.Series
) -> pd.DataFrame:
"""Combine features into an X matrix."""
return pd.concat([feature1, feature2, feature3], axis=1)
def traintestdata(
featurematrix: pd.DataFrame, target: pd.Series, testsize: float, randomstate: int
) -> dict:
"""Split data into train and test."""
Xtrain, Xtest, ytrain, ytest = traintestsplit(
featurematrix, target, testsize=testsize, randomstate=randomstate
)
return {"Xtrain": Xtrain, "Xtest": Xtest, "ytrain": ytrain, "ytest": ytest}
def fittedmodel(traintestdata: dict, nestimators: int) -> RandomForestClassifier:
"""Train a Random Forest."""
model = RandomForestClassifier(nestimators=nestimators, randomstate=42)
model.fit(traintestdata["Xtrain"], traintestdata["ytrain"])
return model
def predictions(fittedmodel: RandomForestClassifier, traintestdata: dict) -> pd.Series:
"""Predict on the test data."""
return pd.Series(fittedmodel.predict(traintestdata["Xtest"]))
def modelaccuracy(predictions: pd.Series, traintestdata: dict) -> float:
"""Compute model accuracy."""
return accuracyscore(traintestdata["ytest"], predictions)
And the script to run it:
# runml.py
from hamilton import base, driver
import mlpipeline
dr = (
driver.Builder()
.withmodules(mlpipeline)
.withadapters(base.DictResult())
.build()
)
results = dr.execute(
["fittedmodel", "modelaccuracy", "predictions"],
inputs={
"datapath": "customers.csv",
"testsize": 0.2,
"randomstate": 42,
"nestimators": 100,
},
)
print("Accuracy:", results["modelaccuracy"])
model = results["fittedmodel"] # model ready to use or save
Let this structure sink in. The entire ML pipeline, from loading data, building features, splitting, training, to evaluation, all becomes small functions connected through a DAG. You can request output at any point. Want just the trained model without evaluation? Request fittedmodel only. Want to see the accuracy? Request modelaccuracy. Hamilton computes exactly what's needed.
Notice too that I use nestimators, testsize, randomstate as inputs. This makes your hyperparameters explicit and easy to change. Want to experiment with nestimators=500? Change one number in the inputs, the DAG stays the same. This makes experiments reproducible and tidy.
And of course, you can visualize this entire ML pipeline:
dr.visualizeexecution(
final
vars=["modelaccuracy"],
output
filepath="mldag.png",
inputs={"datapath": "customers.csv", "testsize": 0.2,
"randomstate": 42, "nestimators": 100},
)
Imagine showing this diagram to a stakeholder or a new team. They immediately understand the pipeline flow without having to read the code line by line. This is a huge selling point of Hamilton for team collaboration.
Best Practices
After using Hamilton on a few projects, there are a few best practices I want to share with you so you don't fall into the same holes I did.
1. Keep functions pure. Your transform functions should ideally depend only on their parameters and have no side effects. Don't read files or global variables inside a transform function unless it's actually an input node. Pure functions make the DAG predictable and easy to test. 2. Give clear type hints. Hamilton uses type hints not just for documentation, but also for validation. If you write-> pd.Series, Hamilton can check whether the output is really a Series. Correct type hints catch errors earlier.
3. Function name equals a meaningful feature name. Because the function name becomes the node name, give descriptive names. spendpersignup is far clearer than sps or feature12. This is a small investment that pays off big during maintenance.
4. Split modules by domain. If your pipeline is large, don't pile all functions into one file. Split them into dataloading.py, features.py, model.py. The Driver can accept multiple modules at once:
dr = (
driver.Builder()
.withmodules(dataloading, features, model)
.build()
)
Hamilton will combine all nodes from all modules into one DAG. So you can organize your code neatly without losing the connections between nodes.
5. Take advantage of unit testing. This is Hamilton's biggest strength. Because each function is isolated, write unit tests for the important logic. You'll sleep better knowing each transformation has been validated. 6. Use@config.when for environment variations, not if-else inside functions. If you need different logic for dev vs prod, use the config decorator. This keeps each version a separate, clean function that can be tested on its own.
7. Validate outputs with @checkoutput. Hamilton has a decorator for data validation right at the node:
from hamilton.functionmodifiers import checkoutput
@check
output(range=(0, 1), datatype=float)
def conversion
rate(signups: pd.Series, visits: pd.Series) -> pd.Series:
return signups / visits
If the output falls outside the range you specify, Hamilton will give a warning or error. This is a good safety net for data quality in production.
8. Visualize before deploying. Before you push a pipeline to production, draw the DAG first. I often find accidentally wired dependencies or orphan nodes that aren't used, just by looking at the diagram. 9. Keep config explicit. If your pipeline uses a lot of config and inputs, store them in a separate YAML or JSON file, don't hardcode them in the script. This makes the pipeline easy to manage and version. 10. Start small, grow gradually. You don't need to migrate your entire codebase to Hamilton right away. Start with the messiest part of the pipeline, feel the benefits, then expand. Hamilton is incremental, so take it easy.Conclusion
Alright everyone, we've traveled pretty far into Hamilton. Let me recap the important points. Hamilton is a lightweight Python library from DAGWorks that makes your dataflows and pipelines clean through one simple idea: every function is a node, the function name is the output, the parameters are the dependencies. From there Hamilton builds the DAG automatically, executes only what's needed, and gives you code that is easy to test, read, trace, and reuse.
We covered how to install it (pip install sf-hamilton, remember the import is still hamilton), how to write transform functions in a module, build a Driver using Builder, request outputs with execute, distinguish inputs from config, visualize the DAG, and use decorators like @extractcolumns, @parameterize, @config.when, and @checkoutput. Then we practiced directly by building structured feature engineering and an end-to-end ML pipeline from raw data to a trained model.
In my opinion, Hamilton's greatest strength is not its fancy features, but the discipline it enforces. It makes you write structured data code naturally, without having to think about complicated architecture. If your team often argues about pipelines that are hard to maintain, give Hamilton a try. I'm confident you'll feel the difference, especially when a new team member can immediately understand the data flow just from looking at the DAG.
That's it from me for now. Try it out right away, because learning a library like this is fastest through hands-on coding. If you find an interesting pattern or have questions, don't hesitate to explore Hamilton's official documentation, which I think is very complete. Keep up the learning spirit and build clean pipelines. See you in the next tutorial. Happy coding everyone!