Mage: Building Modern Data Pipelines That Feel Like Coding in a Notebook
Hi friends, it's me again, Ruby Abdullah. This time I want to introduce you to a tool that I find genuinely refreshing in the data engineering world. It is called Mage (the repo is mage-ai). Why do I say refreshing? Because whenever we talk about orchestrating data pipelines, the first thing that comes to mind is usually Airflow. Airflow is incredibly powerful, but honestly the experience of writing code inside it can be exhausting. We have to write DAGs, think about operators, deal with XCom to pass data between tasks, and the development cycle is slow because you have to deploy first before you can see any results.
Mage comes with a different philosophy. It blends two worlds that are usually separated, namely the comfort of interactive coding like in a Jupyter Notebook and the power of a truly production-ready pipeline orchestrator. So you write code block by block, each block can be run immediately and you see its output instantly, but at the same time those blocks already form a pipeline that can be scheduled and deployed. For me this is a middle ground that fits perfectly, especially for data teams that are a mix of data scientists and data engineers.
In this tutorial I will cover everything from scratch. We start with installation, then create our first project, understand the concept of blocks (data loader, transformer, data exporter), play around in the interactive UI, write blocks in both Python and SQL, set up scheduling or triggers, and finally learn how to deploy so the pipeline runs automatically. I will give you plenty of code examples that you can actually run yourself. Let's get started.
Introduction: What Exactly Is Mage
Before we dive into the technical part, I want you to understand where Mage sits in the data ecosystem. Mage is an open-source tool for transforming and integrating data. If you have ever heard the term ETL (Extract, Transform, Load) or ELT, well Mage is the tool for building processes like that. What sets it apart from other tools is that Mage offers a far more pleasant developer experience.
There are a few core concepts you need to grasp first. The first one is the pipeline. A pipeline is like a cooking recipe, it contains a series of steps to process data from source all the way to the final destination. The second one is the block. This is the signature of Mage. A single pipeline is made up of several blocks, and each block is essentially just a code file (usually Python or SQL) that has one specific responsibility.
Blocks in Mage come in a few main types. There is the data loader whose job is to pull data from a source, for example from a database, an API, or a CSV file. There is the transformer whose job is to process and reshape data, for example cleaning dirty data, aggregating, or joining. Then there is the data exporter whose job is to save the processed result to its final destination, for example to a data warehouse like BigQuery or Snowflake. There are other types too like sensor, scratchpad, and custom, but those three are the ones you will use most often.
What I really like is that each block produces an output that can automatically be used by the next block. So you do not have to fuss over how to pass a DataFrame from one step to another, Mage handles all of that. Connected blocks form a graph or DAG (Directed Acyclic Graph), just like Airflow, except the way you write it is far more intuitive.
One more thing you should know, Mage has a web-based UI that becomes the place where you do everything. So you do not have to switch back and forth between a code editor and a terminal. The whole process of developing, running blocks, viewing output, and setting schedules is done through one clean interface. But do not worry, everything you build in the UI is still saved as code files in your project, so it can still go into Git and version control.
Installation: Starting From Zero
Okay now we move into the installation part. The easiest way to install Mage is using pip. I strongly recommend that you create a virtual environment first so the dependencies do not mix with your other projects. This is a best practice that I will keep emphasizing.
# Create the project folder and virtual environment first
mkdir learn-mage
cd learn-mage
Create a virtual environment
python3 -m venv venv
Activate the virtual environment (Linux/Mac)
source venv/bin/activate
On Windows use this instead
venv\Scripts\activate
Now install mage-ai
pip install mage-ai
The standard installation above is enough to start learning. But if you later need integration with a specific database or warehouse, Mage provides extra dependencies you can install as needed. For example if you want to connect to Postgres, BigQuery, or need all the features, you can install them like this.
# Install with Postgres support
pip install "mage-ai[postgres]"
Install with Google BigQuery support
pip install "mage-ai[google-cloud-platform]"
Install all connectors at once (a bit heavy, be careful)
pip install "mage-ai[all]"
After the installation finishes, check first whether the mage command is recognized in your terminal.
mage --version
If a version number shows up, your installation was successful. For those of you who are more comfortable using Docker, Mage also provides an official image. This is a good choice if you want a consistent environment that is easy to share with your team. Here is an example of how to run Mage through Docker.
# Pull and run Mage using Docker
docker run -it -p 6789:6789 \
-v $(pwd):/home/src \
mageai/mageai:latest \
/app/runapp.sh mage start learnmage
The command above maps your working folder into the container and opens port 6789, which is the default port for the Mage UI. But for this tutorial I will focus on the pip installation to make it easier to follow. I mention Docker just so you know the option exists.
Basic Usage: Creating Your First Project and Pipeline
Now the fun part. We are going to create our first Mage project. To start a new project, you just run the mage start command followed by the project name.
# Start a new project named learnmage
mage start learn
mage
Once you run this command, Mage does two things. First it creates the project folder structure in your directory. Second it spins up a web server and makes the UI accessible. By default the UI runs at http://localhost:6789. Open that address in your browser, and you will be greeted by Mage's clean dashboard.
Try looking at the contents of the project folder that was just created. The structure looks roughly like this.
learnmage/
├── mage
data/ # Mage internal data and cache
├── dataloaders/ # Data loader block files
├── transformers/ # Transformer block files
├── dataexporters/ # Data exporter block files
├── pipelines/ # Definition of each pipeline
├── ioconfig.yaml # Connection config to database/warehouse
├── metadata.yaml # Project metadata
└── requirements.txt # Additional project dependencies
From this structure alone you can already see how neatly Mage organizes everything. Each block type has its own folder, and each pipeline definition is stored in the pipelines folder. This is what I like, because everything is transparent and easy to track with Git.
Now let's create our first pipeline through the UI. In the dashboard, click the button to create a new pipeline, choose the standard (batch) type. Give the pipeline a name, for example examplesalesetl. After that you will enter the pipeline editor, the place where you will arrange the blocks.
Let's start with the data loader. Click the add block button, choose Data loader, then pick the generic Python template. Mage will generate a function with the @dataloader decorator. I will fill in the code to pull dummy sales data. In the real world you can change this to read from an API or a database.
import pandas as pd
if 'dataloader' not in globals():
from mageai.datapreparation.decorators import dataloader
@dataloader
def loaddata(args, kwargs):
"""
This block pulls raw sales data.
Here I use dummy data, but it can be swapped for reading a CSV or API.
"""
data = {
'orderid': [1, 2, 3, 4, 5],
'product': ['AI Class', 'Data Class', 'AI Class', 'Ebook', 'Data Class'],
'price': [500000, 350000, 500000, 75000, 350000],
'quantity': [2, 1, 3, 5, 2],
'city': ['Jakarta', 'Bandung', 'jakarta', 'Surabaya', 'Bandung'],
}
df = pd.DataFrame(data)
return df
Notice the @dataloader decorator above the function. That is what tells Mage that this function is a data loader block. The value returned from this function automatically becomes the input for the next block. Now try clicking the run button on that block (usually there is a play icon). Below the block, you will immediately see a preview of the DataFrame. This is the notebook feel I mentioned earlier, it runs right away and the result shows up instantly.
Next we add a transformer. Click add block, choose Transformer, generic Python template. The transformer block will have the @transformer decorator and automatically receives the output of the previous block through its first argument. Here I want to clean the data, especially the city column whose spelling is inconsistent (some are lowercase), and I want to add a total column from price times quantity.
import pandas as pd
if 'transformer' not in globals():
from mageai.datapreparation.decorators import transformer
@transformer
def transform(df, args, *kwargs):
"""
Clean the data and add a derived column.
The 'df' argument is the output of the previous data loader block.
"""
# Normalize the city names to title case
df['city'] = df['city'].str.strip().str.title()
# Add a total column = price quantity
df['total'] = df['price'] df['quantity']
# Drop any potential duplicate rows
df = df.dropduplicates()
return df
Just like before, click run on this transformer block and you will see the result. The city column is now uniform, and there is a new total column. The nice thing is that Mage remembers the relationship between blocks. So when you run the transformer, it automatically pulls the output of the data loader above it as input.
Finally we create the data exporter. This is the block for saving the final result. Click add block, choose Data exporter, generic Python template. For this simple example, I will export the result to a local CSV file.
import pandas as pd
if 'dataexporter' not in globals():
from mageai.datapreparation.decorators import dataexporter
@dataexporter
def exportdata(df, args, *kwargs):
"""
Save the processed result to a CSV file.
The 'df' argument is the output of the previous transformer block.
"""
outputpath = 'outputcleansales.csv'
df.tocsv(outputpath, index=False)
print(f'Data successfully saved to {outputpath}')
print(f'Total rows: {len(df)}')
And that is it. You just built a complete ETL pipeline from load, transform, to export. To run the entire pipeline from start to finish, you can click the run all button in the editor, or later we will set it up to run automatically through a trigger.
I want to emphasize one thing. Every time you click run on a block, Mage temporarily stores that block's output. So you can develop incrementally without having to rerun the entire pipeline every time there is a small change. This is what makes the development cycle so much faster compared to having to deploy first like in traditional Airflow.
Advanced Usage: SQL Blocks, Scheduling, and Deploying
Now that you understand the basics, let's level up. I will cover a few more advanced features that will be really useful in real work.
Writing Blocks Using SQL
Not every transformation is pleasant to write in Python. Sometimes for a large aggregation query, SQL is far more concise and fast, especially when the data actually lives in a database. Mage supports SQL blocks natively. So you can write SQL directly, and Mage will execute it against the connection you specify.
Before using a SQL block that connects to a database, you need to configure the connection in the ioconfig.yaml file. Its contents look roughly like this for a Postgres connection.
# Example ioconfig.yaml contents
version: 0.1.1
default:
POSTGRES
CONNECTTIMEOUT: 10
POSTGRES
DBNAME: yourdatabase
POSTGRES
SCHEMA: public
POSTGRESUSER: youruser
POSTGRESPASSWORD: yourpassword
POSTGRESHOST: localhost
POSTGRESPORT: 5432
Once the connection is ready, you can add a SQL block to the pipeline. Choose a data loader or transformer of the SQL type, specify its connection (for example PostgreSQL) and the default profile. Then you just write the query. Here is an example SQL transformer block for aggregating sales by city.
-- SQL Block: aggregate total sales per city
-- Mage automatically injects the previous block's output as a table
SELECT
city,
COUNT(orderid) AS transactioncount,
SUM(total) AS totalrevenue
FROM {{ df1 }}
GROUP BY city
ORDER BY totalrevenue DESC
Notice the {{ df1 }} syntax there. That is how Mage connects the output of the previous block to your SQL query using templating. So you can mix Python and SQL blocks within the same pipeline. For example load with Python, do heavy transformation with SQL, export again with Python. Super flexible, right.
Parameters and Runtime Variables
In a production pipeline, we very often need pipelines that are dynamic. For example pulling data only for today. Mage provides access to runtime variables through kwargs. The one used most often is executiondate. Here is an example data loader that uses the execution date.
if 'dataloader' not in globals():
from mageai.datapreparation.decorators import dataloader
@dataloader
def loaddailydata(args, kwargs):
"""
Pull data based on the pipeline execution date.
kwargs['executiondate'] is automatically filled by Mage at run time.
"""
executiondate = kwargs.get('executiondate')
print(f'Running pipeline for date: {executiondate}')
# Here you can filter your query based on that date
datestr = executiondate.strftime('%Y-%m-%d') if executiondate else 'none'
print(f'Filtering data for: {datestr}')
import pandas as pd
return pd.DataFrame({'date': [datestr], 'status': ['loaded']})
Scheduling and Triggers
This is the part that makes Mage a real orchestration tool, not just a notebook. After your pipeline is done, you surely want it to run automatically, whether every hour, every day, or triggered by a certain event. In Mage this concept is called a trigger.
There are three main types of triggers in Mage. The first is schedule, which runs the pipeline based on a time schedule, similar to cron. The second is event, which runs the pipeline when a certain event comes from another system. The third is API*, which runs the pipeline through an HTTP endpoint call.
To create a schedule trigger, in the UI you go to the Triggers tab of your pipeline, click New trigger, and choose the Schedule type. You can set its frequency, for example every day at 9 in the morning, or even use a custom cron expression. For example if you want it to run every day at 6 in the morning, you enter the cron expression 0 6 .
Besides through the UI, triggers can also be defined through a triggers.yaml file inside the pipeline folder. This is really nice because it can go into version control. Here is an example.
# triggers.yaml file inside the pipeline folder
triggers:
- name: dailysalesschedule
scheduletype: time
scheduleinterval: "0 6 "
starttime: 2026-08-24 06:00:00
status: active
settings:
skipifpreviousrunning: true
The skipifpreviousrunning: true configuration is important. It ensures that if the previous run has not finished, a new run will not pile up. This is a best practice so your pipeline does not overlap and create duplicate data.
Deploying to Production
After everything runs smoothly locally, it is time to deploy so the pipeline runs automatically 24/7 without your laptop having to stay on all the time. There are several ways to deploy Mage.
The easiest way is using Docker on a server. You just run the Mage image on your server, map the project folder, and expose port 6789. The principle is the same as running locally, only on a machine that is always on.
# Simple deployment using Docker on a server
docker run -d --name mage-production \
-p 6789:6789 \
-v /path/to/project:/home/src \
--restart unless-stopped \
mageai/mageai:latest \
/app/runapp.sh mage start learnmage
The -d flag makes it run in the background, and --restart unless-stopped ensures the container comes back up automatically if the server reboots. For a larger scale, Mage also provides official Terraform templates for deploying to cloud providers like AWS, GCP, or Azure. So if your team already uses cloud infrastructure, the integration has been thought through.
For production, there is also a feature you should consider, namely authentication. By default the Mage UI has no login. So if you expose it to the internet, you MUST enable the user authentication feature through an environment variable so not just anyone can access your pipelines.
# Enable authentication when deploying
export REQUIREUSERAUTHENTICATION=1
export MAGEACCESSTOKEN=yoursecrettoken
Best Practices: Keeping Your Pipelines Clean and Durable
After using Mage for a few jobs, there are a few good habits I want to share with you. These are what separate a pipeline that is easy to maintain from one that gives you headaches down the road.
First, one block, one responsibility. The temptation to put all your logic in one giant block is strong, especially when you are in a hurry. But resist it. Break it into small blocks with clear functions. Separate load from transform, and split heavy transformation into several steps. This makes it easy for you to debug, because if there is an error you immediately know which block is wrong.
Second, make use of the data validation feature. Mage supports adding tests inside a block. You can write a function with the @test decorator to make sure a block's output matches expectations. For example checking there are no null values in an important column.
if 'test' not in globals():
from mageai.datapreparation.decorators import test
@test
def testoutputnotempty(df, args) -> None:
"""
Make sure the transformation result is not empty and key columns are filled.
"""
assert df is not None, 'Output must not be None'
assert len(df) > 0, 'DataFrame must not be empty'
assert df['total'].notnull().all(), 'The total column must not have nulls'
print('All validations passed')
Tests like this will automatically run each time a block finishes. If one fails, the pipeline stops and you get a warning. This is a real lifesaver for maintaining data quality.
Third, do not store credentials inside your code. This is a big sin I often see. Database passwords, API keys, none of those should ever be written directly in a block. Use the ioconfig.yaml file for connections, or use environment variables, and call them using templating. Mage supports this, so make use of it.
Fourth, commit to Git diligently. Because everything you build in Mage is stored as code files, you can and should put it into Git. Make your Mage project a repository. That way you have a change history, can collaborate with your team, and can roll back if something breaks.
Fifth, manage memory and resources wisely. If your data is large, be careful not to pull everything into memory at once. Make use of chunking or process the data incrementally. Mage supports processing data in chunks, so study this feature if you work with large scale data.
Sixth, give blocks and pipelines descriptive names. Do not leave default names like transformer1 or dataloader2. Give clear names like cleancity or aggregatecityrevenue. Six months from now you will thank yourself when you open this pipeline again.
Conclusion: Wrapping Up and Next Steps
Okay friends, we have reached the end of the tutorial. I hope by now you have a picture of how enjoyable it is to work with Mage. Let's recap our journey. We started from installation using pip install mage-ai, then created our first project with mage start. We learned Mage's core concept, the block, from the data loader for pulling data, the transformer for processing, to the data exporter for saving results.
We also tried writing blocks in both Python and SQL, played around in its interactive UI which makes the development cycle fast because you can run each block and see the output right away. Then we stepped up to a more serious level by covering runtime parameters, scheduling through cron triggers, and how to deploy to production using Docker so the pipeline runs automatically.
In my opinion, Mage's main strength lies in its developer experience. It removes a lot of the friction we usually feel in traditional orchestration tools. For teams that are a mix of data scientists and data engineers, Mage becomes the perfect bridge. Data scientists feel at home because it feels like a notebook, and data engineers are happy because the result is genuinely production-ready.
But I also want to be honest, Mage is not a solution for every problem. If your needs are already super complex with thousands of DAGs and complicated enterprise integrations, then Airflow or Dagster, which are more mature, might still be a better fit. But for the majority of cases, especially small to medium teams that want to move fast, Mage is a very worthy option to consider.
My advice, do not just read this tutorial and then forget it. Go practice right away. Build your own project, try pulling real data from a source you have, process it, then save it. Feel for yourself how fast the development cycle is in Mage. After that try setting up a trigger and watch your pipeline run automatically. From there you will understand better when Mage is a good fit and when it is not.
If you friends have questions or want to learn more about data engineering and AI, just stop by academy.rubythalib.ai, I have plenty of material there. Thank you for staying with me until the end, I hope this tutorial is useful and makes you even more excited to tinker with data. See you in the next tutorial, take care of your health and keep the spirit of learning alive, friends.