Airbyte: A Complete Guide to Open-Source Data Integration from Zero to Custom Connectors

Hey folks, in this tutorial I want to take you on a deeper journey to get to know Airbyte. For those of you whose daily work touches data, whether you are a data engineer, an analyst, or even a backend developer who needs to move data from one place to another, Airbyte is a tool you absolutely need to know about. I personally have used Airbyte quite a lot to pull data from all sorts of sources, from Postgres databases, third-party APIs like Stripe or HubSpot, all the way to CSV files in cloud storage, then dump it all into a single data warehouse so it is easy to analyze.

So in this article I am going to cover Airbyte comprehensively, starting from the core concepts, how to install it using abctl, building a connection through the UI, understanding sync modes, all the way to more advanced stuff like using PyAirbyte to run connectors directly from Python and building a custom connector with the CDK. Relax, I will explain everything slowly so everyone can follow along. Let us get started.

Introduction

What Is Airbyte

Airbyte is an open-source data integration platform that focuses on ELT (Extract, Load, Transform). Unlike traditional ETL where transformation happens before the data lands in its destination, in ELT we extract data from the source, load it raw into the destination, and only then transform it inside the warehouse using tools like dbt. This ELT approach is more flexible because the raw data stays intact, so if your analysis needs change down the road, you do not have to re-pull everything from the source.

What makes Airbyte special is its ridiculously large number of connectors. There are hundreds of connectors, both for sources (where data comes from) and destinations (where data goes). So if you need to pull data from Salesforce, Google Analytics, MySQL, MongoDB, Shopify, or even Google Sheets, chances are the connector already exists. And if it does not exist yet, do not worry, we can build our own using the Connector Development Kit (CDK), which I will cover in the advanced section later.

Why I Chose Airbyte

Before Airbyte, a lot of us wrote data pipelines manually in Python. Build a script to pull data from an API, handle pagination, handle rate limits, handle error retries, then write to a database. It is exhausting, right, and each data source has its own quirks. Airbyte standardizes all of that. We just configure the source and destination, set up a schedule, and Airbyte takes care of all the complexity behind the scenes.

On top of that, Airbyte has an open-source version we can self-host for free. This is really important for those of you who have data privacy concerns or a limited budget. There is also Airbyte Cloud, which is fully managed if you do not want to deal with the infrastructure, but in this tutorial I am focusing on the self-hosted version because that is what most folks in the field actually use.

Core Concepts You Must Understand

Before we jump into practice, there are a few terms you need to grasp first so you do not get confused. These are the foundation.

The first is Source. A source is where your data comes from. It can be a database, an API, a file, anything that acts as an information source. For example, your production Postgres database, or your Stripe account that holds transaction data.

The second is Destination. A destination is where the data is going to land. Usually this is a data warehouse like BigQuery, Snowflake, Redshift, or an analytical database like a Postgres instance dedicated to reporting.

The third is Connection. A connection is the bridge that links one source to one destination. Inside a connection we configure what data gets synced, how often the sync runs, and which sync mode to use.

The fourth is Stream. A stream is an individual unit of data within a source. If the source is a database, one stream usually equals one table. If the source is an API, one stream can be one endpoint, for example a customers endpoint and an orders endpoint are two separate streams.

The fifth is Sync Mode. This determines how data gets synced, whether it pulls all the data every time (full refresh) or only pulls the data that has newly changed (incremental). I will go into detail in the sync modes section.

Installation

Prerequisites

Before installing Airbyte, make sure your laptop or server already has Docker installed and running. Airbyte runs on top of several Docker containers, so Docker is mandatory here. Also, prepare enough resources. I recommend at least 4 GB of RAM and a few GB of free disk, because Airbyte is fairly resource hungry when it runs.

For installation, the most modern and easiest way now is to use abctl, the official Airbyte Command Line Tool. We used to install via docker-compose, but now abctl has become the recommended way because it runs on top of a local Kubernetes cluster (using kind), which is more stable.

Install abctl

The easiest way to install abctl is using the official script from Airbyte. Run the command below in your terminal.

# Install abctl using the official script
curl -LsfS https://get.airbyte.com | bash -

Check the version to make sure the install worked
abctl version

If you are a macOS user and use Homebrew, you can also install via brew for a tidier setup.

# Alternative for macOS users with Homebrew
brew tap airbytehq/tap
brew install abctl

abctl version

Deploy Airbyte Locally

Once abctl is installed, we can now deploy Airbyte. It only takes one command, and abctl handles everything. It downloads the required images, creates a local Kubernetes cluster, and deploys all the Airbyte components. The process is fairly long the first time, it can take 15 to 30 minutes depending on your internet connection, so just be patient.

# Deploy Airbyte locally, this takes a few minutes
abctl local install

Once done, Airbyte will be running and accessible in your browser at http://localhost:8000. For the first login, we need an auto-generated password. Grab the password with the following command.

# Get login credentials to enter the Airbyte UI
abctl local credentials

The command above shows the email and password for logging in. Open http://localhost:8000 in your browser, enter the credentials, and you will land in the Airbyte dashboard. Congratulations, your Airbyte is up and running.

Useful abctl Commands

There are a few other abctl commands I often use to manage my installation. I will share them here so you do not get confused later.

# Check the status of the Airbyte installation
abctl local status

Uninstall Airbyte if you want to clean up
abctl local uninstall

Install with a custom port if port 8000 is already taken
abctl local install --port 8080

Basic Usage

Building a Connection Through the UI

Alright, Airbyte is running, time to build our first data pipeline. I will demonstrate pulling data from a source into a destination. For an easy example, let us use the Faker source, which generates dummy data, then load it into a local Postgres destination. Faker is great for learning because we do not need to set up a real data source.

The first step, in the Airbyte dashboard, click the Sources menu in the left sidebar, then click the New source button. A list of all available source connectors appears. Type "Faker" in the search field, pick the Sample Data (Faker) connector. After that you can configure how many records to generate, say 1000 records, then click Set up source. Airbyte will test the connection first, and if it turns green it means success.

The second step, now we create the destination. Click the Destinations menu, then New destination. Search for "Postgres", pick the Postgres connector. Fill in the connection details like host, port, database name, username, and password matching your Postgres database. If you do not have Postgres, you can use the Local JSON or Local CSV destination, which are simpler for learning. Click Set up destination and wait for the test to succeed.

The third step, this is the most important one, we create the Connection that links the source and destination from before. Click the Connections menu, then New connection. Pick the Faker source from earlier, then pick the Postgres destination from earlier. Airbyte will detect the schema from the source and display the list of available streams. Here you can choose which streams to sync, and for each stream you can set its sync mode.

After that, also configure the sync schedule. There are several options, from Scheduled (for example every 24 hours), Cron (for a more specific schedule using a cron expression), or Manual (run manually whenever we want). For learning, pick Manual first so we stay in control. Click Set up connection, and your connection is done.

Running the First Sync

Once the connection is created, click the Sync now button to run the first sync. Airbyte will pull data from Faker and write it to Postgres. You can watch the progress in real time in the Job History tab, complete with detailed logs showing how many records were read and written. If the status turns Succeeded with a green checkmark, it means your first data pipeline ran successfully. Easy, right.

Try checking the destination Postgres database. There will be new tables filled with dummy data from Faker, usually inside a schema named according to the destination config. The raw data is also usually stored in special columns with extra metadata like airbyteextractedat, which shows when the data was pulled.

One thing you need to know, Airbyte writes several extra metadata columns in every destination table. There is airbyterawid which is a unique identifier for each record, airbyteextractedat which is a timestamp of when the record was extracted, and airbytemeta which stores extra information about the sync process. These columns are important for tracking and debugging, so do not be surprised when you see them in the final result.

# Create a virtual environment first to stay clean
python -m venv venv
source venv/bin/activate

Install PyAirbyte
pip install airbyte

import airbyte as ab

Get the Faker source and auto-install the connector
source = ab.getsource(
    "source-faker",
    config={"count": 1000, "seed": 42},
    installifmissing=True,
)

Check first whether the config and connection are valid
source.check()

See which streams are available from this source
print(source.getavailablestreams())

Select all streams to read
source.selectallstreams()

Read data into the local cache (defaults to DuckDB)
result = source.read()

Access one of the streams as a Pandas DataFrame
usersdf = result["users"].topandas()
print(usersdf.head())

print(f"Total records: {len(usersdf)}")

Cool, right. With just that we can already pull 1000 user records straight into a DataFrame. PyAirbyte automatically stores the data into a local DuckDB-based cache, so if we read again, it does not re-pull from the source.

Now let us try something more realistic, for example pulling data from GitHub. For a source that needs authentication, we pass the credentials via config. I strongly recommend never hardcoding tokens in your code, but reading them from an environment variable instead.

import os
import airbyte as ab

Get the GitHub source with the token from an environment variable
source = ab.getsource(

    "source-github",
    config={
        "repositories": ["airbytehq/airbyte"],
        "credentials": {
            "personalaccesstoken": os.environ["GITHUBTOKEN"],
        },
    },
    installifmissing=True,
)

source.check()

Select only specific streams to save effort, say just issues and stargazers
source.selectstreams(["issues", "stargazers"])


result = source.read()

issuesdf = result["issues"].topandas()

print(f"Total issues pulled: {len(issuesdf)}")

Besides going to a DataFrame, PyAirbyte can also write directly to a real warehouse-backed cache. For example if we want the read result to go straight into BigQuery or Snowflake, we just configure the cache. But for the example, let us use a custom DuckDB cache so the data persists in a local file.

import airbyte as ab
from airbyte.caches import DuckDBCache

Create a DuckDB cache stored in a specific file
cache = DuckDBCache(dbpath="./data/airbytecache.duckdb")

source = ab.getsource(

    "source-faker",
    config={"count": 5000},
    installifmissing=True,

)
source.selectallstreams()


Read and store into the custom cache above
result = source.read(cache=cache)

Since the data is persistent, next time you can access it without re-reading
for name, records in result.streams.items():
    df = records.topandas()
    print(f"Stream {name}: {len(df)} records")

What I love about PyAirbyte is that it makes data integration a perfect fit for orchestration workflows like Airflow, Dagster, or even just a simple cron script. You can embed this Airbyte data-reading logic into a larger Python pipeline seamlessly.

Building a Custom Connector with the CDK

Sometimes there is a data source that does not have a connector in Airbyte yet. This is where the Connector Development Kit or CDK comes in. The CDK provides a framework for building your own connector without coding from scratch. The easiest way to build an API connector now is using the low-code CDK based on a YAML config file, or using the Connector Builder available directly in the Airbyte UI.

For those of you who want to build a connector using the full Python CDK, the flow starts by generating a connector template. Airbyte provides a generator for this.

# Clone the Airbyte repository
git clone https://github.com/airbytehq/airbyte.git
cd airbyte/airbyte-integrations/connector-templates/generator

Run the generator to scaffold a new connector
./generate.sh

That generator asks a few things, like the connector type (pick Python HTTP API Source for example) and the connector name. After that it creates a complete folder structure with boilerplate files ready for us to fill in.

The core of a Python CDK connector is a few classes we must implement. The most important is the Stream class that defines how to pull data from one endpoint. Here is a simple example of a stream that pulls data from a public API.

from typing import Any, Iterable, Mapping, Optional
import requests
from airbytecdk.sources.streams.http import HttpStream



class Customers(HttpStream):
    # Base URL of the API we want to consume
    urlbase = "https://api.example.com/v1/"

    # Primary key to identify each record
    primarykey = "id"


    def path(
        self,
        streamstate: Mapping[str, Any] = None,
        streamslice: Mapping[str, Any] = None,

        nextpagetoken: Mapping[str, Any] = None,

    ) -> str:
        # The specific endpoint for this stream
        return "customers"

    def nextpagetoken(

        self, response: requests.Response
    ) -> Optional[Mapping[str, Any]]:
        # Logic to handle pagination
        data = response.json()
        if data.get("nextcursor"):
            return {"cursor": data["nextcursor"]}

        return None

    def requestparams(
        self,
        streamstate: Mapping[str, Any],

        streamslice: Mapping[str, Any] = None,
        nextpagetoken: Mapping[str, Any] = None,
    ) -> Mapping[str, Any]:
        # Query params sent to the API, including the pagination cursor
        params = {"limit": 100}
        if nextpagetoken:
            params["cursor"] = nextpagetoken["cursor"]
        return params

    def parseresponse(

        self, response: requests.Response, kwargs

    ) -> Iterable[Mapping]:
        # Extract records from the response body
        yield from response.json().get("data", [])

# Test the connector locally, check the connection
python main.py check --config secrets/config.json

Read data from the connector using a defined catalog
python main.py read --config secrets/config.json --catalog integrationtests/configured_catalog.json

First, always choose incremental sync whenever possible. Do not be lazy, take the time to find the right cursor column for each stream. Full refresh is easy to set up, but once your data hits millions of rows, full refresh will take hours and blow up your warehouse costs. Incremental is a small investment up front whose payoff is huge in the long run.

Second, never hardcode credentials in your config or code. Always use environment variables or a secret manager. This is not just about security, it is also about portability. When credentials are separated from code, you can easily move your setup between environments like dev, staging, and production.

Third, start with just a few streams first. When you first build a connection, do not immediately select all streams. Pick a handful of streams you truly need first, test the sync, make sure the data is correct, then gradually add more streams. This makes debugging much easier if something goes wrong.

Fourth, make use of monitoring and alerting features. Airbyte can integrate with notifications via email, Slack, or webhooks to let you know if a sync fails. A data pipeline that silently fails is a nightmare, because you only realize it when your business dashboard shows weird numbers. Set up alerting from the start so you know immediately when there is a problem.

Fifth, separate transformation from syncing. Remember the ELT principle, let Airbyte focus on Extract and Load, then do your data transformation separately using dbt inside the warehouse. Do not try to force complex transformation logic into the Airbyte sync process. This separation of responsibilities makes your pipeline more maintainable and easier to debug.

Sixth, pay attention to resources and scale. If you self-host Airbyte, keep monitoring its memory and CPU usage. Heavy syncs can drain resources, especially if they run at the same time. Schedule your syncs so they do not pile up at the same hour, and give Airbyte enough resources especially if your data is large.

Seventh, document your connections**. The more connections you have, the easier it is to forget which one pulls what and for what purpose. Create simple documentation that notes each connection, its source, its destination, its sync mode, and who is responsible. Trust me, your future self will thank you.

Conclusion

Alright folks, we have traveled quite far into the world of Airbyte. We started from the core concepts like source, destination, connection, and stream, then moved on to how to install using abctl, building a connection through the UI, understanding the various sync modes from full refresh to incremental and CDC, then dove into more advanced stuff like PyAirbyte for pulling data directly from Python, and finally a glimpse at building a custom connector using the CDK.

What I hope you take home from this tutorial is the understanding that Airbyte frees us from the repetitive grind of building data pipelines manually. With hundreds of ready-to-use connectors and the flexibility to build our own, Airbyte can become the backbone of your data infrastructure, whether for a small side project or an enterprise-scale company. And what I love most, the open-source version makes all of this accessible to anyone without licensing costs.

My advice, just go practice. Install Airbyte locally using abctl, try building a Faker to Postgres connection like the one I demonstrated, then play around with PyAirbyte to pull data from a source you use daily. Learning data engineering is fastest when you get your hands dirty with the tool directly. If you run into problems, the Airbyte community on Slack and GitHub is very active and ready to help.

I hope this tutorial is useful for you. If you have any questions or want to discuss further, do not hesitate to reach out to me. Happy coding and may your data pipelines always run smoothly without failed syncs. See you in the next tutorial, folks.