Crawl4AI: AI-Optimized Web Crawling for Data Pipelines

When building modern AI applications, especially those based on Retrieval-Augmented Generation (RAG), one of the biggest challenges is getting high-quality data from the web. Most traditional web scrapers produce messy raw HTML that is difficult for LLMs to process.

Crawl4AI is an open-source solution designed specifically for AI needs. It produces output as clean markdown and structured data ready to be used directly in your AI pipeline.

What Is Crawl4AI?

Crawl4AI is an open-source Python library for web crawling optimized for AI and data pipelines. Its key features include:

• Clean Markdown Output: Automatically converts web pages into clean markdown
• Async Crawling: Asynchronous support for fast parallel crawling
• LLM-Based Extraction: Structured data extraction using LLMs
• CSS-Based Extraction: Data extraction using CSS selectors without LLM
• JavaScript Rendering: Handles JavaScript-rendered pages
• Chunking Strategies: Various strategies for splitting content into optimal chunks
• Session Management: Manages cookies and sessions for complex crawling

Installation

Basic Installation

pip install crawl4ai

Browser Setup (Playwright)

Crawl4AI uses Playwright for web page rendering. After installation, run the browser setup:

# Install Chromium browser
crawl4ai-setup

Or manually via playwright
playwright install chromium

Installation with Full Dependencies

# With all features support
pip install "crawl4ai[all]"

Verify Installation

import crawl4ai
print(f"Crawl4AI version: {crawl4ai.version}")

Basic Crawling

Simple Page Crawling

import asyncio
from crawl4ai import AsyncWebCrawler

async def basiccrawl():

    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun(url="https://example.com")

        # Crawling status
        print(f"Status: {result.success}")
        print(f"Status Code: {result.statuscode}")

        # Content in markdown
        print(f"Markdown length: {len(result.markdown)}")
        print(result.markdown[:500])

        # Clean markdown (without navigation, footer, etc.)
        print(f"\nClean markdown length: {len(result.cleanedhtml)}")


Run
asyncio.run(basiccrawl())

Using Fit Markdown

Crawl4AI provides fitmarkdown which is a filtered version of markdown without unimportant elements:

async def crawlwithfitmarkdown():
    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun(url="https://docs.python.org/3/tutorial/")

        # Raw markdown - includes navigation, sidebar, etc.
        print(f"Raw markdown: {len(result.markdown)} chars")

        # Fit markdown - main content only
        print(f"Fit markdown: {len(result.fitmarkdown)} chars")


        # Use fitmarkdown for LLM input
        print(result.fitmarkdown[:500])


asyncio.run(crawlwithfitmarkdown())

Crawler Configuration

from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, BrowserConfig

async def configuredcrawl():

    # Browser configuration
    browserconfig = BrowserConfig(
        headless=True,
        browsertype="chromium",

        verbose=True
    )

    # Crawling configuration
    runconfig = CrawlerRunConfig(
        wordcountthreshold=10,       # Minimum words per block
        excludeexternallinks=True,   # Ignore external links
        removeoverlayelements=True,  # Remove popups/overlays
        processiframes=False          # Ignore iframes

    )

    async with AsyncWebCrawler(config=browserconfig) as crawler:
        result = await crawler.arun(
            url="https://example.com/blog/article",
            config=runconfig

        )

        print(result.fitmarkdown)