Crawlee Tutorial: Modern Web Scraping Framework for AI/ML Data Collection

Web scraping is a critical step in data pipelines, especially for AI and machine learning projects that require data from various web sources. Crawlee is an open-source web scraping and crawling framework developed by Apify. It is designed to help build reliable, fast, and maintainable scrapers. Crawlee provides built-in features such as auto-scaling, anti-blocking, session management, and proxy rotation out of the box.

In this tutorial, we will learn how to use Crawlee from installation, basic usage with different crawler types, to advanced techniques like proxy management, session handling, and integration with data pipelines for AI/ML use cases.

Why Crawlee?

Before diving into the technical details, here are some reasons why Crawlee is worth considering:

Automatic anti-blocking: Crawlee automatically handles browser fingerprinting, header rotation, and retry logic to avoid being blocked by target websites
Three crawler types: Supports CheerioCrawler (HTTP + Cheerio for fast HTML parsing), PlaywrightCrawler (full browser automation), and JSDOMCrawler (JSDOM for lightweight JavaScript rendering)
Auto-scaling: Adjusts concurrent request count based on system resources and target response times
Request queue: Persistent URL queue management so crawling can resume if errors occur
Session management: Automatically manages cookies and sessions per request
Built-in storage: Dataset and key-value store for structured scraping results
TypeScript-first: Written in TypeScript with full type safety

Installation

Prerequisites

Ensure Node.js version 18 or later is installed on your system.

node --version v18.0.0 or later

Creating a New Project

The fastest way to get started is using the Crawlee CLI:

npx crawlee create my-crawler cd my-crawler

The CLI will present template options. For this tutorial, select the CheerioCrawler template to start.

Manual Installation

If you want to add Crawlee to an existing project:

# For CheerioCrawler (HTTP-based, no browser) npm install crawlee For PlaywrightCrawler (browser automation) npm install crawlee playwright Install browser binaries for Playwright npx playwright install chromium

Project Structure

After installation, your minimal project structure will look like this:

my-crawler/
├── src/
│   ├── main.ts          # Entry point
│   ├── routes.ts        # Route handlers
│   └── types.ts         # Type definitions
├── storage/             # Auto-generated storage
│   ├── datasets/        # Scraping results
│   ├── keyvaluestores/ # Key-value storage
│   └── requestqueues/  # URL queue

├── package.json
└── tsconfig.json

Basic Usage

CheerioCrawler - Fast Scraping
CheerioCrawler is the best choice for websites that don't require JavaScript rendering. It uses HTTP requests and Cheerio (jQuery-like API) for HTML parsing.

import { CheerioCrawler, Dataset } from 'crawlee';

const crawler = new CheerioCrawler({
    maxRequestsPerCrawl: 50,

    async requestHandler({ request, $, enqueueLinks, log }) {
        const title = $('title').text().trim();
        const description = $('meta[name="description"]').attr('content') || '';

        log.info(Scraping: ${title}, { url: request.url });

        const articles = $('article').map((, el) => ({
            title: $(el).find('h2').text().trim(),
            summary: $(el).find('p').first().text().trim(),
            link: $(el).find('a').attr('href'),
        })).get();

        await Dataset.pushData({
            url: request.url,
            title,
            description,
            articles,
            scrapedAt: new Date().toISOString(),
        });

        await enqueueLinks({
            globs: ['https://example.com/blog/'],

            label: 'DETAIL',
        });
    },

    failedRequestHandler({ request, log }) {
        log.error(Request failed: ${request.url});
    },
});

await crawler.run(['https://example.com/blog']);

In the example above:

$ is a Cheerio object that can be used like jQuery

enqueueLinks automatically discovers and adds new links to the queue

Dataset.pushData stores scraping results in local storage

maxRequestsPerCrawl limits the total number of requests

PlaywrightCrawler - Dynamic Websites

For websites that require JavaScript rendering (SPAs, infinite scroll, etc.), use PlaywrightCrawler:

import { PlaywrightCrawler, Dataset } from 'crawlee'; const crawler = new PlaywrightCrawler({ maxRequestsPerCrawl: 20, headless: true, launchContext: { launchOptions: { args: ['--no-sandbox'], }, }, async requestHandler({ request, page, log, enqueueLinks }) { log.info(Opening: ${request.url}); // Wait for elements to appear await page.waitForSelector('.product-card', { timeout: 10000 }); // Scroll to load lazy-loaded content await page.evaluate(async () => { for (let i = 0; i < 3; i++) { window.scrollBy(0, window.innerHeight); await new Promise(r => setTimeout(r, 1000)); } }); // Extract product data const products = await page.$eval('.product-card', (cards) => cards.map((card) => ({ name: card.querySelector('.product-title')?.textContent?.trim(), price: card.querySelector('.product-price')?.textContent?.trim(), rating: card.querySelector('.rating')?.textContent?.trim(), image: card.querySelector('img')?.getAttribute('src'), })) ); await Dataset.pushData({ url: request.url, products, totalProducts: products.length, }); // Find the next page await enqueueLinks({ selector: 'a.next-page', label: 'LISTING', }); }, }); await crawler.run(['https://example.com/products']);

Using Router to Handle Different Pages

In more complex projects, you need to handle different types of pages differently. Crawlee provides Router for this:

import { CheerioCrawler, createCheerioRouter, Dataset } from 'crawlee';

const router = createCheerioRouter();

// Handler for listing pages
router.addHandler('LISTING', async ({ request, $, enqueueLinks, log }) => {
    log.info(Listing page: ${request.url});

    const items = $('div.item').map((, el) => {

        const link = $(el).find('a.detail-link').attr('href');
        return link;
    }).get();

    for (const link of items) {
        if (link) {
            await crawler.addRequests([{
                url: new URL(link, request.url).href,
                label: 'DETAIL',
            }]);
        }
    }

    await enqueueLinks({
        selector: '.pagination a.next',
        label: 'LISTING',
    });
});

// Handler for detail pages
router.addHandler('DETAIL', async ({ request, $, log }) => {
    log.info(Detail page: ${request.url});

    const data = {
        url: request.url,
        title: $('h1.title').text().trim(),
        content: $('div.content').text().trim(),
        date: $('span.date').text().trim(),
        category: $('span.category').text().trim(),
        tags: $('span.tag').map((, el) => $(el).text().trim()).get(),
    };

    await Dataset.pushData(data);
});

// Default handler
router.addDefaultHandler(async ({ request, enqueueLinks, log }) => {
    log.info(Default handler: ${request.url});
    await enqueueLinks({
        globs: ['https://example.com/articles/'],
        label: 'LISTING',
    });
});

const crawler = new CheerioCrawler({
    requestHandler: router,
    maxRequestsPerCrawl: 100,
});

await crawler.run(['https://example.com']);

Advanced Usage

Proxy Management

Proxy rotation is essential for large-scale web scraping to avoid being blocked:

import { CheerioCrawler, ProxyConfiguration } from 'crawlee';

const proxyConfiguration = new ProxyConfiguration({
    proxyUrls: [
        'http://user:pass@proxy1.example.com:8080',
        'http://user:pass@proxy2.example.com:8080',
        'http://user:pass@proxy3.example.com:8080',
    ],
});

const crawler = new CheerioCrawler({
    proxyConfiguration,
    sessionPoolOptions: {
        maxPoolSize: 20,
        sessionOptions: {
            maxUsageCount: 50,
        },
    },

    async requestHandler({ request, $, session, log }) {
        if ($('title').text().includes('blocked') || $('title').text().includes('captcha')) {
            session?.retire();
            throw new Error('Blocked, switching session');
        }

        session?.markGood();

        await Dataset.pushData({
            url: request.url,
            title: $('title').text(),
        });
    },
});

Request Queue and Persistence

Crawlee stores the request queue persistently. This means if the crawler stops mid-run, you can resume from the last position:

import { CheerioCrawler, RequestQueue } from 'crawlee';

// Create a named queue for persistence
const requestQueue = await RequestQueue.open('my-queue');

// Add URLs with metadata
await requestQueue.addRequests([
    {
        url: 'https://example.com/page/1',
        userData: { pageNumber: 1, category: 'tech' },
        label: 'LISTING',
    },
    {
        url: 'https://example.com/page/2',
        userData: { pageNumber: 2, category: 'tech' },
        label: 'LISTING',
    },
]);

const crawler = new CheerioCrawler({
    requestQueue,
    maxConcurrency: 5,
    maxRequestRetries: 3,
    requestHandlerTimeoutSecs: 60,

    async requestHandler({ request, $, log }) {
        const { pageNumber, category } = request.userData;
        log.info(Page ${pageNumber}, Category: ${category});

        await Dataset.pushData({
            page: pageNumber,
            category,
            items: $('div.item').length,
        });
    },
});

await crawler.run();

Custom Headers and Session Rotation

import { CheerioCrawler } from 'crawlee';

const crawler = new CheerioCrawler({
    preNavigationHooks: [
        async ({ request }, gotOptions) => {
            // Custom headers for each request
            gotOptions.headers = {
                ...gotOptions.headers,
                'Accept-Language': 'en-US,en;q=0.9',
                'Accept': 'text/html,application/xhtml+xml',
                'Cache-Control': 'no-cache',
            };

            // Add cookies if needed
            if (request.userData.authCookie) {
                gotOptions.headers['Cookie'] = request.userData.authCookie;
            }
        },
    ],

    postNavigationHooks: [
        async ({ request, response, log }) => {
            if (response && response.statusCode === 403) {
                log.warning(403 Forbidden at: ${request.url});
            }
        },
    ],

    async requestHandler({ request, $, log }) {
        const data = $('body').text().substring(0, 500);
        log.info(Successfully scraped: ${request.url});
    },
});

Exporting Data to Various Formats

Crawlee stores data in JSON format by default, but you can export to other formats:

import { CheerioCrawler, Dataset } from 'crawlee';
import fs from 'fs';

const crawler = new CheerioCrawler({
    async requestHandler({ $, request }) {
        const items = $('tr.data-row').map((, row) => ({

            name: $(row).find('td:nth-child(1)').text().trim(),
            value: $(row).find('td:nth-child(2)').text().trim(),
            date: $(row).find('td:nth-child(3)').text().trim(),
        })).get();

        await Dataset.pushData(items);
    },
});

await crawler.run(['https://example.com/data']);

// Export dataset
const dataset = await Dataset.open();
const { items } = await dataset.getData();

// Export to JSON
fs.writeFileSync('output.json', JSON.stringify(items, null, 2));

// Export to CSV
const headers = Object.keys(items[0] || {});
const csvContent = [
    headers.join(','),
    ...items.map(item =>
        headers.map(h => "${String(item[h] || '').replace(/"/g, '""')}").join(',')
    ),
].join('\n');
fs.writeFileSync('output.csv', csvContent);

console.log(Exported ${items.length} records);

Integration with AI/ML Pipelines

One of the most compelling use cases for Crawlee is as part of a data collection pipeline for AI/ML:

import { CheerioCrawler, Dataset } from 'crawlee';
import fs from 'fs';

interface TrainingData {
    text: string;
    label: string;
    source: string;
    timestamp: string;
}

const crawler = new CheerioCrawler({
    maxRequestsPerCrawl: 200,

    async requestHandler({ request, $, log }) {
        // Scrape articles for NLP training data
        const articleBody = $('article .content, .post-body, .entry-content')
            .text()
            .trim()
            .replace(/\s+/g, ' ');

        if (articleBody.length < 100) {
            log.warning(Content too short, skipping: ${request.url});
            return;
        }

        const category = request.userData.category || 'uncategorized';

        const trainingData: TrainingData = {
            text: articleBody,
            label: category,
            source: request.url,
            timestamp: new Date().toISOString(),
        };

        await Dataset.pushData(trainingData);
        log.info(Collected: ${articleBody.substring(0, 80)}...);
    },
});

// Collect data from various sources
await crawler.run([
    { url: 'https://example.com/tech', userData: { category: 'technology' } },
    { url: 'https://example.com/science', userData: { category: 'science' } },
    { url: 'https://example.com/business', userData: { category: 'business' } },
]);

// Read dataset and prepare for training
const dataset = await Dataset.open();
const { items } = await dataset.getData();

// Save in JSONL format (one JSON per line, common for LLM training)
const jsonlContent = items
    .map(item => JSON.stringify(item))
    .join('\n');

fs.writeFileSync('trainingdata.jsonl', jsonlContent);
console.log(Ready: ${items.length} training data samples);

Rate Limiting and Politeness

Being "polite" while scraping is good practice and reduces the risk of being blocked:

import { CheerioCrawler } from 'crawlee';

const crawler = new CheerioCrawler({
    // Limit speed
    maxConcurrency: 3,
    maxRequestsPerMinute: 30,

    // Add delay between requests
    requestHandlerTimeoutSecs: 30,
    navigationTimeoutSecs: 30,

    // Respect robots.txt
    respectRobotsTxt: true,

    // Retry configuration
    maxRequestRetries: 2,

    async requestHandler({ request, $, log }) {
        log.info(Scraping: ${request.url});

        await Dataset.pushData({
            url: request.url,
            content: $('main').text().trim(),
        });
    },
});

Error Handling and Monitoring

import { CheerioCrawler, Dataset } from 'crawlee';

const stats = {
    success: 0,
    failed: 0,
    skipped: 0,
};

const crawler = new CheerioCrawler({
    maxRequestRetries: 3,

    async requestHandler({ request, $, log }) {
        const title = $('title').text();

        if (!title) {
            stats.skipped++;
            log.warning(Page without title, skipping: ${request.url});
            return;
        }

        await Dataset.pushData({
            url: request.url,
            title,
            contentLength: $('body').text().length,
        });

        stats.success++;
    },

    async failedRequestHandler({ request, log }) {
        stats.failed++;
        log.error(Failed after retries: ${request.url}, {
            errors: request.errorMessages,
        });

        // Store failed URLs for analysis
        await Dataset.pushData({
            url: request.url,
            status: 'FAILED',
            errors: request.errorMessages,
        });
    },
});

await crawler.run(['https://example.com']);

console.log('Crawling statistics:', stats);

Best Practices

1. Choose the Right Crawler

Use CheerioCrawler for static or server-rendered websites. It is much faster and more resource-efficient
Use PlaywrightCrawler only when content is loaded dynamically via JavaScript
Start with CheerioCrawler, then switch to PlaywrightCrawler if needed

2. Manage Resources Wisely

const crawler = new CheerioCrawler({
    maxConcurrency: 5,
    maxRequestsPerMinute: 60,
    maxRequestsPerCrawl: 1000,
    requestHandlerTimeoutSecs: 30,
    maxRequestRetries: 2,
});

Don't be too aggressive with concurrency. Start with small numbers and increase gradually.

3. Use Labels for Routing

await crawler.addRequests([
    { url: 'https://example.com/list', label: 'LISTING' },
    { url: 'https://example.com/item/1', label: 'DETAIL' },
]);

Labels make it easy to manage handlers for different page types.

4. Validate Scraped Data

Always validate data before storing:

async requestHandler({ $, request }) {
    const title = $('h1').text().trim();
    const price = parseFloat($('.price').text().replace(/[^\d.]/g, ''));

    if (!title || isNaN(price)) {
        throw new Error('Invalid data, will retry');
    }

    await Dataset.pushData({ title, price, url: request.url });
}

5. Use Storage Effectively

import { KeyValueStore, Dataset } from 'crawlee';

// Store state or configuration
const store = await KeyValueStore.open();
await store.setValue('last-run', { timestamp: new Date().toISOString() });

// Store structured data
await Dataset.pushData(results);

// Read previous state
const lastRun = await store.getValue('last-run');

6. Clean Storage Before New Runs

import { purgeDefaultStorages } from 'crawlee';

// Clean storage from previous runs
await purgeDefaultStorages();

7. Informative Logging

async requestHandler({ request, $, log }) {
    log.info('Starting scrape', { url: request.url });
    log.debug('Parsing details', { selectorCount: $('div.item').length });
    log.warning('Incomplete content', { url: request.url });
}

Deployment and Automation

Running on a Server

// src/main.ts
import { CheerioCrawler, Dataset } from 'crawlee';

async function main() {
    const crawler = new CheerioCrawler({
        async requestHandler({ request, $, log }) {
            const data = {
                url: request.url,
                title: $('title').text(),
                timestamp: new Date().toISOString(),
            };
            await Dataset.pushData(data);
            log.info(Done: ${data.title});
        },
    });

    await crawler.run(['https://example.com']);

    // Access results
    const dataset = await Dataset.open();
    const { items } = await dataset.getData();
    console.log(Total: ${items.length} pages successfully scraped);
}

main().catch(console.error);

Running as a Cron Job

# Run every day at 02:00
0 2    cd /path/to/my-crawler && node dist/main.js >> /var/log/crawler.log 2>&1

Docker Deployment

FROM node:20-slim RUN npx playwright install-deps chromium WORKDIR /appCOPY package.json ./ RUN npm ci COPY . . RUN npm run build CMD ["node", "dist/main.js"]

Conclusion

Crawlee is a mature, production-ready web scraping framework. With built-in anti-blocking features, session management, auto-scaling, and persistent storage, Crawlee significantly simplifies the process of collecting data from the web.

For AI/ML projects, Crawlee is particularly valuable as the first stage of a data pipeline, collecting raw data from various web sources that can then be processed into training datasets. The combination of CheerioCrawler for static websites and PlaywrightCrawler for dynamic websites provides full flexibility in handling different types of websites.

Key takeaways:

Start with CheerioCrawler, switch to PlaywrightCrawler only when needed
Always respect robots.txt and apply reasonable rate limiting
Validate scraped data before storing
Leverage the request queue for reliable, resumable crawling
Use proxy rotation for large-scale scraping

With the fundamentals covered in this tutorial, you are ready to build reliable and scalable web scrapers for your data collection needs.

Crawlee Tutorial: Modern Web Scraping Framework for AI/ML Data Collection

Why Crawlee?

Installation

Prerequisites

v18.0.0 or later

Creating a New Project

Manual Installation

For PlaywrightCrawler (browser automation)

Install browser binaries for Playwright

Project Structure

Basic Usage

CheerioCrawler - Fast Scraping

PlaywrightCrawler - Dynamic Websites

Using Router to Handle Different Pages