ScrapeGraphAI Tutorial: Intelligent Web Scraping with LLMs and Graph Logic

# Tutorial ScrapeGraphAI: Web Scraping Cerdas dengan LLM dan Graph Logic ## Pendahuluan Web scraping adalah salah satu teknik paling penting dalam dunia data engineering dan data science. Namun, scr...

By Ruby Abdullah · · tutorial
ScrapeGraphAIWeb ScrapingLLMPythonData Engineering

ScrapeGraphAI Tutorial: Intelligent Web Scraping with LLMs and Graph Logic

Introduction

Web scraping is one of the most important techniques in data engineering and data science. However, traditional scraping has many limitations: scripts that break easily when HTML structures change, difficulties extracting data from dynamic pages, and the need to write specific parsers for each website.

ScrapeGraphAI is a modern solution that combines the power of Large Language Models (LLMs) with graph-based pipelines for web scraping. With ScrapeGraphAI, you simply describe what data you want to extract in natural language, and the library will automatically navigate web pages, understand their structure, and extract the requested information.

The library supports various LLM providers including OpenAI, Anthropic Claude, Google Gemini, Ollama for local models, and many more. ScrapeGraphAI also provides several types of graph pipelines that can be customized to fit your scraping needs.

In this tutorial, we will learn how to install, configure, and use ScrapeGraphAI for various scraping scenarios from simple to complex.

Installation

Prerequisites

Make sure you have Python 3.9 or later installed on your system. ScrapeGraphAI also requires some system dependencies.

Basic Installation

pip install scrapegraphai

Installation with Browser Support (for dynamic pages)

pip install scrapegraphai[browsers]

playwright install chromium

Installation with Local Model Support

If you want to use local models through Ollama:

pip install scrapegraphai[local]

Make sure Ollama is installed and the desired model is downloaded:

ollama pull llama3.1

ollama pull nomic-embed-text

Full Installation

To install all optional dependencies:

pip install scrapegraphai[all]

Verify Installation

import scrapegraphai

print(scrapegraphai.version)

Core Concepts

Graph Pipeline Architecture

ScrapeGraphAI works with the concept of graph pipelines. Each scraping operation is represented as a graph consisting of several nodes:

  • FetchNode: Retrieves content from a URL or text input
  • ParseNode: Processes and cleans HTML
  • RAGNode: Uses Retrieval-Augmented Generation for context
  • GenerateAnswerNode: Uses LLM to extract data according to the prompt
  • MergeAnswerNode: Combines results from multiple sources
  • Types of Graphs

    ScrapeGraphAI provides several built-in graphs:

    • SmartScraperGraph: Scrape a single page with natural language prompts
    • SearchGraph: Search the web first, then scrape the results
    • SpeechGraph: Scrape and convert results to audio
    • ScriptCreatorGraph: Generate Python scraping scripts from prompts
    • SmartScraperMultiGraph: Scrape multiple URLs simultaneously

    Basic Usage

    Configuration with OpenAI

    from scrapegraphai.graphs import SmartScraperGraph
    
    

    graphconfig = {

    "llm": {

    "apikey": "sk-your-openai-api-key",

    "model": "openai/gpt-4o-mini",

    },

    "verbose": True,

    "headless": True,

    }

    smartscraper = SmartScraperGraph(

    prompt="Extract all article titles and their descriptions",

    source="https://example.com/blog",

    config=graphconfig,

    )

    result = smartscraper.run()

    print(result)

    Configuration with Anthropic Claude

    from scrapegraphai.graphs import SmartScraperGraph
    
    

    graphconfig = {

    "llm": {

    "apikey": "sk-ant-your-anthropic-key",

    "model": "anthropic/claude-sonnet-4-20250514",

    },

    "verbose": True,

    "headless": True,

    }

    smartscraper = SmartScraperGraph(

    prompt="Extract the list of products with their prices and ratings",

    source="https://example.com/products",

    config=graphconfig,

    )

    result = smartscraper.run()

    print(result)

    Configuration with Local Models (Ollama)

    from scrapegraphai.graphs import SmartScraperGraph
    
    

    graphconfig = {

    "llm": {

    "model": "ollama/llama3.1",

    "temperature": 0,

    "baseurl": "http://localhost:11434",

    },

    "embeddings": {

    "model": "ollama/nomic-embed-text",

    "baseurl": "http://localhost:11434",

    },

    "verbose": True,

    "headless": True,

    }

    smartscraper = SmartScraperGraph(

    prompt="Extract all company names and their addresses",

    source="https://example.com/directory",

    config=graphconfig,

    )

    result = smartscraper.run()

    print(result)

    Scraping from Text or Local HTML

    ScrapeGraphAI can scrape not only from URLs but also from text or local HTML files:

    from scrapegraphai.graphs import SmartScraperGraph
    
    

    htmlcontent = """

    Gaming Laptop X1

    $1,299.99

    RAM 16GB, SSD 512GB, RTX 4060

    Ultrabook Y2

    $899.99

    RAM 8GB, SSD 256GB, Intel Iris

    """

    graphconfig = {

    "llm": {

    "apikey": "sk-your-api-key",

    "model": "openai/gpt-4o-mini",

    },

    }

    smartscraper = SmartScraperGraph(

    prompt="Extract laptop names, prices, and specifications in JSON format",

    source=htmlcontent,

    config=graphconfig,

    )

    result = smartscraper.run()

    print(result)

    Expected output:

    {
    

    "products": [

    {

    "name": "Gaming Laptop X1",

    "price": "$1,299.99",

    "specs": "RAM 16GB, SSD 512GB, RTX 4060"

    },

    {

    "name": "Ultrabook Y2",

    "price": "$899.99",

    "specs": "RAM 8GB, SSD 256GB, Intel Iris"

    }

    ]

    }

    Advanced Usage

    Using Output Schema with Pydantic

    For structured and consistent output, you can define a schema using Pydantic:

    from pydantic import BaseModel, Field
    

    from typing import List

    from scrapegraphai.graphs import SmartScraperGraph

    class Article(BaseModel):

    title: str = Field(description="Article title")

    author: str = Field(description="Author name")

    date: str = Field(description="Publication date")

    summary: str = Field(description="Article summary")

    class ArticleList(BaseModel):

    articles: List[Article] = Field(description="List of articles")

    graphconfig = {

    "llm": {

    "apikey": "sk-your-api-key",

    "model": "openai/gpt-4o-mini",

    },

    "verbose": True,

    }

    smartscraper = SmartScraperGraph(

    prompt="Extract all articles from this page",

    source="https://example.com/news",

    config=graphconfig,

    schema=ArticleList,

    )

    result = smartscraper.run()

    for article in result.get("articles", []):

    print(f"Title: {article['title']}")

    print(f"Author: {article['author']}")

    print(f"Date: {article['date']}")

    print(f"Summary: {article['summary']}")

    print("---")

    Multi-URL Scraping

    To scrape multiple URLs simultaneously:

    from scrapegraphai.graphs import SmartScraperMultiGraph
    
    

    graphconfig = {

    "llm": {

    "apikey": "sk-your-api-key",

    "model": "openai/gpt-4o-mini",

    },

    "verbose": True,

    "headless": True,

    }

    urls = [

    "https://example.com/page1",

    "https://example.com/page2",

    "https://example.com/page3",

    ]

    multiscraper = SmartScraperMultiGraph(

    prompt="Extract the page title and all links present",

    source=urls,

    config=graphconfig,

    )

    result = multiscraper.run()

    print(result)

    SearchGraph: Web Search + Scraping

    SearchGraph combines web search with scraping. You provide a search query, and ScrapeGraphAI will search the web, fetch relevant pages, and extract information:

    from scrapegraphai.graphs import SearchGraph
    
    

    graphconfig = {

    "llm": {

    "apikey": "sk-your-api-key",

    "model": "openai/gpt-4o-mini",

    },

    "maxresults": 5,

    "verbose": True,

    }

    searchscraper = SearchGraph(

    prompt="What are the most popular Python frameworks for machine learning in 2026?",

    config=graphconfig,

    )

    result = searchscraper.run()

    print(result)

    ScriptCreatorGraph: Generate Scraping Scripts

    If you want to get a reusable Python script that can run without an LLM:

    from scrapegraphai.graphs import ScriptCreatorGraph
    
    

    graphconfig = {

    "llm": {

    "apikey": "sk-your-api-key",

    "model": "openai/gpt-4o-mini",

    },

    "verbose": True,

    "library": "beautifulsoup",

    }

    scriptcreator = ScriptCreatorGraph(

    prompt="Create a script to extract all news headlines and their links",

    source="https://example.com/news",

    config=graphconfig,

    )

    result = scriptcreator.run()

    print(result)

    The output is a Python script using BeautifulSoup or other traditional scraping libraries.

    Proxy and Rate Limiting Configuration

    For safer scraping that respects rate limits:

    graphconfig = {
    

    "llm": {

    "apikey": "sk-your-api-key",

    "model": "openai/gpt-4o-mini",

    },

    "loaderkwargs": {

    "proxy": {

    "server": "http://proxy-server:8080",

    "username": "user",

    "password": "pass",

    },

    },

    "verbose": True,

    "headless": True,

    }

    Handling Dynamic Pages with JavaScript

    For pages that require JavaScript rendering:

    graphconfig = {
    

    "llm": {

    "apikey": "sk-your-api-key",

    "model": "openai/gpt-4o-mini",

    },

    "headless": True,

    "browsertype": "chromium",

    "loaderkwargs": {

    "waituntil": "networkidle",

    "timeout": 30000,

    },

    "verbose": True,

    }

    smartscraper = SmartScraperGraph(

    prompt="Extract data from the dynamically loaded table",

    source="https://example.com/dynamic-table",

    config=graphconfig,

    )

    result = smartscraper.run()

    Custom Graph Pipeline

    You can create custom graph pipelines for more complex scraping needs:

    from scrapegraphai.graphs import BaseGraph
    

    from scrapegraphai.nodes import FetchNode, ParseNode, GenerateAnswerNode

    class CustomScraperGraph(BaseGraph):

    def init(self, prompt, source, config):

    self.prompt = prompt

    self.source = source

    self.config = config

    fetchnode = FetchNode(

    input="url | localdir",

    output=["doc"],

    nodeconfig={"verbose": config.get("verbose", False)},

    )

    parsenode = ParseNode(

    input="doc",

    output=["parseddoc"],

    nodeconfig={"chunksize": 4096},

    )

    generatenode = GenerateAnswerNode(

    input="userprompt & parseddoc",

    output=["answer"],

    nodeconfig={

    "llmmodel": config["llm"],

    "verbose": config.get("verbose", False),

    },

    )

    super().init(

    nodes=[fetchnode, parsenode, generatenode],

    edges=[

    (fetchnode, parsenode),

    (parsenode, generatenode),

    ],

    entrypoint=fetchnode,

    )

    Project Example: Scraping Job Listings

    Here is a complete project example for scraping job listings:

    import json
    

    from pydantic import BaseModel, Field

    from typing import List, Optional

    from scrapegraphai.graphs import SmartScraperGraph

    class JobListing(BaseModel):

    title: str = Field(description="Job title")

    company: str = Field(description="Company name")

    location: str = Field(description="Work location")

    salary: Optional[str] = Field(description="Salary range if available")

    requirements: List[str] = Field(description="List of requirements")

    jobtype: str = Field(description="Job type: remote/onsite/hybrid")

    class JobListingResult(BaseModel):

    jobs: List[JobListing] = Field(description="List of job listings")

    graphconfig = {

    "llm": {

    "apikey": "sk-your-api-key",

    "model": "openai/gpt-4o-mini",

    },

    "verbose": True,

    "headless": True,

    }

    scraper = SmartScraperGraph(

    prompt="""

    Extract all job listings from this page.

    For each listing, get the job title, company name,

    location, salary range (if available), key requirements,

    and job type (remote/onsite/hybrid).

    """,

    source="https://example.com/jobs",

    config=graphconfig,

    schema=JobListingResult,

    )

    result = scraper.run()

    with open("joblistings.json", "w", encoding="utf-8") as f:

    json.dump(result, f, indent=2, ensureascii=False)

    print(f"Successfully extracted {len(result.get('jobs', []))} job listings")

    Project Example: E-Commerce Price Monitoring

    import json
    

    from datetime import datetime

    from pydantic import BaseModel, Field

    from typing import List, Optional

    from scrapegraphai.graphs import SmartScraperMultiGraph

    class Product(BaseModel):

    name: str = Field(description="Product name")

    price: str = Field(description="Current price")

    originalprice: Optional[str] = Field(description="Original price before discount")

    discount: Optional[str] = Field(description="Discount percentage")

    rating: Optional[str] = Field(description="Product rating")

    soldcount: Optional[str] = Field(description="Number of items sold")

    class ProductList(BaseModel):

    products: List[Product] = Field(description="List of products")

    graphconfig = {

    "llm": {

    "apikey": "sk-your-api-key",

    "model": "openai/gpt-4o-mini",

    },

    "verbose": True,

    "headless": True,

    }

    categories = [

    "https://example-shop.com/laptops",

    "https://example-shop.com/phones",

    "https://example-shop.com/tablets",

    ]

    multiscraper = SmartScraperMultiGraph(

    prompt="""

    Extract all products from this page including

    name, current price, original price, discount,

    rating, and number sold.

    """,

    source=categories,

    config=graphconfig,

    schema=ProductList,

    )

    result = multiscraper.run()

    timestamp = datetime.now().isoformat()

    output = {

    "scrapedat": timestamp,

    "data": result,

    }

    with open(f"prices{datetime.now().strftime('%Y%m%d')}.json", "w") as f:

    json.dump(output, f, indent=2, ensureascii=False)

    print(f"Price data saved successfully at {timestamp}")

    Best Practices

    1. Choose the Right Model

    For simple scraping tasks, smaller models like gpt-4o-mini are sufficient and more cost-effective. Use larger models like gpt-4o or claude-sonnet only for complex pages or when accuracy is critical.

    2. Use Pydantic Schemas

    Always define output schemas with Pydantic to get consistent and structured results. This also helps the LLM understand the expected output format.

    3. Write Specific Prompts

    Prompts that are too general like "get all data" will produce inconsistent output. Be specific about what data you want to extract.

    # Less effective
    

    prompt = "Get all information from this page"

    More effective

    prompt = "Extract product names, prices in USD, and star ratings (1-5) for each product item"

    4. Handle Errors Gracefully

    from scrapegraphai.graphs import SmartScraperGraph
    
    

    try:

    scraper = SmartScraperGraph(

    prompt="Extract product data",

    source="https://example.com/products",

    config=graphconfig,

    )

    result = scraper.run()

    if result:

    print("Scraping successful:", result)

    else:

    print("No data found")

    except Exception as e:

    print(f"Error during scraping: {e}")

    5. Respect Rate Limits and robots.txt

    Always pay attention to the rate limits of websites you scrape. Add delays between requests if needed, and check robots.txt before scraping.

    6. Use Local Models for High Volume

    If you need to perform high-volume scraping, consider using local models through Ollama. This will drastically reduce API costs.

    7. Cache Scraping Results

    To avoid repeated requests to the same website:

    import json
    

    import hashlib

    import os

    def scrapewithcache(prompt, url, config, cachedir="cache"):

    os.makedirs(cachedir, existok=True)

    cachekey = hashlib.md5(f"{prompt}{url}".encode()).hexdigest()

    cachefile = os.path.join(cachedir, f"{cachekey}.json")

    if os.path.exists(cachefile):

    with open(cachefile, "r") as f:

    return json.load(f)

    scraper = SmartScraperGraph(

    prompt=prompt,

    source=url,

    config=config,

    )

    result = scraper.run()

    with open(cachefile, "w") as f:

    json.dump(result, f, indent=2)

    return result

    8. Logging and Monitoring

    Enable verbose mode during development to understand the pipeline flow:

    graphconfig = {
    

    "llm": {

    "apikey": "sk-your-api-key",

    "model": "openai/gpt-4o-mini",

    },

    "verbose": True,

    }

    Comparison with Other Scraping Libraries

    | Feature | ScrapeGraphAI | BeautifulSoup | Scrapy | Selenium |

    |---------|--------------|---------------|--------|----------|

    | Natural Language Prompt | Yes | No | No | No |

    | No Selector Writing | Yes | No | No | No |

    | Dynamic Pages | Yes | No | Plugin | Yes |

    | Structured Output | Yes (Pydantic) | Manual | Manual | Manual |

    | Multi-LLM Support | Yes | N/A | N/A | N/A |

    | Local Models | Yes (Ollama) | N/A | N/A | N/A |

    | Cost per Request | Yes (LLM) | Free | Free | Free |

    | Speed | Medium | Fast | Fast | Slow |

    Troubleshooting

    Error: Model Not Found

    Make sure the model format uses the provider prefix:

    # Correct
    

    "model": "openai/gpt-4o-mini"

    "model": "anthropic/claude-sonnet-4-20250514"

    "model": "ollama/llama3.1"

    Wrong

    "model": "gpt-4o-mini"

    Error: Playwright Not Installed

    If you encounter browser-related errors:

    pip install playwright
    

    playwright install chromium

    Empty Scraping Results

    • Check if the URL is accessible
    • Try adding headless: True and browsertype: chromium to the config
    • Make sure the prompt is specific enough
    • Check if the website blocks bots

    Timeout Error

    Add a longer timeout:

    graphconfig = {
    

    "llm": {

    "apikey": "sk-your-api-key",

    "model": "openai/gpt-4o-mini",

    },

    "loader_kwargs": {

    "timeout": 60000,

    },

    }

    Conclusion

    ScrapeGraphAI represents a breakthrough in web scraping by combining the ease of natural language processing with the power of graph-based pipelines. Here are its key advantages:

  • Ease of Use: Simply describe the data you want to extract in natural language
  • Flexibility: Supports various LLM providers and local models
  • Structured Output: Integration with Pydantic for consistent output
  • Scalability: Can scrape single or multiple pages simultaneously
  • Adaptability: Not dependent on CSS selectors that can easily change
  • With ScrapeGraphAI, you can focus on what data you want to get, not how to get it. This library is ideal for data scientists, data engineers, and developers who need a fast and adaptive scraping solution.

    For more information, visit the official ScrapeGraphAI documentation on their GitHub repository.

    Related Articles

    Complete Aisuite Tutorial: Unified Multi-LLM Interface by Andrew Ng

    Tutorial Lengkap Aisuite: Unified Multi-LLM Interface dari Andrew Ng Apa itu Aisuite? Aisuite adalah library Python open...

    Complete Jupyter AI Tutorial: Integrate LLMs Directly Into Your Notebooks

    Tutorial Lengkap Jupyter AI: Integrasikan LLM Langsung ke Notebook Anda Jupyter Notebook telah menjadi alat utama bagi d...

    Complete Anthropic Claude API Tutorial: From Basics to Advanced Features

    Tutorial Lengkap Anthropic Claude API: Dari Dasar hingga Fitur Lanjutan Anthropic Claude adalah salah satu Large Languag...

    MergeKit Tutorial: Merging LLM Models to Create Custom AI Models

    Tutorial MergeKit: Menggabungkan Model LLM untuk Membuat Model AI Kustom Model bahasa besar (LLM) open-source terus berk...