ScrapeGraphAI Tutorial: Intelligent Web Scraping with LLMs and Graph Logic
Introduction
Web scraping is one of the most important techniques in data engineering and data science. However, traditional scraping has many limitations: scripts that break easily when HTML structures change, difficulties extracting data from dynamic pages, and the need to write specific parsers for each website.
ScrapeGraphAI is a modern solution that combines the power of Large Language Models (LLMs) with graph-based pipelines for web scraping. With ScrapeGraphAI, you simply describe what data you want to extract in natural language, and the library will automatically navigate web pages, understand their structure, and extract the requested information.
The library supports various LLM providers including OpenAI, Anthropic Claude, Google Gemini, Ollama for local models, and many more. ScrapeGraphAI also provides several types of graph pipelines that can be customized to fit your scraping needs.
In this tutorial, we will learn how to install, configure, and use ScrapeGraphAI for various scraping scenarios from simple to complex.
Installation
Prerequisites
Make sure you have Python 3.9 or later installed on your system. ScrapeGraphAI also requires some system dependencies.
Basic Installation
pip install scrapegraphai
Installation with Browser Support (for dynamic pages)
pip install scrapegraphai[browsers]
playwright install chromium
Installation with Local Model Support
If you want to use local models through Ollama:
pip install scrapegraphai[local]
Make sure Ollama is installed and the desired model is downloaded:
ollama pull llama3.1
ollama pull nomic-embed-text
Full Installation
To install all optional dependencies:
pip install scrapegraphai[all]
Verify Installation
import scrapegraphai
print(scrapegraphai.version)
Core Concepts
Graph Pipeline Architecture
ScrapeGraphAI works with the concept of graph pipelines. Each scraping operation is represented as a graph consisting of several nodes:
Types of Graphs
ScrapeGraphAI provides several built-in graphs:
- SmartScraperGraph: Scrape a single page with natural language prompts
- SearchGraph: Search the web first, then scrape the results
- SpeechGraph: Scrape and convert results to audio
- ScriptCreatorGraph: Generate Python scraping scripts from prompts
- SmartScraperMultiGraph: Scrape multiple URLs simultaneously
Basic Usage
Configuration with OpenAI
from scrapegraphai.graphs import SmartScraperGraph
graphconfig = {
"llm": {
"apikey": "sk-your-openai-api-key",
"model": "openai/gpt-4o-mini",
},
"verbose": True,
"headless": True,
}
smartscraper = SmartScraperGraph(
prompt="Extract all article titles and their descriptions",
source="https://example.com/blog",
config=graphconfig,
)
result = smartscraper.run()
print(result)
Configuration with Anthropic Claude
from scrapegraphai.graphs import SmartScraperGraph
graphconfig = {
"llm": {
"apikey": "sk-ant-your-anthropic-key",
"model": "anthropic/claude-sonnet-4-20250514",
},
"verbose": True,
"headless": True,
}
smartscraper = SmartScraperGraph(
prompt="Extract the list of products with their prices and ratings",
source="https://example.com/products",
config=graphconfig,
)
result = smartscraper.run()
print(result)
Configuration with Local Models (Ollama)
from scrapegraphai.graphs import SmartScraperGraph
graphconfig = {
"llm": {
"model": "ollama/llama3.1",
"temperature": 0,
"baseurl": "http://localhost:11434",
},
"embeddings": {
"model": "ollama/nomic-embed-text",
"baseurl": "http://localhost:11434",
},
"verbose": True,
"headless": True,
}
smartscraper = SmartScraperGraph(
prompt="Extract all company names and their addresses",
source="https://example.com/directory",
config=graphconfig,
)
result = smartscraper.run()
print(result)
Scraping from Text or Local HTML
ScrapeGraphAI can scrape not only from URLs but also from text or local HTML files:
from scrapegraphai.graphs import SmartScraperGraph
htmlcontent = """
Gaming Laptop X1
$1,299.99
RAM 16GB, SSD 512GB, RTX 4060
Ultrabook Y2
$899.99
RAM 8GB, SSD 256GB, Intel Iris
"""
graphconfig = {
"llm": {
"apikey": "sk-your-api-key",
"model": "openai/gpt-4o-mini",
},
}
smartscraper = SmartScraperGraph(
prompt="Extract laptop names, prices, and specifications in JSON format",
source=htmlcontent,
config=graphconfig,
)
result = smartscraper.run()
print(result)
Expected output:
{
"products": [
{
"name": "Gaming Laptop X1",
"price": "$1,299.99",
"specs": "RAM 16GB, SSD 512GB, RTX 4060"
},
{
"name": "Ultrabook Y2",
"price": "$899.99",
"specs": "RAM 8GB, SSD 256GB, Intel Iris"
}
]
}
Advanced Usage
Using Output Schema with Pydantic
For structured and consistent output, you can define a schema using Pydantic:
from pydantic import BaseModel, Field
from typing import List
from scrapegraphai.graphs import SmartScraperGraph
class Article(BaseModel):
title: str = Field(description="Article title")
author: str = Field(description="Author name")
date: str = Field(description="Publication date")
summary: str = Field(description="Article summary")
class ArticleList(BaseModel):
articles: List[Article] = Field(description="List of articles")
graphconfig = {
"llm": {
"apikey": "sk-your-api-key",
"model": "openai/gpt-4o-mini",
},
"verbose": True,
}
smartscraper = SmartScraperGraph(
prompt="Extract all articles from this page",
source="https://example.com/news",
config=graphconfig,
schema=ArticleList,
)
result = smartscraper.run()
for article in result.get("articles", []):
print(f"Title: {article['title']}")
print(f"Author: {article['author']}")
print(f"Date: {article['date']}")
print(f"Summary: {article['summary']}")
print("---")
Multi-URL Scraping
To scrape multiple URLs simultaneously:
from scrapegraphai.graphs import SmartScraperMultiGraph
graphconfig = {
"llm": {
"apikey": "sk-your-api-key",
"model": "openai/gpt-4o-mini",
},
"verbose": True,
"headless": True,
}
urls = [
"https://example.com/page1",
"https://example.com/page2",
"https://example.com/page3",
]
multiscraper = SmartScraperMultiGraph(
prompt="Extract the page title and all links present",
source=urls,
config=graphconfig,
)
result = multiscraper.run()
print(result)
SearchGraph: Web Search + Scraping
SearchGraph combines web search with scraping. You provide a search query, and ScrapeGraphAI will search the web, fetch relevant pages, and extract information:
from scrapegraphai.graphs import SearchGraph
graphconfig = {
"llm": {
"apikey": "sk-your-api-key",
"model": "openai/gpt-4o-mini",
},
"maxresults": 5,
"verbose": True,
}
searchscraper = SearchGraph(
prompt="What are the most popular Python frameworks for machine learning in 2026?",
config=graphconfig,
)
result = searchscraper.run()
print(result)
ScriptCreatorGraph: Generate Scraping Scripts
If you want to get a reusable Python script that can run without an LLM:
from scrapegraphai.graphs import ScriptCreatorGraph
graphconfig = {
"llm": {
"apikey": "sk-your-api-key",
"model": "openai/gpt-4o-mini",
},
"verbose": True,
"library": "beautifulsoup",
}
scriptcreator = ScriptCreatorGraph(
prompt="Create a script to extract all news headlines and their links",
source="https://example.com/news",
config=graphconfig,
)
result = scriptcreator.run()
print(result)
The output is a Python script using BeautifulSoup or other traditional scraping libraries.
Proxy and Rate Limiting Configuration
For safer scraping that respects rate limits:
graphconfig = {
"llm": {
"api
key": "sk-your-api-key",
"model": "openai/gpt-4o-mini",
},
"loaderkwargs": {
"proxy": {
"server": "http://proxy-server:8080",
"username": "user",
"password": "pass",
},
},
"verbose": True,
"headless": True,
}
Handling Dynamic Pages with JavaScript
For pages that require JavaScript rendering:
graphconfig = {
"llm": {
"apikey": "sk-your-api-key",
"model": "openai/gpt-4o-mini",
},
"headless": True,
"browsertype": "chromium",
"loaderkwargs": {
"waituntil": "networkidle",
"timeout": 30000,
},
"verbose": True,
}
smartscraper = SmartScraperGraph(
prompt="Extract data from the dynamically loaded table",
source="https://example.com/dynamic-table",
config=graphconfig,
)
result = smartscraper.run()
Custom Graph Pipeline
You can create custom graph pipelines for more complex scraping needs:
from scrapegraphai.graphs import BaseGraph
from scrapegraphai.nodes import FetchNode, ParseNode, GenerateAnswerNode
class CustomScraperGraph(BaseGraph):
def init(self, prompt, source, config):
self.prompt = prompt
self.source = source
self.config = config
fetchnode = FetchNode(
input="url | localdir",
output=["doc"],
nodeconfig={"verbose": config.get("verbose", False)},
)
parsenode = ParseNode(
input="doc",
output=["parseddoc"],
nodeconfig={"chunksize": 4096},
)
generatenode = GenerateAnswerNode(
input="userprompt & parseddoc",
output=["answer"],
nodeconfig={
"llmmodel": config["llm"],
"verbose": config.get("verbose", False),
},
)
super().init(
nodes=[fetchnode, parsenode, generatenode],
edges=[
(fetchnode, parsenode),
(parsenode, generatenode),
],
entrypoint=fetchnode,
)
Project Example: Scraping Job Listings
Here is a complete project example for scraping job listings:
import json
from pydantic import BaseModel, Field
from typing import List, Optional
from scrapegraphai.graphs import SmartScraperGraph
class JobListing(BaseModel):
title: str = Field(description="Job title")
company: str = Field(description="Company name")
location: str = Field(description="Work location")
salary: Optional[str] = Field(description="Salary range if available")
requirements: List[str] = Field(description="List of requirements")
jobtype: str = Field(description="Job type: remote/onsite/hybrid")
class JobListingResult(BaseModel):
jobs: List[JobListing] = Field(description="List of job listings")
graphconfig = {
"llm": {
"apikey": "sk-your-api-key",
"model": "openai/gpt-4o-mini",
},
"verbose": True,
"headless": True,
}
scraper = SmartScraperGraph(
prompt="""
Extract all job listings from this page.
For each listing, get the job title, company name,
location, salary range (if available), key requirements,
and job type (remote/onsite/hybrid).
""",
source="https://example.com/jobs",
config=graphconfig,
schema=JobListingResult,
)
result = scraper.run()
with open("joblistings.json", "w", encoding="utf-8") as f:
json.dump(result, f, indent=2, ensureascii=False)
print(f"Successfully extracted {len(result.get('jobs', []))} job listings")
Project Example: E-Commerce Price Monitoring
import json
from datetime import datetime
from pydantic import BaseModel, Field
from typing import List, Optional
from scrapegraphai.graphs import SmartScraperMultiGraph
class Product(BaseModel):
name: str = Field(description="Product name")
price: str = Field(description="Current price")
originalprice: Optional[str] = Field(description="Original price before discount")
discount: Optional[str] = Field(description="Discount percentage")
rating: Optional[str] = Field(description="Product rating")
soldcount: Optional[str] = Field(description="Number of items sold")
class ProductList(BaseModel):
products: List[Product] = Field(description="List of products")
graphconfig = {
"llm": {
"apikey": "sk-your-api-key",
"model": "openai/gpt-4o-mini",
},
"verbose": True,
"headless": True,
}
categories = [
"https://example-shop.com/laptops",
"https://example-shop.com/phones",
"https://example-shop.com/tablets",
]
multiscraper = SmartScraperMultiGraph(
prompt="""
Extract all products from this page including
name, current price, original price, discount,
rating, and number sold.
""",
source=categories,
config=graphconfig,
schema=ProductList,
)
result = multiscraper.run()
timestamp = datetime.now().isoformat()
output = {
"scrapedat": timestamp,
"data": result,
}
with open(f"prices{datetime.now().strftime('%Y%m%d')}.json", "w") as f:
json.dump(output, f, indent=2, ensureascii=False)
print(f"Price data saved successfully at {timestamp}")
Best Practices
1. Choose the Right Model
For simple scraping tasks, smaller models like gpt-4o-mini are sufficient and more cost-effective. Use larger models like gpt-4o or claude-sonnet only for complex pages or when accuracy is critical.
2. Use Pydantic Schemas
Always define output schemas with Pydantic to get consistent and structured results. This also helps the LLM understand the expected output format.
3. Write Specific Prompts
Prompts that are too general like "get all data" will produce inconsistent output. Be specific about what data you want to extract.
# Less effective
prompt = "Get all information from this page"
More effective
prompt = "Extract product names, prices in USD, and star ratings (1-5) for each product item"
4. Handle Errors Gracefully
from scrapegraphai.graphs import SmartScraperGraph
try:
scraper = SmartScraperGraph(
prompt="Extract product data",
source="https://example.com/products",
config=graphconfig,
)
result = scraper.run()
if result:
print("Scraping successful:", result)
else:
print("No data found")
except Exception as e:
print(f"Error during scraping: {e}")
5. Respect Rate Limits and robots.txt
Always pay attention to the rate limits of websites you scrape. Add delays between requests if needed, and check robots.txt before scraping.
6. Use Local Models for High Volume
If you need to perform high-volume scraping, consider using local models through Ollama. This will drastically reduce API costs.
7. Cache Scraping Results
To avoid repeated requests to the same website:
import json
import hashlib
import os
def scrapewithcache(prompt, url, config, cachedir="cache"):
os.makedirs(cachedir, existok=True)
cachekey = hashlib.md5(f"{prompt}{url}".encode()).hexdigest()
cachefile = os.path.join(cachedir, f"{cachekey}.json")
if os.path.exists(cachefile):
with open(cachefile, "r") as f:
return json.load(f)
scraper = SmartScraperGraph(
prompt=prompt,
source=url,
config=config,
)
result = scraper.run()
with open(cachefile, "w") as f:
json.dump(result, f, indent=2)
return result
8. Logging and Monitoring
Enable verbose mode during development to understand the pipeline flow:
graphconfig = {
"llm": {
"apikey": "sk-your-api-key",
"model": "openai/gpt-4o-mini",
},
"verbose": True,
}
Comparison with Other Scraping Libraries
| Feature | ScrapeGraphAI | BeautifulSoup | Scrapy | Selenium |
|---------|--------------|---------------|--------|----------|
| Natural Language Prompt | Yes | No | No | No |
| No Selector Writing | Yes | No | No | No |
| Dynamic Pages | Yes | No | Plugin | Yes |
| Structured Output | Yes (Pydantic) | Manual | Manual | Manual |
| Multi-LLM Support | Yes | N/A | N/A | N/A |
| Local Models | Yes (Ollama) | N/A | N/A | N/A |
| Cost per Request | Yes (LLM) | Free | Free | Free |
| Speed | Medium | Fast | Fast | Slow |
Troubleshooting
Error: Model Not Found
Make sure the model format uses the provider prefix:
# Correct
"model": "openai/gpt-4o-mini"
"model": "anthropic/claude-sonnet-4-20250514"
"model": "ollama/llama3.1"
Wrong
"model": "gpt-4o-mini"
Error: Playwright Not Installed
If you encounter browser-related errors:
pip install playwright
playwright install chromium
Empty Scraping Results
- Check if the URL is accessible
- Try adding
headless: Trueandbrowsertype: chromiumto the config - Make sure the prompt is specific enough
- Check if the website blocks bots
Timeout Error
Add a longer timeout:
graphconfig = {
"llm": {
"api
key": "sk-your-api-key",
"model": "openai/gpt-4o-mini",
},
"loader_kwargs": {
"timeout": 60000,
},
}
Conclusion
ScrapeGraphAI represents a breakthrough in web scraping by combining the ease of natural language processing with the power of graph-based pipelines. Here are its key advantages:
With ScrapeGraphAI, you can focus on what data you want to get, not how to get it. This library is ideal for data scientists, data engineers, and developers who need a fast and adaptive scraping solution.
For more information, visit the official ScrapeGraphAI documentation on their GitHub repository.