Python web scraping templates
Build scrapers quickly with templates for Requests, Beautiful Soup, Scrapy, Playwright, and Selenium
Python templates
🦜️🔗 LangGraph AI agent
Example of how to use LangGraph with Apify Actors to create a social media analysis tool-calling agent.
AI Agent
CrewAI agent
Example of how to use CrewAI with Apify Actors to create a social media analysis tool-calling agent.
AI Agent
LlamaIndex agent
LlamaIndex agent to scrape, deduplicate and summarize contact details from a website
AI Agent
PydanticAI agent
A PydanticAI agent designed to generate jokes, showcasing a minimal example using the Pydantic AI framework.
AI Agent
Smolagents agent
An AI news aggregator that fetches and summarizes the latest news based on user-defined interests using DuckDuckGo search and OpenAI models written in Python Smolagents.
AI Agent
Start with Python
Scrape single page with provided URL with HTTPX and extract data from page's HTML with Beautiful Soup.
Starter
BeautifulSoup
Example of a web scraper that uses Python HTTPX to scrape HTML from URLs provided on input, parses it using BeautifulSoup and saves results to storage.
Playwright + Chrome
Crawler example that uses headless Chrome driven by Playwright to scrape a website. Headless browsers render JavaScript and can help when getting blocked.
Selenium + Chrome
Scraper example built with Selenium and headless Chrome browser to scrape a website and save the results to storage. A popular alternative to Playwright.
Empty Python project
Empty template with basic structure for the Actor with Apify SDK that allows you to easily add your own functionality.
Standby Python project
Template with basic structure for an Actor using Standby mode that allows you to easily add your own functionality.
Starter
Scrapy
This example Scrapy spider scrapes page titles from URLs defined in input parameter. It shows how to use Apify SDK for Python and Scrapy pipelines to save results.
Crawlee + BeautifulSoup
Crawl and scrape websites using Crawlee and BeautifulSoup. Start from a given start URLs, and store results to Apify dataset.
Starter
Crawlee + Playwright + Chrome
Crawl and scrape websites using Crawlee and Playwright. Start from a given start URLs, and store results to Apify dataset.
Starter
Apify Universal Scrapers
Universal Scrapers provide you with a solid boilerplate to build fully functioning scrapers directly on the Apify platform. Configure and run your web scrapers manually in a user interface or programmatically via an API.
Cheerio Scraper
apify/cheerio-scraper
Crawls websites using raw HTTP requests, parses the HTML with the Cheerio library, and extracts data from the pages using a Node.js code. Supports both recursive crawling and lists of URLs. This actor is a high-performance alternative to apify/web-scraper for websites that do not require JavaScript.
Apify
7.1k
4.7
Web Scraper
apify/web-scraper
Crawls arbitrary websites using a web browser and extracts structured data from web pages using a provided JavaScript function. The Actor supports both recursive crawling and lists of URLs, and automatically manages concurrency for maximum performance.
Apify
80.1k
4.5
Puppeteer Scraper
apify/puppeteer-scraper
Crawls websites with the headless Chrome and Puppeteer library using a provided server-side Node.js code. This crawler is an alternative to apify/web-scraper that gives you finer control over the process. Supports both recursive crawling and list of URLs. Supports login to website.
Apify
6.1k
5.0
Playwright Scraper
apify/playwright-scraper
Crawls websites with the headless Chromium, Chrome, or Firefox browser and Playwright library using a provided server-side Node.js code. Supports both recursive crawling and a list of URLs. Supports login to a website.
Apify
1.2k
4.3
Vanilla JS Scraper
mstephen190/vanilla-js-scraper
Scrape the web using familiar JavaScript methods! Crawls websites using raw HTTP requests, parses the HTML with the JSDOM package, and extracts data from the pages using Node.js code. Supports both recursive crawling and lists of URLs. This actor is a non jQuery alternative to CheerioScraper.
Matthias Stephens
456
BeautifulSoup Scraper
apify/beautifulsoup-scraper
Crawls websites using raw HTTP requests. It parses the HTML with the BeautifulSoup library and extracts data from the pages using Python code. Supports both recursive crawling and lists of URLs. This Actor is a Python alternative to Cheerio Scraper.
Apify
812
4.4