Python web scraping templates

Build scrapers quickly with templates for Requests, Beautiful Soup, Scrapy, Playwright, and Selenium

Python templates

This example Scrapy spider scrapes page titles from URLs defined in input parameter. It shows how to use Apify SDK for Python and Scrapy pipelines to save results.

Start with Python

Scrape single page with provided URL with HTTPX and extract data from page's HTML with Beautiful Soup.

Starter

BeautifulSoup

Example of a web scraper that uses Python HTTPX to scrape HTML from URLs provided on input, parses it using BeautifulSoup and saves results to storage.

Playwright + Chrome

Crawler example that uses headless Chrome driven by Playwright to scrape a website. Headless browsers render JavaScript and can help when getting blocked.

Selenium + Chrome

Scraper example built with Selenium and headless Chrome browser to scrape a website and save the results to storage. A popular alternative to Playwright.

Empty Python project

Empty template with basic structure for the Actor with Apify SDK that allows you to easily add your own functionality.

Standby Python project

Template with basic structure for an Actor using Standby mode that allows you to easily add your own functionality.

Starter

Crawlee + BeautifulSoup

Crawl and scrape websites using Crawlee and BeautifulSoup. Start from a given start URLs, and store results to Apify dataset.

Starter

Crawlee + Playwright + Chrome

Crawl and scrape websites using Crawlee and Playwright. Start from a given start URLs, and store results to Apify dataset.

Starter

Apify Universal Scrapers

Universal Scrapers provide you with a solid boilerplate to build fully functioning scrapers directly on the Apify platform. Configure and run your web scrapers manually in a user interface or programmatically via an API.

Cheerio Scraper

apify/cheerio-scraper

Crawls websites using raw HTTP requests, parses the HTML with the Cheerio library, and extracts data from the pages using a Node.js code. Supports both recursive crawling and lists of URLs. This actor is a high-performance alternative to apify/web-scraper for websites that do not require JavaScript.

Apify

5.8k

Web Scraper

apify/web-scraper

Crawls arbitrary websites using the Chrome browser and extracts data from pages using JavaScript code. The Actor supports both recursive crawling and lists of URLs and automatically manages concurrency for maximum performance. This is Apify's basic tool for web crawling and scraping.

Apify

72.4k

248

Puppeteer Scraper

apify/puppeteer-scraper

Crawls websites with the headless Chrome and Puppeteer library using a provided server-side Node.js code. This crawler is an alternative to apify/web-scraper that gives you finer control over the process. Supports both recursive crawling and list of URLs. Supports login to website.

Apify

4.7k

Playwright Scraper

apify/playwright-scraper

Crawls websites with the headless Chromium, Chrome, or Firefox browser and Playwright library using a provided server-side Node.js code. Supports both recursive crawling and a list of URLs. Supports login to a website.

Apify

881

Vanilla JS Scraper

mstephen190/vanilla-js-scraper

Scrape the web using familiar JavaScript methods! Crawls websites using raw HTTP requests, parses the HTML with the JSDOM package, and extracts data from the pages using Node.js code. Supports both recursive crawling and lists of URLs. This actor is a non jQuery alternative to CheerioScraper.

Matthias Stephens

425

BeautifulSoup Scraper

apify/beautifulsoup-scraper

Crawls websites using raw HTTP requests. It parses the HTML with the BeautifulSoup library and extracts data from the pages using Python code. Supports both recursive crawling and lists of URLs. This Actor is a Python alternative to Cheerio Scraper.

Apify

736

Already have your own solution?

Sign up for a free Apify account and deploy your code to the platform in just a few minutes! If you need help deploying your code, check out our documentation.

Import your code Check Apify docs