A template for web scraping data from websites starting from provided URLs using Python. The starting URLs are passed through the Actor's input schema, defined by the input schema. The template uses Crawlee for Python for efficient web crawling, making requests via headless browser managed by Playwright, and handling each request through a user-defined handler that uses Playwright API to extract data from the page. Enqueued URLs are managed in the request queue, and the extracted data is saved in a dataset for easy access.

Included features

Apify SDK - a toolkit for building Apify Actors in Python.
Crawlee for Python - a web scraping and browser automation library.
Input schema - define and validate a schema for your Actor's input.
Request queue - manage the URLs you want to scrape in a queue.
Dataset - store and access structured data extracted from web pages.
Playwright - a library for managing headless browsers.

Resources

Video introduction to Python SDK
Webinar introducing to Crawlee for Python
Apify Python SDK documentation
Crawlee for Python documentation
Python tutorials in Academy
Integration with Make, GitHub, Zapier, Google Drive, and other apps
Video guide on getting scraped data using Apify API
A short guide on how to build web scrapers using code templates:

Getting started

For complete information see this article. To run the Actor use the following command:

$apify run

Deploy to Apify

Connect Git repository to Apify

If you've created a Git repository for the project, you can easily connect to Apify:

Go to Actor creation page
Click on Link Git Repository button

Push project on your local machine to Apify

You can also deploy the project on your local machine to Apify without the need for the Git repository.

Log in to Apify. You will need to provide your Apify API Token to complete this action.
```
$apify login
```
Deploy your Actor. This command will deploy and build the Actor on the Apify Platform. You can find your newly created Actor under Actors -> My Actors.
```
$apify push
```

Documentation reference

To learn more about Apify and Actors, take a look at the following resources:

On this page

Share Actor:

Website Content Crawler

apify/website-content-crawler

Crawl websites and extract text content to feed AI models, LLM applications, vector databases, or RAG pipelines. The Actor supports rich formatting using Markdown, cleans the HTML, downloads files, and integrates well with 🦜🔗 LangChain, LlamaIndex, and the wider LLM ecosystem.

Apify

76K

4.6

CarGurus Zipcode Search Scraper

tri_angle/cargurus-zipcode-search-scraper

Scraper to find all CarGurus car listings for a given zipcode in the United States of America

Tri⟁angle

3.6

Cypress Test Runner

valek.josef/cypress-test-runner

Run your Cypress tests on the Apify Platform effectively and easily. Just set up your test environment using a user-friendly UI and let the platform do the rest.

Josef Válek

Content & Listings Monitor

cyberfly/ListingMonitor

Monitor anything, anywhere. Receive mobile alerts or execute actions on updates of interest. Useful for listings, news, articles, products and more.

Vasek Codey Vlcek

Upwork Latest Jobs Scraper

bytebrains/upwork-job-scraper

Scrape the latest job listings from Upwork with powerful filters. Extract full job details including title, budget, client info, and skills. Supports login via cookies to access restricted jobs. Perfect for freelancers, agencies, and market researchers.

Louis Pham

IAAI Vehicle Detail Scraper

easyapi/iaai-vehicle-detail-scraper

Scrape detailed vehicle information and high-resolution images from IAAI (Insurance Auto Auctions) vehicle listings. Get comprehensive data including vehicle specs, damage details, auction info, and HD photos.

EasyApi

5.0

Details of runs

useful-tools/details-of-runs

This Actor is capable to list and store your Runs into the Dataset with all the necessary Run related details (like stats, usage, current pricing).

Useful tools

Facebook Games Scraper

apify/facebook-games-scraper

Scrape and download Facebook Games names and live streams; followers count, concurrent viewer count, URL, and more. Get data from Facebook Games simply using one search query. Export scraped data, run the scraper via API, schedule and monitor runs, or integrate with other tools.

Apify

4.8

Hacker News Data Scraper & Activity Monitoring

lucen_data/hacker-news-data-scraper-activity-monitoring

Unofficial Hacker News API to extract data from all Hacker News categories. You can filter posts based on specific keywords, track trends using the unique activity monitoring feature, and receive updates directly in your Slack channel.