Crawls websites using Chrome browser and extracts data from pages using a provided front-end JavaScript code. The actor supports both recursive crawling and lists of URLs, and automatically scales the crawling for maximum performance. This is Apify's primary tool for web scraping and crawling.

web-scraper

Web Scraper

Crawls Google Search result pages (SERPs) and extracts a list of organic and paid results, ads, snap packs and more. Supports selection of custom country or language, and extraction of custom attributes.

google-search-scraper

Google Search Scraper

Sends an email to a specific address. This actor is useful for notifications and reporting.

send-mail

Send Email

Replacement for the legacy Apify Crawler product with a backward-compatible interface. The actor uses PhantomJS headless browser to recursively crawl websites and extract data from them using a piece of front-end JavaScript code.

legacy-phantomjs-crawler

Legacy PhantomJS Crawler

Crawls websites with the headless Chrome and Puppeteer library using a provided server-side Node.js code. This actor is an alternative to apify/web-scraper that gives you a low-level control of the crawling process. Supports both recursive crawling and list of URLs.

puppeteer-scraper

Puppeteer Scraper

Crawls websites using raw HTTP requests, parses the HTML with the Cheerio library and extracts data from the pages using a provided Node.js code. Supports both recursive crawling and lists of URLs. This actor is a high-performance alternative to apify/web-scraper for websites that do not require JavaScript.

cheerio-scraper

Cheerio Scraper

Run Scrapy spiders written in Python on the Apify platform.

scrapy-executor

Scrapy Executor

Returns a difference of two given images as JPEG or PNG image.

image-diff

Image Diff

An example of a simple Apify actor. It takes a greeting message on input, prints it and stores a new message to output.

hello-world

Hello World

Generates a screenshot of a webpage on a specified URL. The screenshot is stored to the default key-value store as the output.

screenshot-url

Webpage screenshot downloader

Example showing how to use headless Chromium with Puppeteer to open a web page, determine its dimensions, save a screenshot and print the page to PDF.

For more information about Puppeteer, please see https://github.com/GoogleChrome/puppeteer

This act must use image with Puppeteer (Node.js 8 + Puppeteer on Debian).

example-puppeteer

Example Puppeteer

Performs analysis of a webpage to figure the best way how to scrapes its data. On input, it takes an URL and array of strings to search for, and on output, it returns a definition of a crawler.

page-analyzer

Page Analyzer

Example act using PHP as the main language.

example-php

Example Php

Act to upload results from Apify crawler to AWS S3. It is designed to run from crawler finish webhook.

crawler-results-to-s3

Crawler Results To S3

Contains a basic boilerplate of an Apify actor with Node.js source code and a custom Dockerfile. The purpose of this actor is to help you get started quickly to create your own actors.

quick-start

Quick Start

Example of loading a web page in headless Chrome using Selenium Webdriver.

example-selenium

Example Selenium

Example act that opens a webpage with Golden Gate webcam stream. It takes a screenshot from the stream and saves it as output to key-value store. You can easily use it as API that returns a screenshot with: https://api.apify.com/v2/acts/apify~example-golden-gate-webcam/run-sync?token=[YOUR_API_TOKEN]

example-golden-gate-webcam

Example Golden Gate Webcam

This actor serves as an example of a crawling run using the Live View feature. It crawls through Hacker News page by page and you may inspect any of the pages' screenshot or HTML in the Live View panel.

example-live-view

Example Live View

Example act that iterates through all results from a crawler run and counts them. The act shall be called from the crawler's finish webhook. To do so, simply add the following URL to the finish webhook of your crawler:

https://api.apify.com/v2/acts/apify~example-process-crawl-results/runs?token=<YOUR_API_TOKEN>

You can use this act as a starting point to develop custom post-processing of data from the crawler.

example-process-crawl-results

Example Process Crawl Results

Generates a HTTP Archive (HAR) file for web pages specified in a list of URLs. Optionally, the pages can be loaded using proxies from a specific country - to use this feature, you'll need access to Apify Proxy. 

On input, the actor accepts JSON object with the following fields:
- sourceUrls: An array specifying which web pages to load. The format is same as the 'source' parameter of the RequestList in Apify SDK for Node.js. It can combined URLs as well as a link to a CSV file with the list of URLs.
- country: Optional country code

On output, the actor stores the resulting HAR files to the run's default key-value store, and also stores a table into the dataset with information about each web page visited and a link to the corresponding HAR file, or description of errors.

har-files-for-url-list

Har Files For Url List

A simple example showing how to call another actor from an actor. Note that this actor doesn't accept any input and doesn't generate any output.

example-call

Example Call

This example demonstrates how to use web server in actor as communication channel with outer world. Read article about this crawler in Apify knowledge base: https://kb.apify.com/actor/running-a-web-server

example-web-server

Example Web Server

This act simply counts from one up. In each run it prints one number. Its state (counter position) is stored in named key-value store. Name of the store is example-counter and you can find in Apify app under the Storages.

example-counter

Example Counter

This actor simply tests given array of URLs against selected proxy URLs or Apify proxy groups.

proxy-test

Proxy Test

Example of an Apify actor with source code hosted in a GitHub Gist. For example, this is useful if you want to have public code examples accompanied by a working API that anyone can try straight away.

example-github-gist

Example using GitHub Gist

The Indexer crawls recursively a website using the Puppeteer browser (headless Chrome) and indexes the selected pages to the Algolia index.

algolia-website-indexer

Algolia Website Indexer

apify - The official account of Apify Technologies. It hosts a number of generic actors for the public use.