Back to template gallery

Start with Python

Scrape single page with provided URL with HTTPX and extract data from page's HTML with Beautiful Soup.





Use cases


Web scraping



2This module defines the `main()` coroutine for the Apify Actor, executed from the `` file.
4Feel free to modify this file to suit your specific needs.
6To build Apify Actors, utilize the Apify SDK toolkit, read more at the official documentation:
10# Beautiful Soup - library for pulling data out of HTML and XML files, read more at
12from bs4 import BeautifulSoup
14# HTTPX - library for making asynchronous HTTP requests in Python, read more at
15from httpx import AsyncClient
17# Apify SDK - toolkit for building Apify Actors, read more at
18from apify import Actor
21async def main() -> None:
22    """
23    The main coroutine is being executed using ``, so do not attempt to make a normal function
24    out of it, it will not work. Asynchronous execution is required for communication with Apify platform,
25    and it also enhances performance in the field of web scraping significantly.
26    """
27    async with Actor:
28        # Structure of input is defined in input_schema.json
29        actor_input = await Actor.get_input() or {}
30        url = actor_input.get('url')
32        # Create an asynchronous HTTPX client
33        async with AsyncClient() as client:
34            # Fetch the HTML content of the page.
35            response = await client.get(url, follow_redirects=True)
37        # Parse the HTML content using Beautiful Soup
38        soup = BeautifulSoup(response.content, 'html.parser')
40        # Extract all headings from the page (tag name and text)
41        headings = []
42        for heading in soup.find_all(['h1', 'h2', 'h3', 'h4', 'h5', 'h6']):
43            heading_object = {'level':, 'text': heading.text}
44  'Extracted heading: {heading_object}')
45            headings.append(heading_object)
47        # Save headings to Dataset - a table-like storage
48        await Actor.push_data(headings)

Scrape single-page in Python template

A template for web scraping data from a single web page in Python. The URL of the web page is passed in via input, which is defined by the input schema. The template uses the HTTPX to get the HTML of the page and the Beautiful Soup to parse the data from it. The data are then stored in a dataset where you can easily access them.

The scraped data in this template are page headings but you can easily edit the code to scrape whatever you want from the page.

Included features

  • Apify SDK for Python - a toolkit for building Apify Actors and scrapers in Python
  • Input schema - define and easily validate a schema for your Actor's input
  • Request queue - queues into which you can put the URLs you want to scrape
  • Dataset - store structured data where each object stored has the same attributes
  • HTTPX - library for making asynchronous HTTP requests in Python
  • Beautiful Soup - library for pulling data out of HTML and XML files

How it works

  1. Actor.get_input() gets the input where the page URL is defined
  2. httpx.AsyncClient().get(url) fetches the page
  3. BeautifulSoup(response.content, 'html.parser') loads the page data and enables parsing the headings
  4. This parses the headings from the page and here you can edit the code to parse whatever you need from the page
    for heading in soup.find_all(["h1", "h2", "h3", "h4", "h5", "h6"]):
  5. Actor.push_data(headings) stores the headings in the dataset


Already have a solution in mind?

Sign up for a free Apify account and deploy your code to the platform in just a few minutes! If you want a head start without coding it yourself, browse our Store of existing solutions.