Python HTTP Scraper Template (curl_cffi + BeautifulSoup)

A clean and efficient Apify Actor template for building browserless HTTP scrapers in Python.

It uses:

• curl_cffi for high-success HTTP requests with browser impersonation
• BeautifulSoup for HTML parsing
• Apify SDK for Actor lifecycle, datasets, and proxy management

---

Overview

This template is designed for straightforward scraping tasks where full browser automation is not required.

It:

• performs a warm-up request to the target’s origin to establish a session
• reuses cookies and connections via a shared curl_cffi.requests.AsyncSession
• supports Apify Proxy via the standard proxyConfiguration input field
• extracts listings using multiple extraction strategies (JSON-LD + generic DOM selectors)
• supports simple pagination (rel="next", “Next” buttons, ?page=N pattern)
• saves structured data into the Apify Dataset

---

Stack

• Python (async, runs on apify/actor-python:3.13)
• Apify SDK (Python) – Actor lifecycle, input, dataset, proxy config
• curl_cffi – fast HTTP client with browser TLS fingerprint impersonation
• BeautifulSoup – HTML parsing and selector-based extraction

---

Features

• Apify SDK – run scalable Actors on the Apify platform
• curl_cffi AsyncSession – shared session with cookies + connections reused
• Browser impersonation – realistic TLS fingerprints via impersonate="chrome120"
• Proxy support – uses Apify proxy or custom proxies via proxyConfiguration
• Warm-up request – hits the origin once to prime cookies and session state
• Multi-strategy extraction:
  • JSON/JSON-LD <script> blocks (structured data)
  • generic CSS selectors for listings (.listing, .result, li.product, article, …)
• Pagination support – rel="next", “Next/›/»” links, and ?page=N increment
• Monetization hooks – emits outputrecord charge events per pushed item (if available)
• Structured output – saves extracted data to the default Apify Dataset

---

How It Works

1. Input

• Reads the Actor input via Actor.get_input():
  • startUrls – list of URLs to crawl
  • queries – free-text queries turned into search URLs via build_search_url
  • limit – max records to save
  • proxyConfiguration – standard Apify proxy editor object

2. Seeding

   • Normalizes URLs (removes fragments, sorts query params).
   • Deduplicates seeds across startUrls and query-based search URLs.

3. Proxy & Session

   • Creates ProxyConfiguration via Actor.create_proxy_configuration(actor_proxy_input=proxyConfiguration).
   • Generates proxy URLs with fresh session IDs to avoid reusing blocked connections.
   • Creates a curl_cffi.requests.AsyncSession with:
     • Chrome-like headers
     • impersonation (chrome120)
     • proxies
     • timeout and redirect limits

4. Warm-up

   • Derives the origin from the first seed (e.g. https://www.example.com/).
   • Sends a warm-up request to prime cookies and the proxy session.

5. Crawling & Extraction

   • For each seed, paginates using:
     • <a rel="next">
     • links containing “Next”, “›”, “»”
     • incremented ?page=N query param
   • For each page:
     • downloads HTML via curl_cffi
     • parses with BeautifulSoup
     • runs extractors in order:
       1. JSON/JSON-LD script blocks
       2. Generic listing selectors
     • cleans records to a safe public subset of keys

6. Saving

   • Pushes extracted objects to the default Apify Dataset via Actor.push_data.
   • Emits a monetization outputrecord event for each item (if monetization is enabled).
   • Stops automatically once limit is reached.

---

Actor Input

startUrls

• Type: array<string>
• List of URLs to crawl directly. Each one is normalized and deduplicated.

queries

• Type: array<string>
• Free-text queries; each is converted into a search URL using build_search_url().

You should customize build_search_url() in src/main.py per target site.

limit

• Type: integer
• Maximum number of records to save overall.
• Default in code: 1000 (the .actor/input_schema.json can override this).

proxyConfiguration

• Type: object
• Standard Apify proxy editor object (useApifyProxy, apifyProxyGroups, etc.).
• Passed directly to Actor.create_proxy_configuration(actor_proxy_input=...).

---

Adapting the Template for a Specific Target

1. Set the search base URL

   In src/main.py, change:

   base = "https://www.example.com/search"