Pricing

Pay per usage

Try for free

Go to Apify Store

BeautifulSoup Scraper

Try for free

Crawls websites using raw HTTP requests. It parses the HTML with the BeautifulSoup library and extracts data from the pages using Python code. Supports both recursive crawling and lists of URLs. This Actor is a Python alternative to Cheerio Scraper.

Pricing

Pay per usage

Rating

5.0

(6)

Developer

Apify

Actor stats

Bookmarked

Total users

Monthly active users

7 days ago

Last modified

How it works

You give the scraper two things: where to start and how to extract data.

It adds your Start URLs to the crawling queue.
It fetches each URL and builds a BeautifulSoup DOM from the HTML.
It runs your Page function on the page and stores the returned data.
Optionally, it follows links matching your Link selector / Link patterns and enqueues them for recursive crawling.

Page function

Python code run for every page. It receives a BeautifulSoupCrawlingContext and returns the data to store:

from typing import Any
from crawlee.crawlers import BeautifulSoupCrawlingContext

def page_function(context: BeautifulSoupCrawlingContext) -> Any:
    return {
        'url': context.request.url,
        'title': context.soup.title.string if context.soup.title else None,
    }

The code runs on Python 3.14 and may only import modules already installed in the Actor.

Proxy configuration

A proxy is required. Set proxyConfiguration to use Apify Proxy (automatic or selected groups) or your own custom proxy URLs:

{
  "useApifyProxy": true,           // use Apify Proxy
  "apifyProxyGroups": [],          // optional: specific groups
  "proxyUrls": []                  // or custom "scheme://user:pass@host:port" URLs
}

Output

Results returned by your page function land in the run's default dataset. Download them as JSON, CSV, XML, or Excel from Apify Console, or via the API:

https://api.apify.com/v2/datasets/[DATASET_ID]/items?format=json&clean=true

Limitations

The Actor uses raw HTTP requests, so it can't render JavaScript. For dynamic sites use Web Scraper instead. To add Python modules not bundled here, open an issue or PR at github.com/apify/actor-beautifulsoup-scraper.

Latest News MCP Server - Live Global Updates for AI

mrbridge/latest-news-mcp-server

14 MCP tools aggregating 27 free APIs: global news from Reuters, AP, BBC, CNN, Al Jazeera, Bloomberg, GDELT (65+ languages), crypto markets, weather, earthquakes, Reddit, Hacker News, Wikipedia trends, predictions & more. No API keys needed. Works with Claude Desktop, Claude Code & Cursor.

MrBridge

5.0

Cambodia Property Scraper

mai_amm/cambodia-property-scraper

Scrape public Cambodia property listings from Realestate.com.kh and Khmer24 Property with prices, locations, agent contacts, images, and normalized lead fields.

wiseld_squid

JobsDB Thailand Scraper

mai_amm/jobsdb-thailand-scraper

Scrape public job listings from th.jobsdb.com with optional detail enrichment.

wiseld_squid

Singapore Port Congestion & Trade Signal Radar

mai_amm/singapore-port-congestion-trade-signal-radar

Monitor Singapore port congestion pressure, trade activity, and shipping disruption signals using official statistics, optional AIS data, and optional news signals.

wiseld_squid

DDProperty Scraper | Fast & Reliable

fatihtahta/ddproperty-scraper

Extracts detailed real estate listings in Thailand from any search URL, handling pagination automatically. Gathers prices, addresses, agent info, and photos The DDProperty scraper works fast and delivers clean, structured data for your market research.

Fatih Tahta

5.0

Thailand Used Car Market Scraper

mai_amm/thai-used-car-scraper

Scrapes and normalizes used car listings from One2car, Carsome Thailand, Taladrod, and Kaidee Auto.

wiseld_squid

Cheerio Scraper

apify/cheerio-scraper

Crawls websites using raw HTTP requests, parses the HTML with the Cheerio library, and extracts data from the pages using a Node.js code. Supports both recursive crawling and lists of URLs. This actor is a high-performance alternative to apify/web-scraper for websites that do not require JavaScript.

Apify

17K

4.6

Waze Route & City Monitor

mai_amm/waze-route-city-monitor

Monitor Waze traffic alerts and jams across multiple city areas and route corridors with one dataset item per incident, jam, or route summary.

wiseld_squid

Fastwork Thailand Scraper: Services, Prices & Reviews

mai_amm/fastwork-thailand-scraper

Scrape public Fastwork Thailand services, freelancers, packages, prices, ratings, reviews, seller metrics, and monitoring changes for market research and vendor sourcing.

wiseld_squid

DDProperty TH $1💰 Powerful Filters + Deep Search

abotapi/ddproperty-scraper

From $1/1K. Extract property listings from ddproperty.com Thailand at scale. Get comprehensive data including prices, features, images, agent contacts, coordinates, nearby transit (BTS/MRT), and more. Perfect for Thai real estate analytics, market research, and investment analysis.