Pricing

from $2.10 / 1,000 pages

Try for free

Go to Apify Store

Website Content Crawler

Try for free

Website content crawler for RAG and LLMs: each page as clean Markdown, menus and footers removed, optional chunks. $3 per 1,000 saved pages ($2.10 on Business); refused and failed pages are free. HTML only: no JavaScript rendering, proxy or PDF. Max 5,000 pages per website.

Pricing

from $2.10 / 1,000 pages

Rating

0.0

(0)

Developer

Pumpkin Farm

Actor stats

Bookmarked

Total users

Monthly active users

6 hours ago

Last modified

Categories

Developer tools

Automation

Features

You can access the Website Content Crawler programmatically from your own applications by using the Apify API. You can also choose the language preference from below. To use the Apify API, you’ll need an Apify account and your API token, found in API & Integrations in Apify Console.

Python

JavaScript

CLI

OpenAPI

HTTP

MCP

1from apify_client import ApifyClient
2
3# Initialize the ApifyClient with your Apify API token
4# Replace '<YOUR_API_TOKEN>' with your token.
5client = ApifyClient("<YOUR_API_TOKEN>")
6
7# Prepare the Actor input
8run_input = {
9    "startUrls": ["https://docs.apify.com/academy/scraping-basics-javascript"],
10    "maxPages": 20,
11}
12
13# Run the Actor and wait for it to finish
14run = client.actor("pumpkinapi/website-content-crawler").call(run_input=run_input)
15
16# Fetch and print Actor results from the run's dataset (if there are any)
17print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
18for item in client.dataset(run.default_dataset_id).iterate_items():
19    print(item)
20
21# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

Website Content Crawler for RAG: Markdown & Chunks API in Python

The Apify API client for Python is the official library that allows you to use Website Content Crawler API in Python, providing convenience functions and automatic retries on errors.

Install the apify-client

$pip install apify-client

Other API clients include:

Website Content Crawler API in JavaScript

Website Content Crawler API through CLI

Website Content Crawler OpenAPI definition

Website Content Crawler API

Google Search Scraper ($0.15 per 1000 results)

igolaizola/google-search-scraper-ppe

$0.15 per 1000 results 💸 Fast & Low Cost Google Search scraper that returns clean, structured SERP data. Get position, title, URL and snippet. Control region, language and domain, with built-in Apify proxy support for reliable, geo-accurate results.

Iñigo Garcia Olaizola

3.2K

4.1

(6)

Youtube Transcript Scraper

pintostudio/youtube-transcript-scraper

Looking for a reliable way to extract transcripts from YouTube videos? 🎥✨ Look no further! The YouTube-Transcript-Scraper has you covered. 🚀 It effortlessly retrieves transcripts while offering additional valuable insights. Ready to start? Let’s scrape away! 🕵️‍♂️💻

Pinto Studio

28K

3.7

(49)

YouTube Comments Scraper

streamers/youtube-comments-scraper

This alternative YouTube Data API has no limits or quotas. Extract YouTube comments data from one or multiple YouTube videos: full comment text, posting date, author username, video title, videoId. Download YouTube comments in JSON, CSV, and Excel.

Streamers

30K

4.9

(42)

YouTube Thumbnail Downloader

moving_beacon-owner1/my-actor-73

Download high-quality thumbnails from YouTube videos. Bulk processing allows simultaneous downloads from multiple URLs. Choose from quality options: maxresdefault, sddefault, hqdefault, or mqdefault. Automatic fallback switches to a lower quality if necessary.

Jamshaid Arif

573

Website Content Crawler

mikolabs/website-content-crawler

Deep-crawl websites to extract clean text, Markdown, or HTML for AI/LLM apps, RAG pipelines, and vector databases. Supports adaptive crawling, HTML cleaning, file downloads, and structured dataset output. Easily integrates with LangChain, LlamaIndex, and other LLM tools.

Mikolabs

5.0

(1)

Fast Website Content Crawler

6sigmag/fast-website-content-crawler

A high-performance web scraper that rapidly extracts and analyzes content from multiple websites simultaneously. Perfect for competitive research, content aggregation, and website structure analysis.

David

4.8K

4.9

(7)

Website Content Crawler – Markdown for AI, Emails & Contacts

locomint/website-crawler-content-contacts

Website content crawler that turns a whole site into clean Markdown for LLM and RAG pipelines, with the role emails, social profiles and WhatsApp numbers found on each page. Seeded from the sitemap, honours robots.txt, plain HTTP with no browser.

Locomint

Website Content Crawler - Clean Text, Markdown & RAG Chunks

topapi/website-content-crawler

Crawl any website or URL list for $1.20 per 1,000 pages: one row per page with clean text, Markdown, HTML, metadata, links, headings, tables and RAG chunks. Plain HTTP first, headless Chrome only for JavaScript pages. Sitemaps, llms.txt, PDF/DOCX text, Bing/DuckDuckGo search, change monitor.

Top Api

Facebook Groups Scraper

apify/facebook-groups-scraper

Extract data from one or multiple public Facebook groups. Get group and post URLs, post text, comments, timestamp, likes and comments count, and basic commentator info. Download the data in JSON, CSV, and Excel and use it in apps, spreadsheets, and reports.