Pricing

from $0.01 / result

Website Content Crawler

Crawl websites and extract text content to feed AI models, LLM applications, vector databases, or RAG pipelines. The Actor supports rich formatting using Markdown, cleans the HTML, and the wider LLM ecosystem.

Pricing

from $0.01 / result

Rating

0.0

(0)

Developer

yun qing

Actor stats

Bookmarked

Total users

Monthly active users

8 hours ago

Last modified

Features

Crawl from one or more start URLs
Crawl from sitemap URLs in sitemap mode
Follow links recursively with maxDepth
Keep the crawl inside the same start URL scope
Filter out PDFs and other non-HTML files by extension
Extract cleaned page content from main, article, or a custom selector
Output markdown, text, or html
Store OUTPUT_SUMMARY, FAILED_PAGES, SKIPPED_PAGES, and CLEAN_HTML_INDEX
Store cleaned HTML separately in key-value store records like CLEAN_HTML_000001

Local Development

pnpm actor:dev websiteContentCrawler --example 0 --force-input
pnpm actor:dev websiteContentCrawler --example 2 --force-input

Notes:

input-examples.json is used by local actor:dev
Apify platform automated testing uses the prefill values from .actor/input_schema.json
The schema now uses a public default URL so automated testing can pass without relying on localhost

Build

$pnpm actor:build websiteContentCrawler

Publish

pnpm actor:push websiteContentCrawler
pnpm actor:push websiteContentCrawler --dry-run

Dataset Output

Each dataset item includes:

url
title
description
content
contentFormat
cleanHtml
markdown
text
html
wordCount
language
canonicalUrl
depth
httpStatusCode
crawledAt

Crawl Modes

website: start from startUrls, then follow links recursively
sitemap: load URLs from sitemapUrls or fallback origin + /sitemap.xml

Separate Clean HTML Storage

CLEAN_HTML_INDEX stores the mapping between page URL and KVS record key
Individual cleaned HTML records are stored as CLEAN_HTML_000001, CLEAN_HTML_000002, and so on

Website Content Crawler

apify/website-content-crawler

Apify

111K

4.3

(178)

Fast Website Content Crawler

6sigmag/fast-website-content-crawler

A high-performance web scraper that rapidly extracts and analyzes content from multiple websites simultaneously. Perfect for competitive research, content aggregation, and website structure analysis.

David

3.5K

4.9

(7)

🔥 FireScrape AI Website Content Markdown Scraper

mohamedgb00714/fireScraper-AI-Website-Content-Markdown-Scraper

Advanced web scraper powered by Crawlee and Puppeteer — extracts website content, converts it to Markdown, and structures it for LLM training datasets.

mohamed el hadi msaid

280

1.9

(2)

Deep Website Content Crawler

6sigmag/deep-website-content-crawler

Scrape Failed Killer! A high-performance web scraper that rapidly extracts and analyzes content from multiple websites simultaneously. Perfect for competitive research, content aggregation, and website structure analysis.

David

3.0

(1)

AI Website Content Markdown Scraper

quaking_pail/ai-website-content-markdown-scraper

This Apify Actor, "Website Content Crawler with Markdown Extraction," is designed to perform a comprehensive crawl of specified websites, extract their text content, convert it into Markdown format, and store it in a structured dataset. The extracted content is suitable for feeding LLMs.

AI_Builder

907

2.3

(3)

Website Content Crawler Pro

datascoutapi/website-content-crawler-pro

Crawl websites and extract clean, structured content in Markdown, JSON, or plain text for AI models, LLMs, vector DBs, or RAG pipelines. Fast, reliable, and stealthy, with bulk processing, advanced metadata extraction, and seamless integration with LangChain, LlamaIndex, and AI workflows.

halam

508

2.1

(3)

🧪High-Volume Website Content & Media Scraper

caring_dizi/blog-content-scraper-fixed

🧪Crawling Done Right! Let me now what you think, what or where or how i can improve my actor, and i am all for constructive criticism. So please message if you have any questions. Enjoy and have a good day.

Jeff Halverson

139

5.0

(2)

Website Content Crawler

alizarin_refrigerator-owner/website-crawler

Crawl websites for SEO audits. Extracts HTML, title, meta tags, headings, links, & text content from pages. Automatic sitemap detection & parsing Extracts metadata (title, description, OG tags) Heading structure (H1, H2, H3) Internal & external link analysis Image extraction w/alt text Word count

The Howlers

Quick Website Content Scraper ( Extract Text for RAG & LLMs )

automateitplease/ai-web-content-scraper-extract-text-for-rag-llms

Extract clean text from any website for AI/LLM applications. Supports both static and JavaScript-rendered sites (React, Vue, Angular). Perfect for RAG systems, chatbot training, and content analysis.