Pricing

from $0.01 / result

Website Content Crawler

Crawl websites and extract text content to feed AI models, LLM applications, vector databases, or RAG pipelines. The Actor supports rich formatting using Markdown, cleans the HTML, and the wider LLM ecosystem.

Pricing

from $0.01 / result

Rating

0.0

(0)

Developer

yun qing

Actor stats

Bookmarked

Total users

Monthly active users

11 hours ago

Last modified

Categories

Developer tools

News

Social media

Start URLs

startUrls

Required

One or more website URLs to crawl.

Type:array

Crawl Mode

crawlMode

Optional

Choose whether to crawl directly from website pages or from sitemap URLs.

Type:string

Default:website

Options:

websitesitemap

Sitemap URLs

sitemapUrls

Optional

Optional sitemap URLs. Used in sitemap mode. If empty, the actor tries /sitemap.xml from each start URL origin.

Type:array

Max Pages

maxPages

Optional

Maximum number of pages to crawl in one run.

Type:integer

Minimum:1

Default:50

Max Depth

maxDepth

Optional

How deep the crawler can follow links from the start URLs.

Type:integer

Minimum:0

Default:2

Output Format

outputFormat

Optional

Choose the main content format stored in the content field.

Type:string

Default:markdown

Options:

markdowntexthtml

Stay In Start URL Scope

sameDomainOnly

Optional

When enabled, only crawl links within the same origin and path scope as the start URLs.

Type:boolean

Default:true

Content Selector

contentSelector

Optional

Optional CSS selector for the content root. Falls back to main/article/body.

Type:string

Remove Selectors

removeSelectors

Optional

Optional CSS selectors to remove before extracting content.

Type:array

Include URL Globs

includeUrlGlobs

Optional

Optional glob patterns for links you want to keep.

Type:array

Exclude URL Globs

excludeUrlGlobs

Optional

Optional glob patterns for links you want to skip.

Type:array

Exclude File Extensions

excludeFileExtensions

Optional

File extensions to skip, for example pdf, jpg, png, zip.

Type:array

Min Text Length

minTextLength

Optional

Skip dataset output when extracted text is shorter than this number of characters.

Type:integer

Minimum:0

Default:200

Wait For Selector

waitForSelector

Optional

Optional CSS selector to wait for before extracting content.

Type:string

Navigation Timeout (secs)

navigationTimeoutSecs

Optional

Timeout for page navigation and optional selector waiting.

Type:integer

Minimum:15

Default:45

Save Clean HTML

saveCleanHtml

Optional

Store cleaned HTML separately in key-value store records and index them in CLEAN_HTML_INDEX.

Type:boolean

Default:true

Proxy Configuration

proxyConfiguration

Optional

Optional Apify proxy configuration.

Type:object

Website Content Crawler

apify/website-content-crawler

Apify

111K

4.3

(178)

Fast Website Content Crawler

6sigmag/fast-website-content-crawler

A high-performance web scraper that rapidly extracts and analyzes content from multiple websites simultaneously. Perfect for competitive research, content aggregation, and website structure analysis.

David

3.5K

4.9

(7)

🔥 FireScrape AI Website Content Markdown Scraper

mohamedgb00714/fireScraper-AI-Website-Content-Markdown-Scraper

Advanced web scraper powered by Crawlee and Puppeteer — extracts website content, converts it to Markdown, and structures it for LLM training datasets.

mohamed el hadi msaid

280

1.9

(2)

Deep Website Content Crawler

6sigmag/deep-website-content-crawler

Scrape Failed Killer! A high-performance web scraper that rapidly extracts and analyzes content from multiple websites simultaneously. Perfect for competitive research, content aggregation, and website structure analysis.

David

3.0

(1)

AI Website Content Markdown Scraper

quaking_pail/ai-website-content-markdown-scraper

This Apify Actor, "Website Content Crawler with Markdown Extraction," is designed to perform a comprehensive crawl of specified websites, extract their text content, convert it into Markdown format, and store it in a structured dataset. The extracted content is suitable for feeding LLMs.

AI_Builder

907

2.3

(3)

Website Content Crawler Pro

datascoutapi/website-content-crawler-pro

Crawl websites and extract clean, structured content in Markdown, JSON, or plain text for AI models, LLMs, vector DBs, or RAG pipelines. Fast, reliable, and stealthy, with bulk processing, advanced metadata extraction, and seamless integration with LangChain, LlamaIndex, and AI workflows.

halam

508

2.1

(3)

🧪High-Volume Website Content & Media Scraper

caring_dizi/blog-content-scraper-fixed

🧪Crawling Done Right! Let me now what you think, what or where or how i can improve my actor, and i am all for constructive criticism. So please message if you have any questions. Enjoy and have a good day.

Jeff Halverson

139

5.0

(2)

Website Content Crawler

alizarin_refrigerator-owner/website-crawler

Crawl websites for SEO audits. Extracts HTML, title, meta tags, headings, links, & text content from pages. Automatic sitemap detection & parsing Extracts metadata (title, description, OG tags) Heading structure (H1, H2, H3) Internal & external link analysis Image extraction w/alt text Word count

The Howlers

Quick Website Content Scraper ( Extract Text for RAG & LLMs )

automateitplease/ai-web-content-scraper-extract-text-for-rag-llms

Extract clean text from any website for AI/LLM applications. Supports both static and JavaScript-rendered sites (React, Vue, Angular). Perfect for RAG systems, chatbot training, and content analysis.