Pricing

from $2.49 / 1,000 results

Try for free

Go to Apify Store

Website Content Crawler

Try for free

Website Content Crawler SD - Website Content Crawler is a web crawling tool that extracts the main content of every page of a website as clean Markdown and text, with title, language and metadata, for LLMs, RAG, vector databases and search indexes.

Pricing

from $2.49 / 1,000 results

Rating

0.0

(0)

Developer

Neuro Scraper

Actor stats

Bookmarked

Total users

Monthly active users

5 days ago

Last modified

Categories

Developer tools

Start URLs

startUrls

Optional

Pages where the crawl starts, e.g. https://crawlee.dev/python/docs/guides or https://example.com. Links are followed inside the crawl scope below.

Type:array

Max pages

maxPages

Optional

Maximum number of pages saved to the dataset (one row per page).

Type:integer

Minimum:1

Maximum:1000000

Default:20

Max crawl depth

maxCrawlDepth

Optional

How many links away from a start URL the crawler may go. 0 = only the start URLs.

Type:integer

Minimum:0

Maximum:1000

Default:5

Crawl scope

crawlScope

Optional

Which links are followed. 'Under the start URL path' (default): https://site.com/docs/ only follows https://site.com/docs/...; start from the homepage to crawl the whole site. 'Same domain': any page on the start URL's host (www. ignored). 'Domain and subdomains': also blog.site.com, docs.site.com... Include URL globs replace this setting.

Type:string

Default:startUrlPath

Options:

startUrlPathsameDomainsameDomainAndSubdomains

Include URL globs

includeUrlGlobs

Optional

Only follow links matching one of these globs (replaces the crawl scope). ** = anything, * = anything except '/'. Example: https://crawlee.dev/python/docs/**

Type:array

Exclude URL globs

excludeUrlGlobs

Optional

Never follow links matching these globs. Example: https://crawlee.dev/**/api/** or /tag/

Type:array

Also use the site's sitemaps

useSitemaps

Optional

Read the sitemaps listed in robots.txt (or /sitemap.xml) and queue their in-scope URLs, so pages that are not linked from the start page are found too.

Type:boolean

Default:false

Respect robots.txt

respectRobotsTxt

Optional

Skip pages that the site's robots.txt disallows and honour its Crawl-delay (capped at 10 s). Turn off only for sites you are allowed to crawl.

Type:boolean

Default:true

Main content extraction

htmlTransformer

Optional

trafilatura (default) keeps the article/documentation text and drops menus, footers and cookie banners; it falls back to readability, then to the whole cleaned page, when it finds too little. 'readability' = Mozilla Readability port. 'none' = the whole page minus navigation, header, footer and banners.

Type:string

Default:trafilatura

Options:

trafilaturareadabilitynone

Save Markdown

saveMarkdown

Optional

Add the main content as Markdown (headings, lists, tables, code blocks) in the 'markdown' field.

Type:boolean

Default:true

Save plain text

saveText

Optional

Add the main content as plain text in the 'text' field.

Type:boolean

Default:true

Save cleaned HTML

saveHtml

Optional

Add the main content as cleaned HTML in the 'html' field.

Type:boolean

Default:false

Keep links in Markdown

keepMarkdownLinks

Optional

Keep hyperlinks as text in the Markdown. Off = link text only (less noise for LLMs).

Type:boolean

Default:false

Delay between requests to one site (ms)

requestDelayMs

Optional

Pause between two requests to the same host. Pages of one site are fetched one at a time.

Type:integer

Minimum:0

Maximum:60000

Default:1000

Max concurrency

maxConcurrency

Optional

How many pages are processed at once across different hosts (one host is always fetched one page at a time).

Type:integer

Minimum:1

Maximum:10

Default:2

Proxy configuration

proxyConfiguration

Optional

Apify Proxy (datacenter by default). Choose RESIDENTIAL for sites that block datacenter IPs.

Type:object

Default:

{
  "useApifyProxy": true
}

Fast Website Content Crawler

6sigmag/fast-website-content-crawler

A high-performance web scraper that rapidly extracts and analyzes content from multiple websites simultaneously. Perfect for competitive research, content aggregation, and website structure analysis.

David

4.7K

4.9

(7)

Website Content Crawler – Markdown for AI, Emails & Contacts

locomint/website-crawler-content-contacts

Website content crawler that turns a whole site into clean Markdown for LLM and RAG pipelines, with the role emails, social profiles and WhatsApp numbers found on each page. Seeded from the sitemap, honours robots.txt, plain HTTP with no browser.

Locomint

Deep URL Content Crawler

6sigmag/deep-url-content-crawler

Scrape Failed Killer! A high-performance web scraper that rapidly extracts and analyzes content from multiple URLs simultaneously. Perfect for competitive research, content aggregation, and website structure analysis.

David

115

Deep Website Content Crawler

6sigmag/deep-website-content-crawler

Scrape Failed Killer! A high-performance web scraper that rapidly extracts and analyzes content from multiple websites simultaneously. Perfect for competitive research, content aggregation, and website structure analysis.

David

1.1K

3.0

(1)

Facebook Reels Scraper

neuro-scraper/facebook-reels-scraper

Facebook Reels Scraper SD - Facebook Reels Scraper is a social media data tool that extracts reels of public Facebook pages with captions, plays, durations, audio, owners and MP4 links, plus optional comments and transcripts - Facebook reels extractor.

Neuro Scraper

5.0

(1)

Website Content Crawler - Clean Text, Markdown & RAG Chunks

topapi/website-content-crawler

Crawl any website or URL list for $1.20 per 1,000 pages: one row per page with clean text, Markdown, HTML, metadata, links, headings, tables and RAG chunks. Plain HTTP first, headless Chrome only for JavaScript pages. Sitemaps, llms.txt, PDF/DOCX text, Bing/DuckDuckGo search, change monitor.

Top Api

Stealth Website Crawler

nocturne/stealth-website-crawler

Crawl websites protected by Cloudflare, DataDome, and other anti-bot systems. Extract clean text or markdown for AI/LLM pipelines. Like Website Content Crawler, but for sites that block you.

Nocturne

5.0

(1)

Website Content Extractor for RAG: Markdown, HTML, Text

nezha/website-content-crawler

Turn docs sites, help centers, blogs, and websites into clean markdown, text, or HTML for RAG, AI knowledge bases, and internal search. Crawl from start URLs or sitemaps and keep the crawl in scope.

nezha

Website Content Crawler Pro

datascoutapi/website-content-crawler-pro

Crawl websites and extract clean, structured content in Markdown, JSON, or plain text for AI models, LLMs, vector DBs, or RAG pipelines. Fast, reliable, and stealthy, with bulk processing, advanced metadata extraction, and seamless integration with LangChain, LlamaIndex, and AI workflows.

halam

575

3.7

(3)

Website Main Content Extractor

sync-network/website-main-content-extractor

Extract the main content from any webpage — clean text without navigation, ads, footers, or boilerplate. Perfect for AI/LLM pipelines, RAG systems, content analysis, and data extraction. Returns title, description, author, publish date, and clean body text.