Website Content Crawler
Pricing
from $1.00 / 1,000 results
Crawls websites and extracts clean text, markdown, or HTML content. Ideal for LLM training data, RAG pipelines, and knowledge base building.
Pricing
from $1.00 / 1,000 results
Rating
0.0
(0)
Developer
Crawler Bros
Maintained by CommunityActor stats
4
Bookmarked
46
Total users
12
Monthly active users
3 months ago
Last modified
Categories
Share
One or more URLs to start crawling from. The crawler will follow links from these pages.
Select the browser engine for crawling. Chromium handles most sites well. Firefox may work better on some sites. HTTP mode is fastest but only works with static (server-rendered) pages.
Maximum number of links to follow from the start URL. 0 means only the start URLs will be crawled.
Maximum total number of pages to crawl. The crawler stops after reaching this limit.
Maximum number of pages to load in parallel. Higher values are faster but use more memory.
Only crawl URLs matching these glob patterns. Leave empty to crawl all URLs on the same domain.
[]Skip URLs matching these glob patterns. Useful for excluding login pages, media files, etc.
[]Format of the extracted content. Markdown is ideal for LLMs and RAG. Text gives plain text. HTML preserves the original markup.