Website Content Crawler — AI & RAG Ready
Pricing
Pay per event
Crawl any website and extract clean Markdown and plain text optimized for AI ingestion, RAG pipelines, and LLM context. Readability-style main content extraction removes ads, navs, and footers. Configurable depth, concurrency, and page limits. Pay-per-page.
List of URLs to crawl. Set Max Depth to 0 to fetch only these pages (no link-following). Set to 1+ to also crawl linked pages.
How many link levels deep to crawl from each start URL. 0 = start pages only (no crawling), 1 = start pages + their links, 2 = two levels deep, etc.
Maximum total pages to process across all start URLs combined. Controls your budget.
Maximum pages to crawl per unique domain. Prevents one large site from consuming the entire budget.
Number of parallel workers fetching pages simultaneously. Higher values increase throughput.
When enabled (default), strips navigation, footers, sidebars, and ads using readability-style selectors. Produces cleaner AI/RAG content. Disable to get full page HTML converted to Markdown.
