Website Content Crawler
Pricing
from $0.00005 / actor start
Deep-crawl websites to extract clean text, Markdown, or HTML for AI/LLM apps, RAG pipelines, and vector databases. Supports adaptive crawling, HTML cleaning, file downloads, and structured dataset output. Easily integrates with LangChain, LlamaIndex, and other LLM tools.
One or more URLs of pages where the crawler will start. The Actor crawls sub-pages of these URLs by default.
How pages are fetched. Adaptive auto-switches between raw HTTP and Firefox. Cheerio is fastest. Firefox/Chrome handle JS-rendered sites.
Glob patterns for URLs to crawl. If set, overrides the default start-URL-based scoping. Affects links only, not Start URLs.
[]Glob patterns for URLs to skip. Affects links only, not Start URLs.
[]Maximum link-following depth from start URLs. 0 = start URLs only.
Maximum total pages to crawl, including start URLs. 0 = unlimited.
Maximum number of results (items) to save to the dataset. Unlike maxCrawlPages, this only counts pages with saved content.
Maximum number of parallel page loads.
Initial number of parallel page loads at crawl start. Gradually increases up to maxConcurrency. Useful to warm up without overloading the target.
Maximum number of retry attempts for failed requests.
If enabled, fetches sitemap.xml at each domain's root and enqueues matching URLs.
If enabled, the crawler looks for /llms.txt at domain roots and crawls those URLs. Also enables crawling .md files.
Obey robots.txt exclusion rules.
Include URL fragments when checking for already-visited pages. Useful for single-page apps.
If enabled, ignores the canonical URL reported by the page and uses the actual URL for deduplication.
Wait this many seconds after page load for dynamic content to appear. Only applies to browser crawlers.
How much of the page to scroll before extraction. 100 = scroll through the full page to the last section, 0 disables scrolling. Ignored for raw HTTP crawlers.
Wait for this CSS selector to appear before extracting content. Only for browser crawlers.
Click DOM elements matching this selector to expand collapsed content (e.g. [aria-expanded='false']). Browser only.
Include text content from iframes in the output. Browser crawler only.
Controls how the page HTML is cleaned before text extraction.
Minimum character count for readability extraction to be considered successful. Falls back to full-page extraction if below threshold.
Apply more aggressive HTML pruning to remove low-value content. May remove some relevant content.
CSS selector for elements to remove before extraction. Defaults to nav, header, footer, ads, scripts, etc.
If set, only content matching this CSS selector is kept; everything else is removed.
Attempt to remove cookie consent banners.
Convert page content to Markdown and include in output.
Include the cleaned HTML snippet in the output record.
Save full raw page HTML to the Key-Value Store and add a link in the output record.
Capture a screenshot of each page (Playwright crawlers only) and store in the Key-Value Store.
Download PDF, DOC, DOCX, XLS, XLSX, CSV files linked from crawled pages.
Abort file downloads that stay below this speed for more than 10 seconds.
Configure proxy for crawling. Set useApifyProxy: true to use Apify Proxy.
{ "useApifyProxy": true}Cookies injected on every request (for pages behind login). JSON array of {name, value, domain?, path?}.
[]Extra HTTP headers added to all requests, e.g. {"Authorization": "Bearer token"}.
{}Include extra debug fields in output (raw HTML, response headers).