Pricing
Pay per usage
Crawl websites and extract text content to feed AI models, LLM applications, vector databases, or RAG pipelines. The Actor supports rich formatting using Markdown, cleans the HTML, downloads files, and integrates well with 🦜🔗 LangChain, LlamaIndex, and the wider LLM ecosystem.
One or more URLs of pages where the crawler will start.
By default, the Actor will also crawl sub-pages of these URLs. For example, for start URL https://example.com/blog, it will crawl also https://example.com/blog/post or https://example.com/blog/article. The Include URLs (globs) option overrides this automation behavior.
If enabled, the crawler will look for Sitemaps at the domains of the provided Start URLs and enqueue matching URLs similarly as the links found on crawled pages. You can also reference a sitemap.xml file directly by adding it as another Start URL (e.g. https://www.example.com/sitemap.xml)
This feature makes the crawling more robust on websites that support Sitemaps, as it includes pages that might be not reachable from Start URLs. However, loading and processing Sitemaps can take a lot of time, especially for large sites. Note that if a page is found via Sitemaps, it will have depth of 1.
If enabled, the crawler will look for /llms.txt files at the root of the domains of the provided Start URLs (e.g., https://example.com/llms.txt) and enqueue them for crawling. Note that this also enables crawling other Markdown files and enqueueing links from them.
If enabled, the crawler will consult the robots.txt file for the target website before crawling each page. At the moment, the crawler does not use any specific user agent identifier. The crawl-delay directive is also not supported yet.
Select the crawling engine:
More details about Crawler types are in readme.
Glob patterns matching URLs of pages that will be included in crawling.
Setting this option will disable the default Start URLs based scoping and will allow you to customize the crawling scope yourself. Note that this affects only links found on pages, but not Start URLs - if you want to crawl a page, make sure to specify its URL in the Start URLs field.
For example https://{store,docs}.example.com/** lets the crawler to access all URLs starting with https://store.example.com/ or https://docs.example.com/, and https://example.com/**/*\?*foo=* allows the crawler to access all URLs that contain foo query parameter with any value.
Learn more about globs and test them here.
[]Glob patterns matching URLs of pages that will be excluded from crawling. Note that this affects only links found on pages, but not Start URLs, which are always crawled.
For example https://{store,docs}.example.com/** excludes all URLs starting with https://store.example.com/ or https://docs.example.com/, and https://example.com/**/*\?*foo=* excludes all URLs that contain foo query parameter with any value.
Learn more about globs and test them here.
[]Indicates that URL fragments (e.g. http://example.com#fragment) should be included when checking whether a URL has already been visited or not. Typically, URL fragments are used for page navigation only and therefore they should be ignored, as they don't identify separate pages. However, some single-page websites use URL fragments to display different pages; in such a case, this option should be enabled.
If enabled, the Actor will ignore the canonical URL reported by the page, and use the actual URL instead. You can use this feature for websites that report invalid canonical URLs, which causes the Actor to skip those pages in results.
If enabled, the scraper will ignore HTTPS certificate errors. Use at your own risk.
The maximum number of links starting from the start URL that the crawler will recursively follow. The start URLs have depth 0, the pages linked directly from the start URLs have depth 1, and so on.
This setting is useful to prevent accidental crawler runaway. By setting it to 0, the Actor will only crawl the Start URLs.
The maximum number pages to crawl. It includes the start URLs, pagination pages, pages with no content, etc. The crawler will automatically finish after reaching this number. This setting is useful to prevent accidental crawler runaway.
The initial number of web browsers or HTTP clients running in parallel. The system scales the concurrency up and down based on the current CPU and memory load. If the value is set to 0 (default), the Actor uses the default setting for the specific crawler type.
Note that if you set this value too high, the Actor will run out of memory and crash. If too low, it will be slow at start before it scales the concurrency up.
The maximum number of web browsers or HTTP clients running in parallel. This setting is useful to avoid overloading the target websites and to avoid getting blocked.
Cookies that will be pre-set to all pages the scraper opens. This is useful for pages that require login. The value is expected to be a JSON array of objects with name and value properties. For example:
You can use the EditThisCookie browser extension to copy browser cookies in this format, and paste it here.
Note that the value is secret and encrypted to protect your login cookies.
HTTP headers that will be added to all requests made by the crawler. This is useful for setting custom authentication headers or other headers required by the target website. The value is expected to be a JSON object with name and value properties. For example: { "name": "Authorization", "value": "Basic a1b2c3d4..." }.
{}If enabled, the crawler will sign all HTTP requests using its Web Bot Auth private key. This is necessary if you want to use Website Content Crawler as a Cloudflare Signed Agent.
A declaration of an asynchronous JS function (e.g. async function pageFunction({ page }) { await page.click('.submit-button') }).
The function receives context as the only argument. Context is a JavaScript object containing the following properties:
page: Currently loaded Playwright Page instance.request: The request object that triggered the page load.The function will be executed in the browser context for each crawled page, after the page is loaded (included all dynamic content) and before the content is extracted and cleaned.
Enables loading the websites from IP addresses in specific geographies and to circumvent blocking.
{ "useApifyProxy": true}The maximum number of times the crawler will rotate the session (IP address + browser configuration) on anti-scraping measures like CAPTCHAs. If the crawler rotates the session more than this number and the page is still blocked, it will finish with an error.
The maximum number of times the crawler will retry the request on network, proxy or server errors. If the (n+1)-th request still fails, the crawler will mark this request as failed.
Timeout in seconds for making the request and processing its response. Defaults to 60s.
The minimum viable file download speed in kilobytes per seconds. If the file download speed is lower than this value for a prolonged duration, the crawler will consider the file download as failing, abort it, and retry it again (up to "Maximum number of retries" times). This is useful to avoid your crawls being stuck on slow file downloads.
The maximum time in seconds to wait for dynamic page content to load. By default, it is 10 seconds. The crawler will continue processing the page either if this time elapses, or if it detects the network became idle as there are no more requests for additional resources.
When using the Wait for selector option, the crawler will wait for the selector to appear for this amount of time. If the selector doesn't appear within this period, the request will fail and will be retried.
Note that this setting is ignored for the raw HTTP client, because it doesn't execute JavaScript or loads any dynamic resources. Similarly, if the value is set to 0, the crawler doesn't wait for any dynamic to load and processes the HTML as provided on load.
If set, the crawler will wait for the specified CSS selector to appear in the page before proceeding with the content extraction. This is useful for pages for which the default content load recognition by idle network fails. Setting this option completely disables the default behavior, and the page will be processed only if the element specified by this selector appears. If the element doesn't appear within the Wait for dynamic content timeout, the request will fail and will be retried later. The value must be a valid CSS selector as accepted by the document.querySelectorAll() function.
With the raw HTTP client, this option checks for the presence of the selector in the HTML content and throws an error if it's not found.
If set, the crawler will wait for the specified CSS selector to appear in the page before proceeding with the content extraction. Unlike the waitForSelector option, this option doesn't fail the request if the selector doesn't appear within the timeout (the request processing will continue).
The crawler will scroll down the page until all content is loaded (and network becomes idle), or until this maximum scrolling height is reached. Setting this value to 0 disables scrolling altogether.
Note that this setting is ignored for the raw HTTP client, because it doesn't execute JavaScript or loads any dynamic resources.
An optional CSS selector matching HTML elements that should be preserved in the DOM. If provided, all HTML elements which are not matching the CSS selectors or their descendants are removed from the DOM. This is useful to extract only relevant page content. The value must be a valid CSS selector as accepted by the document.querySelectorAll() function.
This option runs before the HTML transformer option. If you are missing content in the output despite using this option, try disabling the HTML transformer.
A CSS selector matching HTML elements that will be removed from the DOM, before converting it to text, Markdown, or saving as HTML. This is useful to skip irrelevant page content. The value must be a valid CSS selector as accepted by the document.querySelectorAll() function.
By default, the Actor removes common navigation elements, headers, footers, modals, scripts, and inline image. You can disable the removal by setting this value to some non-existent CSS selector like dummy_keep_everything.
If enabled, the Actor will try to remove cookies consent dialogs or modals, using the I don't care about cookies browser extension, to improve the accuracy of the extracted text. Note that there is a small performance penalty if this feature is enabled.
This setting is ignored when using the raw HTTP crawler type.
If the flag is enabled and the Actor is using a headless browser, it will not load images, fonts, stylesheets and videos to improve performance. It will load scripts as usual - that is after all the point of using a headless browser.
By default, the Actor will extract content from iframe elements. If you want to specifically skip iframe processing, disable this option. Works only for the playwright:firefox crawler type.
A CSS selector matching DOM elements that will be clicked. This is useful for expanding collapsed sections, in order to capture their text content. The value must be a valid CSS selector as accepted by the document.querySelectorAll() function.
This is an experimental feature. A CSS selector matching DOM elements that will be prevented from deleting any of their children. This is useful in conjunction with the "Expand clickable elements" option on pages where hidden content is actually removed from the DOM (i.e., some variants of the accordion pattern). Enabling this might corrupt the extracted content, which is why it is disabled by default. It is possible to enable the feature for the whole page with the * selector, or you can target specific elements if the former has unwanted side effects.
Specify how to transform the HTML to extract meaningful content without any extra fluff, like navigation or modals. The HTML transformation happens after removing and clicking the DOM elements.
Readable text with fallback - Uses Mozilla's Readability to extract the main content but falls back to the original HTML if the page doesn't appear to be an article. This is useful for websites with mixed content types (articles, product pages, etc.) as it preserves more content on non-article pages.
Readable text (default) - Uses Mozilla's Readability to extract the main article content, removing navigation, headers, footers, and other non-essential elements. Works best for article-rich websites and blogs.
Extractus - Uses the Extractus article extraction library, which is an alternative content extraction algorithm. May work better than Readability on certain websites, particularly news sites or blogs with specific layouts.
Defuddle - Uses the Defuddle library. Defuddle is more forgiving than Readability, often preserving more uncertain elements and providing consistent output for footnotes, math, and code blocks. It also uses mobile styles to help identify unnecessary elements and extracts additional metadata such as schema.org data.
None - Only applies basic cleaning (removing elements specified via 'Remove HTML elements' option) without any content extraction algorithm. Best when you want to preserve most of the original HTML structure with minimal processing.
You can examine output of all transformers by enabling the debug mode.
A configuration options for the "Readable text" HTML transformer. It contains the minimum number of characters an article must have in order to be considered relevant.
This is an experimental feature. If enabled, the crawler will prune content lines that are very similar to the ones already crawled on other pages, using the Count-Min Sketch algorithm. This is useful to strip repeating content in the scraped data like menus, headers, footers, etc. In some (not very likely) cases, it might remove relevant content from some pages.
If enabled, the Actor will store the output of all types of HTML transformers, including the ones that are not used by default, and it will also store the HTML to Key-value Store with a link. All this data is stored under the debug field in the resulting Dataset.
If enabled, the crawler will store all URLs that were skipped during the crawl in a Key-Value Store record named SKIPPED_URLS. The record will contain a JSON object with reasons for skipping and the URLs that were skipped for each reason. This is useful for debugging and understanding why certain pages were not crawled.
If enabled, the actor log will include debug messages. Beware that this can be quite verbose.
If enabled, the crawler stores full transformed HTML of all pages found to the output dataset under the html field. This option has been deprecated in favor of the saveHtmlAsFile option, because the dataset records have a size of approximately 10MB and it's harder to review the HTML for debugging.
If enabled, the crawler stores full transformed HTML of all pages found to the default key-value store and saves links to the files as htmlUrl field in the output dataset. Storing HTML in key-value store is preferred to storing it into the dataset with the saveHtml option, because there's no size limit and it's easier for debugging as you can easily view the HTML.
If enabled, the crawler converts the transformed HTML of all pages found to Markdown, and stores it under the markdown field in the output dataset.
Deprecated in favor of the saveContentTypes option. Will be removed soon. If enabled, the crawler downloads files linked from the web pages, as long as their URL has one of the following file extensions: PDF, DOC, DOCX, XLS, XLSX, and CSV. Note that unlike web pages, the files are downloaded regardless if they are under Start URLs or not. The files are stored to the default key-value store, and metadata about them to the output dataset, similarly as for web pages.
The crawler downloads files linked from the web pages, as long as their content type matches the provided value. Select predefined Content-type groups to download common file types, or enter custom HTTP Content-type strings, including wildcards (e.g., application/pdf, text/*, image/*) for specific downloads. Note that unlike web pages, the files are downloaded regardless if they are under Start URLs or not. The files are stored to the default key-value store, and metadata about them to the output dataset, similarly as for web pages.
If enabled, the crawler stores a screenshot for each article page to the default key-value store. The link to the screenshot is stored under the screenshotUrl field in the output dataset. It is useful for debugging, but reduces performance and increases storage costs.
Note that this feature only works with the playwright:firefox crawler type.
The maximum number of web pages and files to store. This setting helps prevent an accidental crawler runaway by automatically stopping the crawl once this limit is reached. Note that the crawler skips pages whose canonical URL matches a page that has already been crawled, so it may crawl more pages than the number of stored results. Similarly, there may be more stored results than crawled web pages because downloaded files also count toward results.
Removed in favor of the htmlTransformer option. Will be removed soon.
The least amount of content (as a percentage) change after the initial load required to consider the pages client-side rendered