Pricing
Pay per usage
Crawl websites and extract text content to feed AI models, LLM applications, vector databases, or RAG pipelines. The Actor supports rich formatting using Markdown, cleans the HTML, downloads files, and integrates well with 🦜🔗 LangChain, LlamaIndex, and the wider LLM ecosystem.
All notable changes to this project will be documented in this file.
maxRequestRetries limit to 3 (#419)ignoreHttpsErrors input option (#422)keepElementsCssSelector doesn't affect link enqueuing (#413)SitemapRequestList to avoid excessive timeouts (#415)jpg, png, gif...) (#402)saveHtmlAsFile results (#369)waitForSelector input option (#370)softWaitForSelector (#365)content-type header in FileDownload (#360)title (skip iframes) (#362)latest to Apify with GH actions, add auto changelog bumps (#347)content-type response header (#353)This release only contains documentation changes in readme and input schema.
includeUrlGlobs are now filtered out with a warning log message. To enforce the old behavior (i.e. matching everything), use ** instead.<title> element on the page is prepended to the exported content.startUrls being sitemaps.Input:
removeElementsCssSelector now removes img elements with data: URLs to prevent cluttering the text output.Behaviour:
expandIframes now skips broken iframe elements instead of failing the whole request.keepElementsCssSelector accepts a CSS selector targetting elements to extract from the page to the output.maxCrawlPages limits better.RequestQueue hanging read issue.RequestQueue hanging read issue.maxRequestRetries option.maxRequests (ERR_STREAM_PUSH_AFTER_EOF).useSitemaps option is now pre-filled to true to automatically enable it for new users and in API examples.useSitemaps doesn't block indefinitely on large sitemaps anymore.keepFragmentUrls (URL #fragments identify unique pages) input option to consider fragment URLs as separate pages.<iframe>.html, text and markdown fields are trimmed to the first three million characters each).waitForSelector option allows users to specify a CSS selector to wait for before extracting the page content. This is useful for pages that load content dynamically and break the automatic waiting mechanism.startUrls sanitization.startUrls input.expandIframes (Expand iframe elements) option for extracting content from on-page iframe elements. Available only in playwright:firefox.saveScreenshots option now correctly prints warnings with crawler types that don't support screenshots.Start URLs input.saveHtmlAsFile option is available which enables storing the HTML into a key-value store, replacing their values in the dataset with links to make the dataset value smaller, since there is a hard limit for its size.saveHtml in favor of saveHtmlAsFile.saveHtmlAsFile option saves the URL under a new htmlUrl key in the dataset.saveHtmlAsFile option to get around this problem).RequestQueue race condition fixed.AdaptiveCrawler is the new default (prefill) crawler type.AdaptiveCrawler skips the adaptive scanning to speed up the crawl.metadata.headers with the HTTP response headers of the crawled page.startUrls are now filtered out and don't cause the actor to fail.excludeGlobs and includeGlobs input options, stores filenames, and understands Content-Disposition HTTP headers (i.e. "forced download").og: metatags coverage (article:, movie: etc.).metadata field.useSitemaps toggle for sitemap discovery - leads to more consistent results, scrapes also unreachable webpages.playwright:adaptive crawling modemetadata.openGraph output field for contents of og:* metatags.maxRequestRetries input option for limiting the number of request retries on network, server or parsing errors.maxCrawlDepth > 0), as the memory overflow issue from 0.3.18 is now fixed.metadata.mimeType for downloaded files (only applicable when saveFiles is enabled).maxSessionRotations input option for limiting the number of session rotations when recognized as a bot.401, 403 and 429 HTTP status codes.expandClickableElements utility function.<body> tag when extracting text from HTML.simplifiedBody === null exception.ignoreCanonicalUrl toggle to deduplicate pages based on their actual URLs (useful when two different pages share the same canonical URL).0.3.21.debug mode now stores the results of all the extractors (+ raw HTML) as Key-Value Store objects.debug mode now stores the raw page HTML (without the page preprocessing) under the rawHtml key.proxyConfiguration option (which is now required since 0.3.18). This fixes the actor usage via API, falling back to the default proxy settings if they are not explicitly provided.includeUrlGlobs option to allow explicit control over enqueuing logic (overrides the default scoping logic).requestTimeoutSecs option to allow overriding the default request processing timeout.maxCrawlDepth > 0) as it can lead to memory overflow.debugLog option to enable debug logging.startUrls is now a required input field.<link> tags (rel=next, prev, help, search).crawlerType is now playwright:firefoxsaveMarkdown is trueremoveCookieWarnings is trueexcludeUrlGlobs for skipping certain URLs when enqueueingmaxScrollHeightPixels for scrolling down the page (useful for dynamically loaded pages)maxScrollHeightPixels to 0)crawl.httpStatusCode now contains HTTP response status code.processedHtml under debug objectremoveCookieWarnings option to automatically remove cookie consent modals (via I don't care about cookies browser extension)httpstartUrls as text-fileinitialCookies option for passing cookies to the crawler. Provide the cookies as a JSON array of objects with "name" and "value" keys. Example: [{ "name": "token", "value": "123456" }].textExtractor option is now removed in favour of htmlTransformerunfluff extractor has been completely removedremoveElementsCssSelector option, which now defaults to a larger set of elements, including <nav>, <footer>, <svg>, and elements with role attribute set to one of alert, banner, dialog, alertdialog.htmlTransformer option has been introduced which allows to configure how the simplified HTML is further processed. The output of this is still an HTML, which can be later used to generate markdown or plain text from it.aria-expanded="false" attribute. You can configure this selector via clickElementsCssSelector option.SEC_ERROR_UNKNOWN_ISSUER has been solved by preloading the recognized intermediate TLS certificates into the Docker image..url field now contains the main resource URL without the fragment (#) part.initialConcurrency option visible in the input editor.aggressivePruning option. With this option set to true, the crawler will try to deduplicate the scraped content. This can be useful when the crawler is scraping a website with a lot of duplicate content (header menus, footers, etc.)initialConcurrency. This option sets the initial number of web browsers or HTTP clients running in parallel during the actor run. Increasing this number can speed up the crawling process. Bear in mind this option is hidden and can be changed only by editing the actor input using the JSON editor.maxResults to limit the total number of results. If used with maxCrawlPages, the crawler will stop when either of the limits is reached.saveFiles. This is useful for downloading pdf, docx, xslx... files from the crawled pages. The files are saved to the default key-value store of the run and the links to the files are added to the dataset.textExtractor option readableText. It is generally very accurate and has a good ratio of coverage to noise. It extracts only the main article body (similar to unfluff) but can work for more complex pages.readableTextCharThreshold option. This only applies to readableText extractor. It allows fine-tuning which part of the text should be focused on. That only matters for very complex pages where it is not obvious what should be extracted.Overview that has only url and text for quick output checkwww. are now considered equal to ones without it. This means that the start URL https://apify.com can enqueue https://www.apify.com and vice versa.crawlerType option jsdom for processing with JSDOM. It allows client-side script processing, trying to mimic the browser behavior in Node.js but with much better performance. This is still experimental and may crash on some particular pages.dynamicContentWaitSecs option (defaults to 10s), which is the maximum waiting time for dynamic waiting.crawl.date to crawl.loadedTimecrawl.screenshotUrl to top-level objectmarkdown field was made visiblemetadata.language to metadata.languageCodemetadata.createdAt (for now)metadata.keywordstextExtractor input option to choose how strictly to parse the content. Swapped the previous unfluff for CrawleeHtmlToText as default which in general will extract more text. We chose to output more text rather than less by default.removeElementsCssSelector which allows passing extra CSS selectors to further strip down the HTML before it is converted to text. This can help fine-tuning. By default, the actor removes the page navigation bar, header, and footer.saveMarkdown option is chosendebugMode is set.pageType to the output (only as debug for now), it will be fine-tuned in the future.canonicalUrl. E.g. if more different URLs point to the same canonical URL, they are skipped