Website Content Crawler avatar
Website Content Crawler
Try for free

No credit card required

View all Actors
Website Content Crawler

Website Content Crawler

apify/website-content-crawler
Try for free

No credit card required

Crawl websites and extract text content to feed AI models, LLM applications, vector databases, or RAG pipelines. The Actor supports rich formatting using Markdown, cleans the HTML, downloads files, and integrates well with 🦜🔗LangChain, LlamaIndex, and the wider LLM ecosystem.

0.3.35 (2024-05-23)

  • Behaviour:
    • RequestQueue race condition fixed.
  • Output:
    • The Readable text extractor now correctly handles the article titles.

0.3.34 (2024-05-17)

  • Behaviour:
    • If any of the Start URLs lead to a sitemap file, it is processed and the links are enqueued.
    • Performance / QoL improvements (see Crawlee 3.10.0 changelog for more details)

0.3.33 (2024-04-22)

  • Input:
    • AdaptiveCrawler is the new default (prefill) crawler type.
  • Behaviour:
    • The use of Chrome + Chromium browsers was deprecated. The Actor now uses only Firefox internally for the browser-based crawlers.
    • Reimplementation of the file download feature for better stability and performance.
    • On smaller websites, the AdaptiveCrawler skips the adaptive scanning to speed up the crawl.

0.3.32 (2024-03-28)

  • Output:
    • The Actor now stores metadata.headers with the HTTP response headers of the crawled page.

0.3.31 (2024-03-14)

  • Input:
    • Invalid startUrls are now filtered out and don't cause the actor to fail.

0.3.30 (2024-02-24)

  • Behavior:
    • File download now respects excludeGlobs and includeGlobs input options, stores filenames, and understands Content-Disposition HTTP headers (i.e. "forced download").
  • Output:
    • Better og: metatags coverage (article:, movie: etc.).
    • Stores JSON-LD metatags in the metadata field.

0.3.29 (2024-02-05)

  • Input:
    • useSitemaps toggle for sitemap discovery - leads to more consistent results, scrapes also unreachable webpages.
    • Do not fail on empty globs in input (ignore them instead).
    • Experimental playwright:adaptive crawling mode
  • Output:
    • Add metadata.openGraph output field for contents of og:* metatags.

0.3.27 (2024-01-25)

  • Input:
    • maxRequestRetries input option for limiting the number of request retries on network, server or parsing errors.
  • Behavior:
    • Allow large lists of start URLs with deep crawling (maxCrawlDepth > 0), as the memory overflow issue from 0.3.18 is now fixed.

0.3.26 (2024-01-02)

  • Output:
    • The Actor now stores metadata.mimeType for downloaded files (only applicable when saveFiles is enabled).

0.3.25 (2023-12-21)

  • Input:
    • Add maxSessionRotations input option for limiting the number of session rotations when recognized as a bot.
  • Behavior:
    • Fail on 401, 403 and 429 HTTP status codes.

0.3.24 (2023-12-07)

  • Behavior:
    • Fix a bug within the expandClickableElements utility function.

0.3.23 (2023-12-06)

  • Behavior:
    • Respect empty <body> tag when extracting text from HTML.
    • Fix a bug with the simplifiedBody === null exception.

0.3.22 (2023-12-04)

  • Input:
    • ignoreCanonicalUrl toggle to deduplicate pages based on their actual URLs (useful when two different pages share the same canonical URL).
  • Behavior:
    • Improve the large content detection - this fixes a regression from 0.3.21.

0.3.21 (2023-11-29)

  • Output:
    • The debug mode now stores the results of all the extractors (+ raw HTML) as Key-Value Store objects.
    • New extractor "Readable text with fallback" checks the results of the "Readable text" extractor and checks the content integrity on the fly.
  • Behavior:
    • Skip text-cleaning and markdown processing step on large responses to avoid indefinite hangs.

0.3.20 (2023-11-08)

  • Output:
    • The debug mode now stores the raw page HTML (without the page preprocessing) under the rawHtml key.

0.3.19 (2023-10-18)

  • Input:
    • Add a default for proxyConfiguration option (which is now required since 0.3.18). This fixes the actor usage via API, falling back to the default proxy settings if they are not explicitly provided.

0.3.18 (2023-10-17)

  • Input:
    • Adds includeUrlGlobs option to allow explicit control over enqueuing logic (overrides the default scoping logic).
    • Adds requestTimeoutSecs option to allow overriding the default request processing timeout.
  • Behavior:
    • Disallow using large list of start URLs (more than 100) with deep crawling (maxCrawlDepth > 0) as it can lead to memory overflow.

0.3.17 (2023-10-05)

  • Input:
    • Adds debugLog option to enable debug logging.

0.3.16 (2023-09-06)

  • Behavior:
    • Raw HTTP Client (Cheerio) now works correctly with proxies again.

0.3.15 (2023-08-30)

  • Input:
    • startUrls is now a required input field.
    • Input tooltips now provide more detailed description of crawler types and other input options.

0.3.14 (2023-07-19)

  • Behavior:
    • When using the Cheerio based crawlers, the actor processes links from removed elements correctly now.
    • Crawlers now follow links in <link> tags (rel=next, prev, help, search).
    • Relative canonical URLs are now getting correctly resolved during the deduplication phase.
    • Actor now automatically recognizes blocked websites and retries the crawl with a new proxy / fingerprint combination.

0.3.13 (2023-06-30)

  • Input:
    • The Actor now uses new defaults for input settings for better user experience:
      • default crawlerType is now playwright:firefox
      • saveMarkdown is true
      • removeCookieWarnings is true

0.3.12 (2023-06-14)

  • Input:
    • add excludeUrlGlobs for skipping certain URLs when enqueueing
    • add maxScrollHeightPixels for scrolling down the page (useful for dynamically loaded pages)
  • Behavior:
    • by default, the crawler scrolls down on every page to trigger dynamic loading (disable by setting maxScrollHeightPixels to 0)
    • the crawler now handles HTML processing errors gracefully
    • the actor now stays alive and restarts the crawl on certain known errors (Playwright Assertion Error).
  • Output:
    • crawl.httpStatusCode now contains HTTP response status code.

0.3.10 (2023-06-05)

  • Input:
    • move processedHtml under debug object
    • add removeCookieWarnings option to automatically remove cookie consent modals (via I don't care about cookies browser extension)
  • Behavior:
    • consider redirected start URL for prefix matching
    • make URL deduplication case-insensitive
    • wait at least 3s in playwright to let dynamic content to load
    • retry the start URLs 10 times and regular URLs 5 times (to get around issues with retries on burnt proxies)
    • ignore links not starting with http
    • skip parsing non-html files
    • support startUrls as text-file

0.3.9 (2023-05-18)

  • Input:
    • Updated README and input hints.

0.3.8 (2023-05-17)

  • Input:
    • initialCookies option for passing cookies to the crawler. Provide the cookies as a JSON array of objects with "name" and "value" keys. Example: [{ "name": "token", "value": "123456" }].
  • Behavior:
    • textExtractor option is now removed in favour of htmlTransformer
    • unfluff extractor has been completely removed
    • HTML is always simplified by removing some elements from it, those are configurable via removeElementsCssSelector option, which now defaults to a larger set of elements, including <nav>, <footer>, <svg>, and elements with role attribute set to one of alert, banner, dialog, alertdialog.
    • New htmlTransformer option has been introduced which allows to configure how the simplified HTML is further processed. The output of this is still an HTML, which can be later used to generate markdown or plain text from it.
    • The crawler will now try to expand collapsible sections automatically. This works by clicking on elements with aria-expanded="false" attribute. You can configure this selector via clickElementsCssSelector option.
    • When using Playwright based crawlers, the dynamic content is awaited for based on network activity, rather than webpage changes. This should improve the reliability of the crawler.
    • Firefox SEC_ERROR_UNKNOWN_ISSUER has been solved by preloading the recognized intermediate TLS certificates into the Docker image.
    • Crawled URLs are retried in case their processing timeouts.

0.3.7 (2023-05-10)

  • Behavior:
    • URLs with redirects are now enqueued based on the original (unredirected) URL. This should prevent the actor from skipping relevant pages which are hidden behind redirects.
    • The actor now considers all start URLs when enqueueing new links. This way, the user can specify multiple start URLs as a workaround for the actor skipping some relevant pages on the website.
    • Error with not enqueueing URLs with certain query parameters is now fixed.
  • Output:
    • The .url field now contains the main resource URL without the fragment (#) part.

0.3.6 (2023-05-04)

  • Input:
    • Made the initialConcurrency option visible in the input editor.
    • Added aggressivePruning option. With this option set to true, the crawler will try to deduplicate the scraped content. This can be useful when the crawler is scraping a website with a lot of duplicate content (header menus, footers, etc.)
  • Behavior:
    • The actor now stays alive and restarts the crawl on certain known errors (Playwright Assertion Error).

0.3.4 (2023-05-04)

  • Input:
    • Added a new hidden option initialConcurrency. This option sets the initial number of web browsers or HTTP clients running in parallel during the actor run. Increasing this number can speed up the crawling process. Bear in mind this option is hidden and can be changed only by editing the actor input using the JSON editor.

0.3.3 (2023-04-28)

  • Input:
    • Added a new option maxResults to limit the total number of results. If used with maxCrawlPages, the crawler will stop when either of the limits is reached.

0.3.1 (2023-04-24)

  • Input:
    • Added an option to download linked document files from the page - saveFiles. This is useful for downloading pdf, docx, xslx... files from the crawled pages. The files are saved to the default key-value store of the run and the links to the files are added to the dataset.
    • Added a new crawler - Stealthy web browser - that uses a Firefox browser with a stealthy profile. It is useful for crawling websites that block scraping.

0.0.13 (2023-04-18)

  • Input:
    • Added new textExtractor option readableText. It is generally very accurate and has a good ratio of coverage to noise. It extracts only the main article body (similar to unfluff) but can work for more complex pages.
    • Added readableTextCharThreshold option. This only applies to readableText extractor. It allows fine-tuning which part of the text should be focused on. That only matters for very complex pages where it is not obvious what should be extracted.
  • Output:
    • Added simplified output view Overview that has only url and text for quick output check
  • Behavior:
    • Domains starting with www. are now considered equal to ones without it. This means that the start URL https://apify.com can enqueue https://www.apify.com and vice versa.

0.0.10 (2023-04-05)

  • Input:
    • Added new crawlerType option jsdom for processing with JSDOM. It allows client-side script processing, trying to mimic the browser behavior in Node.js but with much better performance. This is still experimental and may crash on some particular pages.
    • Added dynamicContentWaitSecs option (defaults to 10s), which is the maximum waiting time for dynamic waiting.
  • Output (BREAKING CHANGE):
    • Renamed crawl.date to crawl.loadedTime
    • Moved crawl.screenshotUrl to top-level object
    • The markdown field was made visible
    • Renamed metadata.language to metadata.languageCode
    • Removed metadata.createdAt (for now)
    • Added metadata.keywords
  • Behavior:
    • Added waiting for dynamically rendered content (supported in Headless browser and JSDOM crawlers). The crawler checks every half a second for content changes. When there are no changes for 2 seconds, the crawler proceeds to extraction.

0.0.7 (2023-03-30)

  • Input:
    • BREAKING CHANGE: Added textExtractor input option to choose how strictly to parse the content. Swapped the previous unfluff for CrawleeHtmlToText as default which in general will extract more text. We chose to output more text rather than less by default.
    • Added removeElementsCssSelector which allows passing extra CSS selectors to further strip down the HTML before it is converted to text. This can help fine-tuning. By default, the actor removes the page navigation bar, header, and footer.
  • Output:
    • Added markdown to output if saveMarkdown option is chosen
    • All extractor outputs + HTML as a link can be obtained if debugMode is set.
    • Added pageType to the output (only as debug for now), it will be fine-tuned in the future.
  • Behavior:
    • Added deduplication by canonicalUrl. E.g. if more different URLs point to the same canonical URL, they are skipped
    • Skip pages that redirect outside the original start URLs domain.
    • Only run a single text extractor unless in debug mode. This improves performance.
Developer
Maintained by Apify
Actor metrics
  • 2.2k monthly users
  • 0 stars
  • 99.8% runs succeeded
  • 2.9 days response time
  • Created in Mar 2023
  • Modified 3 days ago