Website Content Crawler
Try for free
No credit card required
View all Actors
Website Content Crawler
apify/website-content-crawler
Try for free
No credit card required
Automatically crawl and extract text content from websites with documentation, knowledge bases, help centers, or blogs. This Actor is designed to provide data to feed, fine-tune, or train large language models such as ChatGPT or LLaMA.
0.3.33 (2024-04-22)
- Input:
AdaptiveCrawleris the new default (prefill) crawler type.
- Behaviour:
- The use of Chrome + Chromium browsers was deprecated. The Actor now uses only Firefox internally for the browser-based crawlers.
- Reimplementation of the file download feature for better stability and performance.
- On smaller websites, the
AdaptiveCrawlerskips the adaptive scanning to speed up the crawl.
0.3.32 (2024-03-28)
- Output:
- The Actor now stores
metadata.headerswith the HTTP response headers of the crawled page.
- The Actor now stores
0.3.31 (2024-03-14)
- Input:
- Invalid
startUrlsare now filtered out and don't cause the actor to fail.
- Invalid
0.3.30 (2024-02-24)
- Behavior:
- File download now respects
excludeGlobsandincludeGlobsinput options, stores filenames, and understandsContent-DispositionHTTP headers (i.e. "forced download").
- File download now respects
- Output:
- Better
og:metatags coverage (article:,movie:etc.). - Stores JSON-LD metatags in the
metadatafield.
- Better
0.3.29 (2024-02-05)
- Input:
useSitemapstoggle for sitemap discovery - leads to more consistent results, scrapes also unreachable webpages.- Do not fail on empty globs in input (ignore them instead).
- Experimental
playwright:adaptivecrawling mode
- Output:
- Add
metadata.openGraphoutput field for contents ofog:*metatags.
- Add
0.3.27 (2024-01-25)
- Input:
maxRequestRetriesinput option for limiting the number of request retries on network, server or parsing errors.
- Behavior:
- Allow large lists of start URLs with deep crawling (
maxCrawlDepth > 0), as the memory overflow issue from0.3.18is now fixed.
- Allow large lists of start URLs with deep crawling (
0.3.26 (2024-01-02)
- Output:
- The Actor now stores
metadata.mimeTypefor downloaded files (only applicable whensaveFilesis enabled).
- The Actor now stores
0.3.25 (2023-12-21)
- Input:
- Add
maxSessionRotationsinput option for limiting the number of session rotations when recognized as a bot.
- Add
- Behavior:
- Fail on
401,403and429HTTP status codes.
- Fail on
0.3.24 (2023-12-07)
- Behavior:
- Fix a bug within the
expandClickableElementsutility function.
- Fix a bug within the
0.3.23 (2023-12-06)
- Behavior:
- Respect empty
<body>tag when extracting text from HTML. - Fix a bug with the
simplifiedBody === nullexception.
- Respect empty
0.3.22 (2023-12-04)
- Input:
ignoreCanonicalUrltoggle to deduplicate pages based on their actual URLs (useful when two different pages share the same canonical URL).
- Behavior:
- Improve the large content detection - this fixes a regression from
0.3.21.
- Improve the large content detection - this fixes a regression from
0.3.21 (2023-11-29)
- Output:
- The
debugmode now stores the results of all the extractors (+ raw HTML) as Key-Value Store objects. - New extractor "Readable text with fallback" checks the results of the "Readable text" extractor and checks the content integrity on the fly.
- The
- Behavior:
- Skip text-cleaning and markdown processing step on large responses to avoid indefinite hangs.
0.3.20 (2023-11-08)
- Output:
- The
debugmode now stores the raw page HTML (without the page preprocessing) under therawHtmlkey.
- The
0.3.19 (2023-10-18)
- Input:
- Add a default for
proxyConfigurationoption (which is now required since 0.3.18). This fixes the actor usage via API, falling back to the default proxy settings if they are not explicitly provided.
- Add a default for
0.3.18 (2023-10-17)
- Input:
- Adds
includeUrlGlobsoption to allow explicit control over enqueuing logic (overrides the default scoping logic). - Adds
requestTimeoutSecsoption to allow overriding the default request processing timeout.
- Adds
- Behavior:
- Disallow using large list of start URLs (more than 100) with deep crawling (
maxCrawlDepth > 0) as it can lead to memory overflow.
- Disallow using large list of start URLs (more than 100) with deep crawling (
0.3.17 (2023-10-05)
- Input:
- Adds
debugLogoption to enable debug logging.
- Adds
0.3.16 (2023-09-06)
- Behavior:
- Raw HTTP Client (Cheerio) now works correctly with proxies again.
0.3.15 (2023-08-30)
- Input:
startUrlsis now a required input field.- Input tooltips now provide more detailed description of crawler types and other input options.
0.3.14 (2023-07-19)
- Behavior:
- When using the Cheerio based crawlers, the actor processes links from removed elements correctly now.
- Crawlers now follow links in
<link>tags (rel=next, prev, help, search). - Relative canonical URLs are now getting correctly resolved during the deduplication phase.
- Actor now automatically recognizes blocked websites and retries the crawl with a new proxy / fingerprint combination.
0.3.13 (2023-06-30)
- Input:
- The Actor now uses new defaults for input settings for better user experience:
- default
crawlerTypeis nowplaywright:firefox saveMarkdownistrueremoveCookieWarningsistrue
- default
- The Actor now uses new defaults for input settings for better user experience:
0.3.12 (2023-06-14)
- Input:
- add
excludeUrlGlobsfor skipping certain URLs when enqueueing - add
maxScrollHeightPixelsfor scrolling down the page (useful for dynamically loaded pages)
- add
- Behavior:
- by default, the crawler scrolls down on every page to trigger dynamic loading (disable by setting
maxScrollHeightPixelsto 0) - the crawler now handles HTML processing errors gracefully
- the actor now stays alive and restarts the crawl on certain known errors (Playwright Assertion Error).
- by default, the crawler scrolls down on every page to trigger dynamic loading (disable by setting
- Output:
crawl.httpStatusCodenow contains HTTP response status code.
0.3.10 (2023-06-05)
- Input:
- move
processedHtmlunderdebugobject - add
removeCookieWarningsoption to automatically remove cookie consent modals (via I don't care about cookies browser extension)
- move
- Behavior:
- consider redirected start URL for prefix matching
- make URL deduplication case-insensitive
- wait at least 3s in playwright to let dynamic content to load
- retry the start URLs 10 times and regular URLs 5 times (to get around issues with retries on burnt proxies)
- ignore links not starting with
http - skip parsing non-html files
- support
startUrlsas text-file
0.3.9 (2023-05-18)
- Input:
- Updated README and input hints.
0.3.8 (2023-05-17)
- Input:
initialCookiesoption for passing cookies to the crawler. Provide the cookies as a JSON array of objects with"name"and"value"keys. Example:[{ "name": "token", "value": "123456" }].
- Behavior:
textExtractoroption is now removed in favour ofhtmlTransformerunfluffextractor has been completely removed- HTML is always simplified by removing some elements from it, those are configurable via
removeElementsCssSelectoroption, which now defaults to a larger set of elements, including<nav>,<footer>,<svg>, and elements withroleattribute set to one ofalert,banner,dialog,alertdialog. - New
htmlTransformeroption has been introduced which allows to configure how the simplified HTML is further processed. The output of this is still an HTML, which can be later used to generate markdown or plain text from it. - The crawler will now try to expand collapsible sections automatically. This works by clicking on elements with
aria-expanded="false"attribute. You can configure this selector viaclickElementsCssSelectoroption. - When using Playwright based crawlers, the dynamic content is awaited for based on network activity, rather than webpage changes. This should improve the reliability of the crawler.
- Firefox
SEC_ERROR_UNKNOWN_ISSUERhas been solved by preloading the recognized intermediate TLS certificates into the Docker image. - Crawled URLs are retried in case their processing timeouts.
0.3.7 (2023-05-10)
- Behavior:
- URLs with redirects are now enqueued based on the original (unredirected) URL. This should prevent the actor from skipping relevant pages which are hidden behind redirects.
- The actor now considers all start URLs when enqueueing new links. This way, the user can specify multiple start URLs as a workaround for the actor skipping some relevant pages on the website.
- Error with not enqueueing URLs with certain query parameters is now fixed.
- Output:
- The
.urlfield now contains the main resource URL without the fragment (#) part.
- The
0.3.6 (2023-05-04)
- Input:
- Made the
initialConcurrencyoption visible in the input editor. - Added
aggressivePruningoption. With this option set totrue, the crawler will try to deduplicate the scraped content. This can be useful when the crawler is scraping a website with a lot of duplicate content (header menus, footers, etc.)
- Made the
- Behavior:
- The actor now stays alive and restarts the crawl on certain known errors (Playwright Assertion Error).
0.3.4 (2023-05-04)
- Input:
- Added a new hidden option
initialConcurrency. This option sets the initial number of web browsers or HTTP clients running in parallel during the actor run. Increasing this number can speed up the crawling process. Bear in mind this option is hidden and can be changed only by editing the actor input using the JSON editor.
- Added a new hidden option
0.3.3 (2023-04-28)
- Input:
- Added a new option
maxResultsto limit the total number of results. If used withmaxCrawlPages, the crawler will stop when either of the limits is reached.
- Added a new option
0.3.1 (2023-04-24)
- Input:
- Added an option to download linked document files from the page -
saveFiles. This is useful for downloading pdf, docx, xslx... files from the crawled pages. The files are saved to the default key-value store of the run and the links to the files are added to the dataset. - Added a new crawler - Stealthy web browser - that uses a Firefox browser with a stealthy profile. It is useful for crawling websites that block scraping.
- Added an option to download linked document files from the page -
0.0.13 (2023-04-18)
- Input:
- Added new
textExtractoroptionreadableText. It is generally very accurate and has a good ratio of coverage to noise. It extracts only the main article body (similar tounfluff) but can work for more complex pages. - Added
readableTextCharThresholdoption. This only applies toreadableTextextractor. It allows fine-tuning which part of the text should be focused on. That only matters for very complex pages where it is not obvious what should be extracted.
- Added new
- Output:
- Added simplified output view
Overviewthat has onlyurlandtextfor quick output check
- Added simplified output view
- Behavior:
- Domains starting with
www.are now considered equal to ones without it. This means that the start URLhttps://apify.comcan enqueuehttps://www.apify.comand vice versa.
- Domains starting with
0.0.10 (2023-04-05)
- Input:
- Added new
crawlerTypeoptionjsdomfor processing with JSDOM. It allows client-side script processing, trying to mimic the browser behavior in Node.js but with much better performance. This is still experimental and may crash on some particular pages. - Added
dynamicContentWaitSecsoption (defaults to 10s), which is the maximum waiting time for dynamic waiting.
- Added new
- Output (BREAKING CHANGE):
- Renamed
crawl.datetocrawl.loadedTime - Moved
crawl.screenshotUrlto top-level object - The
markdownfield was made visible - Renamed
metadata.languagetometadata.languageCode - Removed
metadata.createdAt(for now) - Added
metadata.keywords
- Renamed
- Behavior:
- Added waiting for dynamically rendered content (supported in Headless browser and JSDOM crawlers). The crawler checks every half a second for content changes. When there are no changes for 2 seconds, the crawler proceeds to extraction.
0.0.7 (2023-03-30)
- Input:
- BREAKING CHANGE: Added
textExtractorinput option to choose how strictly to parse the content. Swapped the previousunfluffforCrawleeHtmlToTextas default which in general will extract more text. We chose to output more text rather than less by default. - Added
removeElementsCssSelectorwhich allows passing extra CSS selectors to further strip down the HTML before it is converted to text. This can help fine-tuning. By default, the actor removes the page navigation bar, header, and footer.
- BREAKING CHANGE: Added
- Output:
- Added markdown to output if
saveMarkdownoption is chosen - All extractor outputs + HTML as a link can be obtained if
debugModeis set. - Added
pageTypeto the output (only asdebugfor now), it will be fine-tuned in the future.
- Added markdown to output if
- Behavior:
- Added deduplication by
canonicalUrl. E.g. if more different URLs point to the same canonical URL, they are skipped - Skip pages that redirect outside the original start URLs domain.
- Only run a single text extractor unless in debug mode. This improves performance.
- Added deduplication by
Developer
Apify Maintained by Apify
Actor metrics
- 2k monthly users
- 99.7% runs succeeded
- 2.9 days response time
- Created in Mar 2023
- Modified 1 day ago
Categories
