Smart Article Extractor
Pricing
Pay per usage
📰 Smart Article Extractor extracts articles from any scientific, academic, or news website with just one click. The extractor crawls the whole website and automatically distinguishes articles from other web pages. Download your data as HTML table, JSON, Excel, RSS feed, and more.
Features
navigationWaitUntil input option for browser to allow faster or slower loading depending on the use-caseFeatures
maxArticlesPerStartUrl to input to limit the number of articles per start URLFeatures
onlyArticlesForLastDays to input for easier dynamic date filteringChanges
snapshotUrls output have been replaced by screenshotUrlextendOutputFunction is run after all fields were assigned forfull controlFixes
extendOutputFunction now correctly works with undefined fields for browserFeatures
crawlWholeSubdomain to input so you don't need to set pseudoUrls or linkSelectoronlySubdomainArticles to input to limit articles and enqueueing to the subdomain of the start URLsaveHtmlAsLink to input to save HTML of articles as a link in the outputreferrer, startUrl and depth to outputFeatures
Features
saveSnapshotsOfInvalidArticles input field in favor of new saveSnapshots input field that save for all articles.pageWaitSelector and instead add pageWaitSelectorCategory and pageWaitSelectorArticle inputsFeatures
scrollToBottom, scrollToBottomButtonSelector, scrollToBottomMaxSecsFeatures
saveSnapshotsOfInvalidArticles option to inputFeatures
enqueueFromArticles option to enqueue articles from article pages to get even more articles from the website. You need to enable it in input.scanSitemaps and sitemapUrls parameters. scanSitemaps automatically searches sitemaps for articles for each start URL and sitemapUrls allows you to add the sitemaps manually if necessary. Be careful that scanSitemaps may dump a huge amount of (sometimes old) article URLs into the scraping processFixes
onlyNewArticles and onlyNewArticlesPerDomain was loading duplicate items which caused excess usage of dataset read.Features
onlyNewArticlesPerDomain. This is much more efficient way to deduplicate articles, so use it instead of onlyNewArticles.onlyNewArticlesPerDomain works also on local datasets0.21.0 that sometimes crashed at the start of the runcurrentItem param to extendOutputFunctionarticleUrls directly as input option to parse directly on articles