Smart Article Extractor

Can be main page URL or any category URLs. Article pages are found and enqueued from these. If you want direct article URLs, use `articleUrls` input instead

Direct URLs to the articles to be parsed. No extra pages are enqueued from article pages.

This option is only viable for smaller runs. If you plan to use this at large scale, use `onlyNewArticlesPerDomain` instead. If true, will scrape only new articles each you run it. All URLs you scraped are saved in dataset called `articles-state` and are compared with new ones.

If true, will scrape only new articles each you run it. All URLs you scraped ar and are compared with new ones. Scraped articles are saved in one dataset per each domain, datasets are named 'ARTICLES-SCRAPED-domain'

If true, will scrape only articles that are on the domain from where they are linked.

Normally, the scrapers only enqueues from category pages. This option can help to gather more articles per run.

Will scan different sitemaps of the first URL for articles. Be very careful with this option as it can load a huge amount of (sometimes old) articles and the scrape time/cost will raise.

Optionally you can also provide chosen sitemap URLs that have the articles you need to extract.

Saves full HTML of the article page but makes data less readable.

This option will allow you to bypass protection and/or paywall on some sites. Use with caution as it might get blocked.

Article need to contain at least this amount of words to be extracted

Only articles from this day to present will be scraped. If empty, all articles will be scraped. Format is YYYY-MM-DD, e.g. 2019-12-31, or Number type e.g. 1 week or 20 days

If checked, the article must have a date of release to be considered valid.

JSON settings of what considered a link to an article. If any of them is true, then the link will be opened.

Can be used to enqueue more pages like pagination or categories. Doesn't work for articles, they are recognized by the recognition system.

You can limit tags whose links will be enqueued. By default this is empty, add `a.some-class` to enable it

Maximum depth of crawling. 0 is only start URLs, 1 are first level links etc. Only valid for pseudo URLs

Maximum number of total pages crawled. Includes home page, pagination pages, invalid articles etc.

Maximum number of valid articles scraped. The crawler will stop automatically after reaching this number.

You can limit the speed of the scraper. Don't forget to lower your memory too to save Compute units.

Proxy configuration

Using browser is more expensive but gives you ability to evaluate JavaScript and wait for dynamically loaded data.

How long to wait on each page before extracting data

For what selector to wait on each page before extracting data

A function that allows you to merge your custom extraction with the default one. You have to return an object from this function. This object will be merged/overwrite the default output for each article.

Actor run will finish after reachin certain amount of Compute units.

Emails where should the bellow notifications should be sent.

Actor will send notifications on provide email when it reaches provided CUs.

Actor will send notifications on provide email every provided CUs reached from last notification.