Article Extractor scrapes detailed data from all articles on any website. It automatically recognizes what page is an article. It can be used for extracting news from BBC, CNN, Bloomberg, and other popular news websites.

Direct URLs to the articles to be parsed. No extra pages are enqueued from article pages.

Article URLs

Only articles from this day to present will be scraped. If empty, all articles will be scraped. Format is YYYY-MM-DD, e.g. 2019-12-31, or Number type e.g. 1 week or 20 days

Date from

Normally, the scrapers only enqueues from category pages. This option can help to gather more articles per run.

Enqueue articles from articles

A function that allows you to merge your custom extraction with the default one. You have to return an object from this function. This object will be merged/overwrite the default output for each article.

This is only needed if the default output is not good enough for you. Be careful that you should provide valid JavaScript.

Extend output function

JSON settings of what considered a link to an article. If any of them is true, then the link will be opened.

Is URL article?

You can limit <a> tags whose links will be enqueued. By default this is empty, add `a.some-class` to enable it

Link selector

Maximum number of valid articles scraped. The crawler will stop automatically after reaching this number.

Max articles per crawl

You can limit the speed of the scraper. Don't forget to lower your memory too to save Compute units.

Max concurrency

Maximum depth of crawling. 0 is only start URLs, 1 are first level links etc. Only valid for pseudo URLs

Max depth

Maximum number of total pages crawled. Includes home page, pagination pages, invalid articles etc.

Max pages per crawl

Article need to contain at least this amount of words to be extracted

These settings tell the actor what articles should be scraped. Leaving default values usually works fine.

Minumum words

If checked, the article must have a date of release to be considered valid.

Must have date

Emails where should the bellow notifications should be sent.

Notification emails

Actor will send notifications on provide email when it reaches provided CUs.

Notify after CUs

Actor will send notifications on provide email every provided CUs reached from last notification.

Notify every CUs (periodically)

If true, will scrape only articles that are on the domain from where they are linked.

Only inside domain articles

This option is only viable for smaller runs. If you plan to use this at large scale, use `onlyNewArticlesPerDomain` instead. If true, will scrape only new articles each you run it. All URLs you scraped are saved in dataset called `articles-state` and are compared with new ones.

Only new articles (only for small runs)

If true, will scrape only new articles each you run it. All URLs you scraped ar and are compared with new ones. Scraped articles are saved in one dataset per each domain, datasets are named 'ARTICLES-SCRAPED-domain'

Only new articles (saved per domain, preferable)

How long to wait on each page before extracting data

Wait for on each page

For what selector to wait on each page before extracting data

Wait for selector on each page

Proxy configuration

Can be used to enqueue more pages like pagination or categories. Doesn't work for articles, they are recognized by the recognition system.

Use these settings to enqueue more pages. If you want to enqueue direct article URLs this way, you have to add `{ "label": "article"}` to the userData`

Pseudo URLs

Saves full HTML of the article page but makes data less readable.

Save full HTML

Will scan different sitemaps of the first URL for articles. Be very careful with this option as it can load a huge amount of (sometimes old) articles and the scrape time/cost will raise.

Find articles in sitemaps (dangerous)

Optionally you can also provide chosen sitemap URLs that have the articles you need to extract.

Sitemap URLs (safer)

Can be main page URL or any category URLs. Article pages are found and enqueued from these. If you want direct article URLs, use `articleUrls` input instead

Category URLs

Actor run will finish after reachin certain amount of Compute units.

Stop after CUs

Using browser is more expensive but gives you ability to evaluate JavaScript and wait for dynamically loaded data.

Use browser (Puppeteer)

This option will allow you to bypass protection and/or paywall on some sites. Use with caution as it might get blocked.

Use Google Bot headers

Input schema for Smart article extractor

article-extractor-smart

Scrapes content from all articles on any website. Automatically recognizes articles. Can be used to extract news from BBC, CNN, Bloomberg, and more.

Article Extractor - Get content and metadata

Smart Article Extractor

Article Extractor scrapes detailed data from all articles on any website. It automatically recognizes what page is an article. It can be used for extracting news from BBC, CNN, Bloomberg, and other...

Where next?