Any News Site Article Extractor avatar

Any News Site Article Extractor

Pricing

from $0.49 / 1,000 results

Go to Apify Store
Any News Site Article Extractor

Any News Site Article Extractor

Point this actor at a news website's homepage and get back structured articles: title, authors, publish date, full body text, top image, keywords and a short summary, with keyword and minimum-length filtering.

Pricing

from $0.49 / 1,000 results

Rating

0.0

(0)

Developer

vexaloom-labs

vexaloom-labs

Maintained by Community

Actor stats

0

Bookmarked

2

Total users

1

Monthly active users

a day ago

Last modified

Features

Limited permissions

Share

Turn news pages into structured article rows without writing a scraper per publication. Paste article links you already have, or give the actor a news site and let it find the stories itself.

Two ways to run it

Known links. Put one or more article URLs in articleUrls. Each page is fetched and parsed directly — useful when you have a list of links from a feed, a newsletter, or a search result.

Site discovery. Put a homepage, section page, or newsroom URL in websiteUrl. The actor reads the page's current story links first, then walks the publication's sitemaps (including robots.txt declarations, sitemap indexes, and gzipped sitemap files) to fill the rest of your quota. A section URL such as https://example.com/technology keeps discovery focused on that section.

Both paths can be combined in one run. Known links are processed first, then discovery continues until maxArticles is reached.

What each article contains

  • The canonical article URL and the URL that was requested
  • Headline, author names, publish and last-modified timestamps
  • Full body text with navigation, related-story rails, newsletter prompts, and ad labels removed
  • Word count, detected language, section, and meta description
  • Top image plus every other image inside the article body (related-story and author-card images are excluded)
  • Optional top-10 keyword list and a three-sentence summary
  • Source domain and extraction timestamp

Fields a page genuinely does not publish are left out. The actor never writes null, empty strings, or empty arrays into a dataset row.

How the extraction cascade works

Each page is parsed with three fallbacks, in order:

  1. JSON-LD — NewsArticle, Article, BlogPosting, and related schema.org types, including @graph and mainEntity wrappers.
  2. Open Graph / article meta — og:*, article:published_time, article:author, article:section, og:locale, and canonical links.
  3. The rendered HTML — the most article-specific body container (.entry-content, .article-body, [itemprop=articleBody], and similar), author bylines, <time datetime>, and in-article images.

Structured data wins over meta tags, which win over DOM heuristics. This keeps the output consistent across publishers while still returning clean text on older sites that never adopted JSON-LD.

Inputs

FieldTypeDefaultWhat it does
articleUrlsstring[]–Article links to extract directly
websiteUrlstring–Homepage, section, or newsroom URL to discover articles from
maxArticlesinteger20Total article cap for the run (1–500)
keywordFilterstring–Boolean expression over title + body: AND, OR, NOT, parentheses
minWordCountinteger100Drop articles whose body is shorter than this
concurrencyinteger5Parallel article fetches (1–20)
extractKeywordsbooleantrueAdd a top-10 keyword list to each row
extractSummarybooleantrueAdd a three-sentence summary to each row
languagestringautoLanguage hint / filter — auto or one of 40 ISO codes
autoProxyFallbackbooleantrueRetry a blocked direct fetch through Apify residential proxy

At least one of articleUrls or websiteUrl is required.

Output fields

FieldDescription
articleUrlURL that was fetched
articleCanonicalUrlPublisher's canonical URL, when declared
articleTitleHeadline
articleAuthors[]Author names, cleaned and deduplicated
articlePublishDateISO 8601 publish timestamp
articleModifiedDateISO 8601 last-modified timestamp
articleSectionSection or desk, when declared
articleDescriptionMeta / JSON-LD description
articleTextFull body text
articleWordCountWords in articleText
articleTopImageHero / lead image URL
articleImages[]All article image URLs, hero first
articleLanguageDetected or requested language code
articleSiteNamePublication name, when declared
articleKeywords[]Top content words by frequency (optional)
articleSummaryThree-sentence summary (optional)
sourceDomainHostname of the article
scrapedAtISO 8601 extraction timestamp

If a run finds no usable articles it writes one diagnostic row with type: "news_crawler_error" and a reason, so a scheduled job can tell "empty result" apart from "no output".

Examples

Extract three links directly:

{
"articleUrls": [
"https://www.bbc.com/news/articles/example-one",
"https://apnews.com/article/example-two"
],
"maxArticles": 3
}

Discover the latest stories from a technology publication and keep only AI coverage:

{
"websiteUrl": "https://techcrunch.com",
"maxArticles": 25,
"keywordFilter": "(AI OR \"machine learning\") AND NOT crypto",
"minWordCount": 200
}

Scan a French news site in French:

{
"websiteUrl": "https://www.franceinfo.fr",
"maxArticles": 20,
"language": "fr"
}

Common uses

  • Media monitoring — watch a publication or section and route only matching stories to alerts.
  • Research corpora — collect article text and metadata from many outlets into one dataset.
  • Brand and executive tracking — filter coverage by names, products, or competitors.
  • Digest building — take a list of links from a feed and turn it into clean rows for a newsletter or briefing.
  • RAG pipelines — use the body text and canonical URLs as document chunks with reliable provenance.

FAQ

Does it work on every news site? Most publications that expose article metadata or a sitemap work without special handling. Pages that are fully client-side rendered, paywalled behind a login, or served as a bot challenge may return thin or no body text.

What happens on sites that block datacenter IPs? The direct fetch is tried first. If it is blocked and autoProxyFallback is on, the same requests are retried through Apify residential proxy. Sites that work directly never use proxy bandwidth.

Are duplicate links returned twice? No. URLs are deduplicated before fetching, so the same article appears once even if it is both pasted and discovered.

Why did I get fewer rows than maxArticles? The cap is a ceiling, not a promise. A small site, a shallow sitemap, the keyword filter, and the minimum word count can all reduce the number of qualifying articles.

Can I use the keyword filter with phrases? Yes. Quote multi-word phrases: "artificial intelligence" AND NOT "machine learning". Bare terms match on word boundaries, so AI will not match email.

Is this affiliated with any publisher? No. It is an independent tool that reads publicly accessible pages from any news site. It is not endorsed by or connected to any publication.