News Article Extractor: Text, Date, Image from URLs and Feeds

Give it article addresses, RSS or Atom feeds, news sitemaps or a site's front page, and get one clean row per article: title, publisher, publish and modified dates, language, description, lead image, section, tags, word count, reading time, and the article text as Markdown and as plain text. It uses plain HTTP only: no browser, no proxy, no AI model. It reads robots.txt, speaks to each site politely, flags paywalls and consent pages instead of returning half an article, and has an only-new mode so a scheduled run pays only for articles it has not delivered before.

USD 1.00 per 1,000 articles, plus a tiny start fee per run. Paywalled pages, consent and bot-protection pages, pages that are not articles, failed downloads, robots.txt blocks and articles already seen in only-new mode are free.

What people use it for

• Media monitoring and PR. Point it at the feeds of the outlets and blogs you follow, schedule it daily with onlyNew, and receive only the articles published since the last run, with full text and dates.
• RAG and LLM corpora. Clean Markdown and plain text, language, canonical address and a stable publish date for each article, with navigation, related links, comments, share bars and cookie banners removed.
• Competitor and industry blogs. Track what a company publishes, when, and under which section or tags.
• Research datasets. Build article collections from feeds and sitemaps you choose, with provenance on every row (which feed or sitemap, which address, which extraction method, how confident).
• The "get the full text" step after a news finder. A news search or alert tool finds addresses; this actor turns them into text. It pairs with Global News Monitor (JTPalms/global-news-monitor), which finds articles from open GDELT data. For documentation sites and non-article pages use Website to Markdown Crawler (JTPalms/website-to-markdown-crawler), which returns page Markdown for RAG rather than article fields. Both are named here in plain text because they are private until published.

Sample output

Real run on a public article (Deutsche Welle, 2026-10-06), markdown and text shortened:

{
  "url": "https://www.dw.com/en/munich-up-against-at-least-six-competitors-for-2036-olympics/a-79566678",
  "finalUrl": "https://www.dw.com/en/munich-up-against-at-least-six-competitors-for-2036-olympics/a-79566678",
  "canonicalUrl": "https://www.dw.com/en/munich-up-against-at-least-six-competitors-for-2036-olympics/a-79566678",
  "status": "extracted",
  "title": "Munich was picked as the German city that would bid for the 2036 Olympics. But they have competition.",
  "publisher": "Deutsche Welle",
  "publishedAt": "2026-10-06T14:57:45.638Z",
  "modifiedAt": "2026-10-06T19:54:16.232Z",
  "dateSource": "jsonld",
  "language": "en",
  "description": "The International Olympic Committee has revealed the six other countries intending to bid for the 2036 Olympic Games. This comes after Germany selected Munich as its Olympic candidate.",
  "imageUrl": "https://static.dw.com/image/79443887_804.jpg",
  "section": null,
  "tags": [],
  "wordCount": 472,
  "readingTimeMinutes": 3,
  "markdown": "Munich's bid for the 2036 Summer [Olympics](https://www.dw.com/en/olympics/t-17351918) will be up against at least six candidates from Europe, Asia and Africa. On Tuesday, the International Olympic Committee (IOC) announced that...",
  "text": "Munich's bid for the 2036 Summer Olympics will be up against at least six candidates from Europe, Asia and Africa. On Tuesday, the International Olympic Committee (IOC) announced that...",
  "extractionMethod": "jsonld+readability",
  "confidence": "high",
  "paywalled": false,
  "httpStatus": 200,
  "robotsAllowed": true,
  "redactedContacts": 0,
  "source": "start",
  "domain": "dw.com",
  "fetchedAt": "2026-10-06T22:32:27.928Z"
}

The row has no authors field: author names are off by default (see "Author names" below). Fields that are empty for this page (snippet, skippedReason, reason, error, errorCode, sourceUrl) are shown as null in the real output and left out here.

Quick start

The default input extracts five public articles (BBC, FRANCE 24, Deutsche Welle, NASA, GOV.UK) in a few seconds. To follow feeds on a schedule:

{
  "feedUrls": [
    { "url": "https://www.gov.uk/government/organisations/hm-treasury.atom" },
    { "url": "https://www.nasa.gov/feed/" }
  ],
  "maxArticlesPerSource": 20,
  "onlyNew": true,
  "stateKey": "my-morning-feeds"
}

From Python:

from apify_client import ApifyClient

client = ApifyClient("<YOUR_APIFY_TOKEN>")
run = client.actor("JTPalms/news-article-extractor").call(run_input={
    "feedUrls": [{"url": "https://www.nasa.gov/feed/"}],
    "maxArticlesPerSource": 20,
    "onlyNew": True,
})
for article in client.dataset(run["defaultDatasetId"]).iterate_items():
    if article["status"] == "extracted":
        print(article["publishedAt"], article["title"], article["wordCount"])

Sources you can give it

Input	What happens
startUrls	Article addresses, fetched as they are. Tracking parameters (utm_*, fbclid, traffic_source, at_medium and similar) are removed first so the same article is never fetched twice and robots.txt is checked on the clean address.
feedUrls	RSS 2.0, RSS 1.0, Atom and JSON Feed. Items are taken newest first, up to maxArticlesPerSource. A web page given here that advertises a feed is followed to its feed.
sitemapUrls	Sitemaps, sitemap indexes (the most recently modified child files first), .gz files and Google news sitemaps. URLs are taken newest first; news:publication_date is used as the date when the page has none.
siteUrls	A site address. The actor looks for the feed the front page advertises, then a feed at the usual paths (/feed, /rss, /feed.xml, /rss.xml, /atom.xml), then the news sitemap or sitemap named in robots.txt or at the usual paths. About a dozen requests at most.

Sources are taken in turns, so a small maxArticles still covers every source. Author, staff and profile pages found in feeds and sitemaps are never fetched. Addresses on the same article (http and https, with or without www, trailing slash, tracking parameters) are one article.

How an article is read

1. Fetch with an honest user agent, after robots.txt allows the address. Redirects are followed one hop at a time, with a 5 MB limit and a timeout.
2. Metadata comes from the page's own markup, in this order of trust: schema.org JSON-LD (NewsArticle, Article, BlogPosting and relatives), then Open Graph and article:* tags, then other meta tags, then visible markup (<time datetime>, the first heading, the title tag). The row says which date source it used (dateSource: jsonld, meta, feed, sitemap, time-tag, url).
3. Walls are checked before any text is kept. A page that marks itself not free to read, a paywall marker on a page that shows only a teaser, a consent page, or a bot-protection page gives a flagged free row with metadata only. There is no attempt to get past any of them.
4. Main text comes from Mozilla Readability, checked against the page's own article container, with a plain fallback for pages Readability cannot read and the JSON-LD articleBody when the page text is empty. Related lists, comments, share bars, newsletter boxes, ads, byline lines and photo credits are removed.
5. Fallback twins. If the page is too short or builds its text with JavaScript, and it names an AMP page or a canonical address on the same site, that one address is fetched once (extractionMethod ends in +amp or +canonical). Paywalled pages are never retried.
6. Classification. An article has article markup (JSON-LD article type or og:type=article) or enough text with a date or byline. Listing, category and "all news" pages are recognised and flagged not_article.
7. Language comes from the page, or from the text itself (script and stopword ranking, no network) when the page declares none.

extractionMethod reads like jsonld+readability, og+readability or html+article-root: where the metadata came from, then which text method won. confidence is high, medium or low, from how many of these hold: a title, a date, article markup, at least 150 words, a real article container rather than the page body.

Output fields

Field	Meaning
url, finalUrl, canonicalUrl	The address you gave (cleaned), where it ended after redirects (or the AMP or canonical page the text came from), and the page's canonical address.
status	extracted (billed), paywalled, consent_wall, bot_protection, needs_javascript, not_article, too_short, robots_blocked, duplicate, error (all free).
title, publisher, description, imageUrl, section, tags, language	Publisher's own values. publisher falls back to the domain; imageUrl to the feed's image.
publishedAt, modifiedAt, dateSource	ISO 8601 in UTC; a date without a time stays YYYY-MM-DD.
wordCount, readingTimeMinutes	Words of article text; minutes at 200 words per minute.
markdown, text	Article body as Markdown (headings, lists, tables, quotes, links) and as plain text. outputFormat picks one or both. snippet (300 characters) replaces both when includeFullText is off.
extractionMethod, confidence	How the row was built and how far to trust it.
authors	Only when includeBylines is on.
paywalled, skippedReason, reason, error, errorCode, httpStatus, robotsAllowed	Why a row is not an article, in words and as codes.
redactedContacts, truncated	Emails and phone numbers masked in the text (count); text cut at 200,000 characters.
source, sourceUrl, domain, fetchedAt	Where the address came from (start, feed, sitemap, site), the feed or sitemap, the domain, the fetch time.

A SUMMARY record in the key-value store lists counts (extracted, billed, each kind of flagged row, already seen, filtered out), each source with how it was found, and request statistics.

Only-new mode

Turn on onlyNew and give the task a stateKey. Each article address delivered is remembered (a short hash, for a year, up to 300,000 per name) in a named key-value store in your account. The next run with the same name skips what it has seen before it fetches anything, returns only new articles, and charges only for those. The first run returns everything. Paywalled and not-an-article pages are reported once and then remembered; a page that failed is retried on the next run. Use a different stateKey for each task. Change the name to start over.

Author names

Bylines are personal data of the people who wrote the articles, so they are off by default. With includeBylines on, each row gets an authors list exactly as the publisher published it (from the page's JSON-LD or meta tags, else its byline markup). Byline lines ("By Jane Doe", "Reporting by...") and author boxes are removed from the article text whether or not you ask for the names. The actor never opens author or profile pages, and reads no comments or reader data. Email addresses and phone numbers in article text (for example media contacts at the end of a press release) are masked as [email removed] and [phone removed], and redactedContacts counts them.

What it does not do

• Paywalls and logins. No login, no cookies, no paywall bypass, no search-engine user agent tricks. A page whose JSON-LD says it is not free, or that shows a paywall marker over a teaser, comes back as a free paywalled row with its metadata. The publisher's own statement is respected even when the full text happens to be in the page (some metered sites mark every article this way), so those articles are flagged rather than delivered.
• Bot protection and CAPTCHAs. A site that answers with a challenge or refuses the actor (HTTP 403 or 429) gets a clear free row, and after four refusals in a row the rest of that site is skipped for the run. The actor does not rotate addresses, use proxies, imitate a browser or solve anything. Some large publishers (for example AP News and CBC) refuse plain HTTP clients; they will show as errors.
• JavaScript-only sites. There is no browser. Pages that build their text in the browser come back as free needs_javascript rows unless they offer an AMP or canonical twin.
• Comments, reader data, author pages, emails, phone numbers. Not collected.
• PDFs and video pages. Skipped from their headers (free). Use PDF to Markdown and Text (JTPalms/pdf-to-markdown) for PDFs.
• Legal advice. See the rights note.

Rights and robots

You are responsible for having the right to store, analyse and reuse the article text you collect: many publishers license their text for reading, not for republishing or for AI training, and some say so in their robots.txt or terms. Turn off includeFullText to collect metadata and a 300 character snippet only. The actor identifies itself as Mozilla/5.0 (compatible; news-article-extractor/1.0; +https://apify.com/JTPalms), reads every site's robots.txt before the first page (rules for news-article-extractor first, then *; a robots.txt it cannot read means the site is skipped; Crawl-delay is honoured up to 60 seconds), asks for no more than 2 pages at a time from one site and never more often than every 500 milliseconds, and uses no proxies.

Measured, not promised

All numbers are from real runs on 2026-10-06.

• Default input (5 articles from 5 sites): 3 seconds in the cloud, 5 extracted and billed, USD 0.0003 of platform usage, 75 MB peak memory.
• 40 articles from 7 feeds (BBC, Al Jazeera, Deutsche Welle, Euronews, FRANCE 24, NASA, GOV.UK): 9 seconds, 35 extracted and billed, 5 free rows (3 video pages recognised as not articles, 2 items with only 51 words), no failures, 57 requests, 235 MB peak memory, USD 0.0004 of platform usage. That is about USD 0.01 of platform usage per 1,000 articles at this size, so the compute you pay Apify for is a small fraction of the per-article price.
• Extraction time is 10 to 90 ms per page. The limit on speed is politeness, not computing: one site is read at about two pages a second at most, so 1,000 articles from a single site take about 8 minutes; articles from many sites run in parallel.
• Accuracy. Checked by hand on 7 saved articles from 7 publishers (title exactly the publisher's headline, publish date correct, first and last sentences of the story present, no navigation, share, related-link or byline text), and spot-checked on about 15 more live articles from 11 sites (Ars Technica, TechCrunch, ProPublica, Wired, The Verge, Smashing Magazine and others). A large hand-labelled accuracy set has not been built yet; confidence and extractionMethod are your guide on each row.

Pricing

Tier	Per 1,000 articles	Per article
Free and Bronze	USD 1.00	USD 0.001
Silver	USD 0.90	USD 0.0009
Gold, Platinum, Diamond	USD 0.80	USD 0.0008

Plus USD 0.001 per run (the actor start event). Only extracted rows are charged, once each. Examples at the Free and Bronze price: 5 articles = USD 0.006 including the start fee. 30 feeds with 20 new articles a day in onlyNew mode = 600 articles = USD 0.60 a day. A one-off collection of 10,000 articles = USD 10.00. Set a maximum cost on the run and it stops when it is reached, keeping what it has.

Tips

• Give feeds when you have them: they are the cheapest and most reliable way to find new articles, and the feed date is used when a page has none.
• For a news sitemap use sitemapUrls with publishedWithinDays to keep only recent items.
• A small minWords (the default is 80) suits long articles; lower it for short news briefs.
• Run times depend on how many different sites are involved: one site at a time is read at about two pages a second at most.

Related actors

• Global News Monitor: Articles by Keyword, Tone and Topic: News and media monitoring in 65+ languages by keyword, topic, company or tone from GDELT open data.
• RSS Feed Reader & Scraper: Atom, JSON Feed & New-Item Monitor: RSS feed reader and scraper.
• Website to Markdown Crawler for RAG: Fast, No Browser: Crawl a site into clean Markdown for RAG and LLMs without a browser.
• PDF to Markdown and Text: Tables, Pages, Metadata for RAG: PDF parser and text extractor: turn PDF URLs into clean Markdown, plain text, per-page text and detected tables.