Article Extractor: News & Blog Text to Markdown

What does Article Extractor do?

Article Extractor takes news and blog article URLs, or websites, RSS/Atom feeds or sitemaps to find articles on, and returns one clean row per article: title, author, published date, lead image, language and the main text as plain text and Markdown. Navigation, cookie banners, share buttons and footers are removed.

Input fields · API · Use it from Claude, ChatGPT or Cursor (MCP)

Jump to: Monitor sites · Fields · Price · How to use · Input · Output · Chain after another actor · AI agents (MCP) · Limits · FAQ

Try it in one click: the input comes pre-filled with the Rust blog and 3 articles. That's 3 articles: $0.003, plus the $0.00005 start fee (3 × $0.001 per article). Then replace it with the sites or articles you actually want.

Monitor sites for new articles

Article Extractor can check a list of blogs or news sites on a schedule and give you only the articles published since its last run, with their full text. Set it up once:

1. Save a task. Fill in Websites, feeds or sitemaps, turn on Only new articles since the last run (onlyNewArticles: true in the API), then click Save as a new task at the top right of the actor's page. The first run returns the newest articles up to Max articles per site; later runs return only articles this search hasn't returned before.
2. Schedule it. In Apify Console, open Schedules, click Create new, pick the task and set how often it runs (for example every morning at 07:00, cron 0 7 * * *).
3. Send the new articles where you work, from the task's Integrations tab:
   • Slack: a message in the channel you choose when a run succeeds (you pick the events and can edit the message).
   • Email: Send results email via Gmail sends the run's results, with the dataset attached.
   • Webhook: a POST to your URL on the ACTOR.RUN.SUCCEEDED event. The payload's resource.defaultDatasetId points to the new articles: GET https://api.apify.com/v2/datasets/<defaultDatasetId>/items?format=json with your API token.
   • Make, Zapier or n8n: each has an Apify step that starts on a finished run and reads its dataset.

A run that finds nothing new returns 0 articles and costs only the $0.00005 start fee. Slack and email are sent for every successful run, including a quiet one; to hear only about new articles, use the webhook or Make/Zapier/n8n and skip runs whose dataset is empty. Add the run failed event too, so a failed run doesn't go unnoticed. The first few runs may also return older articles the first run didn't reach (see the FAQ). Keep Find articles from and Article URL patterns fixed: changing them starts a new list.

What data does Article Extractor return?

Field	Example	Notes
title	Announcing a Maintainer in Residence
author	Jakub Beránek	Names only, never profile links or email addresses; null when not stated.
publishedAt, modifiedAt	2026-09-22T00:00:00Z	ISO 8601 UTC.
text	At the end of August, we announced...	Main text as plain text.
markdown	# Announcing a Maintainer in Residence...	Main text as Markdown (headings, lists, tables, code, links).
wordCount	740
leadImageUrl	https://www.rust-lang.org/static/images/rust-social-wide.jpg
language, siteName, description	en	From the page's metadata; null when not stated.
url, canonicalUrl	https://blog.rust-lang.org/2026/09/22/...
discoveredVia, discoveredFrom	feed, https://blog.rust-lang.org/feed.xml	How the article was found: input, feed, sitemap or dataset (with the dataset id).
sourceTitle, sourceIndex	"Rust 1.83.0 released", 4	From the dataset item the article came from (see Chain it after another actor); null otherwise.
id	dcb1e0eadb1f512f0c9b3410	From the article URL, the same on every run.

Every article has the same fields; the full record is under Output.

How much does it cost to extract articles?

You pay per article returned: $1.00 per 1,000 articles, plus $0.00005 each time a run starts. Feeds and sitemaps aren't charged, and neither are articles that fail, are skipped, or were returned already in this run. Pages that need JavaScript, are blocked by robots.txt, opt out of AI use, or turn out not to be articles are not charged either.

It's cheaper on paid Apify plans: $0.90 per 1,000 articles on Starter, $0.80 on Scale and $0.70 on Business. The prices on this page are the Free-plan price, so on a paid plan you pay less than the examples show.

• The example below: 3 articles × $0.001 = $0.003, plus the $0.00005 start fee.
• A month, for example: 20 sites checked daily with only-new on, about 5 new articles per site per day: 20 × 5 × 30 = 3,000 articles × $0.001 = $3.00, plus 30 starts ($0.0015).
• Caps: Max articles per run in the input, and Maximum cost per run in the run options; it stops cleanly at whichever comes first. Each article is counted against the limit before it's fetched (and given back if it fails), so a capped run never fetches articles it can't return.

How to extract articles from a website

1. Open the actor and paste sites (https://blog.rust-lang.org/), feed URLs or sitemap URLs into Websites, feeds or sitemaps, or single pages into Article URLs.
2. Set Max articles per site (newest first).
3. Click Start.
4. Open the Output tab and export the articles as JSON, CSV or Excel, or read them through the API.
5. To follow the sites, save it as a task and schedule it (see Monitor sites for new articles above).

For a site, it reads the site's own feed (found automatically from its home page) or, when there's none, its sitemap, and extracts the newest articles first.

Example: the Rust blog

The pre-filled input, with Only new articles turned on:

{"sites": ["https://blog.rust-lang.org/"], "maxArticlesPerSite": 3, "onlyNewArticles": true, "discoverFrom": "auto"}

The newest of the 3 articles (real output from a local run on 2026-09-25; text, markdown and url shortened):

{
  "id": "dcb1e0eadb1f512f0c9b3410",
  "url": "https://blog.rust-lang.org/2026/09/22/announcing-a-maintainer-in-residence-scott-schafer-...",
  "title": "Announcing a Maintainer in Residence: Scott Schafer for the Cargo team | Rust Blog",
  "author": "Jakub Beránek",
  "publishedAt": "2026-09-22T00:00:00Z",
  "leadImageUrl": "https://www.rust-lang.org/static/images/rust-social-wide.jpg",
  "language": "en",
  "text": "At the end of August, we announced our first Maintainers in Residence, Rust Project contributors who...",
  "wordCount": 740,
  "discoveredVia": "feed",
  "discoveredFrom": "https://blog.rust-lang.org/feed.xml"
}

The site's feed was found from its home page. A second run straight after returned the next 3 older articles, not the same ones: the first run stops at 3 per site, and later runs work through the older ones before settling on newly published posts (see the FAQ).

Ready-to-run examples

Each example opens with the input already filled in. Run it as it is, or change the input first.

• Get a blog's latest articles as clean text and Markdown: Give it a blog or news site and get its newest articles with title, author, date and the full text as Markdown, without menus or ads.
• Monitor tech blogs for new articles: Run daily and get only the articles published since the last run, from several blogs at once, with full text as Markdown.

Input

Field	What it does
Websites, feeds or sitemaps	One per line: a site (https://blog.rust-lang.org/), a feed URL, or a sitemap URL.
Article URLs	Individual article pages, extracted as given (no discovery, no "only new" memory).
Or: article URLs from a dataset (datasetId)	One of your Apify datasets, e.g. RSS & Atom Feed Reader's results: each item's link is extracted, and the two fields above are ignored. See Chain it after another actor.
Field with the article URL (datasetUrlField)	Only with a dataset. Empty (default): found automatically among url, link and articleUrl, then one level down (article.link).
Max articles per site	Newest first, at most this many per site (default 3, up to 500).
Only new articles since the last run	Skip articles this search already returned from that site.
Find articles from	Auto (feed, else sitemap), feed only, or sitemap only.
Article URL patterns	Only discovered URLs matching one of these (/blog/, https://example.com/news/*).
Max articles per run	Cap the total across all inputs.

{
  "sites": ["https://blog.rust-lang.org/", "https://www.djangoproject.com/weblog/"],
  "maxArticlesPerSite": 10,
  "onlyNewArticles": true
}

Output

One row per article, always with these fields (null when neither the page nor its feed/sitemap says):

{
  "id": "dcb1e0eadb1f512f0c9b3410",
  "url": "https://blog.rust-lang.org/2026/09/22/announcing-a-maintainer-in-residence/",
  "canonicalUrl": null,
  "title": "Announcing a Maintainer in Residence",
  "author": "Jakub Beránek",
  "publishedAt": "2026-09-22T00:00:00Z",
  "modifiedAt": "2026-09-22T00:00:00Z",
  "leadImageUrl": "https://www.rust-lang.org/static/images/rust-social-wide.jpg",
  "language": "en",
  "siteName": null,
  "description": "Empowering everyone to build reliable and efficient software.",
  "text": "At the end of August, we announced our first Maintainers in Residence ...",
  "markdown": "# Announcing a Maintainer in Residence\n\nAt the end of August, ...",
  "truncated": false,
  "wordCount": 740,
  "input": "https://blog.rust-lang.org/",
  "discoveredVia": "feed",
  "discoveredFrom": "https://blog.rust-lang.org/feed.xml",
  "sourceTitle": null,
  "sourceIndex": null,
  "scrapedAt": "2026-09-24T12:00:00Z"
}

• id comes from the article URL and is the same on every run: deduplicate on it, or use it as a vector-store key.
• Metadata is read from the page's schema.org JSON-LD, Open Graph and article:* tags, common publisher meta tags and microdata; the feed (or a Google News sitemap) fills in the title, author and date when the page doesn't say.
• text and markdown are capped at 500,000 characters each (truncated). Pages over 5 MB of HTML are skipped.

Run it on a schedule, or from your own code

• Schedule: save the input as a task and add it to a schedule, as in Monitor sites for new articles above.
• API: fetch a task's latest results with GET https://api.apify.com/v2/actor-tasks/<task id>/runs/last/dataset/items?status=SUCCEEDED&format=csv and your API token (format can also be json or xlsx).
• Webhooks and automation: a webhook when a run succeeds, or Make, Zapier or n8n through Apify's integrations.

Chain it after another actor

To get the full text of what another actor found, run Article Extractor right after it finishes with an integration (any actor whose results have a link per article, e.g. RSS & Atom Feed Reader, GDELT News Search or a news scraper):

1. Open the other actor or its saved task, go to the Integrations tab and add Apify actor → Article Extractor, triggered when a run succeeds.
2. Set "datasetId": "{{resource.defaultDatasetId}}" in the input (Apify fills in the finished run's dataset) and add Max articles per run if you want a cap.

Every successful feed or news run then produces a dataset of full articles, with no code. Each item's url is extracted (or link, articleUrl, or the field you name in datasetUrlField), and Websites, feeds or sitemaps and Article URLs are ignored, defaults included, so a chained run never fetches the Rust blog. Items without a link are skipped, and an article listed twice is extracted and charged once. Each article carries sourceTitle and sourceIndex (its row in the other dataset, 0 = the first) to join it back. From the API it's the same input: pass the finished run's defaultDatasetId as datasetId. A run reads at most the first 20,000 items and 10,000 article URLs of a dataset, with your own account's access, read-only; RUN_STATS.dataset says how many were read and skipped.

To pay only for new articles, turn on the feed reader's own Only new items so each of its runs lists just what's new; Article Extractor's Only new articles applies to sites, not to a dataset.

Can I use Article Extractor from an AI agent (MCP)?

Yes, through Apify's MCP server, from Claude, ChatGPT, Cursor or any other MCP client. Add this to your client's MCP configuration (or let the agent find it with the server's actor search); your client signs you in to Apify:

{
  "mcpServers": {
    "apify": {
      "url": "https://mcp.apify.com?tools=humble-echidna/article-extractor"
    }
  }
}

To use an Apify API token instead of signing in, add "headers": {"Authorization": "Bearer <APIFY_TOKEN>"} next to url.

The agent passes the same input JSON as above, usually with Article URLs for pages it already has. A field left out takes its default, and Websites, feeds or sitemaps defaults to the Rust blog, so for article URLs only pass {"articleUrls": ["https://..."], "sites": []}. An agent that has just run a feed reader or news scraper can pass its dataset instead: {"datasetId": "<dataset id>"} (the other fields' URLs are then ignored). Keep Max articles per site low so answers stay small; markdown is the field to hand to the model.

Who it's for

Anyone who follows a set of blogs or news sites and needs the full articles, not just the headlines: media monitoring, competitor and industry news tracking, newsletters and research digests, and AI pipelines that summarise or index new posts. Give it the sites once, schedule it with Only new articles, and each run returns just what was published since the last one, with author, date and clean text.

Why this one?

• One flat price, no usage bill. You pay per result at the price above and nothing else: no compute, proxy or storage charges on top, apart from the $0.00005 run-start fee, so you know what a run costs before it starts. Results you don't get aren't charged.
• Only new articles since the last run. Each site's articles are remembered in your own Apify account (per site, per search), so a scheduled run never returns (or charges for) the same article twice.
• Finds the articles for you. Give it a blog or news site: it discovers its RSS, Atom or JSON feed, or falls back to its sitemap (including sitemap indexes and Google News sitemaps), and orders the articles newest first. URL patterns keep a sitemap to the article pages.
• A stable schema. Every row has the same fields, always in the same format: dates in ISO 8601 UTC, authors as names (never profile links or email addresses), absolute URLs. A field the page and its feed don't state is null, never guessed.
• Clean text, twice. Plain text for search and NLP, Markdown (headings, lists, tables, code, links) for LLMs and RAG.
• Honest charging. Pages that fail, need JavaScript, are blocked by robots.txt, opt out of AI use, or turn out not to be articles are not charged, and neither is an article listed twice.
• Fast and cheap. Plain HTTP, no browser: $1.00 per 1,000 articles.

Limits

• No JavaScript. Pages whose text is built in the browser come back as "needs JavaScript" and aren't charged.
• No logins or paywalls, and no getting around bot protection.
• No presets for sites whose terms forbid crawling (Medium and Substack, for example).
• robots.txt and politeness. It checks each site's robots.txt before every request, including feeds, sitemaps and every redirect hop, and skips what it disallows (blockedByRobots in RUN_STATS, not charged). It identifies itself as HumbleEchidnaApify/1.0 (+https://apify.com/humble-echidna); rules for User-agent: HumbleEchidnaApify apply to it. It honours Crawl-delay and Retry-After, and makes at most 2 requests at a time to one site.
• It honours opt-outs from AI use. Its output is pitched for LLMs, so a page that a site's robots.txt disallows for any of the well-known AI crawlers (Google-Extended, Applebot-Extended, CCBot, GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, anthropic-ai, Claude-Web, PerplexityBot, Perplexity-User) isn't fetched (optedOutOfAI, not charged). Blocking only one company's crawler (Bytespider, Meta-ExternalAgent, Amazonbot and the like) isn't an opt-out of AI use.
• Sizes: text and markdown are capped at 500,000 characters each; pages over 5 MB of HTML are skipped.

FAQ

Why did a site return nothing, or the wrong pages?

Without a feed, the sitemap lists every page of a site, not just articles: add Article URL patterns (e.g. /news/), or give the news sitemap's URL directly. Pages under 50 words (tag pages, listings) are skipped. Sitemap entries are taken newest first by the date the sitemap gives (a news publication date or lastmod), else by a date in the URL (/2026/09/22/), else in the order listed. RUN_STATS says, per input, where the articles came from and why any were skipped.

How does "only new" decide what's new?

An article is new when this search hasn't returned its URL from that site before, under the link the feed or sitemap lists or the address that link redirects to (http to https, a trailing slash): a redirected article is never returned or charged twice. The first run returns the newest ones up to your per-site limit; older articles it didn't reach are returned by later runs once nothing newer is waiting. Changing the discovery mode or URL patterns starts a new list.

Why did an article come back as "needs JavaScript"?

Its text is built in the browser, and this actor doesn't run one. It's not charged.

Why does it refuse localhost, a private address or a URL with a port?

It only fetches public web pages on the standard ports (80 and 443), and checks every address (and redirect) first.

How is this different from other article extractors?

The most-used article extractor in the Store is free and more configurable (its own crawler, many options). Pick this one if you want scheduled "only new" monitoring of sites by their feeds or sitemaps with one predictable output format, at a low per-article price.

Is it legal to extract articles with it?

It fetches only the pages, feeds and sitemaps of the sites you name, as a normal logged-out visitor, and follows each site's robots.txt, including its opt-outs for AI crawlers. Feeds and sitemaps are published by site owners for machines to read. It doesn't log in or get around any protection. What you do with the content (copyright, the site's terms) is your responsibility, as with any page you save.

Related actors

Actor	Use it when
RSS & Atom Feed Reader	The feed already carries what you need (title, link, date, often the full text): it's cheaper and doesn't open the article pages. Use Article Extractor when feeds only give a summary (chain it after the feed reader to get the full text of each item), or a site has no feed.
Website & Page to Markdown for RAG and LLMs	You want documentation and other non-article pages, whole sites, or heading-based chunks for RAG. Article Extractor is for dated articles with an author.

Feedback and support

Found a bug, or need a field that isn't here? Open an issue on the Issues tab with the input you used.

Versions

Current version: 1.1. See the Changelog tab for what changed in each version.