# Article Extractor – News & Blog Articles to JSON & Markdown (`gazidev/article-extractor`) Actor

Extract news and blog articles to clean JSON: title, author, published date, lead image, language, tags, text and Markdown. Give article URLs or a site URL: articles are found via RSS feeds, news sitemaps and section pages. Date filter and only-new mode for scheduled feeds. HTTP-only, fast, cheap.

- **URL**: https://apify.com/gazidev/article-extractor.md
- **Developed by:** [Cemal Atakli](https://apify.com/gazidev) (community)
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.30 / 1,000 article extracteds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Article Extractor – News & Blog Articles to JSON & Markdown

Extract **news articles and blog posts** to clean, structured JSON: **title, author, published and modified date, lead image, language, tags, section, clean text and Markdown**, plus word count, reading time and canonical URL. Paste **article URLs**, or just a **news site or blog URL** and the articles are found for you through its **RSS/Atom feeds, Google News sitemaps, XML sitemaps and section pages**. Add a date filter or the **only-new-articles mode** to get a scheduled news feed for media monitoring, AI/RAG pipelines and content research.

No browser, so it is fast and costs **$0.30 per 1,000 articles**. Pages that turn out not to be articles are free.

### What it does

- **Two inputs, mixed freely.**
  - **Article URLs** are extracted directly.
  - **Site or section URLs** (`https://github.blog`, `https://www.bbc.com/news`, `techcrunch.com`) are crawled for their newest articles:
    1. RSS / Atom / JSON feeds (`<link rel="alternate">`, `/feed`, `/rss.xml`, ...), newest first;
    2. Google News sitemaps and XML sitemaps listed in robots.txt (gzip and sitemap indexes supported; for a section URL like `/news` the matching sitemap is preferred);
    3. article-looking links on the start page.
- **Article detection.** Every discovered page is checked: schema.org `NewsArticle` / `BlogPosting` / `Article` JSON-LD, `og:type=article`, a publish date, enough words, and not a headline list. Home, tag, category, author and other listing pages are skipped and **not charged**.
- **Rich metadata** from JSON-LD (including `@graph` and author references), OpenGraph / `article:*` tags, Parse.ly and Dublin Core meta tags, `<time>` elements, bylines and author links, with the date in the URL as the last fallback (`dateSource` tells you where the date came from).
- **Clean body.** The article body container is located (`itemprop=articleBody`, `.entry-content`, `<article>` ...); navigation, ads, share bars, cookie banners and "Most popular / Related / Read more" blocks are removed. Output as plain `text` and as `markdown` (headings, lists, tables, code, quotes, optional links and images).
- **Date filter** (`publishedAfter`: `2026-10-01` or `7 days`). Old entries in feeds and sitemaps are dropped before downloading, so they cost nothing.
- **Only-new mode** for scheduled runs. Processed URLs are remembered in a named key-value store; the next run extracts only articles that were not seen before.
- **Per-URL errors instead of crashes.** 404s, PDFs, robots.txt blocks, JavaScript-only pages and paywalled articles become rows with a `status` and an `error` message. They are free.
- **Polite crawling.** Honest User-Agent with a contact address, robots.txt and Crawl-delay respected, 2 parallel requests per site by default, retries with backoff on 429/5xx.

### Use cases

- **AI / RAG / LLM pipelines:** feed clean Markdown with titles, dates and authors into a vector database or an LLM summarizer.
- **Media monitoring and news alerts:** schedule hourly with `onlyNew` and send new articles to Slack, email, Google Sheets or a webhook.
- **Content and SEO research:** publishing frequency, word counts, authors, tags and sections of competitor blogs.
- **Datasets:** build a news or blog corpus with consistent metadata for classification, sentiment or trend analysis.
- **Newsletters and digests:** collect this week's articles from a list of publishers in one table.

### Input example

```json
{
  "startUrls": [
    { "url": "https://github.blog" },
    { "url": "https://www.bbc.com/news" },
    { "url": "https://blog.cloudflare.com/post-quantum-geo-key-manager/" }
  ],
  "maxArticlesPerSite": 10,
  "publishedAfter": "3 days",
  "onlyNew": false
}
```

| Field | Default | Meaning |
|---|---|---|
| `startUrls` | – | Article URLs and/or site / section URLs |
| `mode` | `auto` | `auto`: article URLs are extracted, other URLs are crawled as sites. `site`: always discover. `urls`: extract exactly these URLs, no discovery |
| `maxArticlesPerSite` | 20 | Articles per site URL (newest first when dates are known) |
| `maxArticles` | 0 | Cap for the whole run (0 = none) |
| `publishedAfter` | – | Absolute or relative date |
| `onlyNew` / `stateKey` | false | Monitoring memory between runs |
| `useFeeds` / `useSitemaps` / `useLinks` | true | Discovery sources for site URLs |
| `minWords` | 150 | Discovered pages with fewer words are not articles |
| `includeText` / `includeMarkdown` | true | Which body formats to return |

### Output example

One row per article (text and Markdown shortened):

```json
{
  "url": "https://techcrunch.com/2026/10/25/grok-releases-the-first-seven-minutes-of-its-ai-generated-odyssey/",
  "canonicalUrl": "https://techcrunch.com/2026/10/25/grok-releases-the-first-seven-minutes-of-its-ai-generated-odyssey/",
  "status": "ok",
  "title": "Grok releases the first seven minutes of its AI-generated ‘Odyssey’",
  "description": "The footage looks okay, as long as you're not seeking artistry, inspiration, or human interest.",
  "author": "Anthony Ha",
  "authors": ["Anthony Ha"],
  "publishedAt": "2026-10-25T21:31:16Z",
  "modifiedAt": "2026-10-26T02:24:45Z",
  "dateSource": "json-ld",
  "leadImage": "https://techcrunch.com/wp-content/uploads/2026/03/grok-getty.jpg",
  "language": "en",
  "siteName": "TechCrunch",
  "section": "AI",
  "tags": ["Elon Musk", "SpaceX", "xAI"],
  "wordCount": 540,
  "readingTimeMinutes": 3,
  "text": "Grok releases the first seven minutes of its AI-generated ‘Odyssey’\n\nThree months after Elon Musk declared ...",
  "markdown": "# Grok releases the first seven minutes of its AI-generated ‘Odyssey’\n\nThree months after Elon Musk [declared](https://x.com/...) ...",
  "isArticle": true,
  "articleType": "NewsArticle",
  "detection": ["json-ld:NewsArticle", "og:type=article", "date:json-ld", "words:540"],
  "discoveredVia": "feed",
  "site": "techcrunch.com",
  "httpStatus": 200,
  "contentHash": "a8a01344a9377446…",
  "fetchedAt": "2026-10-26T11:59:09Z"
}
```

Rows that are not extracted articles have a `status` such as `failed`, `blocked`, `robotsBlocked`, `skipped` (not HTML), `empty`, `needsBrowser`, `paywalled`, `noArticlesFound` or `invalidUrl`, and an `error` text. The `OUTPUT` record in the key-value store has a per-start-URL summary: feeds and sitemaps used, candidates, pages fetched, articles, non-article pages, date-filtered and already-seen counts.

Dataset views: **Articles** (table with lead image), **Text & Markdown**, **Metadata**.

### Pricing

Pay per event. You pay only for extracted articles.

| Event | Price |
|---|---|
| Article extracted | **$0.0003** ($0.30 / 1,000) |
| Actor start | $0.00005 |

Free: non-article pages found during discovery, pages older than your date filter, already-seen URLs, errors, blocked or paywalled pages.

Compared with other Store Actors (public Store prices, 2026-10-26):

| Actor | What you pay per 1,000 articles |
|---|---|
| **Article Extractor (this Actor)** | **$0.30**, no compute charges |
| tropical_prune/article-news-extractor | $1.00 |
| automation-lab/news-article-extractor | $2.30 + $0.005 per run |
| lukaskrivka/article-extractor-smart | Free Actor, but you pay the platform compute of a browser-capable crawler |

Set **Maximum cost per run** in the run options to cap spending. The Actor stops cleanly when the limit is reached and never fetches more articles than the budget allows.

### FAQ

**Which sites work?**
Server-rendered news sites, magazines, company blogs, WordPress, Ghost, Substack-style and docs blogs: most of the web's articles. Pages that need JavaScript to show their text return `needsBrowser` and are not charged.

**How are the newest articles chosen?**
Feed entries and Google News sitemap entries carry dates, so they are sorted newest first. Regular sitemaps are ordered by `lastmod`. Links from the start page come next. With `publishedAfter`, entries known to be older are dropped before they are downloaded.

**What about paywalls?**
The Actor does not bypass paywalls or logins. Articles that declare `isAccessibleForFree: false` are returned as `paywalled` rows (free) without text. Metered sites may return only the public teaser.

**Which sites are refused?**
Medium (including Medium custom domains), Reddit, Google News and social networks (X/Twitter, Facebook, Instagram, LinkedIn, TikTok, YouTube, Pinterest, Threads, Quora), because their terms do not allow automated extraction. Such URLs return a `blocked` row and are not charged.

**How does only-new mode work?**
URLs that were extracted, skipped as non-articles or filtered by date are stored in a named key-value store (`article-extractor-…`, derived from your start URLs or from `stateKey`). On the first run you get the newest `maxArticlesPerSite` articles. The rest of that run's feed and sitemap entries are also marked as seen, so later runs return only articles published since, without back-filling the archive.

**Article URL mode.**
With `mode: "urls"` every URL you give is extracted as-is and charged when it has readable text, even if it does not look like an article (`isArticle` tells you what the detector thinks).

**Is it legal?**
The Actor only reads pages you point it to, identifies itself, and respects robots.txt and crawl delays. You are responsible for how you use the content (copyright, the site's terms, personal data in bylines). For republishing, get the publisher's permission.

### Limits

- No JavaScript rendering, logins, CAPTCHAs or anti-bot bypass. Sites behind a Cloudflare challenge return `failed` (HTTP 403).
- Language is taken from the page (`lang`, JSON-LD, `og:locale`), not detected from the text.
- Author detection relies on markup; a few sites only show the author in images or scripts.
- Discovery reads at most 10 sitemap files per site and a page budget of `3 × maxArticlesPerSite + 10` candidates (`maxPagesPerSite` to change it).

### Use with AI agents / Apify MCP

- **Apify MCP server:** add `gazidev/article-extractor` to your MCP client (Claude Desktop, Cursor, VS Code) via `https://mcp.apify.com?actors=gazidev/article-extractor`. An agent can call it with `{"startUrls":["https://www.bbc.com/news"],"maxArticlesPerSite":5,"publishedAfter":"1 day"}` and read today's articles as Markdown.
- **API:** `POST https://api.apify.com/v2/acts/gazidev~article-extractor/run-sync-get-dataset-items?token=...` with the input JSON returns the articles directly. It works as a "read this article" tool for LangChain, LlamaIndex or n8n.
- **Scheduled news feed:** create a Task with `onlyNew: true`, schedule it, and connect the Slack, Google Sheets or webhook integration.

### More from the website toolkit

- [RSS Feed Reader](https://apify.com/gazidev/rss-feed-reader): any RSS, Atom or JSON Feed to JSON, OPML import, only-new items.
- [Website to Markdown](https://apify.com/gazidev/website-to-markdown): any URL, sitemap or whole site to Markdown with RAG chunks and llms.txt.
- [Sitemap Extractor](https://apify.com/gazidev/sitemap-url-extractor): every URL from sitemap.xml, with status checks and only-new mode.

# Actor input Schema

## `startUrls` (type: `array`):

Article URLs and/or news or blog site URLs, mixed freely. For a site or section URL (e.g. https://github.blog or https://example.com/technology) articles are discovered through its RSS/Atom feeds, Google News and XML sitemaps and article links on the page. Medium, Reddit, Google News and social networks are not supported.

## `mode` (type: `string`):

'Auto' extracts a start URL directly when it is an article, otherwise discovers the articles of that site. 'Sites' always discovers articles (the start page is never output). 'Article URLs only' extracts exactly the URLs you give, without discovery.

## `maxArticlesPerSite` (type: `integer`):

For site URLs: how many articles to extract per site, newest first when the feed or sitemap has dates.

## `maxArticles` (type: `integer`):

Hard cap for the whole run (0 = no cap). Your cost is at most this many x the article price.

## `publishedAfter` (type: `string`):

Keep only articles published on or after this date. Absolute (2026-10-01) or relative (24 hours, 7 days). Older articles are skipped for free.

## `keepUndated` (type: `boolean`):

When a date filter is set: keep articles whose publish date cannot be found.

## `onlyNew` (type: `boolean`):

For scheduled runs: remembers every URL already processed (in a named key-value store derived from the start URLs) and extracts only articles that were not seen before. Seen and non-article pages are skipped without fetching and are free.

## `stateKey` (type: `string`):

Optional. By default the memory is tied to the exact list of start URLs. Set a name to keep the same memory when you edit the list.

## `useFeeds` (type: `boolean`):

Site URLs: read the site's feeds (link rel=alternate, /feed, /rss.xml ...). Fast and newest first.

## `useSitemaps` (type: `boolean`):

Site URLs: read Google News sitemaps and XML sitemaps listed in robots.txt (or the common locations).

## `useLinks` (type: `boolean`):

Site URLs: also take article-looking links from the start page (useful for section pages such as /technology).

## `maxPagesPerSite` (type: `integer`):

Safety cap on article candidates fetched per site, including pages that turn out not to be articles (those are free). 0 = automatic (3 x max articles + 10).

## `minWords` (type: `integer`):

Discovered pages with fewer words are treated as non-articles (not output, not charged).

## `includeText` (type: `boolean`):

Add the clean article text (no Markdown syntax) as 'text'.

## `includeMarkdown` (type: `boolean`):

Add the article body as Markdown (headings, lists, tables, code, quotes) as 'markdown'. Good for LLM / RAG pipelines.

## `keepLinksInMarkdown` (type: `boolean`):

Render links as [text](URL). Turn off for fewer tokens.

## `includeImagesInMarkdown` (type: `boolean`):

Render in-article images as ![alt](URL). The lead image is always returned in 'leadImage'.

## `respectRobots` (type: `boolean`):

Skip URLs disallowed by robots.txt and honour Crawl-delay (max 10 s). Strongly recommended.

## `maxConcurrency` (type: `integer`):

Parallel requests in total.

## `maxConcurrencyPerHost` (type: `integer`):

Parallel requests to one host. Keep it low to be polite.

## `requestTimeoutSecs` (type: `integer`):

Per-request timeout.

## `maxRetries` (type: `integer`):

Retries for timeouts, connection errors, 429 and 5xx responses (with backoff).

## `userAgent` (type: `string`):

Custom User-Agent header. Leave empty for the default, which identifies this Actor with a contact address.

## `proxyConfiguration` (type: `object`):

Optional. Not needed for most news sites and blogs.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://github.blog"
    },
    {
      "url": "https://blog.cloudflare.com/post-quantum-geo-key-manager/"
    }
  ],
  "mode": "auto",
  "maxArticlesPerSite": 3,
  "maxArticles": 0,
  "keepUndated": true,
  "onlyNew": false,
  "useFeeds": true,
  "useSitemaps": true,
  "useLinks": true,
  "maxPagesPerSite": 0,
  "minWords": 150,
  "includeText": true,
  "includeMarkdown": true,
  "keepLinksInMarkdown": true,
  "includeImagesInMarkdown": false,
  "respectRobots": true,
  "maxConcurrency": 10,
  "maxConcurrencyPerHost": 2,
  "requestTimeoutSecs": 30,
  "maxRetries": 2,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}
```

# Actor output Schema

## `overview` (type: `string`):

No description

## `text` (type: `string`):

No description

## `results` (type: `string`):

No description

## `summary` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://github.blog"
        },
        {
            "url": "https://blog.cloudflare.com/post-quantum-geo-key-manager/"
        }
    ],
    "maxArticlesPerSite": 3
};

// Run the Actor and wait for it to finish
const run = await client.actor("gazidev/article-extractor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": [
        { "url": "https://github.blog" },
        { "url": "https://blog.cloudflare.com/post-quantum-geo-key-manager/" },
    ],
    "maxArticlesPerSite": 3,
}

# Run the Actor and wait for it to finish
run = client.actor("gazidev/article-extractor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://github.blog"
    },
    {
      "url": "https://blog.cloudflare.com/post-quantum-geo-key-manager/"
    }
  ],
  "maxArticlesPerSite": 3
}' |
apify call gazidev/article-extractor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,gazidev/article-extractor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/ODmb8ih2OLIFLztoH/builds/EgUWSpSDQkdakTrju/openapi.json