# AI Article Extractor (`headlessagent/ai-article-extractor`) Actor

Turn article URLs, sitemaps and RSS/Atom feeds into clean Markdown/text/HTML with metadata and optional RAG chunks. Browser-free HTTP extraction.

- **URL**: https://apify.com/headlessagent/ai-article-extractor.md
- **Developed by:** [Headless Agent](https://apify.com/headlessagent) (community)
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.50 / 1,000 results

This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## AI Article Extractor

Turn public web pages into clean article text you can read, save, or feed into an AI tool. Give it article links, a sitemap, or an RSS/Atom feed, and it returns one tidy row per article with the title, author, date, language, word count, and the full article body. It is for anyone who wants the useful part of a page without menus, ads, cookie banners, or other clutter.

### What you get

One result row per article, with these fields:

- **Link** — the address you asked for, plus the page's own preferred address for the same article when it provides one.
- **Title** — the article headline.
- **Author** — the byline, or several names separated by commas.
- **Published date** — in a standard date format when the page gives a readable date, otherwise the date as shown on the page.
- **Site name** — the publication or website name.
- **Language** — a short language code such as `en` or `en-US`.
- **Word count** — counted by spaces in the cleaned article text.
- **Article body** — the main text only. You choose Markdown (the default, with working links, headings, lists, tables, and code blocks), plain text, or cleaned HTML.
- **Chunks** — optional smaller pieces of the article, each with a piece number, the text, an approximate token count (a token is a small unit of text used by AI tools), and a character count.
- **Status** — `ok` means the main article was found; `fallback` means a simpler reading method found it; `failed` means no usable article text was found, but the row is still kept so you can see what was missed.
- **Reason** — a short explanation when a page failed or only partly worked.
- **Fetched at** — when the row was created.

### How to use

1. Open the actor in Apify Console.
2. Fill in the input form. You need at least one source: article links, a sitemap, or a feed.
   - **Article URLs** — paste full web addresses, one per line. You can combine these with a sitemap or feed. If this is empty, the other sources are used; if all three source fields are empty, the run stops and tells you to add a source.
   - **Sitemap URL** — paste the address of an XML sitemap. It can be a sitemap that points to other sitemaps; up to 25 sitemap files are followed, three levels deep. Leave it empty to skip sitemaps. Links that look like files, such as images, PDFs, or ZIP archives, are ignored.
   - **RSS/Atom feed URL** — paste the address of an RSS or Atom feed to use its entries. Leave it empty to skip feeds.
   - **Max articles** — the total number of articles to process, from 1 to 1000. The default is 10. Empty means 10. Sources are read in this order: article links, then sitemap, then feed. Once the limit is reached, later sources are not read.
   - **Output format** — choose Markdown, plain text, or HTML. Empty means Markdown.
   - **Chunk size** — optional. Set a number from 32 to 8000 to split each article into smaller pieces of roughly that many tokens (small units of text used by AI tools). Empty means no chunks.
   - **Request timeout** — how long to wait for one page, from 1 to 300 seconds. Empty means 30 seconds.
   - **Max requests per minute** — a politeness limit from 1 to 600 requests. Empty means 60.
   - **Respect robots.txt** — on by default. When on, pages the site's robots.txt rules disallow are skipped and do not appear in the results. Leave it on unless you own the site.
3. Press **Start**.
4. Wait for the run to finish. The run shows how many articles it has processed. Large runs take longer because requests are spread out.
5. Open the **Output** or **Dataset** tab. Each article is one row. Download the results as JSON, CSV, or Excel.
6. You can also fetch results through the Apify API or connect them to other tools, without writing code.

### Example input

```json
{
    "urls": ["https://blog.apify.com/what-is-web-scraping/"],
    "maxArticles": 1,
    "chunkSize": 200
}
```

### Example output

The example below is shortened: the article text and the chunk texts are cut, and the author's name is replaced with a fake name.

```json
{
    "url": "https://blog.apify.com/what-is-web-scraping/",
    "canonicalUrl": "https://blog.apify.com/what-is-web-scraping/",
    "title": "What is web scraping?",
    "author": "Jane Doe",
    "publishedAt": "2024-10-15T14:32:00.000Z",
    "siteName": "Apify Blog",
    "language": "en",
    "wordCount": 1427,
    "outputFormat": "markdown",
    "extractionStatus": "ok",
    "extractionError": null,
    "fetchedAt": "2026-10-27T06:00:13.627Z",
    "markdown": "Web scraping is the process of automatically extracting data from a website. You use a program called a web scraper to access a web page, interpret the data, and extract what you need. The data is…",
    "chunks": [
        {
            "index": 0,
            "text": "Web scraping is the process of automatically extracting data from a website. You use a program called a web scraper to access a web page, interpret the data, and extract what you need. The data is…",
            "tokenCount": 165,
            "charCount": 659
        },
        {
            "index": 1,
            "text": "Web scraping is the **automated process of extracting data from a website.** Web scraping is also known as **web harvesting**, **data extraction**, **web data extraction, data scraping,** and **data…",
            "tokenCount": 94,
            "charCount": 376
        }
    ]
}
```

This output comes from a real run of the actor on that link.

### Tips and limits

- **Public pages only.** Login-only, private, and paywalled content cannot be read.
- **Some pages need extra steps to show their text.** The actor reads the page as it is first delivered, so those pages may come back empty. It never invents article content.
- **Some sites block automated visits** or show different content to them. A blocked request is retried twice, then its row is marked `failed` with the reason.
- **Missing details stay empty.** Some pages do not state an author, date, or language. A date may be the page's last-updated date rather than the original publish date.
- **Images, videos, and formulas are left out of Markdown and plain text.** HTML output keeps images. Links, tables, lists, and code blocks are kept.
- **Duplicate links are compared after cleanup.** A trailing `#section` and common tracking parameters are removed first. The same article under two different addresses still produces two rows.
- **Pages disallowed by robots.txt are skipped without a row.** This is why the number of rows can be lower than the number of links you supplied.
- **Sitemaps are taken in their listed order**, not newest first. Up to 25 sitemap files are read, three levels deep.
- **Word and token counts are estimates.** The word count uses spaces, so languages that do not separate words with spaces are undercounted. Chunk token counts are approximate and do not match any one AI model exactly.
- **A large run can be slow.** The default limit is 60 requests per minute, so 1,000 articles can take a while. Start with a small number to check the output.
- **The actor does not summarize, translate, or rewrite articles.** It collects and cleans what the page already contains. It does not follow links inside an article or crawl beyond the sources you provide.
- **Use content responsibly.** Respect each site's terms, robots.txt rules, and copyright. Extracted pages may contain personal data, so collect only what you need and follow the privacy rules that apply to you.

### FAQ

**Do I need to know how to code?**
No. Fill in the form, press Start, and download the results from the Output or Dataset tab.

**Why did an article fail?**
The page may be blocked, paywalled, empty, or only show its text after extra page steps. Open the row's reason field to see what happened. If a page was disallowed by robots.txt, it is skipped and has no row at all.

**Can I process a whole blog?**
Yes, use its sitemap. The actor follows up to 25 sitemap files, three levels deep, and the max articles setting caps the total. For a few pages, pasting article links is simpler.

**Can I get images in the output?**
Not in Markdown or plain text. Choose HTML if you need the cleaned article with its images.

**Are the chunks ready to use in my AI tool?**
They are plain pieces of the article, split near paragraphs or sentences and never overlapping. The token counts are estimates, so check them against your tool's limits before uploading very large sets.

# Actor input Schema

## `urls` (type: `array`):

Direct article URLs to extract. Combined with sitemapUrl/feedUrl when both are provided.

## `sitemapUrl` (type: `string`):

XML sitemap or sitemap index. Nested sitemap indexes are followed (up to 25 sitemaps, maxArticles caps the result).

## `feedUrl` (type: `string`):

RSS 2.0 or Atom feed used to discover article URLs.

## `maxArticles` (type: `integer`):

Upper bound on the number of articles processed per run.

## `outputFormat` (type: `string`):

Body field emitted per article. Markdown preserves links and code; text strips markup; html is cleaned article HTML with absolute links.

## `chunkSize` (type: `integer`):

Optional. Splits the article body into RAG-friendly chunks of roughly this many tokens (paragraph/sentence boundaries preferred, no overlap). Leave empty to skip chunking.

## `requestTimeoutSecs` (type: `integer`):

Timeout for a single page request.

## `maxRequestsPerMinute` (type: `integer`):

Politeness limit for page requests.

## `respectRobotsTxt` (type: `boolean`):

Skip pages disallowed by robots.txt (Crawlee checks them). Leave enabled unless you own the target site.

## Actor input object example

```json
{
  "urls": [
    "https://en.wikipedia.org/wiki/Web_scraping",
    "https://blog.apify.com/what-is-web-scraping/"
  ],
  "maxArticles": 10,
  "outputFormat": "markdown",
  "requestTimeoutSecs": 30,
  "maxRequestsPerMinute": 60,
  "respectRobotsTxt": true
}
```

# Actor output Schema

## `results` (type: `string`):

Dataset with one record per article: metadata, Markdown/text/HTML body, and optional RAG chunks.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "urls": [
        "https://en.wikipedia.org/wiki/Web_scraping",
        "https://blog.apify.com/what-is-web-scraping/"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("headlessagent/ai-article-extractor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "urls": [
        "https://en.wikipedia.org/wiki/Web_scraping",
        "https://blog.apify.com/what-is-web-scraping/",
    ] }

# Run the Actor and wait for it to finish
run = client.actor("headlessagent/ai-article-extractor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "urls": [
    "https://en.wikipedia.org/wiki/Web_scraping",
    "https://blog.apify.com/what-is-web-scraping/"
  ]
}' |
apify call headlessagent/ai-article-extractor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,headlessagent/ai-article-extractor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/TcbdiKN9jQi02opZB/builds/tcWI6gqCX5lTJHna2/openapi.json