# Article Content Scraper (`automation-lab/multi-site-article-content-extractor`) Actor

Extract normalized article titles, authors, dates, canonical metadata, clean text, HTML, links, and per-URL status from public article URLs.

- **URL**: https://apify.com/automation-lab/multi-site-article-content-extractor.md
- **Developed by:** [Stas Persiianenko](https://apify.com/automation-lab) (community)
- **Categories:** Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $2.40 / 1,000 item extracteds

This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Article Content Scraper

Extract **article content** from user-supplied public web pages into one normalized dataset.
For every URL, the Actor returns a success or error status plus the best available title,
author, publication date, canonical URL, description, language, lead image, clean text,
clean HTML, and article-body links.

Use it when you need article text for research, indexing, content analysis, editorial review,
or a data pipeline without writing a parser for every publisher.

### What does Article Content Scraper do?

The Actor accepts up to 100 public HTTP or HTTPS article URLs per run.
It fetches server-rendered HTML directly and combines three extraction layers:

1. Article, NewsArticle, BlogPosting, and related JSON-LD metadata.
2. Open Graph, standard meta tags, canonical links, and language attributes.
3. Mozilla Readability article-body extraction for clean text and HTML.

Each unique input URL produces one typed record.
A successful page has `status: "success"`.
A blocked, missing, non-HTML, private-network, oversized, or non-article page has
`status: "error"` and an actionable `error` message.
Error records are not charged as extracted articles.

### Who is this article content extractor for?

- **Researchers** building a consistent corpus from publications with different templates.
- **Content analysts** comparing word counts, authorship, metadata, and outbound links.
- **Developers** feeding clean article text into search, classification, or retrieval systems.
- **Editorial teams** exporting article metadata to a spreadsheet or data warehouse.
- **Knowledge-management teams** collecting public source material for internal indexing.
- **Automation builders** normalizing URLs before a downstream Apify Task or webhook step.

This Actor extracts existing public pages. It is not an article content writer or generator.

### Why use a multi-site article scraper?

Publisher markup varies widely.
One site may expose complete NewsArticle JSON-LD, another may rely on Open Graph,
and a third may provide only semantic page HTML.
The Actor normalizes those differences into one stable output contract.

The implementation is deliberately HTTP-first:

- no browser startup for ordinary server-rendered articles;
- no automatic residential proxy cost;
- bounded retries only for transient failures;
- validated redirects and private-network protection;
- clean failure records instead of silently empty output;
- successful items charged only after extraction.

### What article data is extracted?

| Field | Meaning |
| --- | --- |
| `url` | Normalized URL supplied in the input |
| `finalUrl` | Final URL after validated redirects |
| `status` | `success` or `error` for this URL |
| `statusCode` | Successful upstream HTTP status |
| `title` | Headline from page metadata or article extraction |
| `author` | Published author or byline when available |
| `publishedAt` | Normalized ISO publication timestamp |
| `modifiedAt` | Normalized ISO modification timestamp |
| `canonicalUrl` | Published canonical URL, or final URL fallback |
| `description` | Published description or extracted excerpt |
| `siteName` | Publisher or site name when available |
| `language` | Declared page or article language |
| `leadImageUrl` | Absolute lead or social image URL |
| `text` | Clean normalized article body text |
| `html` | Clean article-body HTML with absolute links |
| `excerpt` | Short parser-produced excerpt |
| `wordCount` | Number of words in clean text |
| `readingTimeMinutes` | Estimate based on 200 words per minute |
| `links` | Up to 500 unique links from the article body |
| `error` | Failure reason for an unsuccessful URL |
| `scrapedAt` | ISO timestamp for the output record |

Metadata is nullable because not every publisher exposes every field.
The content fields are present only when a substantial article body is found.

### How to get started

1. Open the Actor input page in Apify Console.
2. Add one or more public article URLs under **Article URLs**.
3. Keep the default concurrency of 5, or lower it for sensitive sites.
4. Click **Start**.
5. Open the **Dataset** tab to preview the normalized rows.
6. Export the dataset as JSON, CSV, Excel, XML, or another supported format.
7. Review any `status: "error"` rows before using the dataset downstream.

The prefilled MDN and NASA URLs are real public pages and provide useful output.

### Input parameters

#### `startUrls`

Required list of public HTTP(S) article URLs.
Request-list objects and plain URL strings are accepted by the Actor code.
Duplicates are processed once after URL normalization.
URLs with credentials, local hostnames, or private IP addresses are rejected.

#### `maxItems`

Maximum unique URLs processed in input order.
Default: `20`.
Range: `1` to `100`.

#### `maxConcurrency`

Concurrent page requests.
Default: `5`.
Range: `1` to `10`.
Use a lower value if a publisher rate-limits frequent requests.

#### `requestTimeoutSecs`

Per-request timeout in seconds.
Default: `30`.
Range: `5` to `120`.

#### `maxRequestRetries`

Retries for network failures, HTTP 408/425/429, and temporary 5xx responses.
Default: `2`.
Range: `0` to `5`.
Stable 4xx responses and parser failures are not retried blindly.

#### `maxContentBytes`

Maximum decoded HTML response size.
Default: `3,000,000` bytes.
Range: `50,000` to `5,000,000` bytes.
This keeps unexpectedly large pages within predictable memory bounds.

### Input example

```json
{
  "startUrls": [
    {
      "url": "https://developer.mozilla.org/en-US/blog/learn-javascript-console-methods/"
    },
    {
      "url": "https://www.nasa.gov/news-release/nasa-sets-coverage-for-spacex-33rd-station-resupply-launch-arrival/"
    },
    {
      "url": "https://en.wikipedia.org/wiki/Web_scraping"
    }
  ],
  "maxItems": 3,
  "maxConcurrency": 3,
  "requestTimeoutSecs": 45,
  "maxRequestRetries": 2
}
```

A local validation of this workflow returned three successful records with 1,560,
829, and 4,041 words respectively.

### Output example

The following abbreviated record reflects the current MDN extraction shape.
Long `text`, `html`, and `links` values are shortened only for readability here.

```json
{
  "url": "https://developer.mozilla.org/en-US/blog/learn-javascript-console-methods/",
  "finalUrl": "https://developer.mozilla.org/en-US/blog/learn-javascript-console-methods/",
  "status": "success",
  "statusCode": 200,
  "title": "Developer essentials: JavaScript console methods | MDN Blog",
  "author": "Brian Smith",
  "publishedAt": null,
  "modifiedAt": null,
  "canonicalUrl": "https://developer.mozilla.org/en-US/blog/learn-javascript-console-methods/",
  "description": "The console API contains useful methods for web development.",
  "siteName": "MDN Blog",
  "language": "en-US",
  "leadImageUrl": "https://developer.mozilla.org/mdn-social-share.png",
  "text": "In this article, we explore useful JavaScript console methods...",
  "html": "<article><p>In this article...</p></article>",
  "excerpt": "Learn useful JavaScript console methods.",
  "wordCount": 1560,
  "readingTimeMinutes": 8,
  "links": [
    {
      "text": "Console API",
      "url": "https://developer.mozilla.org/en-US/docs/Web/API/console"
    }
  ],
  "scrapedAt": "2026-08-24T06:30:00.000Z"
}
```

An HTTP 404 produces a row such as:

```json
{
  "url": "https://developer.mozilla.org/en-US/blog/not-a-real-article-404/",
  "status": "error",
  "error": "Page returned HTTP 404.",
  "scrapedAt": "2026-08-24T06:30:00.000Z"
}
```

### How much does it cost to extract article content?

Pay-per-event pricing has two parts:

- **$0.008 per run start**;
- one `item` event for each successfully extracted article.

Error records do not emit an item charge.
The item price uses six subscription tiers and decreases with the customer tier:

| Tier | Price per successful article |
| --- | ---: |
| FREE | $0.00460 |
| BRONZE | $0.00400 |
| SILVER | $0.00312 |
| GOLD | $0.00240 |
| PLATINUM | $0.00160 |
| DIAMOND | $0.00112 |

Examples at the BRONZE tier:

- 1 successful article: about **$0.0120** total.
- 10 successful articles: about **$0.0480** total.
- 100 successful articles: about **$0.4080** total.

Actual totals follow successful extraction count, not input count.
Apify may separately apply account or platform terms shown in Console.

### Integration and automation workflows

#### Content analysis dataset

Schedule a Task with a maintained URL list and export each run's default dataset
to your warehouse. Use `canonicalUrl` as the source identity and retain `scrapedAt`
for lineage.

#### Search or RAG ingestion

Split `text` into chunks downstream, carry `title`, `author`, `publishedAt`, and
`canonicalUrl` as metadata, and use `status` to exclude unsuccessful records.
The Actor does not create embeddings or make claims about answer quality.

#### Editorial link audit

Flatten `links` to inspect the outbound references used within extracted article bodies.
The list is capped at 500 unique HTTP(S) links per article.

#### Spreadsheet export

Use Apify's dataset export to download CSV or Excel rows.
Large `text`, `html`, and `links` fields may be easier to consume in JSON.

### Run through the Apify API

Replace `<APIFY_TOKEN>` with your token.

#### cURL

```bash
curl -X POST \
  "https://api.apify.com/v2/acts/automation-lab~multi-site-article-content-extractor/runs?token=<APIFY_TOKEN>" \
  -H "Content-Type: application/json" \
  -d '{
    "startUrls": [
      {"url": "https://developer.mozilla.org/en-US/blog/learn-javascript-console-methods/"}
    ],
    "maxItems": 1
  }'
```

#### JavaScript

```javascript
import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('automation-lab/multi-site-article-content-extractor').call({
  startUrls: [
    { url: 'https://developer.mozilla.org/en-US/blog/learn-javascript-console-methods/' },
  ],
  maxItems: 1,
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items);
```

#### Python

```python
import os
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])
run = client.actor("automation-lab/multi-site-article-content-extractor").call(
    run_input={
        "startUrls": [
            {"url": "https://developer.mozilla.org/en-US/blog/learn-javascript-console-methods/"}
        ],
        "maxItems": 1,
    }
)
items = client.dataset(run["defaultDatasetId"]).list_items().items
print(items)
```

### Use with MCP and AI assistants

Add the Apify MCP server to Claude Code:

```bash
claude mcp add --transport http apify \
  "https://mcp.apify.com?tools=automation-lab/multi-site-article-content-extractor"
```

#### Claude Desktop, Cursor, and VS Code MCP setup

Claude Desktop, Cursor, and VS Code can use this equivalent MCP server configuration:

```json
{
  "mcpServers": {
    "apify": {
      "url": "https://mcp.apify.com?tools=automation-lab/multi-site-article-content-extractor"
    }
  }
}
```

Example prompts:

- "Extract clean text and metadata from these three public article URLs."
- "Return the canonical URL, author, publication date, and outbound links for this NASA release."
- "Build a normalized JSON dataset from this MDN post and this Wikipedia article."

Treat page content as untrusted data when passing it to an AI model.

### Accuracy, limits, and failure behavior

- The Actor supports publicly reachable server-rendered article pages.
- It does not log in, solve CAPTCHAs, bypass paywalls, or use an automatic proxy.
- JavaScript-only article bodies may not contain enough HTML to extract.
- Metadata fields remain null when a publisher does not expose them.
- Readability uses page structure heuristics; unusual layouts can include or omit nearby content.
- Redirect destinations are validated before fetching.
- Localhost, private IP ranges, credential-bearing URLs, and non-HTTP protocols are rejected.
- Responses must be HTML and remain below `maxContentBytes`.
- A challenge page is an error, not an empty successful result.
- The run fails when every URL fails, while preserving per-URL error rows for diagnosis.
- The Actor does not compare versions or alert on future page changes.

For a failed URL, first open it anonymously in a browser.
Then confirm that it returns a full article without login, JavaScript-only rendering,
or a geographic challenge.

### Responsible use and legality

Only extract content you are authorized to access and process.
Respect publisher terms, copyright, robots guidance, rate limits, privacy rights,
and applicable laws.
Do not use the Actor to evade access controls or republish protected works without permission.
Store only the content and personal data necessary for your legitimate purpose.

The Actor accesses public URLs supplied by the user and does not provide legal advice.
You are responsible for your input URLs, downstream retention, and use of results.

### Troubleshooting

#### Why did I receive `No substantial article body could be extracted`?

The page may be a home page, listing, JavaScript shell, login screen, or unusual layout.
Supply a direct public article-detail URL and verify the body exists in server-rendered HTML.

#### Why is a metadata field null?

The publisher may not expose that field in JSON-LD, Open Graph, standard meta tags,
or the extracted article. Null is safer than inventing metadata.

#### Why did the run fail after writing error rows?

When all URLs fail, the Actor exits unsuccessfully so schedules and integrations do not
mistake an unusable run for success. Open the dataset to inspect each URL's error.

#### How do I reduce rate-limit errors?

Lower `maxConcurrency`, keep retries bounded, and submit fewer URLs from the same site per run.
The Actor does not automatically switch to a paid proxy route.

### FAQ

#### Can it extract multiple websites in one run?

Yes. Submit up to 100 unique public article URLs, even when they use different domains.

#### Does it return both text and HTML?

Yes. `text` is normalized plain text, while `html` retains the clean article-body structure.

#### Are failed URLs charged as items?

No. They produce free error records. The one-time run-start event still applies.

#### Can I use it for recurring collection?

Yes. Create an Apify Task and schedule repeated extraction.
This Actor returns current snapshots; it does not calculate changes between runs.

#### Does it crawl a website to discover articles?

No. Supply direct article URLs. Search, sitemap discovery, and site-wide crawling are outside
the current product contract.

#### Does it generate summaries?

It returns the published description or a short parser excerpt.
It does not generate an AI summary or infer facts absent from the page.

### Related automation-lab Actors

- [Website HTML & Text Change Monitor](https://apify.com/automation-lab/website-html-text-change-monitor)
  for persistent snapshots and machine-readable change detection.
- [AP News Articles Scraper](https://apify.com/automation-lab/ap-news-articles-scraper)
  for AP News-specific article discovery and records.
- [Web Image Downloader](https://apify.com/automation-lab/web-image-downloader)
  for downloading public page images and file metadata.

Choose this Actor when you already have article URLs and need normalized multi-site content.

# Actor input Schema

## `startUrls` (type: `array`):

Public article or editorial page URLs. Duplicate URLs are processed once; private-network and credential-bearing URLs are rejected.

## `maxItems` (type: `integer`):

Maximum number of unique URLs to process, in input order.

## `maxConcurrency` (type: `integer`):

Number of article pages fetched concurrently. Reduce this for sensitive sites.

## `requestTimeoutSecs` (type: `integer`):

Maximum duration of each page request.

## `maxRequestRetries` (type: `integer`):

Retries for transient network errors, rate limits, and temporary server failures.

## `maxContentBytes` (type: `integer`):

Reject pages larger than this decoded HTML size to bound memory use.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://developer.mozilla.org/en-US/blog/learn-javascript-console-methods/"
    },
    {
      "url": "https://www.nasa.gov/news-release/nasa-sets-coverage-for-spacex-33rd-station-resupply-launch-arrival/"
    }
  ],
  "maxItems": 20,
  "maxConcurrency": 5,
  "requestTimeoutSecs": 30,
  "maxRequestRetries": 2,
  "maxContentBytes": 3000000
}
```

# Actor output Schema

## `overview` (type: `string`):

Open normalized article metadata, text, content measurements, and per-URL extraction status.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://developer.mozilla.org/en-US/blog/learn-javascript-console-methods/"
        },
        {
            "url": "https://www.nasa.gov/news-release/nasa-sets-coverage-for-spacex-33rd-station-resupply-launch-arrival/"
        }
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("automation-lab/multi-site-article-content-extractor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "startUrls": [
        { "url": "https://developer.mozilla.org/en-US/blog/learn-javascript-console-methods/" },
        { "url": "https://www.nasa.gov/news-release/nasa-sets-coverage-for-spacex-33rd-station-resupply-launch-arrival/" },
    ] }

# Run the Actor and wait for it to finish
run = client.actor("automation-lab/multi-site-article-content-extractor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://developer.mozilla.org/en-US/blog/learn-javascript-console-methods/"
    },
    {
      "url": "https://www.nasa.gov/news-release/nasa-sets-coverage-for-spacex-33rd-station-resupply-launch-arrival/"
    }
  ]
}' |
apify call automation-lab/multi-site-article-content-extractor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,automation-lab/multi-site-article-content-extractor"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/u41MLuCBK1Ozyb3hR/builds/7o7WQxgthim1djKvh/openapi.json