Article Content Scraper

Extract article content from user-supplied public web pages into one normalized dataset. For every URL, the Actor returns a success or error status plus the best available title, author, publication date, canonical URL, description, language, lead image, clean text, clean HTML, and article-body links.

Use it when you need article text for research, indexing, content analysis, editorial review, or a data pipeline without writing a parser for every publisher.

What does Article Content Scraper do?

The Actor accepts up to 100 public HTTP or HTTPS article URLs per run. It fetches server-rendered HTML directly and combines three extraction layers:

1. Article, NewsArticle, BlogPosting, and related JSON-LD metadata.
2. Open Graph, standard meta tags, canonical links, and language attributes.
3. Mozilla Readability article-body extraction for clean text and HTML.

Each unique input URL produces one typed record. A successful page has status: "success". A blocked, missing, non-HTML, private-network, oversized, or non-article page has status: "error" and an actionable error message. Error records are not charged as extracted articles.

Who is this article content extractor for?

• Researchers building a consistent corpus from publications with different templates.
• Content analysts comparing word counts, authorship, metadata, and outbound links.
• Developers feeding clean article text into search, classification, or retrieval systems.
• Editorial teams exporting article metadata to a spreadsheet or data warehouse.
• Knowledge-management teams collecting public source material for internal indexing.
• Automation builders normalizing URLs before a downstream Apify Task or webhook step.

This Actor extracts existing public pages. It is not an article content writer or generator.

Why use a multi-site article scraper?

Publisher markup varies widely. One site may expose complete NewsArticle JSON-LD, another may rely on Open Graph, and a third may provide only semantic page HTML. The Actor normalizes those differences into one stable output contract.

The implementation is deliberately HTTP-first:

• no browser startup for ordinary server-rendered articles;
• no automatic residential proxy cost;
• bounded retries only for transient failures;
• validated redirects and private-network protection;
• clean failure records instead of silently empty output;
• successful items charged only after extraction.

What article data is extracted?

Field	Meaning
url	Normalized URL supplied in the input
finalUrl	Final URL after validated redirects
status	success or error for this URL
statusCode	Successful upstream HTTP status
title	Headline from page metadata or article extraction
author	Published author or byline when available
publishedAt	Normalized ISO publication timestamp
modifiedAt	Normalized ISO modification timestamp
canonicalUrl	Published canonical URL, or final URL fallback
description	Published description or extracted excerpt
siteName	Publisher or site name when available
language	Declared page or article language
leadImageUrl	Absolute lead or social image URL
text	Clean normalized article body text
html	Clean article-body HTML with absolute links
excerpt	Short parser-produced excerpt
wordCount	Number of words in clean text
readingTimeMinutes	Estimate based on 200 words per minute
links	Up to 500 unique links from the article body
error	Failure reason for an unsuccessful URL
scrapedAt	ISO timestamp for the output record

Metadata is nullable because not every publisher exposes every field. The content fields are present only when a substantial article body is found.

How to get started

1. Open the Actor input page in Apify Console.
2. Add one or more public article URLs under Article URLs.
3. Keep the default concurrency of 5, or lower it for sensitive sites.
4. Click Start.
5. Open the Dataset tab to preview the normalized rows.
6. Export the dataset as JSON, CSV, Excel, XML, or another supported format.
7. Review any status: "error" rows before using the dataset downstream.

The prefilled MDN and NASA URLs are real public pages and provide useful output.

Input parameters

startUrls

Required list of public HTTP(S) article URLs. Request-list objects and plain URL strings are accepted by the Actor code. Duplicates are processed once after URL normalization. URLs with credentials, local hostnames, or private IP addresses are rejected.

maxItems

Maximum unique URLs processed in input order. Default: 20. Range: 1 to 100.

maxConcurrency

Concurrent page requests. Default: 5. Range: 1 to 10. Use a lower value if a publisher rate-limits frequent requests.

requestTimeoutSecs

Per-request timeout in seconds. Default: 30. Range: 5 to 120.

maxRequestRetries

Retries for network failures, HTTP 408/425/429, and temporary 5xx responses. Default: 2. Range: 0 to 5. Stable 4xx responses and parser failures are not retried blindly.

maxContentBytes

Maximum decoded HTML response size. Default: 3,000,000 bytes. Range: 50,000 to 5,000,000 bytes. This keeps unexpectedly large pages within predictable memory bounds.

Input example

{
  "startUrls": [
    {
      "url": "https://developer.mozilla.org/en-US/blog/learn-javascript-console-methods/"
    },
    {
      "url": "https://www.nasa.gov/news-release/nasa-sets-coverage-for-spacex-33rd-station-resupply-launch-arrival/"
    },
    {
      "url": "https://en.wikipedia.org/wiki/Web_scraping"
    }
  ],
  "maxItems": 3,
  "maxConcurrency": 3,
  "requestTimeoutSecs": 45,
  "maxRequestRetries": 2
}

A local validation of this workflow returned three successful records with 1,560, 829, and 4,041 words respectively.

Output example

The following abbreviated record reflects the current MDN extraction shape. Long text, html, and links values are shortened only for readability here.

{
  "url": "https://developer.mozilla.org/en-US/blog/learn-javascript-console-methods/",
  "finalUrl": "https://developer.mozilla.org/en-US/blog/learn-javascript-console-methods/",
  "status": "success",
  "statusCode": 200,
  "title": "Developer essentials: JavaScript console methods | MDN Blog",
  "author": "Brian Smith",
  "publishedAt": null,
  "modifiedAt": null,
  "canonicalUrl": "https://developer.mozilla.org/en-US/blog/learn-javascript-console-methods/",
  "description": "The console API contains useful methods for web development.",
  "siteName": "MDN Blog",
  "language": "en-US",
  "leadImageUrl": "https://developer.mozilla.org/mdn-social-share.png",
  "text": "In this article, we explore useful JavaScript console methods...",
  "html": "<article><p>In this article...</p></article>",
  "excerpt": "Learn useful JavaScript console methods.",
  "wordCount": 1560,
  "readingTimeMinutes": 8,
  "links": [
    {
      "text": "Console API",
      "url": "https://developer.mozilla.org/en-US/docs/Web/API/console"
    }
  ],
  "scrapedAt": "2026-08-24T06:30:00.000Z"
}

An HTTP 404 produces a row such as:

{
  "url": "https://developer.mozilla.org/en-US/blog/not-a-real-article-404/",
  "status": "error",
  "error": "Page returned HTTP 404.",
  "scrapedAt": "2026-08-24T06:30:00.000Z"
}

How much does it cost to extract article content?

Pay-per-event pricing has two parts:

• $0.008 per run start;
• one item event for each successfully extracted article.

Error records do not emit an item charge. The item price uses six subscription tiers and decreases with the customer tier:

Tier	Price per successful article
FREE	$0.00460
BRONZE	$0.00400
SILVER	$0.00312
GOLD	$0.00240
PLATINUM	$0.00160
DIAMOND	$0.00112

Examples at the BRONZE tier:

• 1 successful article: about $0.0120 total.
• 10 successful articles: about $0.0480 total.
• 100 successful articles: about $0.4080 total.

Actual totals follow successful extraction count, not input count. Apify may separately apply account or platform terms shown in Console.

Integration and automation workflows

Content analysis dataset

Schedule a Task with a maintained URL list and export each run's default dataset to your warehouse. Use canonicalUrl as the source identity and retain scrapedAt for lineage.

Search or RAG ingestion

Split text into chunks downstream, carry title, author, publishedAt, and canonicalUrl as metadata, and use status to exclude unsuccessful records. The Actor does not create embeddings or make claims about answer quality.

Editorial link audit

Flatten links to inspect the outbound references used within extracted article bodies. The list is capped at 500 unique HTTP(S) links per article.

Spreadsheet export

Use Apify's dataset export to download CSV or Excel rows. Large text, html, and links fields may be easier to consume in JSON.

Run through the Apify API

Replace <APIFY_TOKEN> with your token.

cURL

curl -X POST \
  "https://api.apify.com/v2/acts/automation-lab~multi-site-article-content-extractor/runs?token=<APIFY_TOKEN>" \
  -H "Content-Type: application/json" \
  -d '{
    "startUrls": [
      {"url": "https://developer.mozilla.org/en-US/blog/learn-javascript-console-methods/"}
    ],
    "maxItems": 1
  }'

JavaScript

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('automation-lab/multi-site-article-content-extractor').call({
  startUrls: [
    { url: 'https://developer.mozilla.org/en-US/blog/learn-javascript-console-methods/' },
  ],
  maxItems: 1,
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items);

Python

import os
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])
run = client.actor("automation-lab/multi-site-article-content-extractor").call(
    run_input={
        "startUrls": [
            {"url": "https://developer.mozilla.org/en-US/blog/learn-javascript-console-methods/"}
        ],
        "maxItems": 1,
    }
)
items = client.dataset(run["defaultDatasetId"]).list_items().items
print(items)

Use with MCP and AI assistants

Add the Apify MCP server to Claude Code:

claude mcp add --transport http apify \
  "https://mcp.apify.com?tools=automation-lab/multi-site-article-content-extractor"

Claude Desktop, Cursor, and VS Code MCP setup

Claude Desktop, Cursor, and VS Code can use this equivalent MCP server configuration:

{
  "mcpServers": {
    "apify": {
      "url": "https://mcp.apify.com?tools=automation-lab/multi-site-article-content-extractor"
    }
  }
}

Example prompts:

• "Extract clean text and metadata from these three public article URLs."
• "Return the canonical URL, author, publication date, and outbound links for this NASA release."
• "Build a normalized JSON dataset from this MDN post and this Wikipedia article."

Treat page content as untrusted data when passing it to an AI model.

Accuracy, limits, and failure behavior

• The Actor supports publicly reachable server-rendered article pages.
• It does not log in, solve CAPTCHAs, bypass paywalls, or use an automatic proxy.
• JavaScript-only article bodies may not contain enough HTML to extract.
• Metadata fields remain null when a publisher does not expose them.
• Readability uses page structure heuristics; unusual layouts can include or omit nearby content.
• Redirect destinations are validated before fetching.
• Localhost, private IP ranges, credential-bearing URLs, and non-HTTP protocols are rejected.
• Responses must be HTML and remain below maxContentBytes.
• A challenge page is an error, not an empty successful result.
• The run fails when every URL fails, while preserving per-URL error rows for diagnosis.
• The Actor does not compare versions or alert on future page changes.

For a failed URL, first open it anonymously in a browser. Then confirm that it returns a full article without login, JavaScript-only rendering, or a geographic challenge.

Responsible use and legality

Only extract content you are authorized to access and process. Respect publisher terms, copyright, robots guidance, rate limits, privacy rights, and applicable laws. Do not use the Actor to evade access controls or republish protected works without permission. Store only the content and personal data necessary for your legitimate purpose.

The Actor accesses public URLs supplied by the user and does not provide legal advice. You are responsible for your input URLs, downstream retention, and use of results.

Troubleshooting

Why did I receive No substantial article body could be extracted?

The page may be a home page, listing, JavaScript shell, login screen, or unusual layout. Supply a direct public article-detail URL and verify the body exists in server-rendered HTML.

Why is a metadata field null?

The publisher may not expose that field in JSON-LD, Open Graph, standard meta tags, or the extracted article. Null is safer than inventing metadata.

Why did the run fail after writing error rows?

When all URLs fail, the Actor exits unsuccessfully so schedules and integrations do not mistake an unusable run for success. Open the dataset to inspect each URL's error.

How do I reduce rate-limit errors?

Lower maxConcurrency, keep retries bounded, and submit fewer URLs from the same site per run. The Actor does not automatically switch to a paid proxy route.

FAQ

Can it extract multiple websites in one run?

Yes. Submit up to 100 unique public article URLs, even when they use different domains.

Does it return both text and HTML?

Yes. text is normalized plain text, while html retains the clean article-body structure.

Are failed URLs charged as items?

No. They produce free error records. The one-time run-start event still applies.

Can I use it for recurring collection?

Yes. Create an Apify Task and schedule repeated extraction. This Actor returns current snapshots; it does not calculate changes between runs.

Does it crawl a website to discover articles?

No. Supply direct article URLs. Search, sitemap discovery, and site-wide crawling are outside the current product contract.

Does it generate summaries?

It returns the published description or a short parser excerpt. It does not generate an AI summary or infer facts absent from the page.

Related automation-lab Actors

• Website HTML & Text Change Monitor for persistent snapshots and machine-readable change detection.
• AP News Articles Scraper for AP News-specific article discovery and records.
• Web Image Downloader for downloading public page images and file metadata.

Choose this Actor when you already have article URLs and need normalized multi-site content.