# Article Extractor: Paywall, Split Pages & Tables (`aiqlabs/article-extractor`) Actor

Extracts article text and Markdown, and reports three things other extractors stay quiet about: a paywall that cut the body short, an article that continues on later pages, and tables and links that plain-text output destroys.

- **URL**: https://apify.com/aiqlabs/article-extractor.md
- **Developed by:** [Ai-Q Labs](https://apify.com/aiqlabs) (community)
- **Categories:** AI, Developer tools, Automation
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $5.00 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Article Extractor: Paywall, Split Pages & Tables

Give it article URLs. It returns the body as Markdown and plain text, with the
metadata — and it tells you three things the other extractors do not.

### What is different here

Every extractor on this store returns "clean article text". So does this one. The
difference is what it says about that text.

#### 1. Whether a paywall cut the body short

An extractor that returns the first three paragraphs of a metered article and calls
it a success is worse than one that fails, because the failure is invisible. Feed a
thousand of those into a corpus and it looks complete.

This Actor reports two separate fields, because they come apart constantly:

| Field | Meaning |
|---|---|
| `paywallDetected` | The page runs a paywall |
| `bodyTruncated` | **The text you are holding is short of what the page itself claims** |

Publishers who want to stay in Google ship the whole article in the HTML and hide it
with CSS, so a page can be paywalled with the body complete. A metered site on your
last free view ships three paragraphs with no paywall markup at all. Only the second
field is the one that should stop your pipeline.

Measured on a live New York Times article: `paywallDetected: true` from three
signals (`isAccessibleForFree: false`, `hasPart` naming `.meteredContent`, and a
`meteredcontent` class in the markup), `bodyTruncated: false`, 2,757 words extracted.

**It does not bypass anything.** The page is fetched the way any reader's browser
fetches it, and the only output is a verdict on what arrived. There is no login, no
cookie injection and no cache trick.

Truncation is proven against the page's own `articleBody` or `wordCount` in JSON-LD
when it publishes one — `coverageRatio` and `coverageBasis` show the arithmetic — and
otherwise inferred from named signals you can read in `paywallEvidence`.

#### 2. Whether the article continues on later pages

Long articles get split, and `rel="next"` is used for two unrelated things: the rest
of this article, and the next page of an index. Following it blindly appends a
different article to the body.

So a continuation has to earn the merge:

- same site, never a cross-domain hop
- both pages claim the same title, after `Page 2 of 5` decoration is removed
- the source page is not itself an index

When a merge is declined the reason is in `issues`, not swallowed. Real refusals from
the test run:

```
not the same article: titles differ ("html 4 01 specification" vs "about the html 4 specification")
this page is an index, so its "next" link is site pagination rather than a continuation
the link holds unrendered template syntax (/'.$next.')
```

That last one is a live WordPress blog whose `rel="next"` is literally
`'.$next.'` — valid as a relative URL, so nothing but a look at the string catches it.

When parts are merged, the header and footer each part repeats are dropped
(`repeatedBlocksRemoved`), and only at the edges of a part — a sentence repeated in
the middle of a body is the author, not furniture.

#### 3. Tables, links and code, kept

"Clean text" is where a table goes to die. A 40-row financial table becomes a run of
loose numbers with no columns, and the article's whole point goes with it.

Markdown output keeps:

- **tables** as Markdown tables, with the header row taken from the document's own
  `<thead>` or `<th>` rather than guessed, and `colspan` widened so values stay under
  their own column
- **links** as `[text](absolute-url)`, resolved against the page
- **code blocks** with their line breaks and language, from the `<pre>` as written
- nested lists, block quotes, figure captions, definition lists

`tableCount`, `linkCount` and `codeBlockCount` let you tell "this article has three
tables" from "this article had three tables".

#### 4. And: published or last updated

Everyone returns "publish date". A page rewritten two years after publication looks
identical in that output. This Actor returns `publishedAt` and `modifiedAt`
separately, plus `updatedAfterPublish` and `daysBetweenPublishAndUpdate`. Stamps
written minutes apart are the CMS saving both on publish, not a revision, so they do
not count.

### Also included

- **Is it even an article?** `pageKind` separates `article` from `listing`. A category
  page with eight teasers is not an article; a blog post with a "you may also like"
  rail is, even though both trip the same signals.
- **`extractionConfidence`** (`high` / `medium` / `low`) with `extractedVia` naming
  what chose the body: an `<article>` element, `itemprop="articleBody"`, or paragraph
  scoring.
- Title taken by **vote** across JSON-LD `headline`, `og:title`, `<title>` and `<h1>`
  rather than by rank. Wikipedia puts the Wikidata one-line description in `headline`,
  so ranking it first titles the web scraping article "data scraping used for
  extracting data from websites".
- Author, site name, section, language, description, hero image, tags, canonical,
  schema.org types, word count, reading time.
- **CJK counted per character**, so a Japanese article is not measured as one word.
- **robots.txt is honoured.** A disallowed page is reported with the rule that
  blocked it rather than dropped silently — and not downloaded.

### Input

```jsonc
{
  "urls": ["https://example.com/post"],
  "followSplitPages": true,        // merge continuation pages
  "maxPagesPerArticle": 10,
  "outputFormats": ["markdown", "text"],   // also "html" for the cleaned article HTML
  "respectRobotsTxt": true,
  "skipTruncatedByPaywall": false, // drop partial bodies instead of returning them
  "maxConcurrency": 8
}
```

Set `skipTruncatedByPaywall` when the output feeds a corpus that must not contain
half articles. Off by default so the finding stays visible.

### Output

One row per input URL. `SUMMARY` in the key-value store holds the totals: how many
were paywalled, how many came back cut short, how many were merged from several
pages, how many were not articles.

### Cost

Free to run — you pay only Apify platform usage. One request per article, plus one
robots.txt per host, plus one request per continuation page actually merged. No
browser, no JavaScript execution, no proxy.

### Verification

- 69 unit tests
- 26 checks against the live web on every release, covering a blog post, a
  reference page heavy with tables and links, a paywalled news article, a section
  page, a blog index, a Japanese WordPress post and a chaptered specification —
  including the negative controls that an index is not merged as one article and a
  free post is not reported as paywalled

Defects these live checks found and that the unit tests did not: a blog index whose
six pagination pages were merged into "one article", a WordPress post reported as
`not_an_article` while holding a full body, Wikipedia titled by its JSON-LD
description, MediaWiki maintenance notices treated as body text, and a crash in
Node's own HTTP client when a keep-alive socket closes (`assert(!this.paused)` in
undici's HTTP/1 parser) — measured across eight consecutive runs per approach and
guarded explicitly, since it fires from a socket event where nothing can catch it.

# Actor input Schema

## `urls` (type: `array`):

Article, blog post or news pages to extract. Full URLs (https://example.com/post) or bare hosts both work, and one entry may hold several URLs separated by newlines, commas or spaces.

## `followSplitPages` (type: `boolean`):

When a page declares a continuation (rel="next", or a "Page 2 of 5" style pager), fetch the rest and return one merged body. The repeated header and footer of each part are removed before merging.

## `maxPagesPerArticle` (type: `integer`):

Upper bound on how many continuation pages one article may pull in, so a mislabelled pager cannot walk a whole site.

## `outputFormats` (type: `array`):

Which representations of the body to include. Markdown keeps tables, links, lists, quotes and code; text is plain reading order; html is the cleaned article HTML.

## `respectRobotsTxt` (type: `boolean`):

Skip pages that robots.txt disallows. The URL is still reported, with the rule that blocked it, instead of being dropped silently.

## `skipTruncatedByPaywall` (type: `boolean`):

Leave partial articles out of the dataset instead of returning them. Off by default so the finding stays visible; turn it on when the output feeds a corpus that must not contain half articles.

## `maxConcurrency` (type: `integer`):

How many articles to fetch at the same time. Lower this for small servers.

## `requestTimeoutSecs` (type: `integer`):

How long to wait for one response before giving up on it.

## `maxUrls` (type: `integer`):

Hard cap on how many input URLs are processed in one run, counted after duplicates are removed.

## `maxHtmlKbPerPage` (type: `integer`):

How much of each page's HTML to read. Raise it if a row reports "html\_read\_limit\_reached" — that means the end of the document, and anything in it, was cut off. Lower it to cap memory on very large crawls.

## Actor input object example

```json
{
  "urls": [
    "https://blog.apify.com/what-is-ethical-web-scraping-and-how-do-you-do-it/",
    "https://en.wikipedia.org/wiki/Web_scraping"
  ],
  "followSplitPages": true,
  "maxPagesPerArticle": 10,
  "outputFormats": [
    "markdown",
    "text"
  ],
  "respectRobotsTxt": true,
  "skipTruncatedByPaywall": false,
  "maxConcurrency": 8,
  "requestTimeoutSecs": 20,
  "maxUrls": 5000,
  "maxHtmlKbPerPage": 6144
}
```

# Actor output Schema

## `results` (type: `string`):

No description

## `csv` (type: `string`):

No description

## `summary` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "urls": [
        "https://blog.apify.com/what-is-ethical-web-scraping-and-how-do-you-do-it/",
        "https://en.wikipedia.org/wiki/Web_scraping"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("aiqlabs/article-extractor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "urls": [
        "https://blog.apify.com/what-is-ethical-web-scraping-and-how-do-you-do-it/",
        "https://en.wikipedia.org/wiki/Web_scraping",
    ] }

# Run the Actor and wait for it to finish
run = client.actor("aiqlabs/article-extractor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "urls": [
    "https://blog.apify.com/what-is-ethical-web-scraping-and-how-do-you-do-it/",
    "https://en.wikipedia.org/wiki/Web_scraping"
  ]
}' |
apify call aiqlabs/article-extractor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,aiqlabs/article-extractor"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/am63LbH3KYqNEy6Cf/builds/mBLEKkQU8hMgrUTKA/openapi.json