Article Extractor: Paywall, Split Pages & Tables

Give it article URLs. It returns the body as Markdown and plain text, with the metadata — and it tells you three things the other extractors do not.

What is different here

Every extractor on this store returns "clean article text". So does this one. The difference is what it says about that text.

1. Whether a paywall cut the body short

An extractor that returns the first three paragraphs of a metered article and calls it a success is worse than one that fails, because the failure is invisible. Feed a thousand of those into a corpus and it looks complete.

This Actor reports two separate fields, because they come apart constantly:

Field	Meaning
paywallDetected	The page runs a paywall
bodyTruncated	The text you are holding is short of what the page itself claims

Publishers who want to stay in Google ship the whole article in the HTML and hide it with CSS, so a page can be paywalled with the body complete. A metered site on your last free view ships three paragraphs with no paywall markup at all. Only the second field is the one that should stop your pipeline.

Measured on a live New York Times article: paywallDetected: true from three signals (isAccessibleForFree: false, hasPart naming .meteredContent, and a meteredcontent class in the markup), bodyTruncated: false, 2,757 words extracted.

It does not bypass anything. The page is fetched the way any reader's browser fetches it, and the only output is a verdict on what arrived. There is no login, no cookie injection and no cache trick.

Truncation is proven against the page's own articleBody or wordCount in JSON-LD when it publishes one — coverageRatio and coverageBasis show the arithmetic — and otherwise inferred from named signals you can read in paywallEvidence.

2. Whether the article continues on later pages

Long articles get split, and rel="next" is used for two unrelated things: the rest of this article, and the next page of an index. Following it blindly appends a different article to the body.

So a continuation has to earn the merge:

• same site, never a cross-domain hop
• both pages claim the same title, after Page 2 of 5 decoration is removed
• the source page is not itself an index

When a merge is declined the reason is in issues, not swallowed. Real refusals from the test run:

not the same article: titles differ ("html 4 01 specification" vs "about the html 4 specification")
this page is an index, so its "next" link is site pagination rather than a continuation
the link holds unrendered template syntax (/'.$next.')

That last one is a live WordPress blog whose rel="next" is literally '.$next.' — valid as a relative URL, so nothing but a look at the string catches it.

When parts are merged, the header and footer each part repeats are dropped (repeatedBlocksRemoved), and only at the edges of a part — a sentence repeated in the middle of a body is the author, not furniture.

3. Tables, links and code, kept

"Clean text" is where a table goes to die. A 40-row financial table becomes a run of loose numbers with no columns, and the article's whole point goes with it.

Markdown output keeps:

• tables as Markdown tables, with the header row taken from the document's own <thead> or <th> rather than guessed, and colspan widened so values stay under their own column
• links as [text](absolute-url), resolved against the page
• code blocks with their line breaks and language, from the <pre> as written
• nested lists, block quotes, figure captions, definition lists

tableCount, linkCount and codeBlockCount let you tell "this article has three tables" from "this article had three tables".

4. And: published or last updated

Everyone returns "publish date". A page rewritten two years after publication looks identical in that output. This Actor returns publishedAt and modifiedAt separately, plus updatedAfterPublish and daysBetweenPublishAndUpdate. Stamps written minutes apart are the CMS saving both on publish, not a revision, so they do not count.

Also included

• Is it even an article? pageKind separates article from listing. A category page with eight teasers is not an article; a blog post with a "you may also like" rail is, even though both trip the same signals.
• extractionConfidence (high / medium / low) with extractedVia naming what chose the body: an <article> element, itemprop="articleBody", or paragraph scoring.
• Title taken by vote across JSON-LD headline, og:title, <title> and <h1> rather than by rank. Wikipedia puts the Wikidata one-line description in headline, so ranking it first titles the web scraping article "data scraping used for extracting data from websites".
• Author, site name, section, language, description, hero image, tags, canonical, schema.org types, word count, reading time.
• CJK counted per character, so a Japanese article is not measured as one word.
• robots.txt is honoured. A disallowed page is reported with the rule that blocked it rather than dropped silently — and not downloaded.

Input

{
  "urls": ["https://example.com/post"],
  "followSplitPages": true,        // merge continuation pages
  "maxPagesPerArticle": 10,
  "outputFormats": ["markdown", "text"],   // also "html" for the cleaned article HTML
  "respectRobotsTxt": true,
  "skipTruncatedByPaywall": false, // drop partial bodies instead of returning them
  "maxConcurrency": 8
}

Set skipTruncatedByPaywall when the output feeds a corpus that must not contain half articles. Off by default so the finding stays visible.

Output

One row per input URL. SUMMARY in the key-value store holds the totals: how many were paywalled, how many came back cut short, how many were merged from several pages, how many were not articles.

Cost

Free to run — you pay only Apify platform usage. One request per article, plus one robots.txt per host, plus one request per continuation page actually merged. No browser, no JavaScript execution, no proxy.

Verification

• 69 unit tests
• 26 checks against the live web on every release, covering a blog post, a reference page heavy with tables and links, a paywalled news article, a section page, a blog index, a Japanese WordPress post and a chaptered specification — including the negative controls that an index is not merged as one article and a free post is not reported as paywalled

Defects these live checks found and that the unit tests did not: a blog index whose six pagination pages were merged into "one article", a WordPress post reported as not_an_article while holding a full body, Wikipedia titled by its JSON-LD description, MediaWiki maintenance notices treated as body text, and a crash in Node's own HTTP client when a keep-alive socket closes (assert(!this.paused) in undici's HTTP/1 parser) — measured across eight consecutive runs per approach and guarded explicitly, since it fires from a socket event where nothing can catch it.