Article Extractor — Any Site, Sitemaps & Crawl, Text + Markdown

Article Extractor — Any Site, Sitemaps & Crawl, Clean Text + Markdown

Extract article text from any news site or blog as clean rows: give direct article URLs, or point it at a front page, category page or sitemap and it finds the articles itself. Every row has the title, author, publish and modified dates, clean body text and Markdown, description, lead image, tags, section, headings, word count, reading time, language and canonical URL — with per-field provenance and a confidence score. Filter by minimum words, publish date or last-N-days; remember what was delivered so a scheduled run returns only new articles. Pay per extracted article; pages that are paywalled, empty, too short, too old or blocked are reported and never billed. Built by NexGen Watch on Apify.

• Three ways in: urls (direct), startUrls (discover article links on a page), sitemapUrls (sitemap index or urlset, newest first by lastmod).
• Clean extraction: a content extractor (trafilatura) with a metadata fallback — not a raw HTML dump.
• Scheduler-ready: onlyArticlesForLastDays + onlyNewArticles give a daily feed of fresh articles per site.
• Honest and polite: robots.txt checked per site, identified NexGenWatchBot user agent, no logins, no paywall circumvention, no browser tricks.

Billing: $0.02 per run start, then $0.05 per extracted article (article_extraction_check). Discovery pages and skipped pages cost nothing.

Who uses this

• AI / RAG teams building clean article corpora from news and blog URLs (Markdown ready for chunking).
• Media-monitoring, PR and research teams tracking what a set of publications publish each day.
• SEO and content teams auditing competitors' article structure, length and cadence.

📊 Sample Output

Real rows from run OMxSckXQ1YQ2Ws6d0 on build 0.1.28 (2026-10-18), the Quick start input below (long text shortened):

url	outcome	title	author	published_date	word_count	source_type
https://blog.python.org/2024/10/python-3130-fina…	answer	Python 3.13.0 (final) released	Python …		2024-10-07	551

The run finished with the status message: NORMAL: 1 billable verdict(s), 0 blocked (unbilled status), 0 unreachable, 0 unusable (filtered/paywalled/empty, unbilled), 0 unparseable of 1 target(s) | article_extraction_check billable=1 | article_extraction_check charged=1

✅ What you get

One article_record row per article (outcome: answer rows are billed; every other row is a free status row):

• title, author, published_date, modified_date, description, site_name, section, tags, image_url, language, canonical_url.
• main_text (clean text) and main_markdown (headings, lists and tables kept), headings (outline), word_count, reading_time_min.
• url, final_url, http_status, source_type (direct / start_url / sitemap), discovered_from.
• extraction_confidence, extractor (trafilatura or regex fallback), partial (author or date missing), provenance (where each field came from), faults.
• html_key / html_url when saveHtml is on.
• outcome: answer (billed) · unusable (paywalled, bodyless, below minWords, before dateFrom, no date when required — not billed) · blocked (robots / 403 / 429 / 5xx / private host — not billed) · unreachable (DNS, timeout — not billed) · discovery (one free row per start URL or sitemap: how many article links it yielded).

⚙️ Sample inputs

1. Quick start — direct article URLs (the Store example)

{
  "urls": ["https://blog.python.org/2024/10/python-3130-final-released"]
}

About $0.07 on the Free tier ($0.02 start + 1 × $0.05).

2. Find and extract the latest articles from a front page

{
  "startUrls": ["https://blog.python.org/"],
  "maxArticles": 20,
  "minWords": 150
}

3. A whole site from its sitemap, last 30 days only, daily schedule delivering only new ones

{
  "sitemapUrls": ["https://example-news.com/sitemap.xml"],
  "onlyArticlesForLastDays": 30,
  "mustHaveDate": true,
  "onlyNewArticles": true,
  "maxArticles": 200
}

4. Only a section, with your own URL rule, saving the HTML too

{
  "startUrls": ["https://example-news.com/technology"],
  "articleUrlPattern": "/technology/20\\d\\d/",
  "excludeUrlPattern": "/video/|/live/",
  "saveHtml": true,
  "maxArticles": 50
}

🧾 JSON sample record

{
  "url": "https://blog.python.org/2024/10/python-3130-final-released",
  "final_url": "https://blog.python.org/2024/10/python-3130-final-released/",
  "http_status": 200,
  "title": "Python 3.13.0 (final) released | Python Insider",
  "author": null,
  "published_date": "2024-10-07",
  "modified_date": null,
  "canonical_url": "https://blog.python.org/2024/10/python-3130-final-released/",
  "description": "Python 3.13.0 is now available https://www.python.org/downloads/release/python-3130/ This is the stable release of Python 3.13.0 Python 3.13.0 is the newest ...",
  "site_name": "Python Insider",
  "section": null,
  "tags": null,
  "image_url": "https://blog.python.org/og/python-3130-final-released.png",
  "headings": "[{\"level\":1,\"text\":\"Python 3.13.0 (final) released\"},{\"level\":1,\"text\":\"Python 3.13.0 is now available\"},{\"level\":1,\"text\":\"This is the stable release of Python…",
  "main_text": "Python 3.13.0 (final) released\nPython 3.13.0 is now available\npython.org/downloads/release/python-3130\nThis is the stable release of Python 3.13.0\nPython 3.13.0…",
  "main_markdown": "# Python 3.13.0 (final) released\n\n# Python 3.13.0 is now available\n\npython.org/downloads/release/python-3130\n\n# This is the stable release of Python 3.13.0\n\nPyt…",
  "word_count": 551,
  "reading_time_min": 2,
  "language": "en",
  "extraction_confidence": "0.83",
  "extractor": "trafilatura",
  "partial": true,
  "source_type": "direct",
  "discovered_from": null,
  "html_key": null,
  "html_url": null,
  "provenance": "{\"title\":\"og:title\",\"canonical_url\":\"link/og:url\",\"main_text\":\"trafilatura\",\"published_date\":\"trafilatura\"}",
  "faults": "missing author",
  "record_type": "article_record",
  "source": "buyer-submitted article, category and sitemap URLs",
  "outcome": "answer",
  "input": "https://blog.python.org/2024/10/python-3130-final-released",
  "outcome_reason": null
}

🔧 How it works

Discovery. Each start URL is fetched once and its links are kept when they look like articles (dated paths, /news/ /blog/ /article/ segments, long hyphenated slugs or numeric ids — or your articleUrlPattern), same domain by default. Sitemaps and sitemap indexes are followed; entries are ordered newest-first by lastmod and skipped before fetching when older than dateFrom. Up to maxArticlesPerStartUrl per page and maxArticles per run.

Extraction. The page is fetched through the gate (public-address check, robots.txt per site, bounded body, redirects walked hop by hop). trafilatura extracts the body as text and Markdown plus metadata; JSON-LD / Open Graph / meta tags fill title, author, dates, image, tags and section when the extractor leaves them empty. A page with no title or under 50 words is unusable; so is one under minWords, before dateFrom, or undated when mustHaveDate is on — none of those bill.

Only new articles. With onlyNewArticles, delivered URLs are remembered in a named key-value store (one per set of domains, or dedupeStoreName) and skipped on later runs.

Billing. Each article is pushed and charged in one call; a row that cannot be charged is not delivered.

Limits. 1,000 articles per run; 50 start URLs; sitemap walks capped at 200 files / 2,000 entries each; body text capped at maxTextChars (default 20,000). No JavaScript rendering: sites that only render articles client-side come back unusable.

💰 Pricing example

Event	Free	Bronze	Silver	Gold
Actor Start (apify-actor-start)	$0.02	$0.02	$0.02	$0.02
Extracted article (article_extraction_check)	$0.05	$0.045	$0.04	$0.0335

• 20 articles: $0.02 + 20 × $0.05 = $1.02
• 200 articles: $0.02 + 200 × $0.05 = $10.02

⚖️ Legal & ToS

Reads public pages as any visitor would, honours each site's robots.txt, identifies itself with a contact User-Agent and never logs in or bypasses paywalls. Extracted text remains the publisher's copyright: use it in line with each site's terms and your local law (indexing, analysis, monitoring and quotation are the intended uses).

❓ FAQ

Q: It found 0 articles on my start URL. A: Check the free discovery row's reason. Some front pages load their links with JavaScript (not rendered here); use the site's sitemap instead, or set articleUrlPattern if the site's URLs don't look like articles.

Q: Paywalled sites? A: A page with a paywall marker and no readable body is unusable and not billed. The actor does not bypass paywalls.

Q: Can I get the HTML? A: saveHtml: true stores each article's HTML in the run's key-value store and links it in html_url.

Q: How do I get only new articles each day? A: Schedule the actor with onlyNewArticles: true (and usually onlyArticlesForLastDays). The first run seeds the memory; later runs deliver only unseen URLs.

Q: What formats can I export? A: JSON, CSV, Excel or XML from the Dataset tab or API; main_markdown drops straight into a RAG pipeline.

🆘 Troubleshooting

• outcome = unusable, "bodyless/near-empty" → the page is JavaScript-rendered or not an article; try the sitemap or a different URL.
• outcome = blocked, robots → the site disallows that path for crawlers; it will not be fetched.
• Status says PARTIAL → some targets were unusable/blocked/unreachable; the counts are in the message and every row says why.
• Run TIMED-OUT → lower maxArticles or raise the run timeout; delivered rows are kept.

🔗 Related Actors

• RSS Feed Health Checker — validate and read any RSS/Atom feed.
• Company Newsroom Press Releases Scraper — press releases from company newsrooms.
• Website Screenshot API — a dated capture of any page.
• 🏢 About NexGen Watch — more than 300 public-data actors, pay-per-result: apify.com/nexgenwatch.

⭐ Found this useful?

A quick review on the Apify Store helps other teams find it. Feature request? Open it from the Issues tab — every one is read.