Article Extractor: Text, Markdown, Author, Date from Any URL

Article Extractor turns article and blog post URLs into clean text, Markdown or HTML, with the metadata you need next to it: title, author, publish and update dates, site name, description, language, lead image, tags and categories, plus word count and reading time. Paste a list of URLs, or point it at any Apify dataset of links (a sitemap run, an RSS run, a scraper's results), and each URL becomes one row. Menus, ads, cookie banners, related-story boxes and footers are left out.

Why this Actor

• Text, Markdown and HTML in one run. Pick any of the three. Markdown keeps headings, lists, bold and italics, code blocks and tables (and links and images if you ask for them), so it's ready for LLMs, RAG and search indexes.
• Metadata from every source the page has. The main text and first-pass metadata come from trafilatura. Missing or imprecise values are filled from the page's JSON-LD (schema.org NewsArticle, BlogPosting…), OpenGraph and <meta> tags: all authors (authors list), full publish and update timestamps in UTC, publisher name, keywords and sections.
• You only pay for real articles. A row is charged only when its main text has at least Minimum words (default 50). Home pages, category pages, product grids, PDFs, 404s, timeouts, pages blocked by robots.txt and invalid URLs get a free row that says why.
• Respects robots.txt by default, checked for every URL and every redirect hop, with a polite per-site limit (2 requests at a time) and a User-Agent that names the tool.
• Character sets handled properly. Pages in windows-1252, Shift_JIS, GB18030, KOI8-R and other encodings are decoded from the HTTP header, the <meta charset> tag or the bytes, so accented and non-Latin text comes out intact.
• Enrich a dataset. With Keep the item's original fields on, every item keeps its fields and gains the article columns.
• Safe with untrusted link lists. Requests to private, loopback, link-local and cloud-metadata addresses are refused, on the first request and on every redirect. Pages are capped at 10 MB while downloading, compressed pages included.

Use cases

• RAG and LLM ingestion: turn blog posts, docs pages and news into Markdown, then load them into a vector database.
• News and content monitoring: get new post URLs from an RSS feed or sitemap every day and extract only those.
• Media and competitor research: collect titles, authors, dates, tags and word counts of articles across many sites.
• Content audits and SEO: find thin pages (wordCount), missing authors or dates, and outdated posts (modifiedAt).
• Datasets for analysis: build a clean text corpus from a list of URLs, with the language and publish date of each page.

How to use it

1. Add Article URLs, and/or pick a Dataset to enrich (set URL field if the links aren't in url; for the RSS reader use link).
2. Choose the Output formats (default: text and Markdown). Optional: Include reader comments, Include links, Include images, Only these languages.
3. Run it. Export as JSON, CSV or Excel, or chain it with an integration.

Input example

{
  "startUrls": [
    "https://en.wikipedia.org/wiki/Web_scraping",
    "https://peps.python.org/pep-0008/",
    "https://peps.python.org/pep-0020/"
  ],
  "outputFormats": ["text", "markdown"],
  "includeComments": false,
  "includeLinks": false,
  "includeImages": false,
  "minWords": 50,
  "respectRobotsTxt": true
}

Output example (shortened; values are illustrative)

{
  "url": "https://news.example.com/local/2026/09/bike-lanes-main-street",
  "finalUrl": "https://news.example.com/local/2026/09/bike-lanes-main-street",
  "statusCode": 200,
  "result": "extracted",
  "title": "City council approves new bike lanes on Main Street",
  "author": "Maria Lopez; Tom Becker",
  "authors": ["Maria Lopez", "Tom Becker"],
  "publishedAt": "2026-09-03T18:05:00Z",
  "modifiedAt": "2026-09-04T12:30:00Z",
  "siteName": "Riverside Courier",
  "description": "The council voted 7-2 to add protected bike lanes along a two-mile stretch of Main Street.",
  "language": "en-US",
  "pageType": "NewsArticle",
  "image": "https://cdn.example.com/photos/main-street.jpg",
  "tags": ["transport", "cycling", "city council"],
  "categories": ["Local News"],
  "text": "City council approves new bike lanes on Main Street\nThe Riverside city council voted 7-2 on Wednesday night ...",
  "markdown": "# City council approves new bike lanes on Main Street\n\nThe Riverside city council voted 7-2 ...\n\n## Supporters point to safety\n\n...",
  "wordCount": 253,
  "readingTimeMinutes": 2,
  "truncated": false,
  "error": null,
  "extractedAt": "2026-10-21T12:00:00Z"
}

• result says what happened: extracted (charged), or one of the free outcomes not-article, robots-blocked, not-html, language-skipped, failed, invalid-url. Free rows have error set to a plain-English reason, for example HTTP 404 Not Found, Blocked by robots.txt: … or Not an HTML page (application/pdf); use PDF & Image to Text: ….
• not-article and language-skipped rows have the title, language and word count, but no text and no other metadata.
• publishedAt and modifiedAt are ISO 8601 in UTC (2026-09-03T18:05:00Z) when the page gives a time, and a plain date (2026-09-03) when it gives only a day. Times without a time zone are read as UTC.
• html appears with the Clean HTML format; comments with Include reader comments; links (url, text) and images (url, alt) with their options.
• readingTimeMinutes assumes 200 words a minute. Chinese and Japanese characters count as one word each.
• A run summary (counts per result, duplicates skipped, why the run stopped) is saved as OUTPUT in the key-value store.

How much does it cost?

Pay per event: $0.001 per extracted article, plus Apify's $0.00005 run start. Apify platform usage is included.

• An extracted article is one output row with result: "extracted": a page whose main text has at least Minimum words words (default 50).
• Free: rows for pages that aren't articles (too few words), pages in a language you filtered out, pages blocked by robots.txt, non-HTML files (PDFs, images, JSON…), HTTP errors (404, 403, 500…), timeouts, DNS errors, blocked private addresses and invalid URLs. Repeats skipped by Process each URL once get no row at all. The $0.00005 run start applies to every run, even one where nothing is extracted.

Apify Store discounts: the price above is for Apify's Free plan. Paid plans pay less per extracted article: $0.0009 on Starter, $0.0008 on Scale, $0.0007 on Business. The "from" price at the top of the Store page is the Business price.

Worked examples:

Examples use the Free-plan rate. Where an example doesn't add it, each run also has its run-start charge, counted per GB of Actor memory (minimum one; one at the default 512 MB).

• 1,000 article URLs in one run: 1,000 × $0.001 + $0.00005 = $1.00005.
• A sitemap of 5,000 URLs where 4,200 are posts and 800 are tag or category pages with under 50 words: 4,200 × $0.001 = $4.20 (+ run start). The 800 short pages are free.
• 100 links from a scraper, of which 10 are PDFs and 5 return 404: 85 × $0.001 + $0.00005 = $0.08505.
• A daily run on the 30 newest posts of an RSS feed: 30 × $0.001 + $0.00005 = $0.03005 per run, about $0.90 per month (30 runs).

If you set a maximum cost per run, the Actor stops cleanly when the next article wouldn't fit, and the summary says "stoppedBy": "max cost". No article is stored that wasn't paid for, and nothing is charged that wasn't stored. If a run is restarted (for example after a migration or a timeout), it continues after the rows already in its dataset, so nothing is charged twice.

FAQ and limits

Does it run JavaScript?

No. It reads the HTML the server sends, like a search engine's first pass. Most news sites, blogs and documentation sites send their articles in the HTML, so this works well and fast. Pages that build their content in the browser (some single-page apps) come back as not-article (free), and comments loaded by widgets such as Disqus aren't seen.

Does it use proxies or get past bot protection?

No proxies are used. Requests come from Apify's servers with a User-Agent that names the tool (ArticleExtractor/1.0; +https://apify.com/charlesmarkette). Sites behind heavy bot protection (CAPTCHAs, browser challenges) or that block data-centre IP addresses return a free error row, usually HTTP 403 Forbidden or HTTP 429, or a challenge page that comes back as not-article.

What about paywalls and logins?

There's no way to log in, and no cookies are sent. On paywalled sites you get what the page shows to a visitor without an account. If that's a teaser of at least Minimum words words, it's extracted and charged like any other article, so check a few URLs from a paywalled site before running a long list.

How accurate is the extraction?

The main text comes from trafilatura, which does well on typical article layouts but can miss a paragraph or keep a caption or a "read more" line on unusual layouts. If trafilatura finds nothing, readability-lxml is tried as a fallback. Author, dates and tags are only as good as what the page declares; many pages declare no author or only a date without a time, and those fields are then null or a plain date. Raise Minimum words if short non-article pages slip through; lower it (or set 0) for short posts.

How does the robots.txt check work?

With Respect robots.txt on (the default), each site's /robots.txt is read once per run and its rules for our token ArticleExtractor (or for *) are applied to each URL and each redirect hop, following RFC 9309: a missing robots.txt (404) allows everything; one that can't be read because of HTTP 429, a 5xx error or a network failure blocks the site for now (it's tried again after 10 minutes). Blocked URLs get a free robots-blocked row and are never requested. Crawl-delay lines aren't read; instead, each site gets at most Parallel pages per site requests at a time (default 2).

How is the language filter applied?

It uses the language the page declares (HTML lang, og:locale, JSON-LD inLanguage, or the Content-Language header); there's no statistical language detection. Pages that declare no language are kept and processed.

Are there size limits?

Pages over 10 MB (after decompression) get a free error row. Each text field is capped at 1,000,000 characters, or at Max text length if you set it, with truncated: true; wordCount is always for the full text. A truncated html field can end mid-tag.

Does it crawl a whole site?

No: one URL, one row. To extract a whole blog, get its post URLs with the Sitemap URL Extractor or the RSS reader below and pass that dataset here.

Is it allowed to extract articles?

Respecting robots.txt keeps you within what sites ask of bots, but articles are usually copyrighted. Check the site's terms and your local law before you republish or redistribute extracted text.

Related Actors

Actor	Use it to
Sitemap URL Extractor & New Page Monitor	Get every post URL from a site's sitemap, then extract them here. Its monitor mode gives only new pages.
RSS & Atom Feed Reader	Get only the new posts of blogs and news feeds, then extract their full text here (set URL field to link).
Merge & Deduplicate Datasets	Keep only URLs you haven't extracted before, so each article is charged once across runs.
Dataset Filter & Transform	Drop free rows (result isn't extracted), keep only the columns you need, rename fields.
Vector DB Sync: Pinecone & Qdrant RAG Loader	Chunk, embed and load the article text into a vector database (its defaults read text and url).
Google Sheets Sync	Send titles, authors, dates and word counts to a Google Sheet.
HTTP Request & API Calls	Pull article URLs from a CMS or news API first, or post the results to your own endpoint.
PDF & Image to Text	Extract text from the PDFs and images this Actor skips (white papers, reports, scans).

Built with open-source software under permissive licences: trafilatura and htmldate (Apache-2.0) for main-text and date extraction, readability-lxml (Apache-2.0) as the fallback, and lxml (BSD).