# Article Extractor: Text, Markdown, Author, Date from Any URL (`charlesmarkette/article-extractor-text-markdown-author-date`) Actor Extract clean article text and Markdown from news and blog URLs, with title, author, publish date, language, tags and word count. Works on lists or a dataset column, respects robots.txt, no API key. Non-articles, errors and blocked pages are free rows. - **URL**: https://apify.com/charlesmarkette/article-extractor-text-markdown-author-date.md - **Developed by:** [Charles Markette](https://apify.com/charlesmarkette) (community) - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $0.70 / 1,000 extracted articles This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have. Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event ## What's an Apify Actor? An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md. ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run. For examples already wired to this Actor's own input schema, see the [API](#api) section below. Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`). # README **Article Extractor** turns article and blog post URLs into clean **text**, **Markdown** or **HTML**, with the metadata you need next to it: **title, author, publish and update dates, site name, description, language, lead image, tags and categories**, plus **word count** and **reading time**. Paste a list of URLs, or point it at any Apify dataset of links (a sitemap run, an RSS run, a scraper's results), and each URL becomes one row. Menus, ads, cookie banners, related-story boxes and footers are left out. ### Why this Actor - **Text, Markdown and HTML in one run.** Pick any of the three. Markdown keeps headings, lists, bold and italics, code blocks and tables (and links and images if you ask for them), so it's ready for LLMs, RAG and search indexes. - **Metadata from every source the page has.** The main text and first-pass metadata come from [trafilatura](https://github.com/adbar/trafilatura). Missing or imprecise values are filled from the page's **JSON-LD** (schema.org `NewsArticle`, `BlogPosting`…), **OpenGraph** and `` tags: all authors (`authors` list), full publish and update timestamps in UTC, publisher name, keywords and sections. - **You only pay for real articles.** A row is charged only when its main text has at least *Minimum words* (default 50). Home pages, category pages, product grids, PDFs, 404s, timeouts, pages blocked by robots.txt and invalid URLs get a free row that says why. - **Respects robots.txt** by default, checked for every URL and every redirect hop, with a polite per-site limit (2 requests at a time) and a User-Agent that names the tool. - **Character sets handled properly.** Pages in windows-1252, Shift_JIS, GB18030, KOI8-R and other encodings are decoded from the HTTP header, the `` tag or the bytes, so accented and non-Latin text comes out intact. - **Enrich a dataset.** With *Keep the item's original fields* on, every item keeps its fields and gains the article columns. - **Safe with untrusted link lists.** Requests to private, loopback, link-local and cloud-metadata addresses are refused, on the first request and on every redirect. Pages are capped at 10 MB while downloading, compressed pages included. ### Use cases - **RAG and LLM ingestion:** turn blog posts, docs pages and news into Markdown, then load them into a vector database. - **News and content monitoring:** get new post URLs from an RSS feed or sitemap every day and extract only those. - **Media and competitor research:** collect titles, authors, dates, tags and word counts of articles across many sites. - **Content audits and SEO:** find thin pages (`wordCount`), missing authors or dates, and outdated posts (`modifiedAt`). - **Datasets for analysis:** build a clean text corpus from a list of URLs, with the language and publish date of each page. ### How to use it 1. Add **Article URLs**, and/or pick a **Dataset to enrich** (set **URL field** if the links aren't in `url`; for the RSS reader use `link`). 2. Choose the **Output formats** (default: text and Markdown). Optional: **Include reader comments**, **Include links**, **Include images**, **Only these languages**. 3. Run it. Export as JSON, CSV or Excel, or chain it with an integration. **Input example** ```json { "startUrls": [ "https://en.wikipedia.org/wiki/Web_scraping", "https://peps.python.org/pep-0008/", "https://peps.python.org/pep-0020/" ], "outputFormats": ["text", "markdown"], "includeComments": false, "includeLinks": false, "includeImages": false, "minWords": 50, "respectRobotsTxt": true } ``` **Output example** (shortened; values are illustrative) ```json { "url": "https://news.example.com/local/2026/09/bike-lanes-main-street", "finalUrl": "https://news.example.com/local/2026/09/bike-lanes-main-street", "statusCode": 200, "result": "extracted", "title": "City council approves new bike lanes on Main Street", "author": "Maria Lopez; Tom Becker", "authors": ["Maria Lopez", "Tom Becker"], "publishedAt": "2026-09-03T18:05:00Z", "modifiedAt": "2026-09-04T12:30:00Z", "siteName": "Riverside Courier", "description": "The council voted 7-2 to add protected bike lanes along a two-mile stretch of Main Street.", "language": "en-US", "pageType": "NewsArticle", "image": "https://cdn.example.com/photos/main-street.jpg", "tags": ["transport", "cycling", "city council"], "categories": ["Local News"], "text": "City council approves new bike lanes on Main Street\nThe Riverside city council voted 7-2 on Wednesday night ...", "markdown": "# City council approves new bike lanes on Main Street\n\nThe Riverside city council voted 7-2 ...\n\n## Supporters point to safety\n\n...", "wordCount": 253, "readingTimeMinutes": 2, "truncated": false, "error": null, "extractedAt": "2026-10-21T12:00:00Z" } ``` - `result` says what happened: `extracted` (charged), or one of the free outcomes `not-article`, `robots-blocked`, `not-html`, `language-skipped`, `failed`, `invalid-url`. Free rows have `error` set to a plain-English reason, for example `HTTP 404 Not Found`, `Blocked by robots.txt: …` or `Not an HTML page (application/pdf); use PDF & Image to Text: …`. - `not-article` and `language-skipped` rows have the title, language and word count, but no text and no other metadata. - `publishedAt` and `modifiedAt` are ISO 8601 in UTC (`2026-09-03T18:05:00Z`) when the page gives a time, and a plain date (`2026-09-03`) when it gives only a day. Times without a time zone are read as UTC. - `html` appears with the *Clean HTML* format; `comments` with *Include reader comments*; `links` (`url`, `text`) and `images` (`url`, `alt`) with their options. - `readingTimeMinutes` assumes 200 words a minute. Chinese and Japanese characters count as one word each. - A run summary (counts per result, duplicates skipped, why the run stopped) is saved as `OUTPUT` in the key-value store. ### How much does it cost? Pay per event: **$0.001 per extracted article**, plus Apify's **$0.00005 run start**. Apify platform usage is included. - An **extracted article** is one output row with `result: "extracted"`: a page whose main text has at least *Minimum words* words (default 50). - **Free:** rows for pages that aren't articles (too few words), pages in a language you filtered out, pages blocked by robots.txt, non-HTML files (PDFs, images, JSON…), HTTP errors (404, 403, 500…), timeouts, DNS errors, blocked private addresses and invalid URLs. Repeats skipped by *Process each URL once* get no row at all. The $0.00005 run start applies to every run, even one where nothing is extracted. **Apify Store discounts:** the price above is for Apify's Free plan. Paid plans pay less per extracted article: $0.0009 on Starter, $0.0008 on Scale, $0.0007 on Business. The "from" price at the top of the Store page is the Business price. **Worked examples:** *Examples use the Free-plan rate. Where an example doesn't add it, each run also has its run-start charge, counted per GB of Actor memory (minimum one; one at the default 512 MB).* - 1,000 article URLs in one run: 1,000 × $0.001 + $0.00005 = **$1.00005**. - A sitemap of 5,000 URLs where 4,200 are posts and 800 are tag or category pages with under 50 words: 4,200 × $0.001 = **$4.20** (+ run start). The 800 short pages are free. - 100 links from a scraper, of which 10 are PDFs and 5 return 404: 85 × $0.001 + $0.00005 = **$0.08505**. - A daily run on the 30 newest posts of an RSS feed: 30 × $0.001 + $0.00005 = $0.03005 per run, about **$0.90 per month** (30 runs). If you set a maximum cost per run, the Actor stops cleanly when the next article wouldn't fit, and the summary says `"stoppedBy": "max cost"`. No article is stored that wasn't paid for, and nothing is charged that wasn't stored. If a run is restarted (for example after a migration or a timeout), it continues after the rows already in its dataset, so nothing is charged twice. ### FAQ and limits #### Does it run JavaScript? No. It reads the HTML the server sends, like a search engine's first pass. Most news sites, blogs and documentation sites send their articles in the HTML, so this works well and fast. Pages that build their content in the browser (some single-page apps) come back as `not-article` (free), and comments loaded by widgets such as Disqus aren't seen. #### Does it use proxies or get past bot protection? No proxies are used. Requests come from Apify's servers with a User-Agent that names the tool (`ArticleExtractor/1.0; +https://apify.com/charlesmarkette`). Sites behind heavy bot protection (CAPTCHAs, browser challenges) or that block data-centre IP addresses return a free error row, usually `HTTP 403 Forbidden` or `HTTP 429`, or a challenge page that comes back as `not-article`. #### What about paywalls and logins? There's no way to log in, and no cookies are sent. On paywalled sites you get what the page shows to a visitor without an account. If that's a teaser of at least *Minimum words* words, it's extracted and charged like any other article, so check a few URLs from a paywalled site before running a long list. #### How accurate is the extraction? The main text comes from trafilatura, which does well on typical article layouts but can miss a paragraph or keep a caption or a "read more" line on unusual layouts. If trafilatura finds nothing, readability-lxml is tried as a fallback. Author, dates and tags are only as good as what the page declares; many pages declare no author or only a date without a time, and those fields are then `null` or a plain date. Raise *Minimum words* if short non-article pages slip through; lower it (or set 0) for short posts. #### How does the robots.txt check work? With *Respect robots.txt* on (the default), each site's `/robots.txt` is read once per run and its rules for our token `ArticleExtractor` (or for `*`) are applied to each URL and each redirect hop, following RFC 9309: a missing robots.txt (404) allows everything; one that can't be read because of HTTP 429, a 5xx error or a network failure blocks the site for now (it's tried again after 10 minutes). Blocked URLs get a free `robots-blocked` row and are never requested. Crawl-delay lines aren't read; instead, each site gets at most *Parallel pages per site* requests at a time (default 2). #### How is the language filter applied? It uses the language the page declares (HTML `lang`, `og:locale`, JSON-LD `inLanguage`, or the `Content-Language` header); there's no statistical language detection. Pages that declare no language are kept and processed. #### Are there size limits? Pages over 10 MB (after decompression) get a free error row. Each text field is capped at 1,000,000 characters, or at *Max text length* if you set it, with `truncated: true`; `wordCount` is always for the full text. A truncated `html` field can end mid-tag. #### Does it crawl a whole site? No: one URL, one row. To extract a whole blog, get its post URLs with the Sitemap URL Extractor or the RSS reader below and pass that dataset here. #### Is it allowed to extract articles? Respecting robots.txt keeps you within what sites ask of bots, but articles are usually copyrighted. Check the site's terms and your local law before you republish or redistribute extracted text. ### Related Actors | Actor | Use it to | |---|---| | [Sitemap URL Extractor & New Page Monitor](https://apify.com/charlesmarkette/sitemap-url-extractor-new-page-monitor) | Get every post URL from a site's sitemap, then extract them here. Its monitor mode gives only new pages. | | [RSS & Atom Feed Reader](https://apify.com/charlesmarkette/rss-atom-feed-reader-only-new-items) | Get only the new posts of blogs and news feeds, then extract their full text here (set *URL field* to `link`). | | [Merge & Deduplicate Datasets](https://apify.com/charlesmarkette/merge-deduplicate-datasets-only-new-items) | Keep only URLs you haven't extracted before, so each article is charged once across runs. | | [Dataset Filter & Transform](https://apify.com/charlesmarkette/dataset-filter-transform-rename-flatten-fields) | Drop free rows (`result` isn't `extracted`), keep only the columns you need, rename fields. | | [Vector DB Sync: Pinecone & Qdrant RAG Loader](https://apify.com/charlesmarkette/vector-db-sync-pinecone-qdrant-rag-loader-delta-updates) | Chunk, embed and load the article text into a vector database (its defaults read `text` and `url`). | | [Google Sheets Sync](https://apify.com/charlesmarkette/google-sheets-sync) | Send titles, authors, dates and word counts to a Google Sheet. | | [HTTP Request & API Calls](https://apify.com/charlesmarkette/http-request-api-calls-bulk-pagination) | Pull article URLs from a CMS or news API first, or post the results to your own endpoint. | | [PDF & Image to Text](https://apify.com/charlesmarkette/pdf-image-to-text-markdown-ocr) | Extract text from the PDFs and images this Actor skips (white papers, reports, scans). | Built with open-source software under permissive licences: [trafilatura](https://github.com/adbar/trafilatura) and htmldate (Apache-2.0) for main-text and date extraction, [readability-lxml](https://github.com/buriy/python-readability) (Apache-2.0) as the fallback, and [lxml](https://lxml.de/) (BSD). # Actor input Schema ## `startUrls` (type: `array`): One per line. Links to news articles, blog posts, documentation pages or any other web page with a main text. Each URL becomes one output row. ## `datasetId` (type: `string`): Optional. A dataset from another run that has article links (a sitemap or RSS run, a scraper's results…). Each item's URL is extracted and the article columns are added next to the item's fields. Use {{resource.defaultDatasetId}} in an integration. ## `urlField` (type: `string`): The item field that holds the article URL. Dotted paths work (link.href). Default: url. ## `keepItemFields` (type: `boolean`): On: each output row is the original item plus the article columns (article columns win if a name clashes). Off: only the article columns. ## `outputFormats` (type: `array`): Which versions of the main text to return. Text: plain text, one paragraph per line. Markdown: headings, lists, bold and italics, code blocks and tables (plus links and images when those options are on); the best input for LLMs and RAG. HTML: the cleaned main content as simple HTML. ## `includeComments` (type: `boolean`): Adds a comments field with the text of reader comments when the page shows them in its HTML. Comments loaded later by JavaScript (Disqus and similar) aren't seen. ## `includeLinks` (type: `boolean`): Adds a links list (url and text) of the links inside the article text, and keeps them as [text](url) in the Markdown. ## `includeImages` (type: `boolean`): Adds an images list (url and alt text) of the images inside the article, and keeps them as ![alt](url) in the Markdown. The lead image is always in the image field. ## `language` (type: `string`): Optional. Comma-separated language codes, e.g. en, de. Pages that declare another language (in their HTML lang attribute, og:locale, JSON-LD or Content-Language header) get a free 'language-skipped' row without text. Pages that declare no language are kept. ## `minWords` (type: `integer`): Pages whose main text has fewer words are reported as 'not-article' without text and aren't charged (home pages, category pages, product grids, login walls). Chinese and Japanese characters count as one word each. ## `maxTextLength` (type: `integer`): Optional. Cuts each text field (text, markdown, html, comments) to this many characters and sets truncated to true. wordCount is always for the full text. Without a value, fields are capped at 1,000,000 characters. ## `respectRobotsTxt` (type: `boolean`): On (recommended): each site's robots.txt is read first, and pages it disallows for bots get a free 'robots-blocked' row instead of being fetched. If robots.txt can't be read because the site is down or overloaded (HTTP 429 or 5xx), the site is treated as off-limits, as the robots.txt standard (RFC 9309) says. Turn off only for sites you are allowed to fetch. ## `concurrency` (type: `integer`): How many pages to fetch at the same time (1–20). ## `perHostConcurrency` (type: `integer`): At most this many requests to the same host at a time (1–10), to stay polite to each site. ## `timeoutSecs` (type: `integer`): How long one request may take (5–300). A page, with its redirects, retries and robots.txt check, gets at most twice this long. ## `dedupe` (type: `boolean`): Skip repeats of a URL already processed in this run (URLs that differ only by #fragment count as the same). Skipped repeats get no row and are free. ## `maxResults` (type: `integer`): Optional cap on output rows (each row is one URL, extracted or not). ## Actor input object example ```json { "startUrls": [ "https://en.wikipedia.org/wiki/Web_scraping", "https://peps.python.org/pep-0008/", "https://peps.python.org/pep-0020/" ], "keepItemFields": true, "outputFormats": [ "text", "markdown" ], "includeComments": false, "includeLinks": false, "includeImages": false, "minWords": 50, "respectRobotsTxt": true, "concurrency": 5, "perHostConcurrency": 2, "timeoutSecs": 45, "dedupe": true } ``` # Actor output Schema ## `items` (type: `string`): No description ## `summary` (type: `string`): No description # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "startUrls": [ "https://en.wikipedia.org/wiki/Web_scraping", "https://peps.python.org/pep-0008/", "https://peps.python.org/pep-0020/" ] }; // Run the Actor and wait for it to finish const run = await client.actor("charlesmarkette/article-extractor-text-markdown-author-date").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "startUrls": [ "https://en.wikipedia.org/wiki/Web_scraping", "https://peps.python.org/pep-0008/", "https://peps.python.org/pep-0020/", ] } # Run the Actor and wait for it to finish run = client.actor("charlesmarkette/article-extractor-text-markdown-author-date").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}") for item in client.dataset(run.default_dataset_id).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "startUrls": [ "https://en.wikipedia.org/wiki/Web_scraping", "https://peps.python.org/pep-0008/", "https://peps.python.org/pep-0020/" ] }' | apify call charlesmarkette/article-extractor-text-markdown-author-date --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "type": "http", "url": "https://mcp.apify.com/?tools=fetch-actor-details,charlesmarkette/article-extractor-text-markdown-author-date" } } } ``` The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer ` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations). ## OpenAPI specification Download the OpenAPI definition: https://api.apify.com/v2/actors/5jqY5JnGTpr008v7d/builds/tq04pf33q8c09fB4B/openapi.json