Article Scraper & Clean Text Extractor to Markdown for LLM/RAG

Paste a list of URLs and get the main content of every page as clean plain text and Markdown, with title, author, published date, site name, language, description and word count. Menus, sidebars, footers, cookie notices, scripts and styles are removed (using Mozilla Readability, the engine behind Firefox Reader View), so what you get is the article itself, ready to chunk and embed.

Use it to: feed web pages into an LLM or RAG pipeline, build a knowledge base from docs or blog posts, collect articles for summarisation or classification, archive readable copies of pages, or get word counts and metadata for a list of URLs.

It does not use a browser, so it's fast and cheap. The trade-off: it reads the HTML the server sends and does not run JavaScript (see Limitations).

Features

• Bulk: any number of URLs, 5 in parallel by default (up to 20). If one URL fails, the error goes in its row and the run keeps going.
• Markdown for LLMs: headings, lists, links (made absolute), bold/italic, code blocks and simple tables become GitHub-flavoured Markdown. Complex tables (merged cells, no header row, e.g. Wikipedia infoboxes) become Label: value lines instead of raw HTML.
• Plain text with paragraph breaks kept, plus a wordCount.
• Metadata from JSON-LD (schema.org Article/NewsArticle/BlogPosting), OpenGraph and standard meta tags, with Readability's guesses as a fallback. Dates are normalised to ISO 8601 when they parse.
• Retries with backoff for timeouts, network errors and HTTP 408/425/429/5xx.
• Encoding aware: the charset from the HTTP header or <meta charset> is used to decode the page.
• Optional cleaned HTML of the main content (includeHtml).

Input

{
  "urls": ["https://en.wikipedia.org/wiki/Sourdough", "paulgraham.com/greatwork.html"],
  "includeHtml": false,
  "maxConcurrency": 5,
  "maxRetries": 2,
  "timeoutSecs": 30
}

Output (one dataset row per URL)

{
  "url": "https://paulgraham.com/greatwork.html",
  "finalUrl": "https://paulgraham.com/greatwork.html",
  "statusCode": 200,
  "title": "How to Do Great Work",
  "author": null,
  "publishedDate": null,
  "siteName": null,
  "language": null,
  "description": "If you collected lists of techniques for doing great work in a lot of different fields, what would the intersection look like? ...",
  "text": "July 2023\nIf you collected lists of techniques for doing great work ...",
  "markdown": "![How to Do Great Work](https://...gif)July 2023\n\nIf you collected lists of techniques ...",
  "wordCount": 11801,
  "error": null
}

• Fields the page doesn't provide are null (the example page has no author or language tags).
• finalUrl is the URL after redirects. statusCode is the final HTTP status.
• error is null on success. On failure (HTTP 4xx/5xx, DNS error, timeout, non-HTML content such as a PDF, invalid URL) the row still appears with error set, e.g. "HTTP 404" or "fetch failed: ENOTFOUND", and empty content.
• With includeHtml: true an extra html field holds the cleaned main-content HTML.

The Console has two views: Articles (metadata table) and Content (Markdown). Export as JSON, CSV or Excel, or fetch via the API.

Pricing

Pay per event: $1 per 1,000 articles ($0.001 per successfully extracted URL). Failed URLs (error rows) are not charged. Apify platform usage is billed separately as usual. If you set a maximum charge for the run, the actor stops when it is reached.

FAQ

Does it work on any website? It works on pages whose content is in the HTML the server returns: most news sites, blogs, documentation, Wikipedia and similar. It won't see content that only appears after JavaScript runs.

What about home pages or listing pages? They are extracted, but a home page has no single "article", so the result may be a list of headlines, a section of the page, or some navigation text. The actor is built for article and documentation pages.

Can it get past paywalls, logins or CAPTCHAs? No. It sees what a logged-out visitor sees. Bot-protected sites may return an error or a challenge page.

Is the Markdown good for chunking? Yes, that's the goal: headings are #-style, and there are no scripts, styles or layout tables. Split on headings or paragraphs.

Why is title sometimes "Page - Site"? The OpenGraph/<title> value is returned as the site publishes it.

Limitations

• No JavaScript rendering. Single-page apps and pages that load content client-side may come back empty or partial; when fewer than 50 words are found the row gets an error note and is not charged.
• Only HTML (and plain text) pages. PDFs, images and other files get an error row.
• Coverage check: when the result is much shorter than the running text of the page's own <article> (a body cut into several blocks by ads or images, or a teaser picked instead of the body), the blocks are joined and the extraction is repeated. Product galleries and roundups built from cards can still come back partial.
• Data tables (3+ rows) are kept even when they are full of links, e.g. Wikipedia list pages; in text the cells of a row are separated by spaces, in markdown they are a table.
• <aside> callouts and notes inside the article body are kept when they are prose (40+ words, few links); sign-up boxes, link lists and cards promoting other articles are dropped.
• author comes from the page's metadata, else from its byline or author links; null if the page names nobody.
• Pages over 15 MB are skipped with an error.
• Which part of a page counts as "main content" is decided by Readability's heuristics. They are good on articles and can be wrong on unusual layouts.
• Respect the terms of the sites you extract from.

Support

Open an issue on the actor's Issues tab and include the URL and your input. Issues are answered asynchronously.

Not affiliated with Mozilla or any website you extract from. Built and maintained with AI assistance.

Related tools

Other actors by the same developer (same flat pay-per-result pricing, no subscription):

• App Store Reviews Scraper – Apple iOS, Multi-Country
• Job Postings & Career Page Scraper – Workday, Greenhouse, Lever
• Bluesky Posts Scraper – Profiles & Threads, No Login
• Company Enrichment from Domain (Logo, Socials, Tech)
• DNS Lookup & WHOIS Domain Checker — SPF/DMARC, SSL Expiry
• Email Validator & MX Record Checker – Bulk, Disposable, Role
• Bulk PageSpeed Insights, Lighthouse & Core Web Vitals
• PDF Extractor & Parser – Bulk PDF to Text with Metadata
• SEO Audit Crawler & Broken Link Checker – Website Health
• Sitemap Extractor & Bulk URL Status Checker
• Steam Reviews Scraper – Game Reviews & App Details
• Website Technology Detector – Tech Stack, CMS & Analytics
• Website Screenshot API – Bulk Full Page PNG, JPEG & PDF

Integrations

Run it from the Apify API, a schedule, or no-code tools: the Apify apps for Zapier, Make and n8n can start any public actor ("Run Actor") and read its dataset. AI agents can call it through the Apify MCP server.

Ready-made examples

One-click tasks you can run or copy and adapt:

• Convert Paul Graham essays to clean Markdown
• Extract news articles as clean text for LLM summaries
• Save blog posts as Markdown for a knowledge base
• Extract Wikipedia articles as clean text
• Save tech blog posts as clean Markdown (Willison, Julia Evans)
• Convert documentation pages to clean Markdown for RAG
• Extract clean article HTML (reader mode) from blog posts