Article Content Extractor — Clean Text, Markdown & Metadata

Turn any article, blog post or news URL into clean, structured content. Paste a list of URLs and get back the main article — title, author, publish date, full text, LLM‑ready Markdown, main image, language, word count and reading time — with ads, navigation, cookie banners, related‑posts and other boilerplate stripped out.

Built for AI and RAG pipelines, researchers, content and media teams, and anyone who needs the actual article instead of a page full of clutter — as JSON, CSV or Excel.

---

✨ Why use this extractor

• Just the article. The main‑content extraction removes menus, ads, footers, share buttons and "recommended for you" noise, leaving the real story.
• LLM‑ready Markdown. Every article can be returned as clean Markdown — the format retrieval‑augmented generation (RAG) and LLM workflows want. Feed it straight into a vector database or a prompt.
• Rich metadata. Author, publish and modified dates, site name, language, keywords, main image, word count and reading time — pulled from JSON‑LD, Open Graph and meta tags with sensible fallbacks.
• Bring your own URLs. This is a pure extractor, not a crawler: you decide exactly which pages to process, so runs are fast, predictable and cheap.
• Batch at scale. Paste one URL or thousands — upload a file or link a Google Sheet — and process them in parallel.
• Clean, typed output. Numbers as numbers, dates as ISO strings, images as arrays — ready for databases, dashboards and AI tools.
• Export anywhere. JSON, CSV, Excel, HTML or via the Apify API and integrations.

---

🚀 Quick start

1. Paste the article URLs you want to extract (one per line, a file, or a Google Sheet).
2. Choose which content fields you want — Markdown, plain text, HTML, images.
3. Click Start.

Each URL returns one clean record with the article and its metadata.

---

📥 Input

Field	Type	Description
Article URLs (startUrls)	array	The article, blog or news page URLs to extract.
Include Markdown (includeMarkdown)	boolean	Add an LLM‑ready Markdown version of the body. On by default.
Include plain text (includeText)	boolean	Add the clean plain‑text body. On by default.
Include cleaned HTML (includeHtml)	boolean	Also include the cleaned article HTML (larger output). Off by default.
Include image URLs (includeImages)	boolean	Collect image URLs from the article body. On by default.
Concurrency (maxConcurrency)	integer	How many URLs to process in parallel. Recommended 5–12.
Proxy configuration (proxyConfiguration)	object	Apify Proxy is recommended for reliable access.

Example 1 — a few articles

{
  "startUrls": [
    { "url": "https://blog.apify.com/what-is-web-scraping/" },
    { "url": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation" }
  ]
}

Example 2 — Markdown only, for a RAG pipeline

{
  "startUrls": [
    { "url": "https://example.com/blog/post-1" },
    { "url": "https://example.com/blog/post-2" }
  ],
  "includeMarkdown": true,
  "includeText": false,
  "includeImages": false
}

Example 3 — full output with HTML and images

{
  "startUrls": [{ "url": "https://example.com/news/story" }],
  "includeHtml": true,
  "includeImages": true
}

---

📤 Output

Each URL is one dataset item. Example (body fields trimmed for readability):

{
  "url": "https://blog.apify.com/what-is-web-scraping/",
  "title": "What is web scraping?",
  "author": "David Barton",
  "publishedDate": "2024-10-15T14:32:00.000Z",
  "modifiedDate": "2024-11-02T09:10:00.000Z",
  "excerpt": "The basics of web scraping: what it is, how it works, real-world use cases...",
  "siteName": "Apify Blog",
  "lang": "en",
  "keywords": "web scraping, data extraction",
  "mainImage": "https://storage.ghost.io/c/.../cover.jpg",
  "wordCount": 1363,
  "readingTimeMinutes": 6,
  "text": "Full clean article text...",
  "markdown": "# What is web scraping?\n\nFull clean article in Markdown...",
  "images": ["https://.../image-1.jpg"],
  "extractedAt": "2026-08-25T10:00:00.000Z"
}

Field reference

Field	Description
url	The URL you provided.
finalUrl	The final URL after redirects (only when it differs from url).
title	Article title.
author	Author / byline.
publishedDate	Publish date (ISO 8601 when available).
modifiedDate	Last‑modified date (ISO 8601 when available).
excerpt	Short summary / description of the article.
siteName	Publication or site name.
lang	Content language code.
keywords	Article keywords / tags when published.
mainImage	Lead image (Open Graph / structured data).
wordCount	Number of words in the extracted body.
readingTimeMinutes	Estimated reading time.
text	Clean plain‑text body (when enabled).
markdown	Clean Markdown body (when enabled).
html	Cleaned article HTML (when enabled).
images	Image URLs found inside the article body (when enabled).
extractedAt	ISO timestamp of the extraction.
error	Set when a page could not be fetched or no article could be found.

---

💡 Use cases

• RAG & LLM pipelines — turn a list of source URLs into clean Markdown chunks, ready to embed into a vector database and feed to an LLM. No boilerplate polluting your context window.
• AI knowledge bases — build and refresh a knowledge base from trusted articles, keeping only the real content.
• Media & competitive monitoring — extract the full text of articles you track, then run sentiment, keyword or topic analysis on clean input.
• Research & academia — collect and archive article text with author, date and source for citation and analysis.
• Content archiving — snapshot the readable version of pages for a durable, clutter‑free archive.
• Newsletters & curation — pull clean excerpts, titles and images to assemble digests quickly.
• SEO & content teams — audit word counts, reading times, titles and metadata across many URLs at once.

---

🧠 How it works

The extractor loads each page and applies a main‑content extraction pass — the same idea behind browser "reader modes" — to isolate the article body and discard everything around it (navigation, ads, sidebars, comments, related links). It then layers on structured metadata from JSON‑LD, Open Graph and standard meta tags, so the author, dates, site name, language and lead image are as accurate as the page allows. Finally it renders the body as clean plain text and Markdown and computes word count and reading time.

Because you supply the exact URLs, the extractor never wanders off crawling a whole site — it does one focused job per URL, which keeps runs fast and predictable.

---

🗂️ What it works best on

• News articles — national and local news, magazines, press releases.
• Blog posts — company blogs, personal blogs, Medium‑style publications.
• Documentation & encyclopedic pages — reference articles and long‑form guides.
• Long‑form content — essays, reports, guides and tutorials.

Pages that are mostly interactive apps, hard paywalls, or that require login may return little or no article body — those come back with an error or a low wordCount so you can filter them out.

---

🎯 Tips for best results

• Point at article pages, not homepages. A section front or homepage has no single "article"; give the extractor the specific story URL.
• Use Markdown for AI. For RAG and LLM use, keep includeMarkdown on and turn off includeText/includeHtml to keep records lean.
• Filter by wordCount. After export, drop rows with a very low wordCount — those are usually non‑article pages or blocked fetches.
• Batch big jobs. Upload a file or link a Google Sheet of URLs and raise concurrency for large runs.
• Keep the metadata. author, publishedDate and siteName are perfect for citations, dedup and freshness checks.

---

📈 Scaling up

• Paste or upload as many URLs as you like — the extractor processes them in parallel up to your concurrency setting.
• Re‑run on a schedule to refresh content from the same URLs and capture updates via modifiedDate.
• Combine with a crawler or search actor upstream to gather URLs, then feed them here for clean extraction.

---

🤖 For AI & RAG builders

This extractor is built to sit at the front of an AI pipeline:

• Clean Markdown in, better answers out. Feed the markdown field straight into your chunker and embedder. Because it contains only the article — no menus, ads or "related posts" — your embeddings are about the content, not the page furniture.
• Smaller, cheaper context. Stripping boilerplate can cut a page down by 80–95%, so you store fewer tokens and spend less on every LLM call that uses the content.
• Metadata for citations & filters. Keep title, author, siteName, publishedDate and url alongside each chunk to power citations, freshness filters and source ranking in your RAG app.
• Deterministic and fast. You pass the exact URLs; there's no crawl to babysit. Perfect for on‑demand "add this source" flows and for scheduled refreshes.
• Quality signal built in. Use wordCount to skip thin or failed extractions before they ever reach your index.

A typical flow: gather source URLs (from a search actor, a sitemap, or your users) → run this extractor → chunk the Markdown → embed → store with the metadata → retrieve and cite.

---

🆚 Extractor vs. crawler

	This extractor	A website crawler
Input	The exact URLs you want	A start URL + crawl rules
Scope	One clean article per URL	Many pages, discovered automatically
Output	Article body + metadata, boilerplate removed	Whole‑page content, often with navigation
Speed & cost	Fast and predictable	Depends on how much it crawls
Best for	"I have URLs, give me the articles"	"Discover and pull a whole site/section"

Use a crawler or search actor to find URLs, then use this extractor to turn them into clean articles. They complement each other.

---

🌐 Works with any site

The extractor is site‑agnostic — point it at articles from news outlets, company and personal blogs, documentation, encyclopedias, newsletters, release notes, knowledge bases and more, in any language. It reads the same structured metadata publishers already provide (JSON‑LD, Open Graph, meta tags) and falls back to on‑page signals when they're missing, so it adapts to each site automatically instead of needing per‑site rules.

Because different publishers structure pages differently, a small number of pages — highly interactive apps, hard paywalls, or login‑only content — may return little or no body. Those come back clearly marked with an error or a low wordCount, so your pipeline can skip them cleanly.

---

🔌 Run it your way

• Apify Console — paste URLs and click Start.
• API — start runs and pull the dataset programmatically.
• Scheduler — extract on a recurring schedule.
• Integrations — push results to Google Sheets, Zapier, Make, webhooks, vector databases, S3 and more.

---

❓ FAQ

What do I put in as input? The URLs of the specific articles you want. One record comes back per URL.

Is the Markdown good for LLMs / RAG? Yes — that's the point. The Markdown is the clean article body only, so it embeds and prompts cleanly without navigation or ad text.

Does it crawl a whole website? No. It extracts exactly the URLs you give it — fast and predictable. To discover URLs first, pair it with a crawler or search actor.

Will it get the author and date? When the page publishes them (via JSON‑LD, Open Graph or meta tags), yes. Some pages don't expose them, in which case those fields are empty.

What about paywalls or login‑only pages? The extractor reads what's publicly served. Hard paywalls and login‑gated pages may return little or no body; those come back with an error or a low wordCount.

Which languages are supported? Any — the extractor is language‑agnostic and returns the detected lang for each article.

In what format is the data? Structured JSON by default, exportable to CSV, Excel, HTML and more.

Can I run it on a schedule? Yes — use the Apify Scheduler, or start runs through the Apify API and its integrations.

How many URLs can I process at once? As many as you like — paste them, upload a file, or link a Google Sheet. They're processed in parallel up to your concurrency setting.

Do I get images? Yes. mainImage holds the lead image, and images (when enabled) lists the image URLs found inside the article body, resolved to absolute URLs.

What happens on a page that isn't an article? You'll get a record with a low wordCount and often empty body fields — filter those out with a simple wordCount threshold after export.

Is the output good for feeding into a vector database? Yes — that's a primary use case. Keep markdown for the body and the metadata fields for citations and filtering.

Does it slow down on very long pages? Extremely large pages (encyclopedic mega‑articles) take longer to process than typical news or blog posts, but normal articles are fast. Failed or overly slow fetches return an error so a run never hangs.

---

⚖️ Legal & responsible use

This tool extracts publicly available page content for legitimate uses such as research, archiving, monitoring and AI pipelines. You are responsible for how you use the output and for complying with all applicable laws and regulations, the website's terms, copyright, and data‑protection rules (including GDPR/CCPA where relevant). Respect content owners' rights: do not republish extracted content in a way that infringes copyright, and honour opt‑outs where required. This tool is not affiliated with any of the websites whose pages you choose to process; all trademarks belong to their respective owners.

---

Tags

article extractor · content extractor · article content · readability · web content · markdown · rag · llm · ai content · news scraper · blog scraper · text extraction · main content · reader mode · metadata