AI-Ready Webpage Extractor

Turn any public web page, documentation site or blog into clean Markdown, plain text and RAG-ready chunks for LLMs, AI agents and vector databases. Menus, cookie banners, footers and ads are stripped out, so you only pay (in tokens) for the content that matters.

Why use it

• Clean Markdown for LLMs: headings, lists, code blocks (with language), tables and links are kept; page clutter is removed.
• Works on JavaScript sites: pages are fetched fast as plain HTML, and only pages that come back empty (React, Vue, Next.js apps, some bot walls) are opened in a real browser automatically.
• Whole sites in one run: give a homepage and extract the pages it links to, or every page in its sitemap, optionally filtered to /docs/ or /blog/.
• RAG chunks built in: optional heading-aware chunks with token estimates and overlap, ready to embed.
• Rich metadata: title, description, author, published and modified dates, canonical URL, language and schema.org JSON-LD.
• Fair billing: you pay only for pages that were extracted successfully. Failed pages are reported, not charged.
• Use it from your AI tools: call it from Claude, Cursor, ChatGPT or any MCP client through the Apify MCP server, or from n8n, Make, Zapier, LangChain and LlamaIndex.

Use cases

• Feed documentation, help centers or blogs into a RAG pipeline or a custom GPT.
• Give an AI agent a "read this web page" tool that returns compact Markdown instead of raw HTML.
• Build a search index or knowledge base from a competitor's or partner's public docs.
• Collect articles with author and date metadata for research and content analysis.
• Extract pricing tables, specs and other tables as structured rows.

How to use it

1. Paste one or more URLs.
2. Pick a crawl mode:
   • Only these URLs – extract exactly what you list.
   • Follow links on the same site – also extract linked pages, up to the crawl depth.
   • Use the site's sitemap – fastest way to grab a whole docs site or blog.
3. Optionally turn on Split content into RAG chunks.
4. Click Start and download the results as JSON, CSV, Excel, or connect them to your pipeline.

Example: one page

{
  "urls": ["https://docs.apify.com/platform"],
  "outputFormat": "markdown"
}

Example: a whole docs section, chunked for RAG

{
  "urls": ["https://docs.apify.com"],
  "crawlMode": "sitemap",
  "urlMustContain": "/platform/",
  "maxPages": 200,
  "chunkContent": true,
  "chunkSizeTokens": 800,
  "chunkOverlapTokens": 100
}

Input

Field	Description	Default
urls	Public pages to extract (1–100). Start points in crawl modes.	required
crawlMode	none, sameDomain or sitemap.	none
maxPages	Maximum pages per run, including crawled pages (up to 1,000).	100
maxCrawlDepth	Link depth for sameDomain mode.	2
urlMustContain	Only crawl URLs containing this text, e.g. /docs/.	–
outputFormat	markdown, text or both.	markdown
chunkContent	Add a chunks array for RAG.	false
chunkSizeTokens / chunkOverlapTokens	Chunk size and overlap.	800 / 100
removeBoilerplate	Remove menus, headers, footers, cookie banners, share bars.	true
includeLinks / includeTables / includeMetadata	Toggle the extra fields.	true
renderJavaScript	auto, never or always.	auto
includeFailedPages	Also save failed pages as dataset rows (each counts as a result).	false
maxConcurrency	Pages fetched in parallel.	5

Output

One dataset item per page. Example (shortened):

{
  "sourceUrl": "https://docs.apify.com/platform",
  "finalUrl": "https://docs.apify.com/platform",
  "httpStatus": 200,
  "title": "Apify platform",
  "description": "Apify platform documentation",
  "author": null,
  "publishedDate": null,
  "modifiedDate": null,
  "canonicalUrl": "https://docs.apify.com/platform",
  "language": "en",
  "mainContentMarkdown": "# Apify platform\n\nPlatform content...",
  "mainContentText": null,
  "chunks": [
    {
      "index": 0,
      "headingPath": ["Apify platform"],
      "content": "# Apify platform\n\nPlatform content...",
      "estimatedTokens": 412
    }
  ],
  "headings": [{ "level": 1, "text": "Apify platform" }],
  "links": [{ "text": "Actors", "url": "https://docs.apify.com/platform/actors" }],
  "tables": [],
  "images": [],
  "schemaOrgData": [],
  "wordCount": 900,
  "estimatedTokens": 1200,
  "renderedWithBrowser": false,
  "crawlDepth": 0,
  "scrapedAt": "2026-10-06T10:00:00.000Z",
  "extractionStatus": "succeeded",
  "errorMessage": null
}

The dataset has two views: Overview (one row per page) and RAG chunks (one row per chunk).

A run summary with the list of failed pages and their reasons is saved as the OUTPUT record in the run's key-value store.

Pricing

Pay per event: a small fee per run start plus a fee per successfully extracted page. JavaScript rendering, chunking and crawling are included at no extra cost, and you never pay for failed pages. You can cap the cost of any run with the Maximum cost per run option in Apify.

Use with the API

curl -X POST "https://api.apify.com/v2/acts/s3nafps~ai-ready-webpage-extractor/run-sync-get-dataset-items?token=YOUR_APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{ "urls": ["https://docs.apify.com/platform"], "outputFormat": "markdown" }'

Or with the JavaScript client:

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('s3nafps/ai-ready-webpage-extractor').call({
  urls: ['https://docs.apify.com/platform'],
  chunkContent: true
});
const { items } = await client.dataset(run.defaultDatasetId).listItems();

Use with AI agents (MCP)

Add the Apify MCP server to Claude, Cursor, VS Code or any MCP client and enable this Actor (s3nafps/ai-ready-webpage-extractor) as a tool. Your agent can then read any public page as clean Markdown. See mcp.apify.com.

Tips and troubleshooting

• Content looks short or empty: keep renderJavaScript on auto (or set always) so JavaScript pages are opened in a browser.
• Too much navigation text: make sure removeBoilerplate is on.
• Crawl picks up unrelated pages: set urlMustContain (for example /docs/) or lower maxCrawlDepth.
• Site has no sitemap: sitemap mode automatically falls back to following links from your start URL.
• Smaller output: turn off links, tables or metadata, or choose text output.

Limitations

• Public pages only: no logins, cookies, paywalls or CAPTCHA solving.
• Non-HTML files (PDF, images, archives) are skipped. For PDF and Word files, use PDF & Word to Markdown for RAG, which returns the same chunk format with page numbers.
• Main-content detection is heuristic; unusual layouts may include some extra text.
• Token counts are estimates (about 4 characters per token).

Related Actors

• PDF & Word to Markdown for RAG: the same Markdown and chunk format for PDF and Word (.docx) files, with page numbers on every chunk. Load web pages and documents into one knowledge base.
• Webpage to JSON for AI Agents: when you need specific fields (prices, dates, contacts) as typed JSON instead of full page text.

Feedback

Found a page that doesn't extract well, or need a feature? Open an issue in the Issues tab and include the URL. If the Actor saves you time, a short review on its Store page helps other people find it.

This Actor is intended for public data only. Respect each website's terms of use and applicable laws.