Website to Markdown Crawler for AI, LLMs and RAG

Turn any public website into clean Markdown that is ready for large language models, retrieval-augmented generation (RAG), vector databases, chatbots and search indexes. Give it one or more start URLs; it saves the main content of every page as Markdown, with menus, cookie banners, footers, sidebars, ads and scripts removed, and headings, paragraphs, lists, tables, code blocks and links kept. $1 per 1,000 pages fetched directly; $3 per 1,000 pages that need a real browser.

What can this website to Markdown crawler do?

• Convert single pages or crawl whole sites (same website only), with page and depth limits.
• Find pages three ways: links on the pages, the site's sitemap.xml (including sitemap indexes and .xml.gz), and the site's llms.txt file (including the Markdown files it lists). You can also enter a sitemap or llms.txt URL directly as a start URL.
• Stay inside a section: starting at /docs/ only crawls pages under /docs/.
• Include or skip URLs with glob patterns such as https://example.com/docs/**, */blog/* or plain text such as /tag/.
• Remove boilerplate automatically. Three extraction modes: Main content (default: keeps the main area, removes menus, sidebars, footers, banners), Article text (Readability-style extraction with trafilatura, for blogs and news), and Full page.
• Control it with CSS selectors: remove elements (navigation, footers, dialogs and common cookie banners by default) or keep only specific elements such as article or .docs-content.
• Handle JavaScript sites automatically. In Auto mode every page is first fetched directly (fast and cheap); pages that come back empty because they are built with JavaScript are opened in a real headless browser. If most pages of a site need the browser, the crawler goes browser-first for that site. Always forces the browser for every page.
• Convert linked PDF and Word (DOCX) files to Markdown and text, with headings, lists and tables, plus document metadata (optional).
• Chunk for RAG: optionally split every page into chunks of a chosen size and overlap on heading, paragraph and sentence boundaries, each with its heading path.
• Skip duplicates: pages with identical text, or whose canonical URL points to a page already saved, are not saved or charged.
• Crawl incrementally: only fetch sitemap entries changed since a date, and/or remember page hashes in a named key-value store so the next run saves only new and changed pages.
• Rich metadata: title, description, canonical URL, language, author, published and modified dates, keywords, Open Graph tags and useful HTTP headers (Last-Modified, ETag, Content-Type and more).
• Skip thin pages below a word count (not charged).
• Report AI-usage preferences: Content-Signal rules a site declares in robots.txt (for example ai-train=no) are returned with every page so your pipeline can honour them.

What data do you get?

Field	Description
url, loadedUrl	Requested URL and final URL after redirects
title, description	Page title and meta description
markdown	Clean Markdown (headings, lists, tables, links, code blocks)
text	Plain text (Markdown formatting removed); can be turned off
html	Cleaned main-content HTML (only with Save cleaned HTML)
chunks	RAG chunks: index, text, headings (section path), charCount (only with Chunk size above 0)
wordCount, language	Size and language of the content
contentType	html, markdown, pdf or docx
source, referrerUrl, depth	How the URL was found (start, link, sitemap, llms.txt), the page that linked to it, and crawl depth
httpStatus, canonicalUrl	Status code and canonical link
renderedWithBrowser	Whether a browser was needed
metadata	title, description, canonicalUrl, languageCode, author, keywords, publishedAt, modifiedAt, siteName, image, openGraph (all og: tags), headers (Content-Type, Last-Modified, ETag, Date, Cache-Control, X-Robots-Tag); for documents also pageCount and fileSizeBytes
contentHash	Hash of the page text, for change detection and de-duplication
changeStatus	new or changed (only in incremental mode)
crawledAt	When the page was fetched
contentSignals	Site's declared Content-Signal preferences, or null

Main input options

Option	Default	What it does
Start URLs	Python tutorial	Pages, domains, sitemap URLs, llms.txt URLs, or PDF/DOCX links
Follow links / Maximum pages / Maximum link depth	on / 20 / 2	Crawl scope
Stay within the start URL's folder	on	Only crawl under the start path
Only crawl / Skip URLs matching	none	Glob patterns (* matches anything, including /)
Find pages in sitemap.xml / llms.txt	off / off	Extra URL sources
Render JavaScript	Auto	Auto, Always (browser for every page) or Never
Content extraction	Main content	Main content, Article text (trafilatura) or Full page
Remove elements / Keep only elements	nav, footer, aside, dialogs, cookie banners / none	CSS selectors
Save Markdown / plain text / cleaned HTML	on / on / off	Output formats
Chunk size / overlap	0 (off) / 150	RAG chunking, in characters
Convert linked PDF and DOCX files	off	Max file size 20 MB, first 300 PDF pages
Skip duplicate pages	on	By text hash and canonical URL
Only pages modified since	none	Date filter for sitemap entries and page dates
Remember pages in key-value store	none	Incremental mode: save only new and changed pages

Who uses it?

• AI and chatbot builders loading documentation, help centers or websites into a knowledge base.
• Developers feeding docs sites to coding assistants and agents.
• Content and SEO teams exporting a site's copy for review, translation or migration.
• Researchers collecting readable article text from lists of URLs.

How to use it

1. Click Try for free (or Start).
2. Enter Start URLs (example.com works).
3. Keep Follow links on the same website on to crawl, and set Maximum pages and Maximum link depth.
4. Run it and download JSON, JSONL, CSV or Excel, or read the dataset from your RAG pipeline through the API.

Input example

{
  "startUrls": [{ "url": "https://docs.python.org/3/tutorial/index.html" }],
  "crawlLinks": true,
  "maxPages": 200,
  "maxDepth": 3,
  "useSitemaps": true,
  "excludeUrlPatterns": ["**/genindex*", "**/search.html*"],
  "renderJavaScript": "auto",
  "contentExtraction": "smart",
  "chunkSize": 1500,
  "chunkOverlap": 150,
  "downloadFiles": true,
  "incrementalStoreName": "python-docs-state",
  "minWordCount": 50
}

Output example

{
  "url": "https://docs.python.org/3/tutorial/introduction.html",
  "loadedUrl": "https://docs.python.org/3/tutorial/introduction.html",
  "title": "3. An Informal Introduction to Python — Python 3.15.2 documentation",
  "description": "In the following examples, input and output are distinguished by the presence or absence of prompts...",
  "language": "en",
  "wordCount": 2735,
  "depth": 1,
  "httpStatus": 200,
  "canonicalUrl": "https://docs.python.org/3/tutorial/introduction.html",
  "renderedWithBrowser": false,
  "contentType": "html",
  "source": "link",
  "referrerUrl": "https://docs.python.org/3/tutorial/index.html",
  "crawledAt": "2026-10-28T14:49:00+00:00",
  "markdown": "# 3. An Informal Introduction to Python\n\nIn the following examples, input and output are distinguished by the presence or absence of prompts...",
  "text": "3. An Informal Introduction to Python\n\nIn the following examples, input and output are distinguished...",
  "chunks": [
    { "index": 0, "text": "# 3. An Informal Introduction to Python\n\nIn the following examples...", "headings": ["3. An Informal Introduction to Python"], "charCount": 1432 }
  ],
  "metadata": {
    "title": "3. An Informal Introduction to Python — Python 3.15.2 documentation",
    "canonicalUrl": "https://docs.python.org/3/tutorial/introduction.html",
    "languageCode": "en",
    "author": null,
    "publishedAt": null,
    "modifiedAt": null,
    "siteName": "Python documentation",
    "openGraph": { "title": "3. An Informal Introduction to Python", "type": "website", "site_name": "Python documentation" },
    "headers": { "content-type": "text/html", "last-modified": "Tue, 27 Oct 2026 22:10:41 GMT" }
  },
  "contentHash": "5b0f6c3e8f1a4d2c9e7b6a5d4c3b2a19",
  "contentSignals": null
}

Use it with LangChain, LlamaIndex and vector databases

Every record is one page (or one PDF/DOCX file) with its url for citations, so it maps directly onto a "document" in RAG frameworks. Turn on Chunk size to get ready-made chunks with their heading path, or chunk the markdown field yourself.

LangChain (Python). Install langchain-apify and load the results as documents:

from langchain_apify import ApifyWrapper
from langchain_core.documents import Document

apify = ApifyWrapper()  # reads APIFY_API_TOKEN from the environment
loader = apify.call_actor(
    actor_id="plotpointtech/website-to-markdown-crawler",
    run_input={"startUrls": [{"url": "https://docs.python.org/3/tutorial/"}], "maxPages": 100},
    dataset_mapping_function=lambda item: Document(
        page_content=item.get("markdown") or "",
        metadata={"source": item["url"], "title": item.get("title")},
    ),
)
docs = loader.load()

To use the crawler's own chunks instead, read the dataset with apify-client and create one Document per entry in item["chunks"], putting item["url"] and chunk["headings"] in the metadata.

LlamaIndex (Python). Use the Apify reader from llama-index-readers-apify:

from llama_index.core import Document, VectorStoreIndex
from llama_index.readers.apify import ApifyActor

reader = ApifyActor("YOUR_API_TOKEN")
documents = reader.load_data(
    actor_id="plotpointtech/website-to-markdown-crawler",
    run_input={"startUrls": [{"url": "https://docs.python.org/3/tutorial/"}], "maxPages": 100},
    dataset_mapping_function=lambda item: Document(text=item.get("markdown") or "", metadata={"url": item["url"]}),
)
index = VectorStoreIndex.from_documents(documents)

Vector databases (Pinecone, Qdrant, Weaviate, Chroma, pgvector, OpenAI vector stores). Run with Chunk size set (1,000 to 2,000 characters suits most embedding models), then for each item and chunk embed chunk["text"] and upsert it with an ID such as contentHash + "-" + chunk["index"] and metadata url, title and headings. Apify's Pinecone and Qdrant integrations can also take the dataset directly after each run.

Keeping an index fresh. Schedule the crawler with Remember pages in key-value store set to a fixed name. Each run then outputs only pages that are new or changed (changeStatus), so you re-embed just those; unchanged pages are not charged. Use Only pages modified since to skip old sitemap entries entirely on large sites.

How much does it cost?

$1 per 1,000 pages ($0.001 per page or document saved), plus a $0.00005 start fee per run. Pages that need a real browser because they are built with JavaScript cost an extra $0.002 each ($3 per 1,000 in total), because a browser takes many times more computing power than a direct download. In Auto mode the browser is only used when a page needs it, so ordinary sites cost $1 per 1,000. Apify's higher subscription plans get lower per-page prices automatically.

You convert	You pay
A 20-page docs section	$0.02
A 500-page help center	$0.50
10,000 pages	$10.00
1,000 pages of a JavaScript-only site	$3.00

Each converted PDF or DOCX file counts as one page, however many pages it has. Pages that fail to load, are blocked by robots.txt, are not HTML, are duplicates, are unchanged since the last incremental run, or fall below your minimum word count are not saved and not charged.

Use it via API, Make, Zapier, n8n or an AI agent

Apify API. Start a run and get the results back in one HTTP request (replace YOUR_API_TOKEN with the token from Apify Console > Settings > Integrations):

curl -X POST "https://api.apify.com/v2/acts/plotpointtech~website-to-markdown-crawler/run-sync-get-dataset-items?token=YOUR_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"startUrls": [{"url": "https://docs.python.org/3/tutorial/index.html"}], "crawlLinks": true, "maxPages": 20, "maxDepth": 2, "minWordCount": 50}'

For longer jobs, start the run asynchronously and read the dataset when it finishes. Official apify-client libraries exist for Python and JavaScript; the API tab on this page has ready-made code.

No-code automation. Connect the tool to Make, Zapier, n8n, Google Sheets, Slack, Airtable or any webhook with Apify integrations, and use Schedules in Apify Console to run it hourly, daily or weekly.

AI agents (MCP). Give Claude, ChatGPT, Cursor, VS Code or any MCP-compatible agent direct access to this tool through the hosted Apify MCP server. Add this server URL to your client:

https://mcp.apify.com?tools=plotpointtech/website-to-markdown-crawler

The agent can then call the tool with the same input fields shown above and read the results.

FAQ

Is it legal to crawl websites for AI? Collecting public web pages is generally lawful, but rights in the content stay with the website owner. This crawler always respects robots.txt and Crawl-delay, identifies itself as PlotPointBot, and returns each site's declared AI-usage preferences in contentSignals. Make sure you have the right to use the content for your purpose, and honour those preferences.

How do I load the output into LangChain, LlamaIndex or a vector database? Read the dataset through the Apify API (or the Apify LangChain/LlamaIndex loaders) and chunk the markdown field. Each record keeps its url for citations.

Does it include images? No. Images are left out of the Markdown because their alt text is not reliable enough for AI use.

Does it follow subdomains? www.example.com and example.com are treated as the same site; other subdomains are not followed.

Will it work on sites behind a login or bot check? No. Only public pages are supported, and blocking is not bypassed.

Related tools from Plot Point Technologies

• AI Web Scraper - Extract Any Website to JSON in Plain English: describe the fields you want and get JSON from any web page.
• PDF to Text & Tables Extractor - Markdown + OCR, $0.40/1K: PDFs to text, Markdown and CSV tables, with OCR for scans.
• SEO Audit Crawler - Titles, Meta, H1, Broken Pages & Links: on-page SEO check of every page in a website's sitemap.
• Wikipedia Scraper - Wikidata Facts, Summaries & Images: Wikipedia summaries and Wikidata facts for any entity list.

Plot Point Technologies is not affiliated with any website you crawl with this tool, or with Apify.

---

If this tool saved you time, we would be grateful for a short review on this page; it helps other people find it. If something did not work as expected, please open an issue on the Issues tab and we will look into it quickly.