Website to Markdown Crawler (for AI, RAG and LLMs)

Website to Markdown Crawler crawls any website and returns clean Markdown of every page, without menus, footers, cookie banners or scripts, plus the title, description, language and word count. It is a website to Markdown converter for developers and AI teams who feed ChatGPT, Claude, RAG pipelines, vector databases and AI agents. Use it to turn a docs site, help center or blog into LLM-ready text, or to convert a whole website to Markdown in one run. Fast: plain HTTP, no browser.

What does Website to Markdown Crawler do?

You give it one or more start URLs. The crawler downloads each page, follows its links up to the depth and page limits you set, cleans the HTML and converts the main content to Markdown. Every page becomes one row with its URL, metadata and Markdown.

Start URLs can be:

Start URL	What gets crawled
A home page, e.g. https://example.com	The whole website, up to your limits
A section, e.g. https://docs.apify.com/academy/	That section and the pages it links to (turn on "Stay under the start path" to keep it inside the section)
A single page with maxDepth 0	Only that page

URLs without https:// get it added. Several websites in one run are fine, and each one gets its own page limit.

What the cleaning does:

• Removed: scripts, styles, iframes, SVG and canvas, forms and buttons, nav, header, footer and aside blocks, navigation, banner and footer roles, hidden elements, common cookie banner and newsletter boxes, screen-reader-only text, and all images, video and audio.
• Main content first: if the page has a main, article, [role="main"], #content or .content element with more than 200 characters of text, only that part is converted. Otherwise the cleaned page body is used.
• Kept: headings (# style), paragraphs, lists (- bullets), bold and italic text, fenced code blocks, data tables (tables with header cells become Markdown tables) and links, which are made absolute. Layout tables without header cells are flattened into plain text blocks.

What data can you extract?

Field	What it means	Example
url	Final page URL (after redirects)	https://docs.apify.com/academy
title	Page <title>, or the first H1 when there is no title	Apify Academy | Academy | Apify Documentation
description	Meta description (or og:description)	Learn everything about web scraping and automation…
language	Language from the page's lang attribute	en
canonicalUrl	Canonical link as written in the page	https://docs.apify.com/academy
depth	Link depth from the start URL (0 = start page)	0
statusCode	HTTP status code of the page	200
wordCount	Number of words in the Markdown	204
markdown	Clean Markdown of the page's main content	# Apify Academy\n\nLearn everything about…
crawledAt	When the page was crawled	2026-10-11T20:23:30.322Z
error	Only on the error row when no page at all could be loaded	No pages could be loaded

Fields that a page does not have (for example no meta description) come as null.

How to use Website to Markdown Crawler

1. Open Website to Markdown Crawler in Apify Console (sign in or create an Apify account).
2. In Start URLs, add one or more websites or pages, one per line.
3. Set Max pages per website and Max link depth. Turn on Stay under the start path to crawl only one section, such as /docs/.
4. Optionally add CSS selectors in Extra elements to remove, for example .sidebar, #comments.
5. Click Start. The log shows how many pages were saved.
6. Open the Output tab and click Export to download JSON, CSV, Excel or HTML, or load the dataset from the API straight into your RAG pipeline.

Input

Field	Type	Default	Description
startUrls	array of strings (required)	["https://docs.apify.com/academy"]	Websites or pages to start from.
maxPages	integer, 1 to 100,000	10	Pages returned for each start website, counted by domain.
maxDepth	integer, 0 to 20	3	0 = only the start pages, 1 = also the pages they link to, and so on.
sameDomainOnly	boolean	true	Only follow links to the start URLs' domains.
stayUnderStartPath	boolean	false	Only follow links that start with the start URL's folder (for example only /docs/...).
removeSelectors	string	empty	Extra CSS selectors to remove before converting, comma-separated.
proxy	object	{"useApifyProxy": true}	Apify Proxy settings.

Notes on the settings:

• Websites are counted by host name, ignoring www.. Subdomains are separate websites: docs.example.com and example.com each have their own maxPages. If a start URL redirects to another host (for example to a docs. subdomain), that host is crawled as the start website.
• "Start path" means the folder up to the last slash. https://example.com/docs/ keeps the crawl inside /docs/, but https://example.com/docs (no trailing slash) only keeps it under https://example.com/. End your start URL with a slash when you use this option.
• With "Stay on the same website" off, external links are followed too, and each new domain gets its own maxPages limit.

Example input:

{
  "startUrls": [
    "https://docs.apify.com/academy/"
  ],
  "maxPages": 200,
  "maxDepth": 3,
  "sameDomainOnly": true,
  "stayUnderStartPath": true,
  "removeSelectors": ".sidebar, #comments",
  "proxy": {
    "useApifyProxy": true
  }
}

Output example

A real row from the latest run (the Markdown is shortened; the full page had 204 words):

{
  "url": "https://docs.apify.com/academy",
  "title": "Apify Academy | Academy | Apify Documentation",
  "description": "Learn everything about web scraping and automation with our free courses that will turn you into an expert scraper developer.",
  "language": "en",
  "canonicalUrl": "https://docs.apify.com/academy",
  "depth": 0,
  "statusCode": 200,
  "wordCount": 204,
  "markdown": "# Apify Academy\n\nCopy for LLM\n\nLearn everything about web scraping and automation with our free courses that will turn you into an expert scraper developer.\n\n## Beginner courses\n\n[\n\n#### Web scraping basics with JS\n\nLearn how to use JavaScript to extract information from websites in this practical course, starting from the absolute basics.\n\n](https://docs.apify.com/academy/scraping-basics-javascript)\n\n…",
  "crawledAt": "2026-10-11T20:23:30.322Z"
}

How much does it cost?

Website to Markdown Crawler uses pay per event pricing. You pay $1.00 per 1,000 pages returned ($0.001 per page) on the Free plan. Platform usage (compute and proxy) is included in that price.

Apify plan	Price per 1,000 pages
Free	$1.00
Bronze (10% off)	$0.90
Silver (20% off)	$0.80
Gold and higher (30% off)	$0.70

Worked examples on the Free plan:

• A default run (10 pages) = $0.01
• A help center with 1,500 pages = $1.50
• 20 company websites with 500 pages each = 10,000 pages = $10.00 ($7.00 on Gold)

What you are not charged for:

• Pages that fail to load after retries.
• Pages whose Markdown is shorter than 20 characters (for example empty pages). They are skipped, not saved.
• Links to files that are never downloaded (PDF, images, ZIP, Office documents and similar).
• The error row "No pages could be loaded".

Use cases

• RAG knowledge bases. Crawl your docs or help center, split the Markdown into chunks and load them into a vector database for a support chatbot.
• Custom GPTs and Claude projects. Export a site section as Markdown and upload it as knowledge files.
• AI agents. Give an agent the content of a competitor's website, a product catalog or a pricing page as clean text.
• Summaries and analysis. Feed blog posts or product pages into an LLM for summaries, classification or sentiment.
• Content audits. Use wordCount, title, description and language to find thin pages, missing meta descriptions or pages in the wrong language.
• Site migration and archiving. Save a website's text as Markdown before a redesign or CMS move.
• Documentation monitoring. Re-crawl docs on a schedule and compare the Markdown to spot changes.

Tips and limits

• No JavaScript rendering. The crawler uses plain HTTP and does not run a browser. It works best on server-rendered sites (most docs, blogs, marketing and shop pages). Pages that only render with JavaScript may return little text or be skipped.
• No images. Images, video and audio are removed, so the Markdown contains text and links only.
• Files are not converted. Links to PDF, images, ZIP, audio and video, Word, Excel and PowerPoint files, TXT, MD, YAML, JSON, XML, RSS, CSS and JS are not followed.
• Pages are found by following links. The crawler does not read sitemap.xml. Pages that nothing links to are only crawled if you add them as start URLs.
• Same page, different anchors. The #section part of links is removed, so a page is not crawled twice for different anchors.
• Budget per website. The crawler queues at most 2 × maxPages links per website and makes at most 4 × maxPages × (number of start websites) requests in total. On sites where many pages fail or are skipped you can get fewer than maxPages rows.
• Speed and retries. Up to 10 pages load at the same time and each page gets 2 retries. With sameDomainOnly on, the run stops as soon as every start website has reached maxPages.
• Remove what is left. If a site's sidebar or "related posts" box still ends up in the Markdown, add its CSS selector in removeSelectors.
• Error row. When not a single page could be loaded (bad URL, site down or blocking), the run saves one row with error: "No pages could be loaded".

Use it from the API, Python or JavaScript

JavaScript (Node.js) with the official apify-client package:

const { ApifyClient } = require('apify-client');

const client = new ApifyClient({ token: 'YOUR_TOKEN' });
const input = { startUrls: ['https://docs.apify.com/academy/'], maxPages: 50, stayUnderStartPath: true };

(async () => {
    const run = await client.actor('juliob.rangel/website-to-markdown-crawler').call(input);
    const { items } = await client.dataset(run.defaultDatasetId).listItems();
    for (const page of items) console.log(page.url, page.wordCount);
})();

Python with the official apify-client package, saving each page as a .md file:

from apify_client import ApifyClient

client = ApifyClient('YOUR_TOKEN')
run_input = {'startUrls': ['https://docs.apify.com/academy/'], 'maxPages': 50, 'stayUnderStartPath': True}

run = client.actor('juliob.rangel/website-to-markdown-crawler').call(run_input=run_input)
items = client.dataset(run['defaultDatasetId']).list_items().items
for i, page in enumerate(items):
    if page.get('markdown'):
        with open(f'page_{i}.md', 'w', encoding='utf-8') as f:
            f.write(page['markdown'])

Plain HTTP, running the crawler and getting the rows in one call:

curl -X POST "https://api.apify.com/v2/acts/juliob.rangel~website-to-markdown-crawler/run-sync-get-dataset-items?token=YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"startUrls": ["https://docs.apify.com/academy/"], "maxPages": 20}'

The crawler also works with Make, Zapier and n8n, can export results to Google Sheets, can call your webhook when a run finishes, and can be used by AI agents through the Apify MCP server.

FAQ

Does it render JavaScript?

No. It downloads pages over plain HTTP, which makes it fast and cheap. Sites that build their content only with JavaScript may return little text.

Do I need a login or cookies?

No, and it cannot crawl pages that are behind a login. It reads the public pages any visitor can open.

Can I export the Markdown to CSV, Excel or JSON?

Yes. The Markdown is one column of the dataset, so you can export as JSON, CSV, Excel or HTML. For RAG pipelines, JSON or the API keeps the line breaks intact.

How many pages can I crawl?

Up to 100,000 pages per website (maxPages), on as many websites as you add as start URLs.

How do I crawl only one section of a site?

Start from that section with a trailing slash, for example https://example.com/docs/, and turn on stayUnderStartPath.

Is it legal to crawl websites?

The crawler only collects publicly available pages. You are responsible for complying with each website's terms and with privacy and copyright rules, including laws such as GDPR and LGPD when pages contain personal data. Personal data needs a legitimate reason to be collected and processed.

Why did I get fewer pages than maxPages, or an error row?

The site may have fewer reachable pages within maxDepth, many pages may be files or JavaScript-only, or some pages may have failed. If not a single page loaded, you get the error row "No pages could be loaded". Check the URL and try a higher maxDepth.

Related scrapers

• Website Contact Details Scraper (Emails, Phones, Socials)
• YouTube Transcript Scraper (Fast & Cheap)
• Substack Scraper (Posts, Likes, Comments, Full Text)
• Google News Scraper (Real Article URLs)

---

Sites em Markdown para IA (PT-BR)

Rastreie qualquer site e receba o texto limpo de cada página em Markdown, sem menus, rodapés ou banners de cookies, pronto para ChatGPT, Claude e bases de conhecimento (RAG). Você paga só por página retornada.