Website to Markdown for AI, LLMs & RAG

What does Website to Markdown for AI, LLMs & RAG do?

This Actor crawls a website and turns every page into clean Markdown, ready for ChatGPT, Claude, custom GPTs, RAG pipelines and vector databases. Enter a homepage (or a list of URLs) and get one row per page with the main content as Markdown, plus one combined llms-full.md file with the whole site.

Menus, headers, footers, forms, language switchers and cookie banners are removed automatically. Headings, lists, tables, links and images are kept, and relative links become absolute URLs. It reads the HTML directly, without a browser and without proxies, so runs are fast and cheap.

Why use this website to Markdown converter?

• Build a knowledge base for your chatbot: feed your docs, help center or product pages into a custom GPT, Claude Project or RAG app
• Create llms-full.txt for your website: the combined file lists every page with its title and source URL
• Clean input for embeddings: no navigation noise repeated on every page, so vector search finds real content
• Stay focused with URL patterns: crawl only /docs/** or exclude /tag/**
• Pay per page: you control the cost with Max pages
• Run it from the Apify API, Make, Zapier, n8n or LangChain, or schedule it to keep your knowledge base fresh

What data does it return?

Field	Description
url	Final URL of the page
title, description, lang	Page title, meta description and language
markdown	Main content as Markdown
wordCount	Words in the main content
crawlDepth, foundOn	Clicks from the start URL and the page that linked to it
truncated	True if the Markdown was cut at 2 million characters
error	Reason if the page could not be loaded

The key-value store record llms-full.md contains all pages in one file, each starting with its title and source URL. It is limited to 8 MB; the dataset always contains every page in full.

How to convert a website to Markdown

1. Click Try for free.
2. Enter your homepage in Start URLs.
3. Set Max pages (start with 20 to check the result).
4. Optional: limit the crawl with Include URLs, e.g. https://example.com/docs/**.
5. Click Start.
6. Download the dataset as JSON, CSV or Excel, or open llms-full.md in the Storage tab.

How much does it cost?

This Actor uses pay per event: a fixed price per converted page, and platform usage is included. The current price is shown in the Pricing tab. Pages that return an error or can't be reached are not charged. With Max pages you always know the maximum cost in advance.

Input

See the Input tab for all options.

{
    "startUrls": [{ "url": "https://example.com/docs/" }],
    "crawlSite": true,
    "maxPages": 100,
    "includeUrlGlobs": ["https://example.com/docs/**"],
    "excludeUrlGlobs": ["**/changelog/**"],
    "removeSelectors": ".newsletter-box, #related-posts"
}

Output

You can download the dataset in various formats such as JSON, HTML, CSV, or Excel.

{
    "url": "https://example.com/docs/getting-started/",
    "statusCode": 200,
    "crawlDepth": 1,
    "foundOn": "https://example.com/docs/",
    "title": "Getting started | Example Docs",
    "description": "Install Example and run your first project in five minutes.",
    "lang": "en",
    "wordCount": 640,
    "truncated": false,
    "markdown": "# Getting started\n\nInstall the CLI:\n\n```\nnpm i -g example\n```\n\n| Plan | Price |\n| --- | --- |\n| Free | $0 |",
    "error": null
}

Related Actors

• SEO Site Audit & Broken Link Checker: crawl a whole website and find on-page SEO issues and broken pages
• Structured Data, Schema.org & Open Graph Extractor: extract JSON-LD, schema.org types and Open Graph tags from many pages
• PDF to Text with OCR: your knowledge base also has PDFs? Extract their text, scanned pages included
• PageSpeed Insights & Core Web Vitals Bulk Audit: Lighthouse scores and real-user Core Web Vitals for many URLs

FAQ

Does it work with JavaScript-heavy sites?

The Actor reads the HTML the server delivers. Most websites, docs and blogs deliver their content this way. Single-page apps that render everything in the browser return little or no content; for those, use a browser-based crawler.

How does it find the main content?

It uses the page's <main> element (or a single <article>) when there is one. Otherwise it takes the whole page and removes header, footer, navigation, forms and cookie banners. If a site has extra boxes you don't want, add their CSS selectors in Remove elements.

Will it overload my server?

No. It follows robots.txt by default and uses at most 10 parallel requests.

Something does not work

Please open an issue in the Issues tab with the URL and the run ID. You can also start the Actor programmatically from the API tab.