Website to Markdown Crawler for AI & RAG

Turn a whole website, or one section of it, into clean Markdown for your LLM, chatbot or RAG index. Give a start URL such as a documentation site, a blog or a help center, and the Actor crawls its pages and returns each one as Markdown with the main content only, plus title, description, language and dates. Limit the crawl to a section, include or exclude URL patterns, and schedule it to get only the pages that are new or changed.

No browser: the Actor fetches pages over plain HTTP and converts them with a fast HTML-to-Markdown engine that keeps headings, lists, tables, code blocks and absolute links. robots.txt is respected. You pay $0.0005 per page with content.

What you get

One Dataset row per page:

• url, title, description, markdown, and text with Include plain text.
• wordCount, language, author, publishedAt, modifiedAt, canonical, linksCount.
• truncated when the page was cut at Max characters per page, likelyJsRendered when it needs JavaScript and little text was found.
• contentHash and changeStatus (new or changed compared with the previous run of the same start URL).

Pages without text (for example apps that need JavaScript) are returned but not charged.

How to use it

1. Open the Input tab.
2. In Start URLs paste one per line: https://docs.apify.com/platform, https://yourcompany.com/blog.
3. Set Max pages per start URL (default 50, up to 500).
4. Keep Stay within the start URL on to crawl only that section, or use Include URL patterns and Exclude URL patterns with * as a wildcard (for example */docs/guides/*, */changelog*).
5. Click Start and export the rows as JSON or CSV, or send them to your vector database with an Apify integration.
6. To keep a knowledge base fresh, schedule the Actor; filter the rows by changeStatus to re-index only what changed.

Input

{
  "startUrls": ["https://crawlee.dev/js/docs/quick-start"],
  "maxPages": 100,
  "stayInPath": true,
  "includePatterns": ["*/docs/guides/*", "*/docs/quick-start*"],
  "excludePatterns": ["*/changelog*"],
  "mainContent": true,
  "maxChars": 30000,
  "includeText": false
}

Field	Description
startUrls	Websites or sections. Up to 50 per run.
maxPages	Pages per start URL, 1 to 500. Default 50.
stayInPath	Only pages under the start URL. Default on; include patterns take precedence.
includePatterns, excludePatterns	URL patterns with * wildcards.
mainContent	Remove navigation, footers and banners. Default on.
maxChars	Characters per page, 1,000 to 200,000. Default 30,000.
includeText	Also return plain text. Default off.

Output example

{
  "startUrl": "https://crawlee.dev/js/docs/quick-start",
  "url": "https://crawlee.dev/js/docs/guides/configuration",
  "title": "Configuration | Crawlee for JavaScript",
  "description": "Configuring Crawlee parameters",
  "markdown": "# Configuration\n\nAll the configuration options can be set with environment variables...",
  "wordCount": 912,
  "language": "en",
  "author": null,
  "publishedAt": null,
  "modifiedAt": null,
  "canonical": "https://crawlee.dev/js/docs/guides/configuration",
  "truncated": false,
  "likelyJsRendered": false,
  "linksCount": 84,
  "contentHash": "4f7c...",
  "changeStatus": null,
  "timestamp": "2026-10-08T12:00:00.000Z"
}

Pricing

Pay per event, no subscription:

• Page crawled: $0.0005 per page with content. A 200-page documentation site costs $0.10.
• Page changed: $0.001 per page that is new or changed in a scheduled run.

Compare with website content crawlers at $3 per 1,000 pages, or browser-based crawling at $5 per 1,000 pages in platform usage.

Use it as an API

curl -X POST "https://api.apify.com/v2/acts/aolmo~website-to-markdown/run-sync-get-dataset-items?token=YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"startUrls": ["https://docs.apify.com/platform"], "maxPages": 50}'

You can also call it from AI agents through the Apify MCP server. For a single page use aolmo/page-to-markdown; to search the web first, aolmo/rag-web-search.

FAQ

Does it render JavaScript? No. It reads the HTML the server sends, which covers documentation sites, blogs and most company websites. Single-page apps that render everything with JavaScript return little text (likelyJsRendered: true) and are not charged.

Does it follow links to other domains? No, only pages of the start URL's website.

Does it respect robots.txt? Yes. Disallowed pages are skipped.

How long does a crawl take? Roughly 1 to 3 seconds per 10 pages, depending on the website.

Where is the data stored between runs? In a Key-Value Store of your account named micro-saas-state-website-to-markdown, one key per start URL. Delete it to start fresh.