Website to Markdown Crawler for LLMs and RAG

Crawl a website, a documentation section or a blog and get every page as clean Markdown or plain text, ready to load into a vector database, a custom GPT, an AI chatbot or any RAG pipeline. Give it one or more start URLs; it follows links on the same site, extracts the main content of each page (menus, footers, cookie banners and ads removed), and saves one dataset row per page with the URL, title, description, language, canonical URL, link depth and Markdown.

At a glance: $0.80 per 1,000 pages, plus $0.00005 per run start. Pay per result with no subscription, set a maximum cost per run, export to JSON, CSV or Excel, or call it from the Apify API, Make, Zapier or n8n.

Plain HTTP with no browser, so it is fast and cheap. robots.txt is respected (RFC 9309), including crawl delays, on every request and redirect.

Use cases

• Build a knowledge base for a support chatbot from your help center or docs.
• Load a competitor's public docs or a product's changelog into an LLM for analysis (where their terms allow it).
• Migrate a website's content to Markdown for a static site generator.
• Create a text snapshot of a site for search, auditing or change tracking.

How the crawl is scoped

• Only the same host as the start URL is followed (www. is ignored).
• With Stay within the start URL's folder on (default), https://example.com/docs/ crawls only /docs/....
• Optional include and exclude globs, for example */blog/* or *?page=*.
• Images, PDFs, archives, media, CSS and JS links are skipped.
• Tracking parameters (utm_*, gclid, fbclid) and #fragments are removed before de-duplication.

Input

{
  "startUrls": [{"url": "https://docs.python.org/3/tutorial/"}],
  "maxPages": 10,
  "maxDepth": 5,
  "stayWithinPath": true,
  "contentMode": "main",
  "outputFormat": "markdown"
}

Output example

Real output from a 5-page test crawl of the URL above (Markdown shortened):

{
  "url": "https://docs.python.org/3/tutorial/appetite.html",
  "depth": 1,
  "referrerUrl": "https://docs.python.org/3/tutorial/",
  "finalUrl": "https://docs.python.org/3/tutorial/appetite.html",
  "statusCode": 200,
  "title": "1. Whetting Your Appetite — Python 3.15.2 documentation",
  "description": "If you do much work on computers, eventually you find that there’s some task you’d like to automate. ...",
  "language": "en",
  "canonicalUrl": "https://docs.python.org/3/tutorial/appetite.html",
  "markdown": "# 1. Whetting Your Appetite\n\nIf you do much work on computers, eventually you find ...",
  "wordCount": 56,
  "truncated": true,
  "error": null,
  "crawledAt": "2026-10-28T03:50:00+00:00"
}

A RUN_SUMMARY record in the key-value store lists pages saved, failures, URLs blocked by robots.txt and URLs discovered.

Pricing

Pay per saved page: $0.80 per 1,000 pages ($0.0008 per page), plus the standard Apify run start event of $0.00005 per run. Failed, skipped and robots-blocked pages are free. Max pages caps every run, and Maximum cost per run stops it cleanly.

Use with AI agents / MCP

AI agents can run this Actor as a tool through Apify's MCP server. Add this server URL to your MCP client (Claude, Cursor, VS Code, OpenAI Agents SDK, LangChain and others) and sign in with OAuth, or send your Apify token as Authorization: Bearer <token>:

https://mcp.apify.com?tools=fecklesspleb/website-to-markdown-crawler

The tool is named fecklesspleb--website-to-markdown-crawler. Agents can also find it with search-actors and run it with call-actor. The run returns a dataset ID; read the rows with get-dataset-items, and pass fields (for example url,depth,title,markdown) to keep the response small.

Minimal tool input:

{"startUrls": [{"url": "https://docs.python.org/3/tutorial/"}], "maxPages": 3, "maxCharsPerPage": 300}

This input saves 3 rows: the start page (depth 0, referrerUrl null) and two pages it links to (depth 1). Each row has the fields shown in the Output example above.

Tips for agents:

• Cost is $0.0008 per saved page. maxPages (default 10) caps every run, so set it to what you need, and set maxTotalChargeUsd in the run options (callOptions in call-actor) as a second cap.
• Start from the folder you want: with the default stayWithinPath: true, https://example.com/docs/ crawls only /docs/.... Use excludeUrlGlobs to skip tag, archive or paging URLs.
• If no page is saved, the dataset holds one free row with recordType: "status" and a message saying why (failures, robots.txt blocks).
• The RUN_SUMMARY record in the run's key-value store shows pagesSaved, failed, skippedRobots, urlsDiscovered and stoppedAtMaxPages.

Limits

• No JavaScript rendering, so sites that build their content in the browser may return little text. Server-rendered sites (most docs, blogs, news, marketing sites) work well.
• No login support.
• On the Apify platform, pages whose address is private, loopback or link-local (also after a redirect) are not crawled and are free. They count as failed pages (error: "not fetched: private, loopback and link-local addresses are not allowed" when includeFailedPages is on).

Is this legal?

The crawler identifies itself with its own user agent, obeys robots.txt and crawl delays, and spaces requests to each host. It only reads public pages. You are responsible for using the content in line with each site's terms and copyright.