Website to Markdown Crawler: Firecrawl Alternative for RAG

Turn any web page or URL into clean Markdown, or crawl a whole site as a web crawler for Markdown, for AI agents and RAG. It works on pages that don't need JavaScript. You pay only per page, failed pages are free, and there's no subscription.

Website to Markdown use cases

• Load a documentation site, help center or blog into a RAG index or a custom GPT.
• Feed your help center to a support chatbot.
• Give an AI agent the text of one page or a whole section of a site.
• Keep a Markdown copy of your own site's content.

What you get for each page

Give it a website and get back every page as clean Markdown, ready for an LLM, a RAG index or an AI agent. Each page comes with its title, description and language, and the main text only: menus, footers, cookie banners, forms, scripts and images are taken out, and links are kept as full addresses. It obeys robots.txt and paces its requests. $0.002 per page, and failed pages are free. It works for people and for AI agents through Apify's MCP server.

Input

Field	What to put
Start URLs	Where to start, one per line, like https://docs.example.com/guide/. Required.
Max pages	Stop after this many pages. Default 50. Set 1 to fetch just the start URL.
Max link depth	How many links away from a start URL to go. Default 5. 0 saves only the start URLs.
Crawl the whole site	Off by default: the crawl stays under the start URL's folder (start at /docs/ and it stays in /docs/). On: any page on the same site.
Skip URLs matching	Pages to leave out, like https://example.com/blog/tag/**. * matches within one part of the path, ** across parts.
Proxy	Off by default. Turn on Apify Proxy for a site that blocks cloud servers.

Example:

{
  "startUrls": ["https://crawlee.dev/js/docs/quick-start"],
  "maxPages": 20,
  "maxDepth": 3
}

Read Google results

Pass the result URLs from Google Search Scraper as Start URLs with Max link depth 0. This crawler does not run JavaScript, so pages that need it may come back thin.

Output

One row per page. This one came from crawling crawlee.dev on October 15, 2026 (Markdown cut short here):

{
  "type": "page",
  "url": "https://crawlee.dev/js/docs/quick-start",
  "title": "Quick Start | Crawlee for JavaScript · Build reliable crawlers. Fast.",
  "description": "With this short tutorial you can start scraping with Crawlee in a minute or two. To learn more, read the Introduction.",
  "language": "en",
  "canonicalUrl": "https://crawlee.dev/js/docs/quick-start",
  "markdown": "Version: 3.18\n\n# Quick Start\n\nWith this short tutorial you can start scraping with Crawlee in a minute or two. To learn in-depth how Crawlee works, read the [Introduction](https://crawlee.dev/js/docs/introduction), which is a comprehensive step-by-step guide for creating your first scraper.\n\n## Choose your crawler\n\n...",
  "wordCount": 897,
  "depth": 0,
  "statusCode": 200,
  "crawledAt": "2026-10-15T01:48:51.191Z"
}

• Tables stay tables, code stays in code blocks, and lists stay lists.
• A page that redirected has requestedUrl, the address that was asked for.
• A page that failed (a 404, a server error, a file that isn't a web page, like JSON, or a page with no text in its HTML) has error and no Markdown, and isn't charged.

The run's SUMMARY record says how many pages were saved and failed, how many robots.txt kept out, and why the crawl stopped (done, maxPages or chargeLimit).

Price

• $0.002 per page saved, so $2 per 1,000 pages.
• That's the price on Apify's free plan. On a paid Apify plan it's less: $1.80 per 1,000 pages on Starter, $1.60 on Scale and $1.40 on Business, the "from" price at the top of this page.
• Failed pages, pages robots.txt disallows and files that aren't web pages are free.
• You can set a maximum charge for a run. The crawl stops when it's reached and keeps what it already saved.

Good to know

• Polite by design. It reads each site's robots.txt and skips what it disallows. It fetches one page at a time with a one-second pause between pages, so expect about 2.5 seconds a page: 20 pages of crawlee.dev took 53 seconds on Apify, start-up included.
• It reads the HTML the site sends. Most docs sites, blogs and help centers work. A site that draws its text with JavaScript after loading has no text to read: those pages are failed rows, not charged.
• Only web pages and text files are saved. A plain text or Markdown file (a README.md, an llms.txt) comes back as it is. Links to PDFs, images, archives and other files are skipped.
• Each page is saved once. Links to the same page with a different #anchor, or through a redirect, don't make extra rows.
• Redirects are followed, including pages that only forward with a meta refresh. If a start URL moves to another address (like docs.example.com to example.github.io/docs), the crawl carries on there.
• It doesn't log in and only reads public pages.

Use it from n8n, Zapier or Make

You don't need a separate app. Apify's own integration runs this Actor in all three. Use the Actor ID nebulacolors/website-content-markdown and put where to start in startUrls, for example {"startUrls": ["https://docs.example.com/guide/"], "maxPages": 50}.

• n8n: add the Apify node (search "Apify" in the nodes panel) and pick Run an Actor and get dataset. You get one row per page back, with its Markdown, ready for the next node, like a vector store or Google Docs.
• Zapier: add the Apify app and use the Run Actor action. To get the rows, add a second Apify step that reads the run's dataset.
• Make: use the Apify module Run an Actor, then Get Dataset Items with the run's dataset ID.

A common setup is a weekly schedule that crawls your docs site, so your RAG index stays current. Failed pages cost nothing.

Use it from an AI agent

The Actor works through Apify's MCP server like any Store Actor. An agent can find it, read its input and call it. Only startUrls is required. To read one page, set maxPages to 1, which usually finishes within the MCP call's wait. In a test of 10 typical pages (docs, news, blogs, GitHub, a product page, Wikipedia), each fetch took 3 to 8 seconds end to end, and all 10 came back with the page's text. The MCP call waits 30 seconds by default (45 at most). A bigger crawl takes longer, so the call returns the run's status and dataset ID, and the agent reads the rows with get-dataset-items.

FAQ

How do I convert a website to Markdown for RAG? Give it the site's address. It crawls the pages and returns each one as clean Markdown with its title, description and language, ready to chunk and embed.

Can it fetch just one page? Yes. Set Max pages to 1. It usually finishes within an AI agent's wait.

Does it work on sites built with JavaScript? It reads the HTML the site sends. Docs sites, blogs and help centers work. A site that draws its text with JavaScript after loading has no text to read, so those pages come back as failed rows and aren't charged.

Does it respect robots.txt? Yes, and it paces its requests.

More tools from us

• AI Brand Mention Checker: is your brand named by ChatGPT and Gemini, and who comes first.
• Google Ads Transparency Scraper: every Google, YouTube and Display ad a competitor runs.
• Website Screenshot: full page or mobile screenshots of any URL, PNG or JPEG.
• News & Blog Article Extractor: the newest articles from a site or RSS feed, as Markdown.
• Image Downloader: every image on a page, full size, in one ZIP.

Questions or problems

Open an issue on this Actor's Issues tab. I read them. If it saves you time, a short review helps others find it.