Website to Markdown Crawler: Firecrawl Alternative for RAG

Turn a website into clean Markdown for an LLM or RAG index, the job Firecrawl and Jina Reader do, but pay only per page: $2 per 1,000 pages, failed pages free, no plan and no API key.

Website to Markdown use cases

• Load a documentation site, help center or blog into a RAG index or a custom GPT.
• Feed your help center to a support chatbot.
• Give an AI agent the text of one page or a whole section of a site.
• Keep a Markdown copy of your own site's content.

What you get for each page

Give it a website and get back every page as clean Markdown, ready for an LLM, a RAG index or an AI agent. Each page comes with its title, description and language, and the main text only: menus, footers, cookie banners, forms, scripts and images are taken out, and links are kept as full addresses. It obeys robots.txt and paces its requests. $0.002 per page, and failed pages are free. It works for people and for AI agents through Apify's MCP server.

Input

Field	What to put
Start URLs	Where to start, one per line, like https://docs.example.com/guide/. Required.
Max pages	Stop after this many pages. Default 50. Set 1 to fetch just the start URL.
Max link depth	How many links away from a start URL to go. Default 5. 0 saves only the start URLs.
Crawl the whole site	Off by default: the crawl stays under the start URL's folder (start at /docs/ and it stays in /docs/). On: any page on the same site.
Skip URLs matching	Pages to leave out, like https://example.com/blog/tag/**. * matches within one part of the path, ** across parts.
Proxy	Off by default. Turn on Apify Proxy for a site that blocks cloud servers.

Example:

{
  "startUrls": ["https://crawlee.dev/js/docs/quick-start"],
  "maxPages": 20,
  "maxDepth": 3
}

Read Google results

Pass the result URLs from Google Search Scraper as Start URLs with Max link depth 0. This crawler does not run JavaScript, so pages that need it may come back thin.

Output

One row per page. This one came from crawling crawlee.dev on October 15, 2026 (Markdown cut short here):

{
  "type": "page",
  "url": "https://crawlee.dev/js/docs/quick-start",
  "title": "Quick Start | Crawlee for JavaScript · Build reliable crawlers. Fast.",
  "description": "With this short tutorial you can start scraping with Crawlee in a minute or two. To learn more, read the Introduction.",
  "language": "en",
  "canonicalUrl": "https://crawlee.dev/js/docs/quick-start",
  "markdown": "Version: 3.18\n\n# Quick Start\n\nWith this short tutorial you can start scraping with Crawlee in a minute or two. To learn in-depth how Crawlee works, read the [Introduction](https://crawlee.dev/js/docs/introduction), which is a comprehensive step-by-step guide for creating your first scraper.\n\n## Choose your crawler\n\n...",
  "wordCount": 897,
  "depth": 0,
  "statusCode": 200,
  "crawledAt": "2026-10-15T01:48:51.191Z"
}

• Tables stay tables, code stays in code blocks, and lists stay lists.
• A page that redirected has requestedUrl, the address that was asked for.
• A page that failed (a 404, a server error, or a file that isn't a web page, like JSON) has error and no Markdown, and isn't charged.

The run's SUMMARY record says how many pages were saved and failed, how many robots.txt kept out, and why the crawl stopped (done, maxPages or chargeLimit).

Price

• $0.002 per page saved, so $2 per 1,000 pages.
• Failed pages, pages robots.txt disallows and files that aren't web pages are free.
• You can set a maximum charge for a run. The crawl stops when it's reached and keeps what it already saved.

Good to know

• Polite by design. It reads each site's robots.txt and skips what it disallows. It fetches one page at a time with a one-second pause between pages, so expect about 2.5 seconds a page: 20 pages of crawlee.dev took 53 seconds on Apify, start-up included.
• It reads the HTML the site sends. Most docs sites, blogs and help centers work. A site that draws its text with JavaScript after loading gives empty Markdown and a wordCount of 0.
• Only web pages are saved. Links to PDFs, images, archives and other files are skipped.
• Each page is saved once. Links to the same page with a different #anchor, or through a redirect, don't make extra rows.
• Redirects are followed, including pages that only forward with a meta refresh. If a start URL moves to another address (like docs.example.com to example.github.io/docs), the crawl carries on there.
• It doesn't log in and only reads public pages.

Use it from an AI agent

The Actor works through Apify's MCP server like any Store Actor. An agent can find it, read its input and call it. Only startUrls is required. To read one page, set maxPages to 1, which usually finishes within the MCP call's wait (up to 45 seconds). A bigger crawl takes longer, so the call returns the run's status and dataset ID, and the agent reads the rows with get-dataset-items.

FAQ

How do I convert a website to Markdown for RAG? Give it the site's address. It crawls the pages and returns each one as clean Markdown with its title, description and language, ready to chunk and embed.

Can it fetch just one page? Yes. Set Max pages to 1. It usually finishes within an AI agent's wait.

Does it work on sites built with JavaScript? It reads the HTML the site sends. Docs sites, blogs and help centers work. A site that draws its text with JavaScript after loading gives empty Markdown, and the row shows a word count of 0.

Does it respect robots.txt? Yes, and it paces its requests.

More tools from us

• AI Brand Mention Checker: is your brand named by ChatGPT and Gemini, and who comes first.
• Google Ads Transparency Scraper: every Google, YouTube and Display ad a competitor runs.
• Website Screenshot: full page or mobile screenshots of any URL, PNG or JPEG.
• News & Blog Article Extractor: the newest articles from a site or RSS feed, as Markdown.
• Image Downloader: every image on a page, full size, in one ZIP.

Questions or problems

Open an issue on this Actor's Issues tab. I read them.