Website to Markdown for AI & RAG: Fast Crawler

Turn any website into Markdown for RAG and AI agents. Point it at a site and every page comes back as clean Markdown — no navigation, no cookie banners, no boilerplate.

• main content only, without menus, cookie banners or footers
• page metadata, headings and last-modified dates
• optional chunks with overlap, ready for embeddings

Built for RAG pipelines, AI agents, chatbots, knowledge bases and fine-tuning datasets.

How to use

1. Sign up for Apify — the free plan's monthly credit is enough to try this Actor.
2. Open Website to Markdown for RAG, fill in the input (the form is pre-filled with a working example) and click Start.
3. Watch results stream into the dataset table while the run is going.
4. Export as JSON, CSV, Excel, XML, HTML or RSS — or pull the results from your own code with the Apify API.

Why use this Markdown crawler?

• Fast and cheap. Plain HTTP, no headless browser: 50 documentation pages in about 7 seconds in testing. You pay per converted page, and thin or empty pages below your word threshold are free.
• Clean Markdown.
  • Mozilla Readability main-content extraction, GitHub-flavored Markdown tables and fenced code blocks.
  • Absolute links, heading-anchor noise removed.
• Finds the pages that matter.
  • Discovers URLs from robots.txt sitemaps and follows internal links.
  • Stays inside the section you started from (e.g. /docs/), with include/exclude URL globs.
• RAG-ready output.
  • Heading-aware chunking with configurable size and overlap, and token estimates per page and chunk.
  • lastModified from sitemaps for incremental re-indexing.
• Polite by default. Respects robots.txt and removes tracking parameters.

How to convert a website to Markdown

{
  "startUrls": [{ "url": "https://docs.apify.com/platform/actors" }],
  "maxPages": 500,
  "maxCrawlDepth": 5,
  "stayWithinStartPath": true,
  "useSitemaps": true,
  "excludeUrlGlobs": ["**/changelog/**"],
  "extractionMode": "article",
  "includeLinks": true,
  "chunkSize": 1500,
  "chunkOverlap": 200
}

Input reference

Every option at a glance. A minimal input that works as-is:

{
  "startUrls": [
    {
      "url": "https://docs.apify.com/platform/actors"
    }
  ],
  "maxPages": 100,
  "maxCrawlDepth": 5,
  "stayWithinStartPath": true,
  "useSitemaps": true,
  "extractionMode": "article",
  "includeLinks": true,
  "chunkOverlap": 200,
  "minWordCount": 20,
  "respectRobotsTxt": true
}

All fields:

• startUrls (Start URLs) — Websites or sections to convert, e.g. https://docs.example.com/guide/. Pages under the same path are crawled.
• maxPages (Max pages) — Total pages to convert. You are charged per page. Default 100.
• maxCrawlDepth (Max link depth) — How many clicks away from the start URL to follow links (0 = only start URLs and sitemap pages). Default 5.
• stayWithinStartPath (Stay within start path) — Only crawl URLs under the start URL's folder (e.g. /docs/). Turn off to crawl the whole domain. Default true.
• useSitemaps (Use sitemaps) — Discover pages from robots.txt and sitemap.xml, including last-modified dates. Default true.
• includeUrlGlobs (Include URL patterns) — Only convert URLs matching these globs, e.g. https://example.com/blog/**.
• excludeUrlGlobs (Exclude URL patterns) — Skip URLs matching these globs, e.g. /tag/, **/page/*.
• extractionMode (Content extraction) — How much of each page to keep. Options: article (Main content only (removes menus, footers, ads)), fullPage (Full page (minus navigation, header, footer)); default "article".
• includeLinks (Keep links in Markdown) — Keep url links. Turn off for plain text-like Markdown. Default true.
• includeImages (Keep images) — Keep image references. Default false.
• includePlainText (Also output plain text) — Adds a text field without Markdown syntax. Default false.
• chunkSize (Chunk size (characters)) — Split each page into chunks of about this many characters for embeddings (0 = no chunking). ~4 characters ≈ 1 token. Default 0.
• chunkOverlap (Chunk overlap (characters)) — Characters repeated between consecutive chunks. Default 200.
• minWordCount (Skip pages shorter than (words)) — Pages with less content are not returned or charged. Default 20.
• respectRobotsTxt (Respect robots.txt) — Skip URLs disallowed for all crawlers. Default true.
• proxy (Proxy) — Usually not needed. Enable Apify Proxy if a site blocks requests.

What the Markdown output looks like

{
  "url": "https://docs.apify.com/get-started",
  "title": "Get started | Platform | Apify Documentation",
  "description": "Learn how the Apify platform works - what Actors are, how they get to Apify Store, and how to start by running existing Actors or building your own.",
  "language": "en",
  "canonicalUrl": "https://docs.apify.com/get-started",
  "lastModified": "Thu, 17 Sep 2026 12:02:20 GMT",
  "markdown": "Apify is a cloud platform for web scraping, data extraction, and automation. Everything on it revolves around *Actors* - serverless programs that run in the cloud. This page explains how the platform fits together and where to start, whether you want to use ex…",
  "headings": [
    {
      "level": 1,
      "text": "Get started"
    },
    {
      "level": 2,
      "text": "How Apify works"
    },
    {
      "level": 2,
      "text": "How Actors get to Apify Store"
    }
  ],
  "wordCount": 388,
  "tokenEstimate": 721,
  "chunks": [
    {
      "index": 0,
      "text": "Apify is a cloud platform for web scraping, data extraction, and automation. Everything on it revolves around *Actors* -…",
      "tokenEstimate": 293
    },
    {
      "index": 1,
      "text": "rom your code like a regular API.\n\nFor how the platform is built, what happens during a run, and how workloads stay isol…",
      "tokenEstimate": 238
    }
  ],
  "crawlDepth": 1
}

What can you do with a website to Markdown data?

• RAG & chatbots: index product docs, help centers and blogs into vector databases.
• AI agents: give agents up-to-date website knowledge as Markdown.
• Fine-tuning & evaluation datasets: collect clean text corpora from websites.
• Content migration & SEO audits: export site content with headings and metadata.
• Change monitoring: re-crawl with sitemap lastModified dates.

How much does it cost?

Pay per converted page, plus a negligible start fee per run. Pages shorter than Skip pages shorter than (words) are not returned or charged.

Limits you should know about

• Pages that render their content only with client-side JavaScript may produce little text. For those, a browser-based crawler is needed.
• Respect each website's terms of use when crawling and reusing content.

Pricing — what you pay per page

This Actor uses pay-per-event pricing: no monthly rental, you pay for the results you get. Prices per 1,000 results by Apify plan:

Event	Free plan	Bronze	Silver	Gold+
page — One web page converted to Markdown saved to the dataset	$1.50	$1.35	$1.275	$1.20

There is also a small start fee of $0.00005 per GB of memory ($0.00005 per run at the default 1 GB). Platform compute is included in the price — you are not billed separately for run time. Set a maximum total charge on the run if you want a hard cap, and use the input limits to control how much is fetched. The current prices are always on the Pricing tab.

Tips

• Already set up a search on the website? Paste its URL into startUrls and the filters in it are kept.
• Keep the default proxy setting — it is only used as a fallback when a direct request is blocked.
• Rows the Actor cannot read are logged and skipped, never charged — check the run log if a count looks low.

Automate & schedule

Run this Actor on autopilot and pull results into your own stack:

• Apify API — start runs, fetch datasets and manage schedules over REST.
• apify-client for JavaScript and apify-client for Python — official SDKs.
• Schedules — run it on a cron to keep your data fresh.
• Webhooks — trigger downstream actions the moment a run finishes.

JavaScript

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: 'MY_APIFY_TOKEN' });
const run = await client.actor('tindacloud/website-to-markdown').call({
    "startUrls": [
        {
            "url": "https://docs.apify.com/platform/actors"
        }
    ],
    "maxPages": 100,
    "maxCrawlDepth": 5,
    "stayWithinStartPath": true,
    "useSitemaps": true,
    "extractionMode": "article",
    "includeLinks": true,
    "chunkOverlap": 200,
    "minWordCount": 20,
    "respectRobotsTxt": true
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(`Got ${items.length} records`);

Python

from apify_client import ApifyClient

client = ApifyClient("MY_APIFY_TOKEN")
run = client.actor("tindacloud/website-to-markdown").call(run_input={
    "startUrls": [
        {
            "url": "https://docs.apify.com/platform/actors"
        }
    ],
    "maxPages": 100,
    "maxCrawlDepth": 5,
    "stayWithinStartPath": True,
    "useSitemaps": True,
    "extractionMode": "article",
    "includeLinks": True,
    "chunkOverlap": 200,
    "minWordCount": 20,
    "respectRobotsTxt": True
})

for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

cURL — run and get the results in one request:

curl -X POST "https://api.apify.com/v2/acts/tindacloud~website-to-markdown/run-sync-get-dataset-items?token=MY_APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"startUrls": [{"url": "https://docs.apify.com/platform/actors"}], "maxPages": 100, "maxCrawlDepth": 5, "stayWithinStartPath": true, "useSitemaps": true, "extractionMode": "article", "includeLinks": true, "chunkOverlap": 200, "minWordCount": 20, "respectRobotsTxt": true}'

Integrate with any app

Connect the dataset to thousands of apps — no code required:

• Make — multi-step automation scenarios.
• Zapier — push new records straight into your CRM or sheet.
• n8n — self-hosted workflows.
• Slack — get notified when a run finishes or finds something new.
• Google Drive / Sheets — auto-export every run to a spreadsheet.
• Airbyte — pipe results into your data warehouse.
• GitHub — trigger runs from commits or releases.
• Webhooks — call your own endpoint when a run succeeds or fails.

Use with AI assistants (MCP)

The output is clean, LLM-ready JSON. Call this Actor from Claude, ChatGPT, Cursor or any agent framework through the Apify MCP server — ask your assistant to "turn the docs site of this product into Markdown so you can answer questions about it" and let it run this Actor for you.

Agent-ready: autonomous payments (x402 & Skyfire)

This Actor is agent-ready — AI agents can discover it, run it and pay for it on their own. It uses pay-per-event pricing and limited permissions, so it qualifies for Apify's agentic payments:

• x402 — an open, HTTP-native payment protocol. Agents pay in USDC through the Apify MCP server — no Apify account needed.
• Skyfire — agent-to-service payments with pre-funded Skyfire tokens.

Building an AI agent, MCP tool or data pipeline? This Actor is ready to plug in and pay as it goes.

Ready-made examples

Open one, change the fields, press Start — nothing to configure:

• Convert a documentation site to Markdown for RAG
• Markdown for RAG from any website

Frequently asked questions

Do I need an API key or a login?

No. Paste your input, press Start. There is nothing to connect and no account on the target site to create.

How do I export the results?

Every run produces a dataset you can download as JSON, CSV, Excel, XML or HTML, or read through the Apify API from your own code. You can also connect it to Make, Zapier or n8n.

Can I run this on a schedule?

Yes. Apify schedules run it every hour, day or week without you being there. Each run is independent, so a schedule gives you a fresh snapshot every time.

Can I call it from Python or JavaScript?

Yes — the Apify API runs any Actor and returns the dataset, and the official Python and JavaScript clients wrap it in a few lines. AI agents can call it too, through Apify's MCP server.

What does it cost?

You pay per result, not per hour. Filters are applied before anything is charged, so narrowing the input directly lowers the bill. Pages that cannot be read are reported in the log and never charged.

Is this data public?

Everything collected here is what any website shows to anyone without logging in. Check the target site's terms and the rules that apply to you before using the data commercially.

Related scrapers

Other scrapers from the same maker, built the same way — no browser where it isn't needed, one flat price per result:

• Google Ads Transparency Scraper
• Website Contact & Email Scraper