# Website to Markdown Crawler for LLM & RAG (`parth-tools/website-to-markdown-crawler`) Actor

Crawl a website, sitemap or list of URLs and convert pages to clean Markdown for LLMs, RAG and AI agents. Readability main-content extraction, metadata, optional chunking. Fast HTTP crawler, $1 per 1,000 pages.

- **URL**: https://apify.com/parth-tools/website-to-markdown-crawler.md
- **Developed by:** [Parth](https://apify.com/parth-tools) (community)
- **Categories:** AI, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.00 / 1,000 page converteds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Website to Markdown Crawler for LLM & RAG

**Turn any website into clean, LLM-ready Markdown.** Give it a URL, a list of URLs or a sitemap — the Actor crawls the pages, strips menus, cookie banners, footers and ads with **Mozilla Readability** (the engine behind Firefox Reader View), and converts the main content to tidy Markdown with headings, lists, tables, code blocks and absolute links. Optional **chunking** splits each page into overlapping, heading-aware pieces ready for embeddings and vector databases.

It's a lightweight HTTP crawler (no headless browser), so it's fast and **costs just $1 per 1,000 pages** — no subscription, failed pages are free.

### Use cases

- 🤖 **RAG & chatbots** — feed documentation, help centers or blogs into Pinecone, Qdrant, Weaviate, Chroma, OpenAI vector stores, LangChain or LlamaIndex
- 🧠 **AI agents & custom GPTs** — give an agent fresh knowledge of a website
- 📚 **Docs migration & archiving** — export a site's content to Markdown files
- 🔍 **Content analysis** — word counts, metadata and clean text for SEO or research

### Features

- Three modes: **single pages**, **crawl** (follow links with depth and scope control) or **sitemap** (every URL in sitemap.xml)
- **Main-content extraction** (Readability) or **full page** minus navigation/header/footer
- GitHub-flavored Markdown: tables, fenced code, task lists; absolute links; optional images
- **Chunking** for RAG with overlap, split on headings and paragraphs
- Metadata: title, description, author, published date, language, site name, OG image, canonical URL
- `estimatedTokens` per page to plan LLM costs
- Include/exclude URL patterns, query-string de-duplication, robots.txt support
- Remove extra elements with your own CSS selectors

### Input example

```json
{
    "startUrls": [{ "url": "https://crawlee.dev/js/docs/quick-start" }],
    "crawlMode": "crawl",
    "crawlScope": "path",
    "maxPages": 200,
    "maxDepth": 3,
    "contentMode": "main",
    "chunkSize": 2000,
    "chunkOverlap": 200
}
```

### Output example

```json
{
    "url": "https://crawlee.dev/js/docs/quick-start",
    "statusCode": 200,
    "title": "Quick Start | Crawlee for JavaScript",
    "description": "Crawlee helps you build and maintain your crawlers…",
    "language": "en",
    "wordCount": 755,
    "estimatedTokens": 1290,
    "markdown": "# Quick Start\n\nWith this short tutorial you can start scraping with Crawlee in a minute or two…\n\n## Choose your crawler\n\n…",
    "chunks": ["# Quick Start\n\nWith this short tutorial…", "## Installation\n\n…"],
    "crawledAt": "2026-10-23T06:40:00.000Z"
}
```

Download results as JSON, JSONL, CSV or Excel, or read them via the Apify API / Python & JS clients, LangChain's `ApifyDatasetLoader`, Make, Zapier or n8n.

### Pricing

Pay-per-event, no monthly fee:

| Event | Price | When |
|---|---|---|
| Page converted | **$0.001** ($1 per 1,000 pages) | Each page saved with its Markdown |
| Actor start | $0.00005 | Once per run |

Pages that fail to load, return errors, aren't HTML, are blocked by robots.txt or are under your minimum word count are **free**. Set "Max cost per run" to cap spending — the crawler stops cleanly at the limit.

### FAQ

**Does it run JavaScript?** No — it reads the server-rendered HTML, which covers most documentation sites, blogs, news, help centers and marketing sites, and keeps it fast and cheap. Pages that render everything client-side may come back thin.

**How do I crawl only the docs section?** Set *Crawl scope* to *Same path prefix* and start from e.g. `https://example.com/docs/`, or use *Only URLs matching* with `https://example.com/docs/**`.

**How big should chunks be?** 1,000–2,000 characters (~250–500 tokens) with 100–200 overlap is a good default for most embedding models.

**Can I get plain text or HTML too?** Yes, enable *Also output plain text* and/or *Also output cleaned HTML*.

**Is it allowed to crawl any site?** You are responsible for crawling only content you're permitted to use. The Actor respects robots.txt by default.

# Actor input Schema

## `startUrls` (type: `array`):

Pages or websites to convert. In crawl mode, links are followed from these pages.

## `crawlMode` (type: `string`):

<b>Single pages</b>: convert only the URLs you list. <b>Crawl</b>: follow links from the start URLs. <b>Sitemap</b>: take every URL from the site's sitemap.xml.

## `maxPages` (type: `integer`):

Maximum number of pages to convert in total. Each converted page is billed as one <i>Page converted</i> event.

## `maxDepth` (type: `integer`):

How many link hops from the start URLs to follow (crawl mode).

## `crawlScope` (type: `string`):

Which links to follow: same hostname, same registered domain (incl. subdomains), or only URLs under the start URL's path (e.g. <code>/docs/</code>).

## `includeUrlPatterns` (type: `array`):

Glob patterns like <code>https://example.com/docs/\*\*</code> or /regex/.

## `excludeUrlPatterns` (type: `array`):

Glob patterns or /regex/ for URLs to skip.

## `contentMode` (type: `string`):

<b>Main content</b> uses Mozilla Readability (the Firefox Reader View engine) to keep the article/body and drop menus, footers, cookie bars and ads. <b>Full page</b> keeps everything except navigation, header, footer and sidebars.

## `removeSelectors` (type: `string`):

Extra elements to remove before conversion, e.g. <code>.promo, #related-posts</code>.

## `keepLinks` (type: `boolean`):

Keep hyperlinks as Markdown links (absolute URLs). Off = plain text.

## `keepImages` (type: `boolean`):

Keep images as Markdown image tags.

## `chunkSize` (type: `integer`):

Split each page's Markdown into chunks of about this many characters for RAG / vector databases (heading- and paragraph-aware). 0 = no chunking. ~4 characters ≈ 1 token.

## `chunkOverlap` (type: `integer`):

Characters repeated between consecutive chunks.

## `minWordCount` (type: `integer`):

Pages with fewer words are skipped (not saved, not billed).

## `includeText` (type: `boolean`):

Add a <code>text</code> field without Markdown syntax.

## `includeHtml` (type: `boolean`):

Add the cleaned content HTML.

## `ignoreQueryStrings` (type: `boolean`):

Treat URLs that differ only in ?query as the same page.

## `respectRobotsTxt` (type: `boolean`):

Skip URLs disallowed by robots.txt for followed links.

## `userAgent` (type: `string`):

Request identity.

## `maxConcurrency` (type: `integer`):

Parallel requests.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://crawlee.dev/js/docs/quick-start"
    }
  ],
  "crawlMode": "crawl",
  "maxPages": 10,
  "maxDepth": 3,
  "crawlScope": "host",
  "contentMode": "main",
  "keepLinks": true,
  "keepImages": false,
  "chunkSize": 0,
  "chunkOverlap": 200,
  "minWordCount": 0,
  "includeText": false,
  "includeHtml": false,
  "ignoreQueryStrings": true,
  "respectRobotsTxt": true,
  "userAgent": "browser",
  "maxConcurrency": 5
}
```

# Actor output Schema

## `pages` (type: `string`):

No description

## `all` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://crawlee.dev/js/docs/quick-start"
        }
    ],
    "maxPages": 10
};

// Run the Actor and wait for it to finish
const run = await client.actor("parth-tools/website-to-markdown-crawler").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": [{ "url": "https://crawlee.dev/js/docs/quick-start" }],
    "maxPages": 10,
}

# Run the Actor and wait for it to finish
run = client.actor("parth-tools/website-to-markdown-crawler").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://crawlee.dev/js/docs/quick-start"
    }
  ],
  "maxPages": 10
}' |
apify call parth-tools/website-to-markdown-crawler --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,parth-tools/website-to-markdown-crawler"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/BHmlB0tDzcV5ECa7u/builds/NbXX6xEWNEE1hI0nS/openapi.json