# Website to Markdown Crawler | LLM & RAG | $0.5 / 1K (`hridayrungta/website-to-markdown-crawler`) Actor

Convert any website to clean Markdown per page for LLM, RAG and vector databases. Main-content extraction, sitemap + llms.txt, glob filters, no browser. $0.50 per 1,000 pages, JavaScript-only pages are never billed.

- **URL**: https://apify.com/hridayrungta/website-to-markdown-crawler.md
- **Developed by:** [Hriday Rungta](https://apify.com/hridayrungta) (community)
- **Categories:** AI, Automation, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.50 / 1,000 page crawleds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

Convert any website to **clean Markdown, plain text and metadata, one row per page**, ready for an LLM prompt, a RAG pipeline or a vector database. Point it at a docs site, a blog, a help center or a news section. It follows links, reads `sitemap.xml` and `llms.txt` if you ask it to, strips navigation, footers, cookie banners and ads, and keeps headings, lists, tables, links and fenced code blocks intact.

There is **no browser** behind it, so it is fast and cheap: **$0.50 per 1,000 pages** plus a $0.0015 start fee, billed only for pages that came back with real content. Pages that need JavaScript to show anything, pages blocked by a bot wall and empty pages are flagged and **never billed**.

#### What can you use this website crawler for?

- **Website to Markdown for LLMs.** Convert any site into Markdown you can paste into ChatGPT, Claude or Gemini, or save as a context file.
- **Crawl a website for RAG.** Get one clean document per page, with `url`, `title`, `language` and `headings` as metadata for chunking and embedding.
- **Build a vector database knowledge base from a website.** Feed the Markdown into Pinecone, Qdrant, Weaviate, pgvector or any other store.
- **Docs to Markdown.** Export a documentation site, API reference or help center into Markdown files for an AI assistant or a chatbot.
- **Scrape a website for LLM training and fine-tuning data.** Collect article text from blogs and news sections with author, publish date and language when the page exposes them.
- **Crawl from `llms.txt` or `sitemap.xml`.** Let the site tell you which pages matter instead of guessing with link depth.
- **Monitor website content changes.** Turn on content hashes and compare runs to see which pages changed.
- **Website to Markdown API.** Call it from your own code through the Apify API and get JSON rows back.

#### What data do you get?

One dataset row per crawled page:

| Field | Description |
|---|---|
| `url`, `loadedUrl`, `canonical` | Requested URL, final URL after redirects, `<link rel="canonical">` |
| `title`, `description`, `language` | From `<title>`, meta description / Open Graph, `<html lang>` |
| `markdown` | The main content as GitHub-flavoured Markdown (headings, lists, links, tables, code blocks, images) |
| `text` | The same content as plain text |
| `headings` | Array of `{ level, text }` for building chunks by section |
| `wordCount` | Words in the extracted content |
| `depth`, `httpStatus`, `crawledAt` | Crawl depth from the start URL, HTTP status, ISO timestamp |
| `contentStatus`, `needsJavaScript`, `charged` | `ok`, `needs-javascript`, `blocked`, `thin`, `http-error`; whether the page was billed |
| `author`, `publishedTime`, `ogImage` | When the page exposes them |
| `contentHash` | Optional SHA-256 prefix of the text, for change detection between crawls |
| `links` | Optional list of `{ url, text }` for every link on the page |
| `extractionMethod` | How the main content was found (`main`, `article`, `readability`, `body-fallback`) |

#### Input

| Field | Default | What it does |
|---|---|---|
| `startUrls` | - | Where to start. Always crawled, even if they do not match the glob patterns. |
| `maxPages` | 100 | Cost cap. The crawl stops after this many billable pages. |
| `maxDepth` | 3 | Link hops from the start URLs. `0` = only the start URLs. Sitemap and llms.txt pages ignore depth. |
| `crawlScope` | same-hostname | `same-hostname`, `same-domain` (incl. subdomains) or `start-url-path`. |
| `includeUrlGlobs` / `excludeUrlGlobs` | empty | Glob patterns on the full URL, e.g. `https://example.com/docs/**`. |
| `useSitemaps` | false | Reads robots.txt `Sitemap:` lines and `/sitemap.xml`, including sitemap indexes and `.gz` files. |
| `useLlmsTxt` | false | Reads `/llms.txt` and queues the pages it links to. |
| `mainContentOnly` | true | Strip navigation, footers, banners and sidebars. Off = whole body. |
| `includeImages` | true | Keep `![alt](url)` in the Markdown. Off saves tokens. |
| `includeLinks` | false | Add a `links` array to each row. |
| `includeContentHash` | false | Add `contentHash` for change detection. |
| `includeSkippedPages` | false | Also save free rows for skipped pages (JavaScript-only, blocked, thin, errors). Start URLs are always saved so you can see why they failed. |
| `respectRobotsTxt` | true | Skip URLs that robots.txt disallows for all crawlers. |
| `maxConcurrency` | 5 | Parallel requests (1-20). Backs off automatically on HTTP 429. |
| `minWords` | 20 | Pages with fewer extracted words count as empty. |
| `removeSelectors` | empty | Extra CSS selectors to delete before extraction. |
| `proxyConfiguration` | none | Optional. Without it, a 403/429 gets one retry through a free Apify datacenter proxy session. |

Example input:

```json
{
  "startUrls": [{ "url": "https://docs.python.org/3/tutorial/index.html" }],
  "maxPages": 25,
  "maxDepth": 2,
  "crawlScope": "start-url-path",
  "includeImages": false,
  "mainContentOnly": true
}
```

#### Output example

```json
{
  "url": "https://react.dev/learn",
  "loadedUrl": "https://react.dev/learn",
  "depth": 0,
  "httpStatus": 200,
  "crawledAt": "2026-10-14T04:54:34.683Z",
  "canonical": "https://react.dev/learn",
  "title": "Quick Start – React",
  "description": "The library for web and native user interfaces",
  "language": "en",
  "markdown": "# Quick Start\n\nWelcome to the React documentation! This page will give you an introduction to 80% of the React concepts that you will use on a daily basis.\n\n### You will learn\n\n-   How to create and nest components\n-   How to add markup and styles\n...",
  "text": "Quick Start\n\nWelcome to the React documentation! ...",
  "headings": [{ "level": 1, "text": "Quick Start" }, { "level": 2, "text": "Creating and nesting components" }],
  "wordCount": 2195,
  "contentStatus": "ok",
  "needsJavaScript": false,
  "extractionMethod": "main>article",
  "charged": true
}
```

#### How to use the website crawler: website to Markdown in 3 steps

1. Add one or more **Start URLs** (for example `https://docs.example.com/`).
2. Pick a **Crawl scope** (same hostname, same domain, or only below the start URL's folder), set **Max pages** and **Max depth**. Optionally add include / exclude URL glob patterns and tick **sitemap.xml** or **llms.txt** discovery.
3. Run it and download the dataset as JSON, CSV or Excel, or read it through the API. Use the `markdown` field as your chunking input.

#### Pricing

Pay per event. No subscription, no compute bill.

| Event | Price |
|---|---|
| Actor start (once per run) | $0.0015 |
| Page crawled with real content | $0.0005 ($0.50 per 1,000 pages) |

Example costs: 100 pages = $0.0515. 1,000 pages = $0.5015. 10,000 pages = $5.0015. Pages flagged as JavaScript-only, blocked, thin or failed are free and do not count against `maxPages`. Set `maxPages` (and Apify's own max-cost-per-run setting) to cap spend.

#### Integrations and scheduling

- **Vector databases and RAG frameworks**: the dataset is plain JSON, so it loads into LangChain, LlamaIndex or your own ingestion script. Apify also offers vector database integrations, for example [Pinecone](https://apify.com/apify/pinecone-integration).
- **Make, n8n, Zapier, Google Sheets**: connect the Actor through Apify's integrations and push finished runs wherever you need them.
- **Webhooks and API**: trigger runs from code and get a webhook when a run finishes.
- **Apify Schedules**: re-crawl a docs site or blog daily or weekly so your knowledge base stays current.
- **Incremental runs for changed pages**: turn on `includeContentHash`, keep the previous run's hashes, and re-embed only the pages whose hash changed. The comparison is done by your workflow, the Actor provides the hash.
- **AI agents**: call it through the Apify MCP server, the input and output fields are documented in the schemas.

#### Known limits (please read)

- **No JavaScript rendering.** This Actor reads the HTML the server sends. Single-page apps that render everything in the browser come back as `needs-javascript` (and are not billed) instead of empty or junk Markdown. For those sites you need a browser-based crawler.
- **Bot walls.** Sites behind an active Cloudflare, Akamai or DataDome challenge may answer 403 or a "Just a moment" page. These are flagged `blocked` and not billed. A 403/429 is retried once through a free datacenter proxy session; some sites still refuse it.
- **Main-content detection is heuristic.** It looks for `<main>`, `<article>` and common content containers, then falls back to a readability algorithm. On index and listing pages (home pages, category pages) the "content" is mostly a list of links and teasers, and some template text can remain. Review a sample before indexing a whole site.
- No login, cookies or private pages. PDFs, images and other non-HTML files are skipped.
- Sitemap discovery reads up to 25 sitemap files per run.

#### FAQ

##### How do I convert a website to Markdown for an LLM with this website crawler?

Add the site's URL as a start URL, set **Max pages**, and run. Each page comes back with a `markdown` field you can paste into a prompt or save to a file. For one section of a site, use the crawl scope `start-url-path`. For a whole site, turn on `useSitemaps` and `useLlmsTxt` so the crawl is driven by the site's own page lists.

##### How do I crawl a website for RAG?

Use each row as one document: `markdown` as the content, `url`, `title` and `language` as metadata, and `headings` to split the text by section before embedding. Load the rows into your vector database with your own script or an Apify integration.

##### Does it work like Apify's Website Content Crawler?

It covers the HTML-only part of that job (Cheerio-style crawling, no browser) at a fraction of the cost. It does not render JavaScript, so it is not a replacement for browser mode.

##### What are alternatives to this Actor?

The most-used option on the Store is [Website Content Crawler by Apify](https://apify.com/apify/website-content-crawler), which also offers browser-based crawling for JavaScript sites. Choose it when your target needs JavaScript rendering. Choose this Actor when the pages are server-rendered HTML (most docs, blogs and help centers) and you want the lowest cost per page. For a single query rather than a crawl, [RAG Web Browser](https://apify.com/apify/rag-web-browser) searches the web and returns Markdown for the top results.

##### Why use this instead of a hosted website-to-Markdown API or a converter extension?

A browser extension or one-page converter typically handles one page at a time. This Actor crawls many pages in one run, honours robots.txt, applies include and exclude patterns, and returns structured JSON with metadata, so you can run it on a schedule and feed a pipeline. Billing is per page with real content, so empty and blocked pages cost nothing.

##### Does it support `llms.txt`?

Yes, as a way to find pages: with `useLlmsTxt` on, it reads `/llms.txt` and queues the pages it links to. It does not generate an `llms.txt` file for your own site.

##### Can I use the output with LangChain, LlamaIndex or a vector DB?

Yes. Each row is one document: use `markdown` as the page content and `url`, `title`, `language` as metadata. `headings` helps with section-based chunking.

##### How do I detect changed pages?

Turn on `includeContentHash` and compare hashes between runs.

##### Why was a page not returned?

Run again with `includeSkippedPages` on and look at `contentStatus`: `needs-javascript`, `blocked`, `thin` (fewer than `minWords` words), `http-error` or `not-html`.

##### Is it polite?

It honours robots.txt by default, caps concurrency (default 5, max 20), and backs off automatically on 429 and 503 responses.

##### Legal

Crawl only content you are allowed to access and use. Public pages are fetched the way a normal browser would; you are responsible for complying with each site's terms of service, copyright and applicable data-protection law (for example GDPR) when you store or reuse the content.

#### Related Actors

| Actor | What it does |
|---|---|
| [Website Content Crawler by Apify](https://apify.com/apify/website-content-crawler) | The most-used crawler for LLM data, with browser modes for JavaScript sites |
| [RAG Web Browser](https://apify.com/apify/rag-web-browser) | Web search plus Markdown of the top results, for AI agents |
| [Pinecone Integration](https://apify.com/apify/pinecone-integration) | Push crawled content into a Pinecone vector index |
| [YouTube Transcript Scraper](https://apify.com/hridayrungta/youtube-transcript-scraper) | Captions and full text of YouTube videos for LLM and RAG use |
| [Contact Info Scraper](https://apify.com/hridayrungta/contact-info-scraper) | Emails, phone numbers and social links from company websites |
| [Substack Scraper](https://apify.com/hridayrungta/substack-scraper) | Newsletter posts |

# Actor input Schema

## `startUrls` (type: `array`):

Pages to start crawling from. The crawler follows links that stay inside the crawl scope below.

## `maxPages` (type: `integer`):

Cost cap: the crawl stops after this many pages were returned with real content. You are billed once per such page. Pages flagged as JavaScript-only, blocked or empty are never billed and do not count.

## `maxDepth` (type: `integer`):

How many link hops away from the start URLs to follow. 0 = only the start URLs. Pages found through sitemap.xml or llms.txt are not limited by depth.

## `crawlScope` (type: `string`):

Which discovered links to follow. 'Same hostname' stays on the exact host of each start URL. 'Same domain' also allows subdomains. 'Under start URL path' only follows links below the folder of the start URL (good for one docs section).

## `includeUrlGlobs` (type: `array`):

Only follow URLs matching at least one pattern, e.g. https://example.com/docs/\*\*. Leave empty to allow everything in scope. Start URLs are always crawled.

## `excludeUrlGlobs` (type: `array`):

Never follow URLs matching any of these patterns, e.g. https://example.com/blog/tag/\*\*.

## `useSitemaps` (type: `boolean`):

Also read robots.txt Sitemap entries and /sitemap.xml (including sitemap indexes) to find pages that are not linked from the start page.

## `useLlmsTxt` (type: `boolean`):

Also read /llms.txt and queue the pages it links to.

## `mainContentOnly` (type: `boolean`):

Strip navigation, headers, footers, cookie banners, ads and sidebars and return only the main content. Turn off to get the whole page body.

## `includeImages` (type: `boolean`):

Keep images as ![alt](absolute-url) in the Markdown. Turn off for text-only output (fewer tokens).

## `includeLinks` (type: `boolean`):

Add a 'links' array (url + anchor text) with every link found on each page.

## `includeContentHash` (type: `boolean`):

Add a 'contentHash' (SHA-256 prefix of the page text) so you can detect changed pages between crawls.

## `includeSkippedPages` (type: `boolean`):

Also save rows for pages that were skipped (JavaScript-only, blocked, thin, HTTP errors) with their contentStatus. These rows are free. Start URLs are always saved so you can see why they failed.

## `respectRobotsTxt` (type: `boolean`):

Skip URLs that the site's robots.txt disallows for all crawlers.

## `maxConcurrency` (type: `integer`):

Parallel requests (1-20). Lower it for small sites. The crawler backs off automatically on HTTP 429.

## `minWords` (type: `integer`):

Pages with fewer extracted words are treated as empty (flagged, not billed).

## `removeSelectors` (type: `array`):

Optional CSS selectors removed from every page before extraction, e.g. .promo-banner or #comments.

## `proxyConfiguration` (type: `object`):

Optional. By default no proxy is used; if a site answers 403/429, one retry goes through a free Apify datacenter proxy session. Set this to force a specific proxy group for all requests.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://docs.python.org/3/tutorial/index.html"
    }
  ],
  "maxPages": 5,
  "maxDepth": 1,
  "crawlScope": "start-url-path",
  "includeUrlGlobs": [],
  "excludeUrlGlobs": [],
  "useSitemaps": false,
  "useLlmsTxt": false,
  "mainContentOnly": true,
  "includeImages": true,
  "includeLinks": false,
  "includeContentHash": false,
  "includeSkippedPages": false,
  "respectRobotsTxt": true,
  "maxConcurrency": 5,
  "minWords": 20,
  "removeSelectors": []
}
```

# Actor output Schema

## `results` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://docs.python.org/3/tutorial/index.html"
        }
    ],
    "maxPages": 5,
    "maxDepth": 1,
    "crawlScope": "start-url-path",
    "includeUrlGlobs": [],
    "excludeUrlGlobs": [],
    "removeSelectors": []
};

// Run the Actor and wait for it to finish
const run = await client.actor("hridayrungta/website-to-markdown-crawler").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": [{ "url": "https://docs.python.org/3/tutorial/index.html" }],
    "maxPages": 5,
    "maxDepth": 1,
    "crawlScope": "start-url-path",
    "includeUrlGlobs": [],
    "excludeUrlGlobs": [],
    "removeSelectors": [],
}

# Run the Actor and wait for it to finish
run = client.actor("hridayrungta/website-to-markdown-crawler").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://docs.python.org/3/tutorial/index.html"
    }
  ],
  "maxPages": 5,
  "maxDepth": 1,
  "crawlScope": "start-url-path",
  "includeUrlGlobs": [],
  "excludeUrlGlobs": [],
  "removeSelectors": []
}' |
apify call hridayrungta/website-to-markdown-crawler --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,hridayrungta/website-to-markdown-crawler"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/HuHFWNSM2qmjCbAFc/builds/waPPMZjiWd9vU5sIX/openapi.json