# Website to Markdown Crawler - LLM & RAG Ready Content (`digivibetech/website-to-markdown-crawler`) Actor

Crawl documentation sites, blogs and knowledge bases into clean Markdown for LLMs and RAG: main content only, links and tables kept, chunking, sitemap seeding. HTTP-only, fast. $0.0018 per page with a hard page limit; failed pages free. Respects robots.txt.

- **URL**: https://apify.com/digivibetech/website-to-markdown-crawler.md
- **Developed by:** [DigiVibeTech](https://apify.com/digivibetech) (community)
- **Categories:** AI, Developer tools, Automation
- **Stats:** 3 total users, 2 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

$1.79 / 1,000 page converted to markdowns

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Website to Markdown Crawler - LLM & RAG Ready Content

Crawl **documentation sites, blogs and knowledge bases** into clean **Markdown** for LLMs, RAG pipelines and chatbots. Only the main content is kept (menus, footers and cookie bars are dropped); headings, lists, tables and links are preserved. Optional chunking for vector databases.

**$0.0018 per page converted**, no start fee, and a **hard page limit per site**, so you always know the maximum cost before you start (`maxPagesPerSite` × $0.0018). Pages that fail, are not HTML, are empty or are disallowed by robots.txt are **free**.

### Quick start: example input, output and cost

Turn docs sites and blogs into LLM-ready Markdown with chunks. Hard page limit so the cost is known up front, robots.txt respected, failed pages free.

**Example input**

```json
{
  "startUrls": [
    "https://www.rfc-editor.org/rfc/rfc9309.html"
  ]
}
```

**Cost example:** A 300-page docs site x $0.0018 = **$0.54** (the page limit caps it).

### Why this one

- **Predictable cost.** Fixed price per page plus a hard limit, so a run cannot surprise you with a large bill.
- **Fast and light.** Plain HTTP crawling (no browser), so it runs in 512 MB and does not run out of memory in n8n/Make flows. Sites that need JavaScript to show content are out of scope: use a browser-based crawler for those.
- **Deterministic.** Breadth-first crawl, same input = same pages; `stayWithinPath`, `maxDepth`, include/exclude regex, optional sitemap seeding.
- **Robust.** One slow or broken site never fails the others; every skipped URL is listed with a reason in **ERRORS**.
- **Privacy by default.** Email addresses and phone numbers are replaced with `[redacted]` (switch off with `redactContactDetails: false` for your own sites). Social networks and profile sites are not crawled.
- PDFs? Use our *Document to Markdown OCR* actor for linked PDF files.

### Output (one item per page)

`url, requestedUrl, title, markdown, characters, words, chunks[] (if chunkSize > 0), redactions, httpStatus, crawledAt`. Totals in **SUMMARY**.

### Responsible crawling

We follow each site's **robots.txt** for every page and sitemap (disallowed = skipped, free), send at most about one request per second per site (slower if the site asks for a crawl-delay), use an honest User-Agent (`SiteToMarkdown/1.0`), and never log in or bypass bot protection. Crawl sites you own or are allowed to reuse, and respect copyright when you use the content.

### Pricing

| Event | Price |
|---|---|
| Page converted to Markdown | $0.0018 |

A 500-page documentation site costs at most $0.90.

*Maintained by DigiVibeTech. Questions or bugs: use the Issues tab; we reply within 2–5 business days.*

# Actor input Schema

## `startUrls` (type: `array`):

Docs sites, blogs or knowledge bases (up to 200). Each start URL is crawled within its own host (and path, see below).

## `maxPagesPerSite` (type: `integer`):

Hard limit per start URL, so the cost is predictable: at most this many pages x $0.001.

## `maxDepth` (type: `integer`):

0 = only the start URL.

## `stayWithinPath` (type: `boolean`):

For https://site.com/docs/intro only /docs/... pages are crawled.

## `useSitemap` (type: `boolean`):

Also queue in-scope URLs from the site's sitemaps (listed in robots.txt or /sitemap.xml).

## `includeUrlRegex` (type: `string`):

Optional regular expression; other URLs are not crawled.

## `excludeUrlRegex` (type: `string`):

Optional regular expression, e.g. /tag/|/page/\d+

## `chunkSize` (type: `integer`):

0 = off. Otherwise each item also has chunks\[] split on paragraph boundaries. No extra cost.

## `includeLinks` (type: `boolean`):

Keep [text](url) links in the Markdown.

## `redactContactDetails` (type: `boolean`):

Replaces emails and phone numbers in the Markdown with \[redacted] (privacy by default).

## Actor input object example

```json
{
  "startUrls": [
    "https://www.rfc-editor.org/rfc/rfc9309.html"
  ],
  "maxPagesPerSite": 50,
  "maxDepth": 5,
  "stayWithinPath": true,
  "useSitemap": false,
  "chunkSize": 0,
  "includeLinks": true,
  "redactContactDetails": true
}
```

# Actor output Schema

## `results` (type: `string`):

All result items (overview view)

## `errors` (type: `string`):

Inputs that could not be processed, with the reason

## `summary` (type: `string`):

Counts of processed, billed and failed items

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        "https://www.rfc-editor.org/rfc/rfc9309.html"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("digivibetech/website-to-markdown-crawler").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "startUrls": ["https://www.rfc-editor.org/rfc/rfc9309.html"] }

# Run the Actor and wait for it to finish
run = client.actor("digivibetech/website-to-markdown-crawler").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    "https://www.rfc-editor.org/rfc/rfc9309.html"
  ]
}' |
apify call digivibetech/website-to-markdown-crawler --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,digivibetech/website-to-markdown-crawler"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/4cjIykG7t9FeGrrXC/builds/jmnUmM6cq1aEfIJZn/openapi.json