# Website to Markdown Crawler (`properscraper/website-to-markdown-crawler`) Actor

Turn a website into clean Markdown and plain text for AI: crawls the pages of a site, strips menus, footers and scripts, and saves each page's main content with headings, lists, links, tables and code blocks intact. Fast, with no browser, and it obeys robots.txt.

- **URL**: https://apify.com/properscraper/website-to-markdown-crawler.md
- **Developed by:** [jake mcclure](https://apify.com/properscraper) (community)
- **Categories:** AI, Developer tools
- **Features:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.50 / 1,000 page saveds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Website to Markdown Crawler

Turn a website into clean Markdown and plain text, ready for an AI assistant, a RAG pipeline, a knowledge base or a search index.

Give it a start page. It follows links to the other pages of the same website and saves the main content of each one: headings, paragraphs, lists, links, tables and code blocks are kept; menus, headers, footers, sidebars, cookie notices, forms and scripts are dropped.

No browser is used, so it is fast and cheap: a few hundred pages in a couple of minutes.

Use it to:

- feed your documentation, help centre or blog to a chatbot or a custom GPT
- build a RAG or fine-tuning dataset from a list of websites
- keep an AI knowledge base up to date by running it on a schedule
- convert a handful of pages to Markdown (set the page limit to 1)
- archive or migrate the text of a site

### Input

| Field | What it does |
|---|---|
| **Start pages** | One per line: the page to start from on each website, such as `https://docs.example.com/` or just `example.com`. |
| **Pages per website** | The most pages saved for each start page. Default 50. Set it to 1 to convert only the pages you list. |
| **Stay under the start page's folder** | On: starting at `example.com/docs/intro` reads only pages under `example.com/docs/`. Off (default): any page of the website. |
| **Keep images in the Markdown** | Include images as Markdown image links. Off by default. |
| **Obey robots.txt** | On by default: pages a website asks crawlers not to read are skipped. |

```json
{
    "urls": ["https://docs.example.com/"],
    "maxPagesPerWebsite": 200,
    "stayUnderStartPath": true
}
```

### Output

One record per page:

````json
{
    "startUrl": "https://docs.example.com/",
    "url": "https://docs.example.com/guides/install",
    "title": "Install | Example Docs",
    "description": "How to install Example on Linux, macOS and Windows.",
    "language": "en",
    "depth": 2,
    "foundOn": "https://docs.example.com/guides/",
    "wordCount": 412,
    "markdown": "# Install\n\nExample runs on Linux, macOS and Windows.\n\n## Requirements\n\n- Node.js 22 or newer\n- 200 MB of disk space\n\n```\nnpm install example\n```\n\nSee the [quick start](https://docs.example.com/guides/quick-start) next.",
    "text": "Install\n\nExample runs on Linux, macOS and Windows.\n\nRequirements\n\n- Node.js 22 or newer\n- 200 MB of disk space\n\nnpm install example\n\nSee the quick start next.",
    "scrapedAt": "2026-11-18T12:00:00.000Z"
}
````

- `markdown` keeps headings, lists, links (as full addresses), tables, quotes and code blocks. `text` is the same content with no markup at all.
- `depth` is how many clicks the page is from the start page, and `foundOn` is the page that linked to it.
- `title`, `description` and `language` are what the page declares about itself.
- Pages are read breadth first, so the pages nearest the start page come first when the limit cuts a crawl short.

A `SUMMARY` record in the run's key-value store lists, for each website, how many pages were saved and how many were skipped.

### Pricing

Pay per result. You are charged once for each page saved to the dataset, at the per-result price shown on this page.

Pages that could not be read, pages robots.txt does not allow, files such as PDFs and images, and pages with no text are not saved and cost nothing. The run stops as soon as it reaches the maximum charge you set for it.

### Limits

- Pages are read without running their JavaScript. A site that builds its pages entirely in the browser (some single-page apps) yields little or no text; documentation sites, blogs, shops and company sites almost always work.
- Only links to the same website are followed. A subdomain counts as a different website; add it as its own start page.
- The main content is found from the page's own structure (`<main>`, `<article>`, roles and well-known class names). On unusual layouts a stray menu item may be kept, or a side note dropped.
- The pages of one website are read one after another, half a second apart, so 1,000 pages of one site take about ten minutes. Several websites are crawled side by side.
- Some sites refuse automated requests. The Actor identifies itself honestly and does not disguise itself or work around a refusal.
- At most 5 MB of each page is read, and at most 20,000 pages per start page.

### Is this allowed?

The Actor reads public pages the way a search engine's crawler does: it identifies itself in its User-Agent, obeys robots.txt by default and paces its requests to each site. You are responsible for how you use the text; most website content is protected by copyright, so check a site's terms before republishing it or training on it.

# Actor input Schema

## `urls` (type: `array`):

One per line: the page to start from on each website, such as https://docs.example.com/ or example.com. The crawl follows links from there to other pages of the same website.

## `maxPagesPerWebsite` (type: `integer`):

The most pages saved for each start page. Set it to 1 to convert just the pages you list.

## `stayUnderStartPath` (type: `boolean`):

On: starting at example.com/docs/intro reads only pages under example.com/docs/. Off: any page of the website can be read.

## `includeImages` (type: `boolean`):

Include each image as a Markdown image link with its description.

## `respectRobotsTxt` (type: `boolean`):

Skip pages a website's robots.txt asks crawlers not to read.

## Actor input object example

```json
{
  "urls": [
    "https://crawlee.dev/js/docs/quick-start"
  ],
  "maxPagesPerWebsite": 50,
  "stayUnderStartPath": false,
  "includeImages": false,
  "respectRobotsTxt": true
}
```

# Actor output Schema

## `pages` (type: `string`):

Every page saved by the run, one record per page.

## `summary` (type: `string`):

Totals for the run and, for each website, how many results were saved and any error.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "urls": [
        "https://crawlee.dev/js/docs/quick-start"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("properscraper/website-to-markdown-crawler").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "urls": ["https://crawlee.dev/js/docs/quick-start"] }

# Run the Actor and wait for it to finish
run = client.actor("properscraper/website-to-markdown-crawler").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "urls": [
    "https://crawlee.dev/js/docs/quick-start"
  ]
}' |
apify call properscraper/website-to-markdown-crawler --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,properscraper/website-to-markdown-crawler"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/mboi0nqjMqhnkMSWD/builds/Bi1ETK4aStU1asMan/openapi.json
