# Website to Markdown for RAG: Crawl & Changed Pages (`cliqtomedia/website-to-markdown`) Actor

Turn a website or docs section into clean Markdown, one row per page, for RAG and LLM pipelines. Stays inside your start path, respects robots.txt and maxPages exactly. Repeat runs can return only new and changed pages. No login.

- **URL**: https://apify.com/cliqtomedia/website-to-markdown.md
- **Developed by:** [Cliqto Media](https://apify.com/cliqtomedia) (community)
- **Categories:** AI, Developer tools
- **Stats:** 3 total users, 2 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.16 / 1,000 pages

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

Turn a website or a docs section into clean Markdown for RAG and LLM pipelines: one Dataset row per page, without menus, footers or cookie banners.
Add one start URL and a page limit. The smallest run (20 pages of a docs site) takes a few seconds. On repeat runs, `onlyChanged` returns only new, changed and removed pages. No login, no browser.

![Website to Markdown for RAG: start URL, clean Markdown per page, and only new, changed or removed pages on repeat runs](https://api.apify.com/v2/key-value-stores/wGZQYMhNZBQROv8iD/records/README-HERO-20261014.png)

### Table of contents

- [What it does](#what-it-does)
- [Quick start](#quick-start)
- [Use cases](#use-cases)
- [Input](#input)
- [Output](#output)
- [Pricing and limits](#pricing-and-limits)
- [API, export and integrations](#api-export-and-integrations)
- [Troubleshooting](#troubleshooting)
- [FAQ](#faq)
- [Related Actors](#related-actors)
- [Support](#support)

### What it does

The Actor starts from your URLs and follows links inside the same path (for example `/docs/…`). It loads each HTML page over plain HTTP, removes site chrome (navigation, header, footer, sidebars, cookie banners, "copy for LLM" buttons) and converts the main content to Markdown. Each page becomes one Dataset row with `url`, `title`, `markdown`, `wordCount` and a `contentHash`.

It reads `robots.txt` before crawling and skips blocked pages. It waits the site's `Crawl-delay` (up to 10 seconds). It never saves more than `maxPages` pages.

With `onlyChanged: true` the Actor remembers the hash of every page. The next run with the same input saves only pages that are new or changed. Pages from the last run that are no longer linked are checked directly; if one now answers HTTP 404 or 410, you get a free row with `changeStatus: removed`. Unchanged pages are not saved and cost less.

#### Good fit

- Feed a docs site or help center into a vector database or a RAG chatbot.
- Refresh that index every day or week and re-embed only the pages that changed.
- Get LLM-ready Markdown of a blog section, a product guide or an API reference.

#### Not a fit / limits

- JavaScript is not executed. Single-page apps that render content in the browser give empty or short pages, marked `status: partial`.
- No login, cookies, CAPTCHA or paywalled pages. PDF, DOCX and images are skipped.
- Up to 2000 pages per run and up to 50 start URLs. The output is whole pages; the Actor does not split text into chunks or create embeddings.

### Quick start

1. Open the **Input** tab.
2. In **Start URLs** (`startUrls`) keep the prefill `https://docs.apify.com/academy` or add your own page, for example `https://docs.example.com/guide`.
3. Keep **Max pages** (`maxPages`) at the prefill 20. Keep **Crawl scope** at the default `path`.
4. Click **Start**.
5. Open the **Output** tab, view **Markdown**, and check the `markdown` field of the first row.

Expected result: about 20 rows, one per page under `/academy`, each with a title and clean Markdown.

### Use cases

| Buyer job | Input path | Output entity | Key fields | Next integration |
| --- | --- | --- | --- | --- |
| Index a docs site for RAG | `startUrls` + `maxPages` (+ `useSitemap`) | Page | `url`, `title`, `markdown`, `wordCount` | Chunk `markdown`, embed, store with `url` as source |
| Keep the index fresh | same input + `onlyChanged: true` on a schedule | Changed page | `changeStatus`, `url`, `markdown`, `contentHash` | Upsert `new`/`changed`, delete `removed` by `url` |
| One section only | `startUrls` + `includeGlobs` / `excludeGlobs` | Page | `url`, `markdown` | LLM context, knowledge base |
| Collect page links | `includeLinks: true` | Page | `url`, `links` | Site map, link checks |

### Input

#### Input fields

| Field | Plain name | Type | Required | Default / prefill | Allowed values | Effect | Example | Recommendation |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| `startUrls` | Start URLs | array of URLs | yes | prefill `https://docs.apify.com/academy` | 1–50 public `http(s)` URLs | Where crawling starts and what scope means | `["https://docs.example.com/guide"]` | Use the top page of the section you want |
| `maxPages` | Max pages | integer | no | default 50, prefill 20 | 1–2000 | Maximum pages saved and charged | `500` | Start small, then raise |
| `maxDepth` | Max link depth | integer | no | 5 | 0–20 | Clicks away from a start URL; `0` = only start URLs | `2` | Raise for deep docs trees |
| `crawlScope` | Crawl scope | select | no | `path` | `path`, `host`, `domain` | Which links count as in scope | `host` | `path` for docs, `host` for a small site |
| `useSitemap` | Use sitemap.xml | boolean | no | `false` | — | Adds in-scope URLs from the sitemap; sitemap dates fill `lastModified` | `true` | Use when pages are not linked from each other |
| `includeGlobs` | Include URL patterns | array | no | empty | up to 50 patterns, 500 characters each | Crawl only matching URLs (full URL). `*` = inside one path part, `**` = anything | `["https://docs.example.com/**/api/**"]` | Narrow a big site |
| `excludeGlobs` | Exclude URL patterns | array | no | empty | up to 50 patterns | Never crawl matching URLs | `["**/changelog/**"]` | Skip archives, tags, pagination |
| `onlyChanged` | Only changed pages | boolean | no | `false` | — | Saves only new, changed and removed pages | `true` | Use for scheduled refreshes |
| `stateKey` | Delta state name | string | no | derived from start URLs, scope and patterns | letters, digits, `-_.!'()`, up to 200 | Name of the saved state | `my-docs` | Set it if you change patterns but want to keep the history |
| `removeSelectors` | Extra elements to remove | string | no | empty | CSS selectors | Removes more elements before conversion | `.ad, .related-posts` | Use for site-specific widgets |
| `includeLinks` | Include page links | boolean | no | `false` | — | Adds `links` (up to 1000 per page, including links to other sites) | `true` | Off for smaller rows |
| `maxConcurrency` | Parallel requests | integer | no | 5 | 1–10 | Pages loaded at the same time | `3` | Lower for small sites |
| `proxyConfiguration` | Proxy | proxy | no | no proxy | Apify Proxy (automatic or datacenter) or own proxies; residential and SERP groups are not supported | Connection route | `{"useApifyProxy": true}` | Only if a site blocks the default connection |

Start URLs that repeat are crawled once. URLs with `utm_*` parameters and `#fragments` are treated as the same page. Private and local addresses (like `127.0.0.1`) are rejected.

#### Interacting settings

| When | Input A | Input B | Effective behavior | Example | Cost/result effect |
| --- | --- | --- | --- | --- | --- |
| Patterns and scope | `includeGlobs` | `crawlScope` | When set, the patterns replace the path rule, but the crawl stays on the start hosts. Start URLs are always loaded. Pages in between must match too: if a hub page does not match, the pages linked only from it are not found (add the hub pattern or use `useSitemap`) | include `https://docs.example.com/**/api/**` | Only matching pages |
| Sitemap and scope | `useSitemap` | `crawlScope`, globs | Only sitemap URLs inside the scope and patterns are added | — | More pages found, still capped by `maxPages` |
| Delta history | `onlyChanged` | `stateKey` | Same input (or same `stateKey`) continues the same history | — | Unchanged pages cost less |
| Removed pages | `onlyChanged` | `maxPages` | `removed` rows appear for pages of the last run that now answer 404 or 410. Pages not reached by links are re-checked directly while `maxPages` allows; the rest stay in the history | — | `removed` rows are free |

#### Input examples

##### Small docs section (prefill)

```json
{
  "startUrls": ["https://docs.apify.com/academy"],
  "maxPages": 20
}
```

Expected behavior: crawls pages under `/academy`, stops at 20.
Expected output: 20 rows with `url`, `title`, `markdown`, `wordCount`.
Limit or cost note: one run start plus 20 `page` events.

##### Weekly refresh of a whole docs site

```json
{
  "startUrls": ["https://docs.example.com/"],
  "crawlScope": "host",
  "useSitemap": true,
  "maxPages": 1000,
  "excludeGlobs": ["**/changelog/**"],
  "onlyChanged": true
}
```

Expected behavior: the first run saves all pages as `new`. Later runs save only `new`, `changed` and `removed` pages.
Expected output: changed pages with `changeStatus` and `markdown`.
Limit or cost note: unchanged pages are charged as `unchanged-page`, cheaper than `page`.

### Output

#### Output fields

| Field | Meaning | Type | Can be empty? | Example / enum | Source / notes |
| --- | --- | --- | --- | --- | --- |
| `url` | Page URL after redirects | string | no | `https://docs.apify.com/academy` | Normalized, without `#fragment` and `utm_*` |
| `title` | Page title | string | yes | `Apify Academy \| Academy \| Apify Documentation` | `<title>`, then `og:title`, then first `h1` |
| `description` | Meta description | string | yes | `Learn everything about web scraping…` | `null` if the page has none |
| `language` | Page language | string | yes | `en` | `<html lang>` |
| `markdown` | Clean Markdown of the main content | string | only for `removed` | `# Apify Academy…` | Up to 500 000 characters |
| `wordCount` | Words in `markdown` | integer | only for `removed` | `183` | Markup is not counted |
| `contentHash` | SHA-256 of the Markdown (whitespace ignored) | string | only for `removed` | `f5aa6306…` | Same text → same hash |
| `changeStatus` | Delta status | string | yes | `new`, `changed`, `removed` | `null` when `onlyChanged` is off |
| `previousCrawledAt` | When the page was saved before | ISO date | yes | `2026-10-07T10:00:00.000Z` | `null` for new pages and outside delta |
| `lastModified` | Last change date from the site | ISO date | yes | `2026-10-13T19:47:40.000Z` | HTTP `Last-Modified` or sitemap `lastmod`; `null` if the site gives none |
| `httpStatus` | HTTP status | integer | only for `removed` | `200` | |
| `depth` | Clicks from a start URL | integer | yes | `0` | |
| `referrerUrl` | Page where the link was found | string | yes | `null` | `null` for start and sitemap URLs |
| `links` | Links on the page, including links to other sites (whole page, menus too) | array | field absent by default | `["https://…"]` | Only with `includeLinks`; up to 1000 |
| `status` | Row quality | string | no | `complete`, `partial`, `removed` | `partial` = likely needs JavaScript, or page was cut |
| `note` | Why a row is partial or removed | string | yes | `null` | |
| `crawledAt` | When the page was loaded | ISO date | no | `2026-10-14T02:35:12.564Z` | |
| `source` | Source type | string | no | `website` | |

`null` means "unknown or not given by the site". It never means zero.

#### Output entities and surfaces

| Situation | Dataset/API/KVS/live view | Result shape | User action |
| --- | --- | --- | --- |
| Normal page | Dataset, views **Pages** and **Markdown** | One row per page | Use `markdown` and `url` |
| Nothing to crawl (robots.txt blocks all, no HTML) | Dataset empty; KVS `RUN_SUMMARY` `status: not_found` | 0 rows, run succeeded | Check scope, start URL and `robots.txt` |
| Delta with no changes | Dataset empty; `RUN_SUMMARY.counts.unchangedPages` | 0 rows | Nothing to re-index |
| Time or charge limit | Dataset with saved rows; `RUN_SUMMARY` `status: partial`, `reason: TIMEOUT` or `CHARGE_LIMIT` | Saved rows stay | Lower `maxPages` or raise the limit |
| Site fails (DNS, 4xx/5xx) | `RUN_SUMMARY` `status: error`, list `failedPages` (up to 100) | Run fails | Check the URL in a browser; try later or with a proxy |

`RUN_SUMMARY` also has `counts` (pages crawled, saved, new, changed, unchanged, removed, failed, skipped by robots.txt, not HTML, duplicates) and `warnings`.

#### Output example

```json
{
  "url": "https://docs.apify.com/academy",
  "title": "Apify Academy | Academy | Apify Documentation",
  "description": "Learn everything about web scraping and automation with our free courses that will turn you into an expert scraper developer.",
  "language": "en",
  "markdown": "# Apify Academy\n\nLearn everything about web scraping and automation with our free courses…",
  "wordCount": 183,
  "contentHash": "f5aa630681e827a9a830955f9baaaf034734f2c4bc0d0610e82a9d76359723ce",
  "changeStatus": "new",
  "previousCrawledAt": null,
  "lastModified": "2026-10-13T19:47:40.000Z",
  "httpStatus": 200,
  "depth": 0,
  "referrerUrl": null,
  "status": "complete",
  "note": null,
  "crawledAt": "2026-10-14T02:35:12.564Z",
  "source": "website"
}
```

This example is trimmed (`markdown` is shortened). It is the first row of the prefill run with `onlyChanged: true`.

#### Moving from Website Content Crawler

If your pipeline reads the output of `apify/website-content-crawler`, change these paths:

| Website Content Crawler | This Actor |
| --- | --- |
| `url` | `url` |
| `markdown` | `markdown` |
| `crawl.loadedTime` | `crawledAt` |
| `crawl.depth` | `depth` |
| `crawl.referrerUrl` | `referrerUrl` |
| `metadata.title` | `title` |
| `metadata.description` | `description` |
| `metadata.languageCode` | `language` |
| `text`, `html` | not returned (use `markdown`) |

### Pricing and limits

#### Pricing unit

You pay per event (pay per event). Platform usage is included in the price.

| Event | When | Price |
| --- | --- | --- |
| Run start | Once per run (memory is fixed at 1 GB) | $0.002 |
| `page` | A page saved to the Dataset | $0.00019 (Free plan) → $0.00016 (Gold and higher) |
| `unchanged-page` | With `onlyChanged`, a page that did not change (not saved) | $0.00012 |
| `extra-second` | Run time above the included time (see below) | $0.0003 per second |

- Included run time: 5 seconds per run plus 0.4 seconds per charged page. Normal sites stay well inside it (2000 heavy pages took 577 of 805 included seconds), so `extra-second` is not charged.
- `extra-second` appears only on very slow sites or when `robots.txt` sets a long `Crawl-delay`.
- `removed` rows are free.
- Failed pages, skipped pages (robots.txt, not HTML, duplicates) are free.
- A run with no pages (for example, `robots.txt` blocks everything) costs only the run start. Failed runs (site not reachable, invalid input) also pay the run start.
- The Actor stops when it reaches your **Maximum cost per run**. Rows saved before that stay in the Dataset.

#### Cost examples

Free plan prices, memory 1 GB.

| Scenario | Input scope | Expected events | Base charge | Optional charge | Stop/cap behavior |
| --- | --- | ---: | ---: | ---: | --- |
| Prefill | academy, `maxPages` 20 | 1 start + 20 `page` | $0.0058 | none | stops at 20 pages |
| Small site | 100 pages | 1 start + 100 `page` | $0.021 | none | stops at `maxPages` |
| Docs site | 500 pages | 1 start + 500 `page` | $0.097 | none | stops at `maxPages` |
| Large docs | 1000 pages | 1 start + 1000 `page` | $0.192 | none | stops at `maxPages` |
| Maximum | 2000 pages | 1 start + 2000 `page` | $0.382 | none | 2000 is the limit |
| Weekly refresh | 500 pages, 10 changed | 1 start + 10 `page` + 490 `unchanged-page` | $0.0627 | `removed` free | — |
| Nothing found | `robots.txt` blocks all | 1 start | $0.002 | none | run succeeds, 0 rows |
| Slow site | `Crawl-delay: 10`, 1 hour | 1 start + ~360 `page` + ~3450 `extra-second` | $0.070 | ~$1.04 extra time | use a lower `maxPages` or a shorter timeout |

Measured on the platform: 20 pages in about 5 seconds, 1000 pages in about 4–5 minutes, 2000 heavy pages in under 10 minutes.

### API, export and integrations

- **API:** start a run with `POST https://api.apify.com/v2/acts/cliqtomedia~website-to-markdown/runs` and the input JSON. Read rows from the run's default Dataset.
- **Export:** JSON, JSONL, CSV, Excel and other formats from the Dataset. For RAG, JSON or JSONL keeps `markdown` intact.
- **Scheduling:** create a schedule with the same input and `onlyChanged: true`. Each run returns only what changed since the last run.
- **Webhooks / no-code:** on "run succeeded", read the Dataset and upsert rows by `url`; delete rows with `changeStatus: removed`.

#### Use with AI agents (MCP)

- MCP server URL with only this Actor: `https://mcp.apify.com?tools=cliqtomedia/website-to-markdown`.
- Smallest useful input: `{"startUrls": ["https://docs.apify.com/academy"], "maxPages": 20}`.
- Key output fields: `url` (source link), `title`, `markdown` (page text), `wordCount`, `changeStatus` (with `onlyChanged`).
- A 20-page run takes about 5–10 seconds and costs about $0.006. Keep `maxPages` small for interactive use; up to 2000 pages per run.

### Troubleshooting

#### Pages have almost no text

Symptom: rows with `status: partial` and a note about JavaScript.
Likely reason: the site renders content in the browser.
Check: open the page with JavaScript off, or look at the page source.
Action: this Actor does not run JavaScript. Use a browser-based crawler for that site.

#### The run returned 0 rows

Symptom: empty Dataset, `RUN_SUMMARY` `status: not_found`.
Likely reason: `robots.txt` blocks the pages, the start URL is not HTML, or patterns exclude everything.
Check: `RUN_SUMMARY.counts.skippedByRobots`, `skippedNotHtml` and `message`.
Action: change the start URL, `crawlScope` or `includeGlobs`.

#### Fewer pages than expected

Symptom: the run ends before `maxPages`.
Likely reason: there are no more linked pages inside the scope or depth.
Check: `RUN_SUMMARY.stopReason` and `counts`.
Action: turn on `useSitemap`, raise `maxDepth` or use `crawlScope: host`. With `includeGlobs`, check that hub pages between the start URL and the target pages match too.

Note: pages are loaded in parallel, so when `maxPages` cuts a big site, the exact set of pages can differ a little between runs. Set `maxPages` above the site size for a stable set.

#### The run failed

Symptom: run status Failed, `RUN_SUMMARY` `status: error` or `partial` with `reason: SOURCE_ERROR`.
Likely reason: the site returned errors for most pages or could not be reached.
Check: `RUN_SUMMARY.failedPages` (URL, HTTP status, error).
Action: check the site in a browser; try later, lower `maxConcurrency`, or use a datacenter proxy.

### FAQ

#### Do I need an account on the website or an API key?

No. The Actor reads only public pages without login.

#### Does the Actor respect robots.txt?

Yes. It reads `robots.txt` for each host, skips blocked URLs and waits the `Crawl-delay` (up to 10 seconds). If `robots.txt` cannot be read because of a server error, the host is not crawled.

#### Why was I charged `extra-second`?

The site was slow or its `robots.txt` asks to wait between requests (`Crawl-delay`). The Actor waits as asked, and that waiting time above the included time is charged. Check `RUN_SUMMARY.counts.chargedExtraSeconds`. Lower `maxPages`, set a shorter run timeout or a **Maximum cost per run** to cap it.

#### How does "only changed" decide that a page changed?

It compares a hash of the clean Markdown. Changes in menus, ads or whitespace do not count. A page is downloaded each time, but an unchanged page is not saved and costs less. History is kept in your account in the key-value store `website-to-markdown-delta`.

#### When do I get removed pages?

When a page from the last delta run now answers HTTP 404 or 410. Pages that the crawl no longer reaches by links are re-checked directly (they count as normal or unchanged pages). Pages that fail with other errors, or are not re-checked because `maxPages` was reached, stay in the history and are not reported as removed. If no page loads at all in a run (the whole site answers 404), nothing is marked removed. A page that now redirects to another URL is not marked removed either. Up to 1000 `removed` rows per run.

#### Is the Markdown complete?

It is the main content of the HTML page. Very large pages are cut (the row gets `status: partial` and a note). Content that appears only after JavaScript runs is not included.

### Related Actors

- [PDF & Image to Text OCR for AI Agents](https://apify.com/cliqtomedia/pdf-text-ocr) — get text from the PDF files that this Actor skips.

### Support

For bugs, questions or feature requests, open an issue in the **Issues** tab of this Actor. Include the run ID, your input (without secrets) and the error text from `RUN_SUMMARY`. Never send tokens, cookies or passwords.

# Changelog

This Actor's version history is a separate document: https://apify.com/cliqtomedia/website-to-markdown/changelog.md

# Actor input Schema

## `startUrls` (type: `array`):

Public pages to start from (up to 50). By default only pages under the same path are crawled: https://docs.example.com/guide crawls /guide/… but not /blog/. Pages behind a login, private or internal addresses are not crawled.

## `maxPages` (type: `integer`):

Stop after this many HTML pages (1–2000). The run never returns more pages than this.

## `maxDepth` (type: `integer`):

How many clicks away from a start URL to follow. 0 = only the start URLs.

## `crawlScope` (type: `string`):

Which links to follow: under the start URL path (recommended for docs), anywhere on the same host, or on any subdomain of the same domain.

## `useSitemap` (type: `boolean`):

Also add in-scope URLs from the site's sitemap (from robots.txt or /sitemap.xml). Finds pages that are not linked. Sitemap dates fill lastModified.

## `includeGlobs` (type: `array`):

Only crawl URLs matching one of these patterns (full URL). \* matches inside one path segment, \*\* matches anything. Example: https://docs.example.com/**/api/**. When set, replaces the path rule but stays on the start hosts. Pages in between (hubs) must match too, or the pages linked only from them are not found.

## `excludeGlobs` (type: `array`):

Never crawl URLs matching these patterns. Example: **/changelog/** or \*\*/*?page=*.

## `onlyChanged` (type: `boolean`):

Remember each page's content between runs and return only new and changed pages. Unchanged pages are not saved to the dataset and cost less. Pages from the previous run that now answer HTTP 404 or 410 are listed as 'removed' (free); pages the crawl no longer reaches by links are re-checked directly. The first run returns every page as 'new'.

## `stateKey` (type: `string`):

Optional name for the saved delta state (letters, digits, -\_.). By default it is derived from the start URLs and patterns, so the same input always continues the same state. Use different names to keep separate histories.

## `removeSelectors` (type: `string`):

CSS selectors to remove before conversion, e.g. .ad, .related-posts. Navigation, headers, footers, sidebars and cookie banners are always removed.

## `includeLinks` (type: `boolean`):

Add the list of links found on each page, including links to other sites (up to 1000).

## `maxConcurrency` (type: `integer`):

How many pages to load at the same time (1–10). The site's robots.txt Crawl-delay is always respected.

## `proxyConfiguration` (type: `object`):

Usually not needed. Use a proxy only if the site blocks the default connection. Apify Proxy with automatic or datacenter groups, or your own proxy URLs. Residential and SERP proxy groups are not supported (the run fails with an input error).

## Actor input object example

```json
{
  "startUrls": [
    "https://docs.apify.com/academy"
  ],
  "maxPages": 20,
  "maxDepth": 5,
  "crawlScope": "path",
  "useSitemap": false,
  "onlyChanged": false,
  "includeLinks": false,
  "maxConcurrency": 5,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}
```

# Actor output Schema

## `pages` (type: `string`):

One row per page: URL, title, change status, word count.

## `markdown` (type: `string`):

URL, title and the full Markdown of each page.

## `runSummary` (type: `string`):

Status, stop reason, counts (new, changed, unchanged, removed, failed) and failed pages.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        "https://docs.apify.com/academy"
    ],
    "maxPages": 20
};

// Run the Actor and wait for it to finish
const run = await client.actor("cliqtomedia/website-to-markdown").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": ["https://docs.apify.com/academy"],
    "maxPages": 20,
}

# Run the Actor and wait for it to finish
run = client.actor("cliqtomedia/website-to-markdown").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    "https://docs.apify.com/academy"
  ],
  "maxPages": 20
}' |
apify call cliqtomedia/website-to-markdown --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,cliqtomedia/website-to-markdown"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/bk0Hsjd7K1wQcH6Vs/builds/B3s4EW9dLFUvwRYbj/openapi.json