# Website to Markdown Crawler for RAG: Fast, No Browser (`jtpalms/website-to-markdown-crawler`) Actor

Crawl a site into clean Markdown for RAG and LLMs without a browser. Sitemap first, robots.txt honored, main content only, heading chunks with token counts, and changed-pages-only re-crawls. JS-only pages are flagged, not charged. USD 1.20 per 1,000 pages.

- **URL**: https://apify.com/jtpalms/website-to-markdown-crawler.md
- **Developed by:** [JT Palms](https://apify.com/jtpalms) (community)
- **Categories:** AI, Developer tools, Automation
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.96 / 1,000 page converteds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Website to Markdown Crawler for RAG: Fast, No Browser

Give it a website, get one row per page with clean Markdown of the main content, plain text on request, headings, link counts and optional heading-based chunks with token estimates. It uses plain HTTP only: no browser, no proxy, no LLM. It reads the sitemap first, honors robots.txt including Crawl-delay, and has a changed-pages-only mode so a nightly refresh pays only for pages that changed.

**USD 1.20 per 1,000 pages**, plus a tiny start fee per run. Pages that fail, need JavaScript, are blocked by robots.txt, or are unchanged in only-changed mode are free.

### What people use it for

- **A RAG knowledge base that stays in sync.** Documentation sites, help centres and engineering blogs become Markdown with real headings, lists, tables and code blocks. Turn on `chunkMarkdown` for chunks that carry their heading path and a token estimate, then embed them. Schedule it with `onlyChanged` and a refresh costs a few cents.
- **Agent and MCP context.** "Read this whole docs site" as one cheap, deterministic run that returns structured JSON, with no browser to start and nothing to babysit.
- **Content and SEO exports.** Export a site's text for audits, migrations, content gap analysis and translation memory. Each row has the title, description, canonical URL, language, headings outline, internal and external link counts and the schema.org types found on the page.
- **Change tracking.** `contentHash` and `changed` tell you which pages moved since the last run, and the run summary lists the URLs that vanished from the sitemap.

### Sample output

Real run on a public documentation page (`https://crawlee.dev/js/docs/introduction`) with `chunkMarkdown` on; `markdown`, `headings` and `chunks` shortened:

```json
{
  "url": "https://crawlee.dev/js/docs/introduction",
  "finalUrl": "https://crawlee.dev/js/docs/introduction",
  "httpStatus": 200,
  "depth": 0,
  "title": "Introduction | Crawlee for JavaScript · Build reliable crawlers. Fast.",
  "description": "Your first steps into the world of scraping with Crawlee",
  "canonicalUrl": "https://crawlee.dev/js/docs/introduction",
  "language": "en",
  "markdown": "# Introduction\n\nCrawlee covers your crawling and scraping end-to-end and helps you **build reliable scrapers. Fast.**\n\nYour crawlers will appear human-like and fly under the radar of modern bot protections even with the default configuration. Crawlee gives you...",
  "wordCount": 289,
  "tokenEstimate": 529,
  "headings": [
    { "level": 1, "text": "Introduction" },
    { "level": 2, "text": "What you will learn" },
    { "level": 2, "text": "Next steps" }
  ],
  "chunks": [
    { "index": 0, "headingPath": ["Introduction"], "markdown": "# Introduction\n\nCrawlee covers your crawling and scraping end-to-end and helps you **build...", "tokenEstimate": 143 },
    { "index": 1, "headingPath": ["Introduction", "What you will learn"], "markdown": "## What you will learn\n\nThe goal of the introduction is to provide a step-by-step guide to...", "tokenEstimate": 80 }
  ],
  "internalLinkCount": 0,
  "externalLinkCount": 0,
  "jsonLdTypes": ["BreadcrumbList", "ListItem"],
  "lastModified": "2026-10-06T14:19:18.000Z",
  "contentHash": "90161d458bbd9ca6f138bacc59268ef5643a1d67533e47cd34ba5a90e51d2aa2",
  "changed": null,
  "needsJavaScript": false,
  "skippedReason": null,
  "robotsAllowed": true,
  "extractor": "page",
  "error": null,
  "errorCode": null,
  "fetchedAt": "2026-10-06T20:06:26.456Z"
}
```

Code blocks keep every line and their language, and nested lists keep their structure. This is the Markdown of a real tutorial page on `go.dev` (excerpt):

````markdown
### Write some code

Get started with Hello, World.

1. Open a command prompt and cd to your home directory.

   On Linux or Mac:

   ```
   cd
   ```

   On Windows:

   ```
   cd %HOMEPATH%
   ```

2. Create a hello directory for your first Go source code.

   For example, use the following commands:

   ```
   mkdir hello
   cd hello
   ```
````

Tables become pipe tables, with the header row first and `|` inside a cell escaped. This one is from a real documentation page for Python's `json` module:

```markdown
| JSON | Python |
| --- | --- |
| object | dict |
| array | list |
| string | str |
| number (int) | int |
| number (real) | float |
| true | True |
| false | False |
| null | None |
```

### Quick start

The default input crawls 25 pages of the public Crawlee documentation in about 30 seconds. To crawl your own site:

```json
{
  "startUrls": [{ "url": "https://example.com/docs/" }],
  "crawlScope": "sitemap_then_links",
  "maxPages": 200,
  "chunkMarkdown": true
}
```

From Python:

```python
from apify_client import ApifyClient

client = ApifyClient("<YOUR_APIFY_TOKEN>")
run = client.actor("JTPalms/website-to-markdown-crawler").call(run_input={
    "startUrls": [{"url": "https://example.com/docs/"}],
    "maxPages": 200,
    "chunkMarkdown": True,
})
for page in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(page["url"], page["wordCount"], page["changed"])
```

### How it finds pages

| `crawlScope` | What happens |
|---|---|
| `sitemap_then_links` (default) | Reads the sitemap (robots.txt `Sitemap:` lines, then `/sitemap.xml`, `/sitemap_index.xml`, `/wp-sitemap.xml`), expands sitemap indexes and .gz files. If the sitemap lists fewer pages than `maxPages`, it also follows links from the pages it fetched, up to `maxDepth`. |
| `sitemap` | Sitemap pages and your start URLs only. |
| `links_only` | Breadth-first from your start URLs, same site only, up to `maxDepth`. |
| `single_pages` | Exactly the URLs you list. |

- **Scope.** Pages must be on the same host as a start URL (`www.` makes no difference). A start URL with a path, for example `https://example.com/docs/intro`, keeps the crawl inside its folder (`/docs/`); turn off `stayInStartFolder` to crawl the whole site. To crawl another host, add it as a start URL.
- **Nearest first.** Sitemap pages are queued by how many leading path segments they share with your start URL, so a small `maxPages` gets the pages around the one you named.
- **Include and exclude patterns.** `includeUrlPatterns` and `excludeUrlPatterns` take three kinds of pattern, tested against the full URL (and the URL without its query): a glob where `*` matches inside one path segment and `**` across segments (`**/docs/**`); plain text that the URL must contain (`/blog/`); or a regular expression written as `re:^https://example\.com/(en|de)/`. The default excludes skip author, team, people, staff and contact pages, which are mostly personal details. Pages you list as start URLs are always crawled.
- **Links that are not followed.** Links marked `rel="nofollow"` and links on a page whose robots meta says `nofollow` are not followed. Sitemap URLs and start URLs are still read.
- **Duplicates.** `#fragment` variants and tracking parameters (`utm_*`, `gclid`, `fbclid` and similar) are removed before a URL is queued. A page whose canonical link points to a page already delivered (a trailing-slash twin, a `?ref=` variant, a redirect to a page you already have) comes back as a free row with `skippedReason: "duplicate"`. If a page claims a canonical but its content differs, it is still delivered, so a misconfigured canonical cannot hide real pages.
- **Other files.** PDF links are reported once with `skippedReason: "pdf"` and are never downloaded (use PDF to Markdown and Text, `JTPalms/pdf-to-markdown`, for those). Images, archives and other non-page files are ignored. A URL that answers with a non-HTML content type is skipped from its headers alone.

### Input

| Field | What it does |
|---|---|
| `startUrls` | Pages or site roots, one per line. |
| `crawlScope` | `sitemap_then_links` (default), `sitemap`, `links_only`, `single_pages`. |
| `maxPages` | Hard cap on pages requested in the run, across all sites. Default 100, prefill 25, up to 50,000. Pages that fail still count, because they were requested. |
| `maxDepth` | Link depth when following links. Default 3. Start URLs and sitemap pages are depth 0. |
| `stayInStartFolder` | On by default. See scope above. |
| `includeUrlPatterns`, `excludeUrlPatterns` | See above. |
| `contentMode` | `page` (default): the cleaned body or `<main>`, best for documentation. `article`: Mozilla Readability's main article, best for blogs and news; falls back to `page` when no article is found (`extractor` says which one was used). |
| `outputFormat` | `markdown` (default) or `markdown_and_text` (adds `text`, the same reading order with no Markdown syntax). |
| `chunkMarkdown`, `maxChunkTokens` | Add `chunks`: the Markdown split at headings into pieces of at most `maxChunkTokens` estimated tokens (default 500, 50 to 4,000). |
| `includeImages` | Keep images as `![alt](absolute url)`. Off by default. |
| `includeLinks` | Add `links`: the internal and external links in the page content, deduplicated, with link text and an `internal` flag. The counts are always there. |
| `onlyChanged` | Only output pages that are new or changed since your last only-changed run on the same site. See below. |
| `useLlmsTxt` | Also seed from the site's `/llms.txt`. |
| `minWords` | A page with fewer words is not charged. Default 30. |
| `requestsPerSecondPerHost` | Default 1, from 0.1 up to 2. Never more than 2 requests in flight and never closer than 500 ms to one host. |
| `ignoreNoindex` | Include pages marked `noindex` (they are skipped by default). |
| `maxSitemapUrls` | Most URLs read from a site's sitemaps. Default 5,000. |

### Output fields

| Field | Meaning |
|---|---|
| `url`, `finalUrl`, `httpStatus` | The request, where it ended after redirects, and the final status. |
| `depth` | 0 for start URLs and sitemap pages, link depth otherwise. |
| `title`, `description`, `canonicalUrl`, `language` | From the page: `<title>` (or `og:title`, or the first heading), meta description, `rel=canonical` made absolute, and the `lang` attribute. |
| `markdown` | The main content as Markdown: headings, lists, tables, code blocks with language, links made absolute, emphasis. |
| `text` | With `markdown_and_text`: plain text. |
| `wordCount`, `tokenEstimate` | Words in the content (code included; Chinese, Japanese and Korean count two characters per word), and an estimate of the tokens (characters divided by 4, labelled an estimate, not a tokenizer count). |
| `headings` | `[{ level, text }]`, the outline of the content. |
| `chunks` | With `chunkMarkdown`: `[{ index, headingPath, markdown, tokenEstimate }]`. |
| `links` | With `includeLinks`: `[{ url, text, internal }]`. |
| `internalLinkCount`, `externalLinkCount` | Distinct links in the content, same site or not. |
| `jsonLdTypes` | schema.org types found in the page's JSON-LD (`Article`, `FAQPage`, `BreadcrumbList` and so on). |
| `lastModified` | The `Last-Modified` header, else the sitemap's `lastmod`, else `null`. |
| `contentHash` | SHA-256 of the Markdown. |
| `changed` | With `onlyChanged`: `new` or `changed` (unchanged pages are not output). Otherwise `null`. |
| `needsJavaScript` | `true` for an empty app shell (see below). |
| `skippedReason` | Why a row has no content, or `null`. See below. |
| `robotsAllowed` | `false` when robots.txt disallows the URL. |
| `extractor` | `page`, `article`, or `article_fallback_page`. |
| `error`, `errorCode` | Why the fetch failed, otherwise `null`. |
| `fetchedAt` | When the page was fetched. |

The run's key-value store has a `SUMMARY` record: pages requested and billed, unchanged pages, skips by reason, errors, URLs filtered out, the sitemaps read, `deletedUrls`, requests and retries, and why the run stopped.

#### Rows without content (all free)

| `skippedReason` or `errorCode` | Meaning |
|---|---|
| `robots` | robots.txt disallows the URL for this actor (`robotsAllowed: false`). It is never requested. Only start URLs get a row; disallowed links and sitemap entries are counted in the summary. |
| `pdf`, `not_html` | A PDF or another non-HTML file. The body is not downloaded. |
| `noindex` | The page asks not to be indexed (meta robots or `X-Robots-Tag`). Use `ignoreNoindex` to include it. Its links are still followed unless it says `nofollow`. |
| `needs_javascript` | Fewer than `minWords` words and the HTML is an app shell (empty `<div id="root">`, `__NEXT_DATA__`, "enable JavaScript"). `needsJavaScript` is `true` and `markdown` is empty. |
| `below_min_words` | A real page with too little text. The Markdown is delivered, but it is not charged. |
| `duplicate` | The same canonical page was already delivered. |
| `redirect_off_site` | A link redirects outside the crawl scope. |
| `HTTP_ERROR`, `ACCESS_DENIED`, `RATE_LIMITED` | `HTTP 404`, `HTTP 403 (access denied)`, `HTTP 429`. |
| `ROBOTS_UNREADABLE`, `CRAWL_DELAY_TOO_LONG` | robots.txt could not be read (a server error), or it asks for a Crawl-delay over 60 seconds. The site is not crawled. One row per site says why. |
| `TOO_LARGE`, `TIMEOUT`, `DNS`, `TLS`, `CONNECTION_REFUSED`, `PRIVATE_ADDRESS`, `BAD_REDIRECT`, `FETCH_FAILED`, `INTERNAL`, `INVALID_URL` | Fetch problems and bad input, each with a plain message. |

### Refresh a knowledge base nightly (`onlyChanged`)

1. Create a task with your start URLs, `crawlScope: "sitemap"` and `onlyChanged: true`.
2. Run it once. Every page comes back as `changed: "new"`. Load the rows or the `chunks` into your vector database, keyed by `url`, with `contentHash` as the version.
3. Add an Apify schedule, for example `0 3 * * *`. From then on a run outputs only pages that are new or changed, and you pay only for those. A run with nothing changed outputs no rows and costs the start fee.
4. After each run read the `SUMMARY` record: `deletedUrls` lists URLs that were in your last run but are no longer in the sitemap, or that now answer 404 or 410. Remove them from your index.
5. Send the dataset to your vector database with the integration of your choice (Apify Console lists integrations on the task) or a webhook to your own loader.

How it works: state is kept per site in the named key-value store `website-to-markdown-crawler-state` (a short hash, `Last-Modified` and `ETag` per path). Sitemap and list runs send conditional requests, so a server that supports them answers 304 and nothing is downloaded. Changing `contentMode`, `includeImages` or any option that alters the Markdown makes pages look changed once. A page that prints a changing date or counter in its content counts as changed on every run. In `links_only` runs conditional requests are not used (a 304 carries no links to follow), but unchanged pages are still detected by their hash and not charged. Deleted-URL detection runs for sitemap-driven runs only.

### Pricing

| What | Price |
|---|---|
| One page converted (at least `minWords` words of Markdown) | USD 0.0012 (USD 1.20 per 1,000) |
| Run start | USD 0.001 per run |
| Failed, skipped, blocked, duplicate, JavaScript-only and unchanged pages | Free |

Worked examples on the free plan:

- The default input (25 pages of documentation): 25 x 0.0012 + 0.001 = **USD 0.031**.
- A 500-page documentation site, first full run: 500 x 0.0012 + 0.001 = **USD 0.601**.
- The same site refreshed nightly with 10 changed pages: 10 x 0.0012 + 0.001 = **USD 0.013 a night**, about USD 0.39 a month.
- 20,000 pages across several sites, of which 1,500 fail or need JavaScript: 18,500 x 0.0012 = **USD 22.20**.

Higher Apify plans get a discount on the page price (10% on Silver, 20% on Gold and above). You can set a maximum cost per run: the actor stops cleanly when the budget is used and keeps every row already written. There is no proxy cost and no browser compute: this actor sends plain requests from the platform's own network.

### Speed and limits

- **Politeness sets the pace.** One request per second per site by default (up to 2). On one site, 1,000 pages take about 17 minutes at the default; sites on different hosts are crawled in parallel. The default input (25 pages on one site) takes about 30 seconds.
- **Work per page is small.** Parsing, Markdown conversion and chunking take 10 to 30 milliseconds for a typical documentation page (3,000 pages of 72 KB each took 36 seconds of CPU on a laptop, 360 MB peak memory), so the wait is the politeness spacing, not the processing.
- **Run timeout.** The default run timeout is 1 hour. The actor stops about 25 seconds before the platform timeout so the summary and the changed-pages state are saved. Raise the run timeout for big crawls.
- **Page size.** Pages over 5 MB are skipped with `TOO_LARGE`. A row must fit in one Apify dataset item (under 9 MB): a larger row loses `chunks` and `links` first and then has its text cut at a paragraph (`truncated: true`, with a note).
- **Memory.** The default is 1 GB; a few pages are held in memory at once.

### What it does not do

- **No JavaScript.** This is not a browser. A page that builds its content in the browser comes back flagged `needsJavaScript` (and is free) instead of empty and charged. Pages that load part of their content late (tabs, "load more" buttons, content fetched after the page loads) return only what the server sent. For those sites use a browser-based crawler such as [Apify's Website Content Crawler](https://apify.com/apify/website-content-crawler), which is free apart from compute but heavier and slower.
- **No logins, no paywall bypass, no CAPTCHA, no proxies, no fingerprint tricks.** A site that answers an honest crawler with 403 or 429 gets a polite stop (see below), not an arms race. Sites behind bot protection such as Cloudflare often return 403 to any automated client.
- **Content extraction is a heuristic.** `page` mode removes navigation, headers, footers, sidebars, breadcrumbs, cookie banners, share bars, comments, scripts, hidden elements and permalink anchors, by tag, ARIA role, and common class and id words. A site that hides furniture with unusual class names can leak a menu into the Markdown, and a site that puts real content in an element called "sidebar" can lose it. Check a sample of your site. `article` mode suits articles and drops tables and short sections more often.
- **Hidden tab panels are not read.** Elements with the `hidden` attribute or `display:none` in the markup are treated as hidden and dropped.
- **Tables** become pipe tables when they have regular rows. Tables used for page layout are read as ordinary content. Cells that hold lists or code are flattened to one line with `<br>`.
- **Languages.** Any language works; the language of a page is the `lang` it declares.
- **Email addresses and phone numbers** are not extracted as fields and `mailto:` and `tel:` links are reduced to their text. The Markdown is the text of the page as published. The default excludes skip author, team, people, staff and contact pages.

### Rights, robots.txt and politeness

- **You are responsible for having the right to copy a site's content.** This actor fetches public pages at addresses you supply. It does not republish anything itself. Check the site's terms and your own use case before you build a product on copied content.
- **robots.txt is respected** for every host, including after redirects. The `User-agent: *` rules apply to this actor, and so do rules written for the name `website-to-markdown-crawler`. `Crawl-delay` is honored (one request at a time, spaced as asked) up to 60 seconds; a longer delay means the site is skipped with a clear row. If robots.txt cannot be read because of a server error, the site is not crawled (a missing robots.txt, HTTP 404, means no rules). A disallowed URL is never requested and never retried.
- **Honest identification.** Every request uses the User-Agent `Mozilla/5.0 (compatible; website-to-markdown-crawler/1.0; +https://apify.com/JTPalms)`, plain headers, no cookies, no proxies, no browser automation.
- **Gentle by design.** At most 2 requests in flight to one host, at least 500 ms between request starts (1 second by default), retries with back-off for temporary errors, and a stop after 5 consecutive 403 or 429 answers from one site. The crawl never goes outside your start hosts.
- Private and internal addresses (localhost, 10.x, 192.168.x, cloud metadata addresses) are refused.
- Pages marked `noindex` are skipped unless you ask for them.

### Related actors

- [PDF to Markdown and Text: Tables, Pages, Metadata for RAG](https://apify.com/JTPalms/pdf-to-markdown): PDF parser and text extractor: turn PDF URLs into clean Markdown, plain text, per-page text and detected tables.
- [Sitemap URL Extractor: All Pages of Any Website](https://apify.com/JTPalms/sitemap-url-extractor): Get every URL of a website from its XML sitemaps.
- [News Article Extractor: Text, Date, Image from URLs and Feeds](https://apify.com/JTPalms/news-article-extractor): News article scraper and extractor: turn article URLs, RSS or Atom feeds and news sitemaps into clean rows: title, Markdown and plain...
- [SEO Audit & AI Search Readiness Checker (GPTBot, llms.txt)](https://apify.com/JTPalms/seo-ai-readiness-audit): Crawl a site and audit every page for SEO basics (title, meta, H1, canonical, hreflang, noindex, alt text, JSON-LD, status, speed) plus...

### FAQ

**Why was a page not charged?** Fetch errors, skipped pages (robots.txt, PDFs, noindex, duplicates), JavaScript-only shells, pages under `minWords`, and unchanged pages in `onlyChanged` mode are free. The row says why in `skippedReason` or `error`.

**Why did I get fewer pages than `maxPages`?** The sitemap and the links gave fewer pages in your scope, or the rest were filtered (check `urlsFilteredOut` in the summary), or the run hit its time or cost limit (also in the summary).

**Can I crawl a site that needs a login?** No. This actor reads public pages only.

**Does the title contain the site name?** `title` is the page's `<title>` as published. In `article` mode the Markdown starts with the page's own first heading when there is one.

**How are token counts calculated?** Characters divided by 4, rounded up. It is a quick estimate for planning chunk sizes, not the count of a particular model's tokenizer.

# Actor input Schema

## `startUrls` (type: `array`):

Pages or site roots to crawl, one per line. A URL with a path (https://example.com/docs/intro) keeps the crawl inside that folder (/docs/) unless you turn off "Stay in the start folder". The default is one page of the public Crawlee documentation.

## `crawlScope` (type: `string`):

sitemap_then_links: read the sitemap, and follow links only when the sitemap lists fewer pages than your limit. sitemap: sitemap pages only. links_only: follow links from the start URLs. single_pages: only the start URLs.

## `maxPages` (type: `integer`):

Hard cap on pages requested in this run, across all sites (1 to 50,000). Pages that fail still count, because they were requested.

## `maxDepth` (type: `integer`):

How many links deep to follow from a start URL when following links (0 to 20). Sitemap pages and start URLs are depth 0.

## `stayInStartFolder` (type: `boolean`):

When a start URL has a path, only crawl pages under its folder (the start URL https://example.com/docs/intro covers /docs/). Turn off to crawl the whole site. Pages are always on the same host as a start URL.

## `includeUrlPatterns` (type: `array`):

Optional. Crawl only URLs that match at least one pattern. A pattern is a glob (\* matches inside one path segment, \*\* across segments, for example **/docs/**), a plain text that the URL must contain (for example /blog/), or a regular expression written as re:^https://example.com/(en|de)/.

## `excludeUrlPatterns` (type: `array`):

URLs that match any pattern are skipped (same pattern types as above). The defaults skip author, team, people, staff and contact pages, which are mostly personal details. The prefill also skips the old versions of the Crawlee docs.

## `contentMode` (type: `string`):

page: the cleaned page body without navigation, footer, sidebar and cookie banners (best for documentation). article: Readability's main article (best for blogs and news); falls back to page mode when no article is found.

## `outputFormat` (type: `string`):

markdown: the markdown field. markdown_and_text: also a plain text field with the same reading order and no Markdown syntax.

## `chunkMarkdown` (type: `boolean`):

Add chunks: the Markdown split at headings into pieces of at most the token limit below, each with its heading path and a token estimate.

## `maxChunkTokens` (type: `integer`):

Estimated tokens per chunk (characters divided by 4, an estimate, not a tokenizer count). 50 to 4,000.

## `includeImages` (type: `boolean`):

Keep images as Markdown image links with their alt text. Off by default: images are dropped.

## `includeLinks` (type: `boolean`):

Add links: the internal and external links found in the page content (deduplicated, with link text). The counts are always included.

## `onlyChanged` (type: `boolean`):

Skip pages whose Markdown is unchanged since your last only-changed run on the same site. Unchanged pages are not output and not charged. The first run outputs everything as new. State is kept in the key-value store website-to-markdown-crawler-state.

## `useLlmsTxt` (type: `boolean`):

Also seed the crawl from the site's /llms.txt (the curated list of pages some sites publish for AI tools).

## `minWords` (type: `integer`):

A page with fewer words than this is not charged (and with a JavaScript app shell it is flagged needsJavaScript). 0 to 10,000.

## `requestsPerSecondPerHost` (type: `number`):

Politeness limit for each site (0.1 to 2). Never more than 2 requests at once and never closer than 500 ms apart. A Crawl-delay in robots.txt slows this further.

## `ignoreNoindex` (type: `boolean`):

Pages marked noindex (meta robots or X-Robots-Tag) are skipped by default. Turn on to include them. robots.txt is always respected.

## `maxSitemapUrls` (type: `integer`):

Most URLs read from a site's sitemaps (1 to 100,000). Only URLs in scope are kept.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://crawlee.dev/js/docs/introduction"
    }
  ],
  "crawlScope": "sitemap_then_links",
  "maxPages": 25,
  "maxDepth": 3,
  "stayInStartFolder": true,
  "excludeUrlPatterns": [
    "**/author/**",
    "**/author",
    "**/authors/**",
    "**/authors",
    "**/team/**",
    "**/team",
    "**/people/**",
    "**/people",
    "**/staff/**",
    "**/staff",
    "**/contact",
    "**/contact/**",
    "**/contact-us",
    "**/contact-us/**",
    "**/js/docs/3.*/**",
    "**/js/docs/next/**"
  ],
  "contentMode": "page",
  "outputFormat": "markdown",
  "chunkMarkdown": false,
  "maxChunkTokens": 500,
  "includeImages": false,
  "includeLinks": false,
  "onlyChanged": false,
  "useLlmsTxt": false,
  "minWords": 30,
  "requestsPerSecondPerHost": 1,
  "ignoreNoindex": false,
  "maxSitemapUrls": 5000
}
```

# Actor output Schema

## `results` (type: `string`):

One row per page in the default dataset (JSON, CSV, Excel via the format parameter): Markdown, metadata, headings, link counts, chunks when asked.

## `summary` (type: `string`):

Counts (pages requested, billed, unchanged, skipped by reason, errors), the sitemaps read, URLs that vanished from the sitemap, and request statistics.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://crawlee.dev/js/docs/introduction"
        }
    ],
    "maxPages": 25,
    "excludeUrlPatterns": [
        "**/author/**",
        "**/author",
        "**/authors/**",
        "**/authors",
        "**/team/**",
        "**/team",
        "**/people/**",
        "**/people",
        "**/staff/**",
        "**/staff",
        "**/contact",
        "**/contact/**",
        "**/contact-us",
        "**/contact-us/**",
        "**/js/docs/3.*/**",
        "**/js/docs/next/**"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("jtpalms/website-to-markdown-crawler").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": [{ "url": "https://crawlee.dev/js/docs/introduction" }],
    "maxPages": 25,
    "excludeUrlPatterns": [
        "**/author/**",
        "**/author",
        "**/authors/**",
        "**/authors",
        "**/team/**",
        "**/team",
        "**/people/**",
        "**/people",
        "**/staff/**",
        "**/staff",
        "**/contact",
        "**/contact/**",
        "**/contact-us",
        "**/contact-us/**",
        "**/js/docs/3.*/**",
        "**/js/docs/next/**",
    ],
}

# Run the Actor and wait for it to finish
run = client.actor("jtpalms/website-to-markdown-crawler").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://crawlee.dev/js/docs/introduction"
    }
  ],
  "maxPages": 25,
  "excludeUrlPatterns": [
    "**/author/**",
    "**/author",
    "**/authors/**",
    "**/authors",
    "**/team/**",
    "**/team",
    "**/people/**",
    "**/people",
    "**/staff/**",
    "**/staff",
    "**/contact",
    "**/contact/**",
    "**/contact-us",
    "**/contact-us/**",
    "**/js/docs/3.*/**",
    "**/js/docs/next/**"
  ]
}' |
apify call jtpalms/website-to-markdown-crawler --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,jtpalms/website-to-markdown-crawler"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/gpxn2FrxqnXEjPIDH/builds/XscRXYqAduRh6e7eW/openapi.json