# PageMark: Website to LLM-Ready Markdown & RAG Chunks (`1701_engage/pagemark-website-to-markdown`) Actor

Crawl any website into clean Markdown for LLMs and RAG: main content only, code blocks and tables intact, token-counted chunks, only-changed pages. HTTP-first, browser only when needed. $0.45 per 1,000 pages.

- **URL**: https://apify.com/1701_engage/pagemark-website-to-markdown.md
- **Developed by:** [Thomas Jonkergouw](https://apify.com/1701_engage) (community)
- **Categories:** AI, Developer tools, Automation
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.45 / 1,000 pages

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## PageMark: Website to LLM-Ready Markdown & RAG Chunks

PageMark crawls a website and turns every page into **clean Markdown for LLMs, RAG and AI agents**. You get the main content only, with no menus, cookie banners or footers. Code blocks keep their language and indentation, tables stay tables, and headings stay headings. Choose whole pages or **token-counted chunks** that you can load straight into a vector database.

**$0.45 per 1,000 pages.** Pages that need a real browser cost $2.25 per 1,000. You pay only for pages that are saved.

### Why PageMark

- **Correct content.** Code blocks become fenced code with the language tag (` ```python `), with no backslash escapes that break code for an LLM. Doc pages keep their explanations, not only the code samples.
- **Fast and cheap.** PageMark fetches pages over plain HTTP first and only starts a browser for pages that come back empty without JavaScript. A 30-page docs crawl takes about 10 seconds.
- **RAG-ready chunks.** Chunks follow the page headings, every chunk carries its heading path (`Guide > Install > Linux`) and token count (OpenAI cl100k), and a code block is never cut in half. Optional overlap.
- **Only new or changed pages.** On a schedule, PageMark skips pages whose content has not changed since the last run, and you are not charged for skipped pages. Re-embed only what changed.
- **Predictable price.** You pay a fixed price per page, so a heavy site doesn't change your bill.
- **Plays by the rules.** PageMark respects robots.txt (including crawl-delay), identifies itself, and retries politely on 429/5xx. It does not bypass CAPTCHAs or bot protection.

### Benchmark

We ran the same three crawls (30 pages each, 26 October 2026) with PageMark and with Apify's Website Content Crawler on its **default settings**.

| Site | Tool | Cost per 1,000 pages | Time | Fenced code blocks | Backslash escapes | Image links |
|---|---|---|---|---|---|---|
| docs.python.org tutorial | Website Content Crawler | $1.45 (compute) | 105 s | 0 | 2,358 | 0 |
| | **PageMark** | **$0.45** | **7 s** | **344** | **0** | 0 |
| react.dev/learn | Website Content Crawler | $5.36 (compute) | 753 s | 226 | 1,918 | 70 |
| | **PageMark** | **$0.45** | **11 s** | **315** | **0** | 0 |
| bbc.com/news | Website Content Crawler | $3.50 (compute) | 434 s | 0 | 4 | 551 |
| | **PageMark** | **$0.45** | **14 s** | 0 | 0 | 0 |

Website Content Crawler has many settings, and other settings will give different numbers. These are the defaults most people run.

### What you get per page

```json
{
  "url": "https://react.dev/learn/state-a-components-memory",
  "title": "State: A Component's Memory – React",
  "description": "The library for web and native user interfaces",
  "language": "en",
  "markdown": "# State: A Component's Memory\n\nComponents often need to change what’s on the screen as a result of an interaction. ...",
  "stats": { "words": 2802, "tokens": 5038, "codeBlocks": 12, "tables": 0 },
  "crawl": { "requestedUrl": "https://react.dev/learn/state-a-components-memory", "depth": 1, "referrerUrl": "https://react.dev/learn",
             "httpStatus": 200, "renderedWithBrowser": false, "fetchedAt": "2026-10-26T15:16:36+00:00" },
  "contentHash": "cea8cbc84bd2ddf956ec2c5634319c5c"
}
```

In **RAG chunks** mode you get one row per chunk instead:

```json
{ "url": "https://fastapi.tiangolo.com/tutorial/path-params/", "title": "Path Parameters - FastAPI",
  "chunkIndex": 9, "chunkCount": 20,
  "headingPath": ["Path Parameters", "Predefined values", "Create an `Enum` class"],
  "text": "### Create an `Enum` class\n\nImport `Enum` and create a sub-class that inherits from `str` and from `Enum`. ...",
  "tokens": 257 }
```

The run's key-value store has a `SUMMARY` record with counts and the exact pages that failed or were blocked by robots.txt. Nothing is skipped silently.

### How to use it

1. Add one or more **Start URLs**. PageMark follows links that stay under each URL: `https://docs.example.com/guide/` crawls only the guide.
2. Set **Max pages**. You pay only for pages that are saved.
3. Pick **Output**: pages, pages with chunks, or one row per chunk.
4. Optionally turn on **Also use sitemaps** to find pages that no menu links to, or **Only new or changed pages** for scheduled runs.

To fetch just a list of URLs without crawling, set **Max crawl depth** to `0`.

#### Main settings

| Setting | What it does |
|---|---|
| Max crawl depth | How many links deep to go. `0` = only the start URLs. |
| Include / Exclude URLs | Glob patterns, e.g. `https://example.com/blog/**` or `**/tag/**`. |
| JavaScript rendering | **Auto** (browser only when needed), **Never** (cheapest), **Always**. |
| Page formats | Markdown, plain text and/or clean HTML of the main content. |
| Chunk size / overlap | Tokens per chunk (default 512) and optional overlap. |
| Keep links / Include images | Keep `[text](url)` links (on by default); add `![alt](src)` images (off by default). |
| Remove elements | Extra CSS selector to remove. Menus, footers, cookie bars and sidebars are already removed. |

### Pricing

| Event | Price |
|---|---|
| Page saved (HTTP) | $0.00045 ($0.45 per 1,000) |
| Page saved with browser (JavaScript sites) | $0.00225 ($2.25 per 1,000) |

Pages that fail, are blocked by robots.txt, are duplicates, are not HTML, or are unchanged in "only new or changed" mode are **not charged**. You can set a maximum cost per run in the run options, and PageMark stops cleanly when it reaches it.

### Use it with your AI stack

- **LangChain / LlamaIndex:** load the dataset with the Apify dataset loader and use `markdown` (pages) or `text` (chunks) as the document content, with `url` and `headingPath` as metadata.
- **Vector databases (Pinecone, Qdrant, Weaviate…):** run in RAG chunks mode and embed `text`. `contentHash` tells you when a page changed.
- **AI agents (MCP):** PageMark works as a tool through the Apify MCP server.
- **API:** start a run with `POST https://api.apify.com/v2/acts/1701_engage~pagemark-website-to-markdown/run-sync-get-dataset-items`.

### FAQ

**Does it work on JavaScript sites?** Yes. In Auto mode a page that is empty without JavaScript is loaded in a headless browser. Those pages are marked `renderedWithBrowser: true` and billed at the browser price.

**Why did a page fail?** See the `SUMMARY` record. Common reasons are a 403/429 from bot protection, a login wall, or robots.txt. PageMark does not get around bot protection or logins.

**PDFs and other files?** PageMark only converts HTML pages. Links to PDFs, images and archives are skipped and not charged.

**Personal data?** PageMark saves page content as published. Comply with the website's terms and with privacy law (such as the GDPR) for whatever you crawl.

Missing a feature or found a page that converts badly? Open an issue with the URL. Conversion bugs are fixed first.

# Actor input Schema

## `startUrls` (type: `array`):

Pages to start from. PageMark follows links that stay under each start URL (e.g. https://docs.example.com/guide/ crawls only the guide). Set Max crawl depth to 0 to fetch only these exact pages.

## `maxPages` (type: `integer`):

Stop after this many pages are saved. You only pay for saved pages.

## `maxCrawlDepth` (type: `integer`):

How many links deep to follow from the start URLs. 0 = only the start URLs themselves.

## `includeUrlGlobs` (type: `array`):

Optional. Only crawl URLs matching one of these patterns, e.g. https://example.com/blog/\*\*. Replaces the default 'under the start URL' rule.

## `excludeUrlGlobs` (type: `array`):

Skip URLs matching any of these patterns, e.g. **/tag/** or \*\*?page=\*.

## `useSitemaps` (type: `boolean`):

Find pages through the site's sitemap.xml too (from robots.txt). Finds pages that no menu links to.

## `renderJs` (type: `string`):

Auto: fast HTTP first, and a real browser only for pages that come back empty without JavaScript (billed as 'rendered page'). Never: HTTP only, cheapest. Always: every page in a browser.

## `outputMode` (type: `string`):

Pages: one row per page. Pages + chunks: each page also gets a 'chunks' list. Chunks: one row per chunk, ready to load into a vector database.

## `outputFormats` (type: `array`):

Which content fields each page row gets.

## `chunkSizeTokens` (type: `integer`):

Maximum tokens per chunk (OpenAI cl100k tokenizer). Chunks follow headings, and code blocks are never cut in half unless one block alone is larger than this.

## `chunkOverlapTokens` (type: `integer`):

Repeat the last N tokens of a chunk at the start of the next chunk in the same section.

## `onlyChangedSinceLastRun` (type: `boolean`):

For scheduled runs: skip pages whose content is the same as in your previous run with the same start URLs. Skipped pages are not charged.

## `keepLinks` (type: `boolean`):

Keep [text](url) links in the Markdown. Turn off to save tokens.

## `includeImages` (type: `boolean`):

Keep ![alt](url) image references in the Markdown.

## `removeElementsCssSelector` (type: `string`):

Optional extra elements to remove before conversion, e.g. .promo, #cookie-bar. Menus, footers, cookie banners and sidebars are already removed automatically.

## `maxConcurrency` (type: `integer`):

Pages fetched at the same time. robots.txt crawl-delay is always respected.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://docs.apify.com/platform/actors"
    }
  ],
  "maxPages": 100,
  "maxCrawlDepth": 20,
  "useSitemaps": false,
  "renderJs": "auto",
  "outputMode": "pages",
  "outputFormats": [
    "markdown"
  ],
  "chunkSizeTokens": 512,
  "chunkOverlapTokens": 0,
  "onlyChangedSinceLastRun": false,
  "keepLinks": true,
  "includeImages": false,
  "maxConcurrency": 10
}
```

# Actor output Schema

## `pages` (type: `string`):

Saved pages with Markdown and stats.

## `chunks` (type: `string`):

One row per chunk (when Output = RAG chunks).

## `summary` (type: `string`):

Counts, plus pages that failed or were blocked by robots.txt.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://docs.apify.com/platform/actors"
        }
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("1701_engage/pagemark-website-to-markdown").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "startUrls": [{ "url": "https://docs.apify.com/platform/actors" }] }

# Run the Actor and wait for it to finish
run = client.actor("1701_engage/pagemark-website-to-markdown").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://docs.apify.com/platform/actors"
    }
  ]
}' |
apify call 1701_engage/pagemark-website-to-markdown --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,1701_engage/pagemark-website-to-markdown"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/bLKRKdfs9ABr3CdFp/builds/v76FmRrFfkLvXslVU/openapi.json