# Website to Markdown Crawler for AI & RAG — Only Changed Pages (`northbell/website-markdown-crawler`) Actor

Crawl a website over plain HTTP and get clean Markdown for AI and RAG. Sitemap first, no browser, no login, robots.txt respected. Run it again and get only the pages whose main text changed since the last run, so a RAG index stays fresh for cents. $1 per 1,000 pages.

- **URL**: https://apify.com/northbell/website-markdown-crawler.md
- **Developed by:** [Northbell](https://apify.com/northbell) (community)
- **Categories:** AI, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.80 / 1,000 pages

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Website to Markdown Crawler for AI & RAG

Give it one or more website addresses and get **clean Markdown for every page** — title, description, language, headings, word count and a content hash — ready for an AI agent, a RAG index or a knowledge base. It crawls over **plain HTTP with no browser**, reads the **sitemap first**, and respects `robots.txt`. **$1 per 1,000 pages**, no login, no API key.

Run it again — or put it on a daily schedule — and every page says whether it is **new, changed or unchanged** since the last run. Switch on **Return only new and changed pages** and you get (and pay for) only the pages whose main text changed, so keeping a RAG index fresh costs cents. The first run returns every page.

### What you get

One `page` row per crawled page:

| Field | What it is |
|---|---|
| `url`, `canonicalUrl` | the address the page was read from (after redirects) and the address its `<link rel="canonical">` names (the same as `url` when the page names none) |
| `title`, `description`, `language` | `<title>`, the meta description and the `lang` of the page — `null` when the page does not state them |
| `markdown` | the main content as Markdown: headings, paragraphs, lists, links (absolute), bold/italic, code blocks, quotes and tables as GitHub tables |
| `textLength`, `wordCount` | characters and words of the main text (for Chinese, Japanese and Korean, about one word per two characters) |
| `headings` | the h1–h3 headings of the main content as `{level, text}` |
| `linkCount` | how many different links the main content holds |
| `lastModified` | `<lastmod>` from the sitemap, else the `Last-Modified` header, else `null` |
| `contentHash` | SHA-256 of the Markdown (first 32 characters) — the same text gives the same hash |
| `status`, `previousContentHash`, `previousRunAt` | `new`, `changed` or `unchanged` compared with the previous run for this start URL; `previousContentHash` is `null` the first time a page is seen, `previousRunAt` (when the start URL was last crawled) is `null` on its first run |
| `depth`, `startUrl` | clicks from the start page (pages taken from the sitemap count as 1) and the start URL that reached the page |
| `html` | the cleaned main content as HTML — only with *Also return cleaned HTML* |
| `truncated`, `observedAt` | `true` when the Markdown was cut at 1,000,000 characters or the page was too large to read to the end (see Limits); when the run started |

Free rows next to the pages:

- **`summary`** — one per start URL: `pagesFound`, `pagesCrawled`, `new`, `changed`, `unchanged`, `skippedByRobots`, `needsJavascript`, `tooShort`, `failed`, `notCrawled` (found but beyond *Maximum pages per start URL*) and `skippedOther` (not HTML, duplicates, redirected away), plus `firstRun`, `previousRunAt` and `changedOnly`.
- **`note`** — a page that was not returned and why (`reason`): `needs-javascript` (the HTML holds almost no text because the page builds it with JavaScript — `textLength` is tiny), `too-little-text`, `blocked-by-robots` (the start URL itself is disallowed) or `not-html`.
- **`error`** — a page that could not be fetched (`errorKind`: `not-found`, `blocked`, `rate-limited`, `network`; `statusCode`).

### Run it again and again

- **On a schedule** — save your input as a task and add a **Schedule**: the same site every day or week. Send each run to Google Sheets, Slack or email with an integration on the task.
- **Watch for changes** — Turn on **Return only new and changed pages**: from the second run, pages whose main text is the same as last time are not returned and not charged.
- **In bulk** — Put several sites in **Start URLs** in one run.

### Use it for

- **RAG and chatbots** — turn a documentation site or help center into Markdown chunks, then re-run daily with *only new and changed pages* and upsert by `url`
- **AI agents** — give an agent current, readable text of a site without a browser tool
- **Change monitoring** — watch competitors' pricing, docs or terms pages: `status: "changed"` plus `previousContentHash` tells you which page moved
- **Content audits** — word counts, headings and missing descriptions for every page of a site

### Input

```json
{
  "startUrls": ["https://docs.apify.com/academy"],
  "maxPages": 50,
  "maxDepth": 3,
  "scope": "path",
  "useSitemap": true,
  "includeGlobs": [],
  "excludeGlobs": ["**/tag/**"],
  "changedOnly": false,
  "minTextLength": 200,
  "includeHtml": false,
  "includeImages": false
}
```

Globs match the whole address (`https://example.com/blog/**`; `*` stops at `/`, `**` does not) — or only the path when they start with `/` (`/blog/**`). Start URLs are always crawled.

The state for *new / changed / unchanged* is kept per start URL in a named key-value store in your Apify account (`website-markdown-crawler-state`), so the second run of the same start URL already knows the first. Schedule it (Apify → Schedules) for a daily refresh.

### Pricing

**$1 per 1,000 pages** (a `page` row), plus $0.01 per run. Summary, note and error rows are free. With *Return only new and changed pages*, unchanged pages are not returned and not charged: a 500-page documentation site where 12 pages changed costs about $0.02 after the first run.

### Notes

- **Plain HTTP, no JavaScript.** Server-rendered sites (documentation, blogs, news, shops, company sites) work. A page that only shows text after JavaScript runs comes back as a free `needs-javascript` note, never as invented content. Pages behind a login cannot be read. Sites that block automated traffic from data centers (some news and shop sites answer HTTP 403) come back as free `error` rows with `errorKind: "blocked"`; the Actor uses no proxy and does not try to get around the block.
- **robots.txt is respected** for `User-agent: *` and for the groups addressed to Claude crawlers (`ClaudeBot`, `Claude-User`…) — a page is read only if both allow it (`Disallow`, `Allow`, `*` and `$`, the longest rule wins). A disallowed start URL is not fetched. `Crawl-delay` is not applied; the Actor makes at most 5 requests at a time per start URL. If robots.txt cannot be read, the crawl goes on.
- **Where it goes.** The start page first, then the sitemap (from robots.txt, `/sitemap.xml`, sitemap indexes and `.gz` files), then links, breadth first. By default it stays on the same site **and under the start URL's folder** (`https://example.com/docs` reads `/docs` and below); set *Crawl scope* to *Whole site* for everything on the host. Other websites are never followed. With a large sitemap index, reading stops once enough addresses are queued, so `pagesFound` counts what was found so far. *Maximum pages per start URL* counts fetched pages, including notes and errors.
- **Main content.** `<main>`, `<article>` or `role="main"`, otherwise the block that holds most of the text. Navigation, headers and footers, sidebars, forms, cookie banners, share bars, comment sections and text hidden with CSS are left out.
- **No contact data.** E-mail addresses and obvious phone numbers are removed from the text, and `mailto:` and `tel:` links are dropped (an address on example.com is kept — it is not personal data). A link whose address contains an e-mail address is neither followed nor listed. Links to author pages and elements marked as bylines are left out, and pages that exist to show one person (`/author/name/`, `/@name`, `/u/name`) are not crawled unless you give one as a start URL; names that are written inside the body text stay, because that is the page's own text.
- **What "changed" means.** The Markdown of a page differs from the last run. Text that changes on every load (a visit counter, "posted 3 minutes ago") makes a page look changed every time, and so does switching *Keep images in the Markdown* on or off (every page's Markdown changes once). Up to 20,000 pages are remembered per start URL, the oldest dropped first. A page reached under two addresses is returned once when both name the same canonical address and have the same text.
- **Language and location.** Requests are plain and come from Apify's servers with `Accept-Language: en-US`. A site that picks its language, currency or prices by location or browser language may answer with its English or US version — start from the language folder you want (`https://example.com/ja/`).
- **Limits.** A page larger than 3 MB is cut at 3 MB, and a page with more than 60,000 HTML elements is read only as far as that (`truncated: true`; the largest pages we tried had about 22,000); non-HTML addresses (PDF, images) are skipped; addresses on private networks are refused; up to 5,000 pages per start URL and 100 start URLs per run.

For a blog's feed instead of its pages, see [RSS Feed Reader](https://apify.com/northbell/rss-feed-reader). To see what a site is built with, see [Tech Stack Detector](https://apify.com/northbell/tech-stack-detector). For pages that no longer exist, see [Wayback Machine Scraper](https://apify.com/northbell/wayback-machine-scraper).

### For AI agents

`{"startUrls": ["https://docs.apify.com/academy"], "maxPages": 3}` returns up to 3 rows with `type: "page"` (use `markdown`, keyed by `url`) and one `type: "summary"` row. To keep an index fresh, run on a schedule with `"changedOnly": true` and upsert the returned pages by `url`; rows with `type` `note` or `error` are free and explain what was skipped.

# Actor input Schema

## `startUrls` (type: `array`):

One web address per line, for example https://docs.apify.com/academy. The crawl starts here, reads the site's sitemap and follows links to other pages of the same site (see Crawl scope). Each start URL gets its own summary row.

## `maxPages` (type: `integer`):

The most pages fetched for each start URL. Pages are fetched in the order they are found: the start page first, then the sitemap, then links. Short pages that are left out and notes about pages that need JavaScript count towards this limit, but they are never charged.

## `maxDepth` (type: `integer`):

How many clicks away from the start page the crawl may go. 0 reads only the start page itself. Pages taken from the sitemap count as depth 1. Raise it for sites without a sitemap.

## `scope` (type: `string`):

Which pages count as part of the site. "Start URL folder" keeps to the start URL's path (a start URL of https://example.com/docs reads /docs and everything under it). "Whole site" reads any page on the same host (www.example.com and example.com are the same site). Other websites are never followed.

## `useSitemap` (type: `boolean`):

Read robots.txt and sitemap.xml (including sitemap indexes and .gz sitemaps) to find pages before following links. Fast and complete on sites that publish a sitemap. Turn off to follow links only.

## `includeGlobs` (type: `array`):

Optional. Only pages whose address matches one of these patterns are crawled (start URLs are always crawled). \* matches anything except /, \*\* matches anything. Example: https://example.com/blog/**. A pattern that starts with / is matched against the path only, for example /blog/**.

## `excludeGlobs` (type: `array`):

Optional. Pages whose address matches one of these patterns are skipped. Same pattern rules as above. Example: **/tag/** or /login/\*\*.

## `changedOnly` (type: `boolean`):

When on, pages whose main text is the same as in the previous run are not returned and not charged. The first run returns every page. Leave off to get every page every time (each row still says whether it is new, changed or unchanged).

## `minTextLength` (type: `integer`):

Pages with fewer characters of main text than this are left out and reported in a free note row. Pages that only show their text after JavaScript runs are reported as "needs-javascript"; other short pages as "too-little-text". 0 keeps any page that has at least one character.

## `includeHtml` (type: `boolean`):

Add an html field with the main content as cleaned HTML (no scripts, styles, navigation or footer; links and image addresses made absolute).

## `includeImages` (type: `boolean`):

Write images as ![alt](address). Off by default because most images on a page are logos and icons that add noise for AI.

## Actor input object example

```json
{
  "startUrls": [
    "https://docs.apify.com/academy"
  ],
  "maxPages": 50,
  "maxDepth": 3,
  "scope": "path",
  "useSitemap": true,
  "changedOnly": false,
  "minTextLength": 200,
  "includeHtml": false,
  "includeImages": false
}
```

# Actor output Schema

## `pages` (type: `string`):

Title, status since the last run, word count and the Markdown of every returned page.

## `summary` (type: `string`):

Pages found, crawled, new, changed and unchanged, plus what was skipped and why.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        "https://docs.apify.com/academy"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("northbell/website-markdown-crawler").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "startUrls": ["https://docs.apify.com/academy"] }

# Run the Actor and wait for it to finish
run = client.actor("northbell/website-markdown-crawler").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    "https://docs.apify.com/academy"
  ]
}' |
apify call northbell/website-markdown-crawler --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,northbell/website-markdown-crawler"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/eaN6MleAhRiEuP59C/builds/edGPMExpD4irAgkF7/openapi.json