# Website Markdown Crawler - Content & Change Tracking (`benthepythondev/website-markdown-crawler`) Actor

Crawl public HTML within a website section. Export Markdown, text, source URLs and change hashes, with page limits and a URL-level coverage report.

- **URL**: https://apify.com/benthepythondev/website-markdown-crawler.md
- **Developed by:** [Ben](https://apify.com/benthepythondev) (community)
- **Categories:** Developer tools, Automation, SEO tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.80 / 1,000 html pages

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Website Markdown Crawler

Crawl linked pages within a public website section and export Markdown, readable text and a source URL for each page. Start with a documentation index, blog or help center. The crawler follows links within that URL's origin and path subtree, removes common navigation noise, and records exactly which URLs it attempted.

For recurring imports, supply the previous run's content hashes. Changed pages include their new content; unchanged pages return a small metadata record so your downstream workflow can skip reprocessing them. Every exported page costs the same, including unchanged records.

Ordinary HTTP is the default. Enable `renderJavaScript` for public content generated by same-origin JavaScript. Both modes use the same extraction, link scope and change hashes. The Actor does not use login cookies, solve access challenges, download files or search the web.

**Page-based pricing:** **$1 per 1,000 exported pages** on the Free plan, plus the start event. Unchanged metadata-only records are also billable pages. Both HTTP and JavaScript-rendered pages use this price. Supply URLs; web search is not included.

### Try a small crawl

Run the [Python tutorial example](https://apify.com/benthepythondev/website-markdown-crawler/examples/python-tutorial-markdown-pages) with five pages. Inspect **Overview**, switch to **Content** for the Markdown, and open **URL coverage report** in the run output.

```json
{
  "startUrls": ["https://docs.python.org/3/tutorial/"],
  "maxPages": 5,
  "maxDepth": 1
}
```

Start URLs are strings. `maxDepth: 0` processes only those URLs. Depth one also follows links found on them. The default is ten page attempts and depth two; the hard limits are 200 attempts and depth five. A redirect or robots.txt check does not consume an extra page attempt.

If the start URL is `https://example.com/docs/`, `/docs/setup` is in scope and `/blog/` is outside it. Other subdomains, origins and protocols are outside that start URL's scope. Supply their own start URL if you intend to crawl them. A cross-scope redirect stops the run with an explanation; use the final URL shown by your browser.

### Render JavaScript content

When a page builds its content in JavaScript, enable rendering and wait for an element that signals the content is present. This example uses the public Quotes to Scrape practice site:

```json
{
  "startUrls": ["https://quotes.toscrape.com/js/"],
  "maxPages": 2,
  "maxDepth": 1,
  "renderJavaScript": true,
  "waitForSelector": ".quote"
}
```

`waitForSelector` waits for the first matching element, up to ten seconds after the page load. It does not select the exported content; use `contentSelector` for that. With no wait selector, the crawler captures the DOM one second after page load. This does not guarantee that a timer, animation or later request has finished. Keep the same mode and selectors when comparing content hashes.

Rendering supports same-origin scripts, stylesheets and GET-based JSON requests. It blocks cross-origin resources, forms and other non-GET requests, frames, images, fonts, media, WebSockets and service workers. Sites that require those features may return partial content or fail the wait selector. Each page has a 45-second rendering limit, at most 40 resource requests and 20 MB of resource text. Failed allowed resources and exceeded limits fail visibly in `COVERAGE`; intentionally blocked resource requests are counted there. No click, login or challenge-solving step is provided.

The same public-address checks, robots rules and request delays apply to resources. Browser rendering uses more runtime than HTTP, so start with a small page count. The existing Python documentation example remains in HTTP mode at 512 MB.

### Output

One dataset item represents one successfully extracted HTML page. The following fields are available in JSON, CSV and Excel exports:

| Field | Meaning |
|---|---|
| `url`, `requested_url` | Final fetched page and its queued URL before redirects |
| `title`, `canonical_url` | Page title and declared canonical link, when present |
| `markdown`, `text` | Extracted content; null for an unchanged page |
| `content_hash`, `previous_hash` | SHA-256 of the Markdown and the caller's previous value |
| `change_status` | `new`, `changed` or `unchanged` |
| `word_count` | Word count of the current extracted text, also present for unchanged pages |
| `depth`, `source_url` | Link depth and the page that discovered this URL; a seed has depth zero and no source URL |
| `scraped_at` | UTC observation time |

A metadata excerpt from the tested Python tutorial crawl:

```json
{
  "url": "https://docs.python.org/3/tutorial/",
  "title": "The Python Tutorial \u2014 Python 3.14.7 documentation",
  "change_status": "new",
  "word_count": 1057,
  "content_hash": "b216133cb14e3dfcc3d354ef3a83bd77b1d1100d955064e1477e2eac4be3928e",
  "depth": 0
}
```

The full record also includes the Markdown and readable text.

Markdown retains headings, links, emphasis, lists, fenced code and simple tables. Links become absolute URLs so they still work after export. Code blocks retain whitespace. Complex layouts, table spans and visual components may not translate exactly. Images and their bytes are not exported.

The default content selection prefers a `main` element or a main role, then a single `article`, then the document body. When a page contains several articles, it uses the body rather than silently keeping only the first card. Scripts, styles, navigation, sidebars, form controls, footers and heading permalink controls are removed. Text inside a form remains available because some catalogs wrap their product listings in a form.

### Select the content you need

Use `contentSelector` to select a specific element, or `excludeSelectors` to remove repeated material inside it:

```json
{
  "startUrls": ["https://docs.python.org/3/tutorial/"],
  "maxPages": 5,
  "maxDepth": 1,
  "contentSelector": "div.body",
  "excludeSelectors": [".admonition"]
}
```

Both settings use CSS selectors. An invalid selector fails before crawling. The content selector must match exactly one element; zero or multiple matches fail at the affected page instead of quietly choosing a header or the first card. Inspect a small run before increasing the page limit. Link discovery uses the page's links independently of content removal.

### Compare a later run

Build `previousHashes` from the final URLs and hashes in your last dataset:

```python
previous_hashes = {row["url"]: row["content_hash"] for row in previous_rows}
next_input = {
    "startUrls": ["https://docs.python.org/3/tutorial/"],
    "maxPages": 5,
    "maxDepth": 1,
    "previousHashes": previous_hashes,
}
```

Supply `next_input` to a later run. Matching hashes produce `unchanged` records with null Markdown/text. Keep your earlier content for those URLs; replace it only when the new record contains content. Retain the same selectors and crawl scope between comparisons, since changing extraction settings also changes hashes.

The Actor stores no shared monitoring state. Your workflow owns the previous hash map and archived content. A page absent from a bounded crawl is not proof of deletion. Check the coverage report before interpreting a missing URL. Hashes compare the observed Markdown, not the site's publication date or legal availability status.

### API example

The example below uses the standard Python library. Set `APIFY_TOKEN` in your environment, then run it to download five pages. Tokens belong in your environment, never in a public Task or shared input file.

```python
import json
import os
from pathlib import Path
from urllib.request import Request, urlopen

actor = "benthepythondev~website-markdown-crawler"
url = f"https://api.apify.com/v2/acts/{actor}/run-sync-get-dataset-items?timeout=300"
payload = {
    "startUrls": ["https://docs.python.org/3/tutorial/"],
    "maxPages": 5,
    "maxDepth": 1,
}
request = Request(
    url,
    data=json.dumps(payload).encode(),
    headers={
        "Authorization": "Bearer " + os.environ["APIFY_TOKEN"],
        "Content-Type": "application/json",
    },
)
with urlopen(request, timeout=330) as response:
    pages = json.load(response)
Path("pages.json").write_text(json.dumps(pages, indent=2), encoding="utf-8")
print(f"Saved {len(pages)} pages")
```

For longer crawls, start an asynchronous Actor run and wait for its terminal status before reading the dataset. Make, n8n and other Apify integrations can run the same input and retrieve the dataset. Save a Task when you want to reuse a tested configuration; creating a Task does not automatically create a recurring schedule.

### Pricing and limits

Free-plan pricing is **$1 per 1,000 exported pages**, plus **$0.00005 per start at 512 MB**. Ten exported pages cost $0.01005 at that tier. Bronze receives 10% off, Silver 15%, and Gold or higher 20% off both charge events. Apify's current Pricing tab is authoritative.

Unchanged pages still require a fetch and comparison, so their metadata records have the same page charge. Failed requests, non-HTML responses, robots-disallowed pages and duplicate URLs create no page result charge. Earlier successful records remain billable when a later page fails. The start charge still applies. The Actor observes Apify's maximum-charge setting and stops exporting when that limit is reached.

There is no paid upstream API or residential proxy requirement. Each page response is limited to 3 MB and extracted Markdown to 200,000 characters. The crawler stops with an error rather than silently truncating content beyond those limits. Discovery holds at most 5,000 distinct URLs in one run; the coverage report marks that ceiling if reached.

### Coverage and failure behavior

The run's `COVERAGE` record lists attempted URLs, their depth, extraction status and errors. It also records the number exported, pending queued pages and why the crawl stopped. `page_limit` means there are discovered URLs left. Reaching a depth limit can also leave parts of a site unexplored, so a completed queue does not establish complete website coverage.

A source/network/extraction error stops the crawl and fails the run while preserving already exported pages and the coverage record. The crawler does not repeatedly retry denied requests. A run with no exported HTML pages fails visibly instead of treating a blocked or empty export as successful.

robots.txt is respected. A missing robots.txt allows normal crawling; an unavailable or redirected robots.txt is treated conservatively. Requests are sequential and at least half a second apart per origin, with longer declared crawl delays respected. Use only websites and material you are entitled to process, and follow their terms and content licenses.

### Common questions

**Does it work on any website?** No. It needs a public page without login or a challenge. Optional rendering supports the same-origin JavaScript subset described above. PDFs, authenticated pages, cross-origin applications and interactive dashboards need a different permitted source or tool.

**Does it discover every page?** No. It follows links under the supplied URL subtrees within the depth/page limits. It does not consult sitemaps or search engines. Unlinked pages require explicit start URLs.

**Why did my crawl stop after a redirect?** The destination left the configured origin or subtree, repeated another queued URL, or exceeded the redirect limit. Inspect `COVERAGE` and use the intended final public URL.

**Does an unchanged record overwrite the old content?** Your integration decides. Retain your previous content when `change_status` is `unchanged`; the null fields mean no replacement content was sent.

**Can it crawl my local network?** No. It accepts public HTTP(S) websites on standard ports. Private and reserved addresses are rejected.

**How do I report a problem?** Open an issue on this Actor with the run link, a public test URL and the field you expected. Never post tokens, cookies or private documents. A small failing example is easier to diagnose than a large crawl.

For a supplied list of single pages, see [Webpage Text Extractor](https://apify.com/benthepythondev/webpage-text-extractor). For URL discovery from XML sitemaps, see [Sitemap URL Extractor](https://apify.com/benthepythondev/sitemap-url-extractor). This crawler adds linked-page Markdown and content comparisons to those narrower workflows.

# Actor input Schema

## `startUrls` (type: `array`):

One to twenty public HTTP(S) URL strings. Follow links only within these origins and path subtrees. Supply the final HTTPS URL when possible.

## `maxPages` (type: `integer`):

Stop after this many queued page attempts, including skipped pages. Exported pages cannot exceed this limit. Default ten.

## `maxDepth` (type: `integer`):

Zero fetches only the supplied URLs; one also follows their links. Applies within each start URL subtree.

## `contentSelector` (type: `string`):

Optional CSS selector matching exactly one content element. Zero or multiple matches fail visibly. Otherwise selects main, a main role, a single article, or body.

## `excludeSelectors` (type: `array`):

Optional selectors to remove within the content, such as .cookie-banner. Navigation, scripts, styles, footers and sidebars are removed by default.

## `previousHashes` (type: `object`):

Optional map of final page URLs to previous content\_hash values. Matching pages still produce a charged metadata row with unchanged status and null Markdown/text. Keep the same extraction settings for comparisons.

## `renderJavaScript` (type: `boolean`):

Run Chromium for public same-origin JavaScript content. Only checked GET resources are supplied; no login, cross-origin resources, forms, media, WebSockets or access challenges. HTTP remains the default.

## `waitForSelector` (type: `string`):

Optional CSS selector to wait up to ten seconds for after page load. Requires JavaScript rendering. Without it, capture one second after load. Use contentSelector separately to choose the exported content.

## Actor input object example

```json
{
  "startUrls": [
    "https://docs.python.org/3/tutorial/"
  ],
  "maxPages": 10,
  "maxDepth": 2,
  "contentSelector": "",
  "excludeSelectors": [],
  "previousHashes": {},
  "renderJavaScript": false,
  "waitForSelector": ""
}
```

# Actor output Schema

## `results` (type: `string`):

No description

## `coverage` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        "https://docs.python.org/3/tutorial/"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("benthepythondev/website-markdown-crawler").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "startUrls": ["https://docs.python.org/3/tutorial/"] }

# Run the Actor and wait for it to finish
run = client.actor("benthepythondev/website-markdown-crawler").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    "https://docs.python.org/3/tutorial/"
  ]
}' |
apify call benthepythondev/website-markdown-crawler --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,benthepythondev/website-markdown-crawler"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/rdZKGQ3uIm0ENbULV/builds/b13LVcjRdL4Nt58tQ/openapi.json