# Website Content Crawler to Markdown for AI & LLMs (`deepmine/website-content-crawler`) Actor

Website content crawler for AI, RAG and LLMs: give start URLs or a sitemap, get every page's clean Markdown and text with title, description, headings, author, dates, canonical URL and links. Fast HTTP crawling, URL globs, depth and page limits, sitemaps, llms.txt and PDFs. No login.

- **URL**: https://apify.com/deepmine/website-content-crawler.md
- **Developed by:** [DeepMine](https://apify.com/deepmine) (community)
- **Categories:** AI, Developer tools, SEO tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.12 / 1,000 pages

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

Website Content Crawler: Markdown & Text for AI. Give it a site, a docs section or a sitemap, and get one row per page with its clean Markdown and text, title, description, headings, dates, canonical URL and links, ready for RAG, LLM knowledge bases, search indexes and content audits.

| 🏷️ Title | 🔗 Page | 🔢 Words | 📑 Headings | 🪜 Depth |
|---|---|---:|---|---:|
| Web scraping basics for JavaScript devs | Academy | Apify Documentation | https://docs.apify.com/academy/scraping-basics-javascript | 981 | # Web scraping basics for JavaScript devs, ## What we'll do | 1 |
| API scraping | Academy | Apify Documentation | https://docs.apify.com/academy/api-scraping | 890 | # API scraping, ## What's an API? | 1 |
| Expert scraping with Apify | Academy | Apify Documentation | https://docs.apify.com/academy/expert-scraping-with-apify | 489 | # Expert scraping with Apify, ## Preparations | 1 |

<sub>Collected 2026-10-23 from the prefilled run (50 pages of the Apify Academy).</sub>

**Price:** $0.15 per 1,000 pages on Starter ($0.18 Free, $0.13 Scale, $0.12 Business). Pages that need JavaScript and are rendered in headless Chrome add $3.09 per 1,000 (`renderedInBrowser: true`); most sites never need it. The prefilled run (50 pages) costs less than $0.01. You pay only for pages saved: URLs that fail, are blocked or repeat a page you already have are free.

### What you get

- **Clean content**: the page's main content as Markdown (headings, lists, tables, code blocks with their language, lines and indentation as the page shows them, links, images) and as plain text. Menus, footers, cookie banners, pop-ups and button labels are removed.
- **Metadata**: title, meta description, author, keywords, language, canonical URL, Open Graph tags, JSON-LD (raw and its schema.org types), published and modified dates, meta robots.
- **Structure and links**: the headings outline, internal and external links, the page each URL was found on, its link depth and the start URL that led to it.
- **Change tracking**: a content hash of the text, the server's Last-Modified, and the sitemap's lastmod, so you can see what changed between runs.

### How it crawls

1. Each start URL crawls itself and its sub-pages (same path), the way most site crawlers do. Add **Include URL patterns** to go wider and **Exclude URL patterns** to skip sections (`**` matches anything, `*` anything but `/`).
2. Turn on **Also crawl the sitemap** to add the pages a site lists in its sitemap (from robots.txt or /sitemap.xml), or give a `sitemap.xml` / `llms.txt` URL as a start URL to crawl exactly what it lists.
3. **Max pages** caps what you pay for, **Max link depth** how far links are followed, **Max URLs to fetch** the requests themselves.
4. Pages are read over fast plain HTTP with a real browser's network fingerprint through Apify Proxy, many at once (the first row is in the dataset within about a second on most sites).
5. **JavaScript pages**: a page whose HTML is an empty app shell (React, Vue, Swagger UI, ...) is rendered in headless Chrome automatically and saved like any other page (`renderedInBrowser: true`). When a site's app loads each page as JSON from an address built from the page's path, the crawler learns that address from 3 rendered pages and reads the site's other app pages straight from it, without the browser (plain rows, not charged as rendered; every 25th page is rendered again as a check). Set **Rendering** to *HTTP only* to never start a browser, or *Browser for every page* to render pages in the browser: the first 3 pages of each site are compared with their HTML, and when the HTML already holds the same text (server-rendered sites), that site's pages are read over HTTP, not charged as rendered, and re-checked every 20th page (with **Save screenshots** on, every page is rendered). For browser-heavy crawls give the run 2-4 GB of memory (more CPU).
6. A page is saved once per run, even when several URLs lead to it (redirects, http and https, tracking parameters, the same canonical URL).
7. Rows come out in crawl order (breadth-first, links in page order), so the same input gives the same pages in the same order, and a page cap keeps the pages closest to your start URL.

**Refresh runs**: set **Only pages changed since** (a date or e.g. `7 days`) to save only pages updated since then, from the sitemap's lastmod, the page's own dates or the server's Last-Modified.

**Files**: turn on **Include files** to save linked PDFs as rows with their text (one Markdown section per PDF page), and Word, Excel, PowerPoint, OpenDocument, RTF, EPUB and CSV files as rows linking the downloaded file (`fileUrl`; CSV and other text files with their text). Or pick the content types in **File types to save** (`saveContentTypes`): `image/*` for images, `application/*, text/csv` for documents, or any list of types with wildcards. Linked files are saved even when they are outside your start URLs.

**Screenshots**: turn on **Save screenshots** for a JPEG of the first screen of every page rendered in the browser (`screenshotUrl`).

### What isn't saved (and is free)

Every URL that doesn't become a row is listed in the run's **⚠️ URLs not saved** dataset with the reason, never charged:

| Reason | What it means |
|---|---|
| `blocked` | The site answered with a bot check or block page (Cloudflare, DataDome, PerimeterX, Akamai, ...) on several fresh IPs. Try the residential proxy in Advanced. |
| `needsJavaScript` | An app page whose content only appears in a browser, with Rendering set to *HTTP only*, or a page with no readable text even after rendering. |
| `httpError`, `networkError` | 404s, 5xx errors, timeouts. |
| `notAPage`, `tooLarge`, `emptyContent` | Files that aren't pages (images, YAML, ...), responses over 15 MB, pages with no text. |
| `robotsTxtDisallowed` | Only with **Respect robots.txt** on. |

### Inputs from other crawlers work as they are

Inputs saved for Apify's Website Content Crawler run here unchanged: `startUrls`, `includeUrlGlobs`, `excludeUrlGlobs`, `maxCrawlDepth`, `maxCrawlPages`, `maxResults`, `useSitemaps`, `useLlmsTxt`, `respectRobotsTxtFile`, `onlyCrawlPagesModifiedSince`, `keepUrlFragments`, `ignoreCanonicalUrl`, `removeElementsCssSelector`, `keepElementsCssSelector`, `htmlTransformer`, `saveMarkdown`, `saveHtml`, `saveHtmlAsFile`, `saveFiles` / `saveContentTypes`, `saveScreenshots`. `crawlerType`: `cheerio` / `jsdom` = HTTP only; the browser types render only the pages that need JavaScript (the same content, much faster). Its nested `crawl.*` and `metadata.*` fields are flat columns here (`loadedUrl`, `depth`, `httpStatusCode`, `canonicalUrl`, `title`, `languageCode`, `openGraph`, `jsonLd`, `responseHeaders`...).

### Output fields

One row per page, in this order: `url`, `title`, `description`, `textSnippet`, `markdown`, `text`, `html` (opt-in), `htmlUrl` (opt-in), `screenshotUrl` (opt-in), `fileUrl` (files), `headings`, `wordCount`, `image` (og:image), `author`, `keywords`, `languageCode`, `publishedAt`, `modifiedAt`, `canonicalUrl`, `loadedUrl`, `requestedUrl`, `referrerUrl`, `depth`, `source` (startUrl, link, sitemap, llmsTxt), `startUrl`, `httpStatusCode`, `contentType`, `contentLengthBytes`, `renderedInBrowser`, `internalLinks`, `externalLinks`, `linksCount`, `ogTitle`, `ogDescription`, `ogType`, `ogSiteName`, `openGraph`, `jsonLdTypes`, `jsonLd`, `metaRobots`, `responseHeaders`, `sitemapLastModifiedAt`, `contentHash`, `loadMs`, `scrapedAt`. Empty values are `null`. Dates are ISO 8601 UTC.

Dataset views: **📊 Overview**, **📄 Content for AI** (URL, title, Markdown), **🔎 SEO** (title, description, canonical, robots, status, Open Graph, JSON-LD types), **🔗 Links**, **📈 Stats** and **📎 Screenshots & Files**. The run's OUTPUT record summarizes pages, skipped URLs by reason and timings.

### FAQ

**Can it crawl sites that need a login?** No. It reads public pages only.

**Does it render JavaScript?** Yes, when a page needs it: pages are read over fast HTTP first, and only app-shell pages are rendered in headless Chrome. Most documentation, blog, news and company sites serve their content in the HTML and never start the browser.

**How fast is it?** On the Apify platform at the default 512 MB: 50 documentation pages in 3.7-4.9 s, 100 heavy blog pages (650 KB each) in 12.5 s, 600 docs pages in 63 s (measured 2026-10-23).

### Your feedback

Found a page it reads badly, or a field you need? Open an issue on the Actor's page. If it saved you time, a review on the Store helps a lot.

# Actor input Schema

## `startUrls` (type: `array`):

Pages to start from. The crawler follows links to their sub-pages (same path), e.g. https://docs.example.com/guide. A sitemap.xml or llms.txt URL crawls every page it lists.

## `maxResults` (type: `integer`):

Most pages to save (each one is charged once). Empty = the whole site.

## `useSitemaps` (type: `boolean`):

Add the pages listed in each site's sitemap (from robots.txt or /sitemap.xml) that fall under your start URLs, to find pages no link points to.

## `maxCrawlDepth` (type: `integer`):

How many links away from a start URL to go: 0 = only the start URLs, 1 = pages they link to, and so on.

## `includeUrlGlobs` (type: `array`):

Also crawl links that match, e.g. https://example.com/blog/\*\* (\*\* = anything, \* = anything but /). Empty = only sub-pages of the start URLs.

## `excludeUrlGlobs` (type: `array`):

Never crawl links that match, e.g. https://example.com/**/tag/** or \*\*?page=\*. Start URLs are always crawled.

## `maxCrawlPages` (type: `integer`):

Most URLs to request, including ones that fail or are skipped, as a safety stop on huge sites. Empty = no limit.

## `useLlmsTxt` (type: `boolean`):

Also read each site's /llms.txt (an index for AI tools) and the pages it links to.

## `onlyCrawlPagesModifiedSince` (type: `string`):

Save only pages updated after this date (from the sitemap, the page's own dates or the server), for refresh runs, e.g. 7 days. Empty = every page.

## `respectRobotsTxtFile` (type: `boolean`):

Skip URLs the site's robots.txt disallows for all crawlers (listed as free robotsTxtDisallowed rows in the errors dataset).

## `includeFiles` (type: `boolean`):

Also save linked documents: PDFs as rows with their text; Word, Excel, PowerPoint, CSV, RTF and EPUB files stored and linked in the fileUrl column (text files with their text). Off = files are skipped.

## `saveContentTypes` (type: `string`):

Save linked files of these content types, comma separated, wildcards allowed (image/\*, text/csv). One row per file with its fileUrl, even when the file is outside your start URLs. Empty = Include files' documents.

## `renderingMode` (type: `string`):

Auto renders only JavaScript app shells in headless Chrome (renderedInBrowser). Browser compares each site's first pages with their HTML and reads the site over HTTP when the text is the same; screenshots render every page.

## `saveScreenshots` (type: `boolean`):

Saves a screenshot (JPEG, top of the page) of each page rendered in the browser, linked in the screenshotUrl column. With Browser for every page, that's every page.

## `saveMarkdown` (type: `boolean`):

The page's main content as Markdown (headings, lists, tables, code, links) in the markdown column.

## `saveHtml` (type: `boolean`):

Adds the html column: the main content's HTML after cleaning. Makes rows much larger.

## `saveHtmlAsFile` (type: `boolean`):

Saves each page's cleaned HTML to the run's key-value store and links it in the htmlUrl column.

## `keepUrlFragments` (type: `boolean`):

Treat example.com/page#a and example.com/page#b as two pages (single-page apps with # routes). Off = one page.

## `ignoreCanonicalUrl` (type: `boolean`):

Off: pages that name the same canonical URL are saved once. On: every URL is its own page (for sites with a wrong canonical tag).

## `htmlTransformer` (type: `string`):

How to pick the content to keep after the elements below are removed.

## `removeElementsCssSelector` (type: `string`):

Elements to drop before extracting, e.g. .ads, #comments. The default removes navigation, footers, scripts, banners and dialogs.

## `keepElementsCssSelector` (type: `string`):

Extract only matching elements, e.g. article, .post-body. Empty = the main content.

## `proxyConfiguration` (type: `object`):

Leave as is: Apify datacenter proxy, fresh IPs on refusals. Pick residential only for sites that block datacenter IPs.

## `crawlerType` (type: `string`):

Kept for inputs saved from other crawlers: cheerio / jsdom = plain HTTP only; browser types work like Rendering: Auto.

## `maxConcurrency` (type: `integer`):

Most requests at once (default: set from the run's memory).

## `requestTimeoutSecs` (type: `integer`):

Seconds per request.

## `maxRequestRetries` (type: `integer`):

Tries on new IPs after a refusal or network error.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://docs.apify.com/academy"
    }
  ],
  "maxResults": 50,
  "useSitemaps": false,
  "maxCrawlDepth": 20,
  "includeUrlGlobs": [],
  "excludeUrlGlobs": [],
  "useLlmsTxt": false,
  "respectRobotsTxtFile": false,
  "includeFiles": false,
  "renderingMode": "auto",
  "saveScreenshots": false,
  "saveMarkdown": true,
  "saveHtml": false,
  "saveHtmlAsFile": false,
  "keepUrlFragments": false,
  "ignoreCanonicalUrl": false,
  "htmlTransformer": "readableText",
  "removeElementsCssSelector": "nav, footer, script, style, noscript, svg, img[src^='data:'],\n[role=\"alert\"],\n[role=\"banner\"],\n[role=\"dialog\"],\n[role=\"alertdialog\"],\n[role=\"region\"][aria-label*=\"skip\" i],\n[aria-modal=\"true\"]",
  "proxyConfiguration": {
    "useApifyProxy": true
  },
  "requestTimeoutSecs": 30,
  "maxRequestRetries": 3
}
```

# Actor output Schema

## `pages` (type: `string`):

No description

## `errors` (type: `string`):

No description

## `summary` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://docs.apify.com/academy"
        }
    ],
    "maxResults": 50,
    "proxyConfiguration": {
        "useApifyProxy": true
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("deepmine/website-content-crawler").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": [{ "url": "https://docs.apify.com/academy" }],
    "maxResults": 50,
    "proxyConfiguration": { "useApifyProxy": True },
}

# Run the Actor and wait for it to finish
run = client.actor("deepmine/website-content-crawler").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://docs.apify.com/academy"
    }
  ],
  "maxResults": 50,
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}' |
apify call deepmine/website-content-crawler --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,deepmine/website-content-crawler"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/Xx5M7NZa3WoiFsJdb/builds/hWEuHyZqlMESCmgS0/openapi.json