# Website Content Extractor - Markdown for LLM, RAG, Vector DB (`snow_leo_data/website-to-markdown`) Actor

Readable text from 73.1% of 52 live pages over plain HTTP, including 10 of 10 documentation sites. Website to LLM text and a website to markdown converter in one run: clean Markdown, chunks with overlap for a vector database, 46 flat fields, and what changed since the last run.

- **URL**: https://apify.com/snow\_leo\_data/website-to-markdown.md
- **Developed by:** [Snow Leo Data](https://apify.com/snow_leo_data) (community)
- **Categories:** Integrations, Developer tools
- **Stats:** 3 total users, 2 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

$2.20 / 1,000 pages

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Website Text Extractor for AI, RAG and LLM

Give it a list of links or one address plus a crawl depth. Get back one row per
page: the readable text with the menus and footers gone, the same content as
Markdown, the page metadata, and — if you ask for it — the page already split
into chunks with overlap, ready for a vector database.

No browser. No proxies. No account anywhere. Plain HTTP and the Python standard
library, which is why it is cheap, and also why it has one honest limit — read
the next section before you buy.

### The number you should decide on: 73.1 %

This Actor reads what the server sends. It does not run JavaScript. So there is
a part of the web it cannot read, and here is how big that part measured out.

**52 live pages**, seven kinds of site, fetched from Apify servers
on 2026-09-17 (run `7fpElZkFZAnx8DObw`, the same code that ships here):

| outcome | pages | share | what it means |
|---|---|---|---|
| readable text | **38** | **73.1 %** | 120 words or more. HTTP was enough. |
| refused (401/403) | 6 | 11.5 % | apnews, reuters, etsy, ebay, nestle, tesla |
| empty shell | 3 | 5.8 % | **the JavaScript case — we got nothing** |
| thin (under 120 words) | 2 | 3.8 % | a real page, but mostly headlines |
| network error | 2 | 3.8 % | amazon answered 500, bestbuy dropped the connection |
| disallowed by robots.txt | 1 | 1.9 % | sap.com said no, so we did not download it |

The same measurement from a laptop gave 73.1 % as well, so this is not an
Apify-network effect.

**It depends enormously on what you crawl:**

| kind of site | readable | pages |
|---|---|---|
| documentation | **10 / 10** | docs.python.org, MDN, Django, Kubernetes, React, GitHub, FastAPI, Apify, PostgreSQL, nginx |
| reference | 4 / 4 | Wikipedia, arXiv, GitHub repo, Hacker News |
| blogs | 7 / 8 | Simon Willison, Martin Fowler, Cloudflare, danluu, Paul Graham, Rust, jvns |
| news | 7 / 10 | BBC, Guardian, TechCrunch, Ars Technica, Wired, NPR, CNBC |
| government | 3 / 4 | IRS, gov.uk, European Commission |
| corporate | 4 / 8 | Stripe, IBM, Salesforce, Siemens |
| **online shops** | **3 / 8** | the worst case by a distance |

So: **for documentation sites, knowledge bases, blogs and news this Actor is a
straight fit.** For large retail, for sites behind Cloudflare's stricter modes,
and for single-page apps that render everything client-side, use a
browser-based Actor instead — that is what they are for, and we are not going
to pretend otherwise.

Median text on a page that worked: **1480 words**.

### What one row contains

**46 fields.** Flat — no nested objects to unwrap before a CSV or a BigQuery load.

**Address** — `url`, `final_url` (after redirects), `canonical_url`, `domain`,
`depth`, `redirected`, `dedup_key`, `canonical_key`.

**Response** — `http_status`, `content_type`, `charset`, `response_bytes`,
`response_time_ms`, `error`, `scraped_at`.

**Page card** — `title`, `description`, `author`, `published_at`,
`modified_at`, `language`, `site_name`, `page_type`, `image_url`, `keywords`,
`robots_meta`, `og` (every og: and twitter: tag), `jsonld_types`,
`alternate_languages`, `feeds`.

**The content** — `text`, `markdown`, `word_count`, `char_count`,
`reading_time_min`, and `html` when you switch it on.

**Structure** — `headings` (level and text, in document order),
`heading_count`, `links` (url, anchor text, rel), `link_count`, `images` (url,
alt), `image_count`.

**For RAG** — `chunks` and `chunk_count`.

**For monitoring** — `change_type` (NEW / UPDATED / UNCHANGED) and
`content_hash`.

#### Dates and authors, honestly

Measured on the same 52 pages: an author came back on **28.8 %** and a
publication date on **17.3 %**. That is not a parsing failure — most of those
52 are section fronts and documentation pages, which carry no byline and no
date. Where a page states them (`article:published_time`, JSON-LD,
`parsely-pub-date`, or a `<time datetime>` inside the article) they are read
and normalised to ISO-8601, including Unix-epoch values. What is deliberately
**not** done is guessing: "2 days ago" and "March 3" stay empty, because a
guessed date is worse than a missing one — it looks just as trustworthy.

### Chunking, which is the point for a vector database

Turn on **Split into chunks** and every row gains a `chunks` array. The split
follows the document, not a character counter:

- a heading always starts a new chunk;
- then paragraph boundaries, then sentence ends, and only as a last resort a
  space;
- a fenced code block is never cut while it fits in one chunk;
- `chunkOverlap` repeats the tail of the previous chunk, cut at a sentence
  boundary, so a thought that lands on the seam is still findable whole. It is
  capped at half the chunk size.

Each chunk carries `index`, `text`, `chars`, `words`, `start_char`, `url`,
`title` and **`heading_path`** — the chain of headings above it. That last one
is the difference between storing a nameless paragraph about limits and storing
one you can cite as *Pricing → Enterprise → Limits*.

### Monitor mode: re-index only what changed

Turn on **Monitor mode** and the Actor remembers a fingerprint of each page's
text between runs, in a named key-value store. Every row gets `change_type`:

- `NEW` — never seen before;
- `UPDATED` — the text changed;
- `UNCHANGED` — identical, and by default **not delivered and not charged**.

The fingerprint covers the text only. A view counter, a fresh "related posts"
block or a new ad slot is not a change — otherwise a monitor would report
everything as changed every day, and quietly become a plain re-crawl.

For a scheduled pipeline this is money twice over: you do not pay us for the
unchanged page, and you do not pay your embedding provider to vectorise it
again.

### Crawling a site

Leave **Crawl depth** at 0 to open exactly the addresses you gave. Raise it to
walk a site breadth-first from the entry point: depth 1 adds the links on those
pages, depth 2 one level further.

- **Stay on the starting domain** and **Allow subdomains** keep the crawl from
  wandering off a news site into the whole internet.
- **Only crawl URLs matching** / **Never crawl URLs matching** take wildcard
  patterns such as `https://example.com/docs/*`. They apply to links found
  while crawling, never to the addresses you supplied yourself.
- **Maximum pages** is the budget for the run, counted in requests actually
  sent. **Maximum pages per domain** stops one big site from eating it all.
- **Seed the crawl from sitemap.xml** queues the site's own sitemap. This is
  how you reach a site whose menu is drawn by JavaScript: the links are not in
  the HTML, but the sitemap lists every page anyway.

`rel="nofollow"` links are not followed, and duplicate addresses are collapsed
before you are charged: `?utm_source=...`, a trailing slash, `www.`, http vs
https and a declared `canonical_url` all resolve to one row.

### robots.txt is always obeyed

There is no switch. Before a page is requested, the site's `robots.txt` is
read once per host and matched by the RFC 9309 rules — longest matching rule
wins, `Allow` beats `Disallow` on a tie, `*` and `$` supported. A
`Crawl-delay` is obeyed too.

This is not only the correct thing to do. The disallowed sections of a site are
usually search results, carts and calendars — infinite URL spaces where a crawl
budget burns away for nothing.

A page refused this way still gets a row, with `error: robots-disallowed`, so
you can see which address was skipped and why.

### Empty output always names its reason

A green run with nothing in it is the worst thing that can happen to a buyer.
So the Actor never stays silent about it: the `REPORT` record carries an
`empty_reason` and the log carries the same sentence in words.
Eleven named reasons:

`no-start-urls`, `all-urls-invalid`, `nothing-requested`,
`patterns-exclude-start-urls`, `charge-limit-reached`, `all-pages-refused`,
`all-pages-blocked-by-robots`, `no-text-found`, `filtered-out`,
`nothing-changed`, `nothing-collected`.

The first five are decided from your input alone, and are checked **before a
single byte is downloaded** — you are not charged time for pages that the next
line of code would have thrown away.

### Compared with the three Apify Actors in this niche

Measured from the Apify API on 2026-09-17.

| | this Actor | apify/web-fetch | apify/website-content-crawler | apify/rag-web-browser |
|---|---|---|---|---|
| price | see the store listing | $0.0015 per fetch (Free/Bronze plan) | you pay platform usage | you pay platform usage |
| runs JavaScript | **no** | yes | yes | yes |
| proxies, anti-blocking | **no** | yes | yes | yes |
| Google search built in | no | no | no | **yes** |
| crawls a whole site | yes | **no, single URL** | yes | no |
| chunking for a vector DB | **yes, with overlap and heading path** | no | no | no |
| change detection between runs | **yes** | no | no | no |
| publication date and author fields | **yes** | no | no | no |
| heading outline, link and image lists | **yes** | no | partly | no |
| default memory | **1024 MB** | — | 8192 MB | — |
| output shape | flat, 46 fields | formats you pick | nested (`crawl`, `metadata`) | nested |

**Where they are ahead and we are not going to catch up:** JavaScript
rendering, proxy rotation and anti-blocking, screenshots, LLM summaries, file
downloads, and full CSS-selector control. Those need a browser or a paid
service; this Actor has neither, by design. On the 52-page measurement above
that cost us 3 pages out of 52 to empty shells and 6 to outright refusals.

**Where this Actor is ahead:** chunking, change detection, dates and authors,
a flat schema, named empty reasons, a per-domain budget, and a price you can
calculate before the run instead of a compute bill you find out about after it.

`removeElementsSelector` and `keepElementsSelector` accept simple selectors
only — a tag (`aside`), a class (`.cookie-bar`) or an id (`#sidebar`).
Combinators and pseudo-classes are not supported. A half-implemented selector
engine would fail silently and look like a broken Actor, so it is refused
instead.

### Cost and speed

Measured on Apify, 1024 MB, explicit memory on every run:

- the 52-page measurement run took **76 seconds** for 52 pages across 52
  different hosts, at 512 MB;
- pages are fetched in parallel (default 5 at a time), but requests to the
  **same** host are spaced out, and a `Crawl-delay` in robots.txt is respected.

**Maximum rows** caps what you are charged. It limits the output, not the
crawl: rows removed by a filter do not consume it.

### Notes for the fine print

- Non-HTML responses (PDF, images, archives) are recorded with their
  `content_type` and an `error` of `not-text`. They are never parsed as text.
- A `200` response with an empty body is treated as a **failure**, not a
  success — that is what a polite refusal looks like, and a buyer should not
  pay for a row with no words in it.
- Character encoding is taken from the header, then the meta tag, then tried as
  UTF-8 and cp1252. Expired or self-signed certificates do not stop a read: you
  supplied the address, and nothing is sent to it.
- Pages that fail still produce a row, with the status and the error, so a list
  of 1000 addresses comes back with 1000 answers. That row is a row like any
  other, so it is charged like any other — said plainly rather than buried.
  Switch on **Drop pages that did not open** and they are neither delivered nor
  charged. On the 52-page measurement the failures were 15.4 % of the list.

# Actor input Schema

## `startUrls` (type: `array`):

Pages to turn into clean text and Markdown. Paste a list of links, or one address plus a crawl depth below to walk a whole site. A bare domain such as example.com works too.

## `maxItems` (type: `integer`):

Hard cap on delivered rows, and on what you are charged. It limits the OUTPUT, not the crawl: filtered rows do not eat the budget. 0 means no cap.

## `outputFormat` (type: `string`):

Markdown keeps headings, lists, tables and code blocks - the structure a RAG pipeline chunks on. Plain text is smaller. Both is the default.

## `crawlDepth` (type: `integer`):

0 opens only the addresses you gave. 1 also opens the links found on them, 2 goes one level further, and so on. Every crawl obeys the site's robots.txt.

## `maxPages` (type: `integer`):

Budget for the whole run, counted in requests actually sent. Protects you from a site with an infinite calendar. 0 means no budget.

## `maxPagesPerDomain` (type: `integer`):

Keeps one large site from eating the whole budget when you crawl several. 0 means no per-domain limit.

## `stayOnDomain` (type: `boolean`):

Off lets the crawl follow links anywhere, which on a news site means the whole internet. Keep it on unless you know why not.

## `includeSubdomains` (type: `boolean`):

Treats docs.example.com and example.com as the same site. Most documentation lives on a subdomain.

## `includeUrlPatterns` (type: `array`):

Wildcard patterns such as https://example.com/docs/\* . Applied to links found while crawling, never to the URLs you gave yourself.

## `excludeUrlPatterns` (type: `array`):

Wildcard patterns such as */tag/* or */page/* . Applied to links found while crawling.

## `useSitemap` (type: `boolean`):

Reads the site's own sitemap and queues the addresses in it. This is how you reach a site whose menu is drawn by JavaScript: the links are not in the HTML, but the sitemap lists every page. Only used when crawl depth is above 0.

## `chunkText` (type: `boolean`):

Adds a `chunks` field: the page cut on heading, paragraph and sentence boundaries, never mid-word and never inside a code block. Each chunk carries its heading path, so a match can be cited.

## `chunkSize` (type: `integer`):

Target size of one chunk in characters. Roughly four characters per token, so 1200 is about 300 tokens.

## `chunkOverlap` (type: `integer`):

How much of the previous chunk is repeated at the start of the next one, so a thought split across the boundary is still findable whole. Capped at half the chunk size.

## `keepNavigation` (type: `boolean`):

Off (the default) removes navigation, footers, cookie banners and 'related articles' - they repeat on every page of a site and pollute a vector search. On returns the whole page body.

## `keepLinksInMarkdown` (type: `boolean`):

Off strips [text](url) down to the text. The `links` field still lists every link either way.

## `keepImagesInMarkdown` (type: `boolean`):

Off strips ![alt](url) out of the Markdown. The `images` field still lists them.

## `compactOutput` (type: `boolean`):

Drops link lists, image lists, og tags and heading outlines, and removes empty fields. Keeps the row small enough for a model context window.

## `includeHtml` (type: `boolean`):

Adds an `html` field with the page source as downloaded. Rows get much larger, so leave it off unless you post-process the HTML yourself.

## `removeElementsSelector` (type: `array`):

Simple selectors only: a tag name (aside), a class (.cookie-bar) or an id (#sidebar). Combinators, attributes and pseudo-classes are not supported - they would fail silently, so they are refused instead.

## `keepElementsSelector` (type: `array`):

Same simple selectors. When set, the text is taken from these elements only and the automatic article detection is switched off - your choice wins over ours.

## `customHttpHeaders` (type: `object`):

Sent with every request, for example an Accept-Language or an API key your own site expects. Measured on bbb.org: an extra Accept-Language turns a working request into a 403, so add headers only when you know the site wants them.

## `minWordsPerPage` (type: `integer`):

Drops pages with less text than this. Use it to throw away the JavaScript-only pages this Actor cannot read: measured from Apify servers, 3 of 52 live pages came back as an empty shell. 0 keeps everything.

## `skipFailedPages` (type: `boolean`):

By default a page that answered 403 or timed out still gets a row, so you can see which address failed and why. Turn this on to leave them out entirely.

## `onlyLanguages` (type: `array`):

Two-letter codes such as en, de, fr, taken from the page's own language tag. Pages that declare no language are kept: missing data is not a mismatch.

## `urlMustContain` (type: `array`):

Keep only rows whose final URL contains one of these strings.

## `urlMustNotContain` (type: `array`):

Drop rows whose final URL contains any of these strings.

## `onlyChanged` (type: `boolean`):

Remembers a fingerprint of each page's text between runs and marks every row NEW, UPDATED or UNCHANGED. Unchanged pages are not delivered and not charged, so a scheduled run re-indexes only the delta in your vector database.

## `emitUnchanged` (type: `boolean`):

Keeps unchanged rows in the output, still labelled UNCHANGED. Useful for a full re-export while keeping the labels.

## `monitorStoreName` (type: `string`):

Name of the storage holding the fingerprints. Give separate pipelines separate names so they do not share memory.

## `concurrency` (type: `integer`):

How many pages are opened at once. Requests to the same host are spaced out regardless, and a Crawl-delay in robots.txt is obeyed.

## `requestTimeoutSecs` (type: `integer`):

How long to wait for one page before giving up on it.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://docs.apify.com/platform/actors"
    },
    {
      "url": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation"
    }
  ],
  "maxItems": 100,
  "outputFormat": "both",
  "crawlDepth": 0,
  "maxPages": 50,
  "maxPagesPerDomain": 0,
  "stayOnDomain": true,
  "includeSubdomains": true,
  "includeUrlPatterns": [],
  "excludeUrlPatterns": [],
  "useSitemap": false,
  "chunkText": false,
  "chunkSize": 1200,
  "chunkOverlap": 150,
  "keepNavigation": false,
  "keepLinksInMarkdown": true,
  "keepImagesInMarkdown": true,
  "compactOutput": false,
  "includeHtml": false,
  "removeElementsSelector": [],
  "keepElementsSelector": [],
  "customHttpHeaders": {},
  "minWordsPerPage": 0,
  "skipFailedPages": false,
  "onlyLanguages": [],
  "urlMustContain": [],
  "urlMustNotContain": [],
  "onlyChanged": false,
  "emitUnchanged": false,
  "monitorStoreName": "",
  "concurrency": 5,
  "requestTimeoutSecs": 25
}
```

# Actor output Schema

## `results` (type: `string`):

All collected rows

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://docs.apify.com/platform/actors"
        },
        {
            "url": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation"
        }
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("snow_leo_data/website-to-markdown").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "startUrls": [
        { "url": "https://docs.apify.com/platform/actors" },
        { "url": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation" },
    ] }

# Run the Actor and wait for it to finish
run = client.actor("snow_leo_data/website-to-markdown").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://docs.apify.com/platform/actors"
    },
    {
      "url": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation"
    }
  ]
}' |
apify call snow_leo_data/website-to-markdown --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,snow_leo_data/website-to-markdown"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/JaHH75uHuhrLNpHme/builds/2Jjc5Wbb2sovHxdgL/openapi.json