# Website to Markdown: LLM-Ready Page Content (`kelp_forest_labs/website-content-markdown`) Actor

Crawl public web pages and convert them to clean Markdown for RAG and AI agents: headings, links, lists and tables kept; scripts, nav and footer elements removed. Honours robots.txt. Failed and empty pages are free.

- **URL**: https://apify.com/kelp_forest_labs/website-content-markdown.md
- **Developed by:** [kelp forest](https://apify.com/kelp_forest_labs) (community)
- **Categories:** AI, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $2.00 / 1,000 page converted (http)s

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

Turn public web pages into **clean Markdown** for RAG and AI agents: headings, links, lists and tables kept;
scripts and `nav`, `footer` and `aside` elements removed (menus and banners built from plain `div`s stay). Crawl from one or more start URLs on the same site, honour robots.txt, and get one row per
page. Failed and empty pages are free.

### Use cases

- **RAG and knowledge bases:** feed LLM-ready Markdown instead of raw HTML.
- **Documentation mirrors:** crawl a docs site to a depth you choose and export Markdown.
- **Content audits:** keep titles, word counts and link counts beside the Markdown body.
- **Agents:** a simple JSON API of `{url, title, markdown}` without running your own crawler.

### What it does

- Starts from each **Start URL**, crawls that site only, up to `maxDepth` and `maxPages`. "Same site" means the same
  host name **and port** (http and https on their default ports count as one site), plus the host a start URL
  redirects to. `www.` and the bare domain are different hosts unless the start URL redirects from one to the other.
- Fetches HTML over **HTTP** (identified user agent) and converts it to Markdown: `h1`–`h6`, paragraphs, lists,
  tables (as Markdown tables), definition lists, links (also around bold text or images), emphasis, code/pre,
  blockquotes, horizontal rules, and images as Markdown image syntax with alt text and absolute src.
- **Dropped:** scripts, styles, `noscript`, `template`, embedded content (`svg` including its text, MathML `math` formulas, `iframe`, `object` including its fallback text, `canvas`), `dialog` (also when open), `nav`, `footer`, `aside`, form controls (buttons, selects, text areas, and `input` fields with their values), every
  `header` that is not inside `article` or `main`, with its headings (site banners and hero blocks, and also section or card headers on pages without `main`; a `header` inside `article` or `main` is kept), and hidden elements (`hidden`, `aria-hidden="true"`,
  inline `display:none` / `visibility:hidden`). Cookie banners and other visible chrome built from plain `div`s are
  **kept**. Text that looks like Markdown syntax is escaped, and text that looks like HTML (for example `<div>`) is
  written as `&lt;div&gt;` so it stays text when the Markdown is rendered. What else is dropped, exactly: links
  with no visible text (icon-only links), and inside headings only the permalink *mark* itself (a link whose visible
  text is empty or just a mark glyph such as `#`, `¶`, `§` or `🔗`; words such as "Link", "Anchor" or "Permalink"
  are text and are kept). Heading text is always kept, also when the heading text is
  itself a link to its own section (`<a href="#id">Title</a>`, named anchors, FAQ toggles); such text is kept without
  the link. A heading link to another page or site keeps its URL (`## [Title](https://…)`). `aria-label` / `title` text is never added to the Markdown. Media with no Markdown form are not output either: `video`, `audio` and `embed` (their fallback text is kept) and `data:` images (only their alt text is kept). Content is never invented, and apart from these and the size limits below nothing else that is visible in the HTML is removed.
- Lists keep their structure: paragraphs, code blocks and nested lists stay inside their item, and numbered lists
  keep their numbers (`<ol start>`). Code blocks are copied **verbatim** (no Markdown added inside, blank lines kept,
  language tag kept, e.g. `python`). A link around a heading (news cards) becomes a heading that links to the story, followed by the rest of the card.
- Decodes each page with its **charset**: a byte-order mark first, then the HTTP header, then `<meta charset>`,
  falling back to UTF-8 and then windows-1252. Charset labels are read with the WHATWG Encoding Standard label table
  that browsers use (for example `x-sjis`, `windows-31j` → Shift_JIS; `gb2312`, `x-gbk` → GBK; `windows-949`,
  `ks_c_5601-1987` → EUC-KR; `x-x-big5` → Big5; `iso-8859-1` → windows-1252; `iso-8859-9` → windows-1254;
  `tis-620` → windows-874), and legacy encodings decode with their superset codec (Shift_JIS as windows-31J, EUC-KR
  as windows-949, Big5 as Big5-HKSCS, EUC-JP including NEC symbols such as ①). The row's `charset` field is the
  WHATWG canonical name (e.g. `UTF-8`, `Shift_JIS`, `EUC-KR`, `GBK`, `windows-1252`); a label outside that table is
  reported with its Python codec name.
- Writes one dataset row per page: `status` `ok` with `markdown`, `title`, `wordCount`; or a free `error` / `skipped`
  row with a reason.
- Duplicates (same final URL from another start or a redirect) are converted and charged once per run. A failed or
  skipped URL reached from several start URLs can give **one free error/skipped row per start URL**.

### robots.txt and our user agent

- User agent: `KelpForestLabs-WebsiteContentMarkdown/0.6 (+https://apify.com/kelp_forest_labs; identified bot, respects robots.txt and Crawl-delay)`.
- robots.txt product tokens we obey, most specific first: **`KelpForestLabs-WebsiteContentMarkdown`**, then
  **`KelpForestLabs`** (all our crawlers), then `*`. Crawl-delay is honoured (up to 10 s).
- robots.txt is checked for **every redirect hop** before it is requested, using the robots.txt of the host the redirect
  points to. A redirect into a disallowed path is not followed and not charged.
- Following RFC 9309, if robots.txt answers 5xx or 429, or cannot be reached (network error, timeout), we assume the
  whole host is disallowed: its pages become free `skipped` rows. A 404 robots.txt means no restrictions.
- Private and reserved addresses are refused on every request and redirect hop.

### Limits on size

- HTML is read up to **10 MB** per page; bytes beyond that are not read. The row then has `htmlTruncated: true` (the
  page is still converted and charged).
- Pages with extreme nesting are flattened past 50 levels, at most 3 million characters of visible page text are read (spaces and HTML
  indentation do not count; if this limit is reached the row has `truncated: true`, `textCapped: true` and
  `originalTextChars`, and the Markdown ends with `…(truncated)`), and a
  page that takes longer than 30 s to convert becomes a free `error` row ("too complex to convert").
- `wordCount` counts space-separated runs, so Chinese or Japanese text without spaces counts as few "words"
  (informational only; billing does not depend on it).
- Markdown is capped at **500,000 characters** per page. Longer output is cut on a paragraph (or line) boundary and ends
  with `…(truncated)`; the row has `truncated: true` and `originalMarkdownChars`. `wordCount` and `markdownChars`
  always describe the Markdown you receive. A truncated page is charged once.

### Pricing (pay per event)

| Event | Price |
|---|---|
| `page` (one HTML page fetched over HTTP and converted to Markdown, with at least one word of text) | US$0.002 |

**Failed and empty pages are free:** HTTP errors, timeouts, DNS failures, robots.txt blocks (including an
unreachable robots.txt), private targets, non-HTML responses, cross-site redirects and pages with **no text in the
HTML** (`skipped`, "needs JavaScript") give a free row. 1,000 converted HTTP pages cost US$2.00.

Each run also has Apify's run-start charge of US$0.00005, once per run for up to 1 GB of memory plus once per extra GB
(US$0.00005 at the default 512 MB), charged when the run starts. You can set a maximum charge per run in the Apify
Console; the actor stops before the first page it could not be paid for.

### How to use

1. Paste one or more **Start URLs**.
2. Set **Max pages** and **Crawl depth** (0 = start pages only). For more than a few hundred pages, raise the run timeout (default 300 s).
3. Leave **Fetch mode** on `http`.
4. Run the actor and open the **Pages** or **Markdown** view, or export as JSON / CSV.
5. Open the `SUMMARY` record for converted / error / skipped counts and `stopReason`.

### Input example

```json
{"startUrls": [{"url": "https://docs.apify.com/"}], "maxPages": 10, "maxDepth": 1, "fetchMode": "http"}
```

### Output example (fixture)

```json
{"url": "https://example.test/about", "status": "ok", "event": "page", "httpStatus": 200,
 "title": "About", "markdown": "# About\n\nWe make kelp.\n", "wordCount": 4, "fetchMode": "http", "depth": 1}
```

### FAQ

**Does it run JavaScript?**
Not in this version. HTTP mode reads the HTML the server returns. A page whose HTML has no text (typical for JavaScript apps) gives a free `skipped` row saying it needs JavaScript. A browser mode is planned; it is not priced or available yet.

**Will you invent missing text?**
No. Markdown only reformats what was in the HTML. Sparse pages stay sparse; pages with no text are free skipped rows.

**Do you follow other sites?**
No. Only the start URL's own site (same host and port) is crawled. External links are counted, not fetched.

**How do I block or slow it down on my site?**
Use the robots.txt product token `KelpForestLabs-WebsiteContentMarkdown` (or `KelpForestLabs` for all our crawlers) with `Disallow` or `Crawl-delay` lines.

### Limits: what this actor does not do

- It does not log in, use cookies, solve captchas, or use proxies to evade blocks.
- It does not fetch private or reserved network addresses (SSRF guard on every hop).
- It does not render JavaScript yet (browser mode planned).
- It does not scrape sites whose robots.txt disallows our crawler, and it does not ignore Crawl-delay.
- It does not return personal data as a product; page content is whatever the site publishes.
- It does not charge for failures or for pages it skips.

### Changelog

- 0.6.3 (October 2026): documentation only, no code change. The README now lists every element the converter
  drops (`noscript`, `template`, `svg`, `math`, `iframe`, `object`, `canvas`, `dialog`, `input` fields, media without a
  Markdown form, and every `header` outside `article`/`main`); the 0.6.2 list was incomplete.
- 0.6.2 (October 2026): documentation only, no code change. The README and Store description now list every element
  the converter drops (`noscript`, `template`, `svg`, `math`, `iframe`, `object`, `canvas`, `dialog`), say plainly that
  menus and banners built from plain `div`s are kept, and mention the 300 s default run timeout.
- 0.6 (October 2026): words such as "Link", "Anchor" or "Permalink" in a heading are never treated as a permalink mark
  (0.5 could drop them); links inside headings keep their URL unless they point to the same page; charset labels follow
  the WHATWG table (`x-sjis`, `x-gbk`, `windows-949`, `x-euc-jp`, `iso-8859-9`, `tis-620` and more) and `charset`
  now reports the WHATWG canonical name (0.5 reported Python codec names such as `cp932`); EUC-JP pages with NEC
  symbols (①) decode cleanly; word counting is inside the per-page time limit.
- 0.5 (October 2026): heading text that is itself a link to its section (self-links, named anchors, FAQ toggles) is
  kept again (0.4 dropped it); only permalink marks are removed; the page-text limit ignores whitespace and is always
  reported as `truncated`; Shift_JIS, EUC-KR and Big5 pages decode with their superset codecs; unclosed bold/italic
  tags end at the next block, so later headings, lists and links keep their structure.
- 0.4 (October 2026): heading permalink icons and other text-less links no longer add their `aria-label`/`title` to
  the Markdown (and never make a page billable); a hidden table row with omitted end tags no longer hides the rest of
  the table; `<ol reversed>` counts down; a heading inside a table cell stays plain text; deep or hostile nesting is
  bounded with a per-page time limit (free error row); `gb2312` pages decode as GBK; clearer messages when a host
  refuses connections or times out.
- 0.3 (October 2026): list items with paragraphs, code or nested lists stay in the list (no empty bullets); `<ol start>`;
  code blocks verbatim; a hidden element with an omitted end tag no longer hides the rest of the page; `wordCount`
  counts text only; text like `<div>` stays text;
  a byte-order mark beats the HTTP charset.
- 0.2 (October 2026): pages with form inputs and article headers keep their content; tables, definition lists and
  block separation; free "needs JavaScript" skip for empty pages; robots.txt on every redirect hop and RFC 9309
  handling of unreachable robots.txt; charset detection; truncation flags; same site = same host and port.
- 0.1 (October 2026): first version — HTTP crawl to Markdown, robots.txt, PPE `page` event; browser mode planned.

# Actor input Schema

## `startUrls` (type: `array`):

Pages to start from. Only the start URL's own site (its host, plus the host it redirects to) is crawled.

## `maxPages` (type: `integer`):

Total page requests in this run (converted or not). Only pages that are fetched and converted are billed, so this also caps your spend.

## `maxDepth` (type: `integer`):

0 = convert the start pages only; 1 = also the pages they link to; and so on.

## `includeQueryUrls` (type: `boolean`):

Off: links such as /search?q=x or /list?page=2 are not followed (they are often endless filter pages).

## `delaySecs` (type: `number`):

Politeness delay per host. A larger robots.txt Crawl-delay (up to 10 s) takes precedence.

## `timeoutSecs` (type: `integer`):

A page that sends nothing for this long, or does not finish within twice this time, is reported as a free error row.

## `fetchMode` (type: `string`):

http (default): fetch HTML over HTTP and convert to Markdown. Browser mode (render JavaScript) is planned and not available in this version.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://docs.apify.com/"
    }
  ],
  "maxPages": 10,
  "maxDepth": 1,
  "includeQueryUrls": false,
  "delaySecs": 0.5,
  "timeoutSecs": 15,
  "fetchMode": "http"
}
```

# Actor output Schema

## `results` (type: `string`):

One row per requested or skipped page: Markdown content for converted pages. Only converted pages (status ok) are billed; error and skipped rows are free.

## `summary` (type: `string`):

Counts of converted, error and skipped pages, fetch mode, and why the run stopped (stopReason).

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://docs.apify.com/"
        }
    ],
    "maxPages": 10,
    "maxDepth": 1
};

// Run the Actor and wait for it to finish
const run = await client.actor("kelp_forest_labs/website-content-markdown").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": [{ "url": "https://docs.apify.com/" }],
    "maxPages": 10,
    "maxDepth": 1,
}

# Run the Actor and wait for it to finish
run = client.actor("kelp_forest_labs/website-content-markdown").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://docs.apify.com/"
    }
  ],
  "maxPages": 10,
  "maxDepth": 1
}' |
apify call kelp_forest_labs/website-content-markdown --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,kelp_forest_labs/website-content-markdown"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/HQqLeLYyuuIv2R3hE/builds/Ai6viSXkLfM755mCV/openapi.json