# Website to Markdown: HTML to LLM-Ready Markdown (`transparent_meteorite/website-to-markdown`) Actor

Convert any web page to clean Markdown for LLMs and RAG with title, headings, links, token count and chunks. Crawl a site or sitemap. No browser, fast.

- **URL**: https://apify.com/transparent_meteorite/website-to-markdown.md
- **Developed by:** [Detroit Web Agent - Open Data](https://apify.com/transparent_meteorite) (community)
- **Categories:** AI, Developer tools
- **Features:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.55 / 1,000 page converted to markdowns

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Website to Markdown: HTML to LLM-Ready Markdown

Website to markdown, done over plain HTTP with no browser: give it page URLs, a sitemap or a whole site and get clean, LLM-ready Markdown back, one dataset row per page. Each row carries the title, description, author, published date, language, the Markdown itself, the heading outline, optional links and images, word count, approximate token count, a content hash for change detection and optional RAG chunks split by heading.

It is built for the jobs people actually search for when they type "html to markdown" or "website to markdown": feeding docs and blogs into ChatGPT, Claude or a vector database, archiving a site as plain text, migrating content to a static site generator, and cutting the token cost of pasting web pages into prompts.

### Run in 30 seconds

1. Open the **Input** tab. The prefilled input starts at the Python library docs, follows links within that section for 30 pages and also returns 500-token chunks.
2. Click **Start**. Thirty pages finish in a few seconds.
3. Open the **Pages** table in the Output tab, or export as JSON, CSV or Excel. The `markdown` field is the converted page.
4. Replace the URLs with your own, or switch on **Discover pages from sitemap.xml** to convert a whole site.

### What does this Actor do?

For every URL it downloads the HTML, finds the main content (article, documentation body or page text), removes menus, footers, cookie banners and scripts, and converts what is left to Markdown with headings, lists, bold and italic, links, code blocks with their language, quotes and GitHub-style tables. Relative links and image paths are turned into absolute ones, so the Markdown still works when you copy it somewhere else.

Because it never starts a browser, it is fast and cheap. A page converts in well under a second of CPU time and the Actor runs many pages at once (16 by default, up to 50). 100 pages normally finish in far less than a minute on a 1 GB run.

| Group | Fields |
|---|---|
| Request | `url`, `finalUrl`, `statusCode`, `status`, `error`, `fetchedAt` |
| Page facts | `title`, `description`, `author`, `publishedDate`, `language`, `siteName`, `canonicalUrl`, `ogImage` |
| Content | `markdown`, `mainContentOnly`, `headings` |
| Optional | `links`, `images`, `chunks` |
| Size | `wordCount`, `approxTokens`, `contentHash` |

`status` is `ok` for converted pages. Pages that cannot be converted still get a row with a clear reason (`blocked`, `not-found`, `http-error`, `non-html`, `empty`, `failed`) and are **not charged**.

### Main content or the whole page

**Main content only** is on by default. The Actor reads the page twice: once with a reader-mode extractor and once by looking for the `main` or `article` element, and keeps whichever holds the real body, so long reference pages do not lose sections. `mainContentOnly` in each row tells you whether a main body was found. If a page has no clear body (a home page, a link list), the Actor falls back to the whole page with scripts and styles removed, and the flag is `false`. Turn the option off to always get the full page.

### RAG chunks

Set **RAG chunk size (tokens)** to a number such as 500 and every row gets a `chunks` array. Chunks follow your headings first, small sections are merged up to the size, and long sections are split on paragraph breaks. Each chunk has `index`, `heading`, `headingPath` (for example `["Install", "Linux"]`), `text` and `tokens`. Embed the chunk text and keep the heading path as metadata for much better retrieval.

### Crawl a site

- **Follow links on the same site** converts pages linked from your start pages, same domain only, down to **Max crawl depth**.
- **Discover pages from sitemap.xml** reads `robots.txt` and `sitemap.xml`, including sitemap indexes, and adds those pages.
- **Only URLs matching** and **Skip URLs matching** take glob patterns such as `https://example.com/docs/**` or `**/tag/**`.
- **Max pages** is a hard stop for both the crawl and your bill.

Pages you list yourself are always fetched. The filters apply to pages found by crawling or sitemaps.

### Limits

- It reads the HTML the server sends. Pages that build their text with JavaScript after loading (some single-page apps) come back as `empty`. Use a browser-based tool for those.
- No login, no cookies by default. You can pass your own headers, for example a session cookie, in **Extra request headers** for pages you are allowed to read.
- Sites that actively block automated traffic return `blocked` rows. No proxy is used.
- PDFs and other non-HTML files return a `non-html` row.
- `approxTokens` is an estimate (characters divided by four), good for budgeting, not exact for any one model.
- Pages are read up to 6 MB.

### Input

| Field | What it does |
|---|---|
| `urls` | Pages to convert, one per line. Also the start pages for crawling. |
| `mainContentOnly` | Keep only the article or documentation body. Default on. |
| `includeLinks` | Add a `links` array with absolute URL, text and an internal flag. |
| `includeImages` | Add an `images` array with absolute URL and alt text. |
| `chunkTokens` | 0 for none, or 50 to 8000 for RAG chunks. |
| `frontMatter` | Start the Markdown with a YAML block (title, source, author, date). |
| `removeSelectors` | Extra CSS selectors to delete before converting. |
| `crawlSameDomain` | Follow links on the same domain. |
| `useSitemap` | Add pages from sitemap.xml. |
| `maxPages` | Stop after this many converted pages. Default 100. |
| `maxCrawlDepth` | Link hops from the start pages. Default 2. |
| `includeGlobs`, `excludeGlobs` | Glob filters for crawled and sitemap URLs. |
| `concurrency` | Pages fetched in parallel. Default 16. |
| `requestTimeoutSecs` | Per-page timeout. Default 20. |
| `extraHeaders` | Extra headers as `Name: value`. |

### Example input

```json
{
  "urls": ["https://docs.python.org/3/library/json.html"],
  "useSitemap": false,
  "crawlSameDomain": true,
  "maxPages": 50,
  "maxCrawlDepth": 1,
  "includeGlobs": ["https://docs.python.org/3/library/**"],
  "chunkTokens": 500,
  "includeLinks": true
}
```

### Output sample

```json
{
  "url": "https://en.wikipedia.org/wiki/Markdown",
  "finalUrl": "https://en.wikipedia.org/wiki/Markdown",
  "statusCode": 200,
  "status": "ok",
  "error": null,
  "fetchedAt": "2026-11-16T09:12:44.120Z",
  "title": "Markdown - Wikipedia",
  "description": null,
  "author": "Contributors to Wikimedia projects",
  "publishedDate": "2005-08-09T19:56:00Z",
  "language": "en",
  "mainContentOnly": true,
  "markdown": "## History\n\nMarkdown was created by John Gruber ...",
  "headings": [{ "level": 2, "text": "History" }],
  "wordCount": 4659,
  "approxTokens": 7200,
  "contentHash": "9f2c...e1",
  "chunks": [{ "index": 0, "heading": "History", "headingPath": ["History"], "text": "## History ...", "tokens": 480 }]
}
```

The values above are an illustration. Your own run returns the real pages.

### Use cases

- Feed documentation, help centres and blogs into a RAG pipeline or vector store with ready-made chunks.
- Give an AI agent clean page text instead of raw HTML and cut token use by a large factor.
- Archive or migrate a site to Markdown for Hugo, Jekyll, Astro, Obsidian or Notion.
- Detect changed pages by comparing `contentHash` between scheduled runs.
- Build a training or evaluation corpus from a list of URLs with title, author, date and language already attached.

### Pricing and cost control

Pay per page converted: **$0.0009 per page on the free plan, $0.000554 on the Gold plan** (Bronze $0.000837, Silver $0.000715), plus a $0.00005 start fee per run. That is **$0.90 per 1,000 pages**. Blocked, missing, empty and non-HTML pages are not charged. Use **Maximum cost per run** or **Max pages** to cap spend; the run stops cleanly at the limit.

### Integrations and scheduling

Run it on a schedule and compare `contentHash` to see what changed, then send results to Google Sheets, Slack, email, Make, Zapier or n8n with the Apify integrations, or call it through the REST API and the Python and JavaScript clients. Send the dataset straight into a vector database with the Apify integrations for Pinecone, Qdrant and similar stores.

### Use with the API, n8n, Make, Zapier or AI agents

**API** (one call, returns the rows as JSON):

```bash
curl -X POST "https://api.apify.com/v2/acts/transparent_meteorite~website-to-markdown/run-sync-get-dataset-items?token=YOUR_APIFY_TOKEN" -H "Content-Type: application/json" -d '{"urls":["https://example.com"],"chunkTokens":500}'
```

**Python**

```python
from apify_client import ApifyClient
client = ApifyClient("YOUR_APIFY_TOKEN")
run = client.actor("transparent_meteorite/website-to-markdown").call(run_input={"urls": ["https://example.com"], "chunkTokens": 500})
for row in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(row["url"], row["wordCount"], row["markdown"][:200])
```

- **n8n, Make, Zapier:** use the Apify app's "Run Actor" step with actor `transparent_meteorite/website-to-markdown`, paste the input JSON, then map `markdown` to the next step.
- **AI agents (MCP):** connect the Apify MCP server at `https://mcp.apify.com?tools=transparent_meteorite/website-to-markdown` to Claude, Cursor or any MCP client, and your agent can turn any URL into Markdown as a tool call.

### Fair use and legal

The Actor requests public web pages the way a browser does and converts the text it receives. Check the site's terms and robots rules before bulk crawling, keep **Parallel requests** modest on small sites, and respect copyright and privacy rules that apply to the content you collect. This is an independent tool.

### FAQ

**Do I need an API key or proxy?** No. It uses plain HTTP and no proxy.

**Does it run JavaScript?** No. That is why it is fast and cheap. Pages that render their text only in the browser return `empty`.

**Why does a page have fewer words than I expected?** With **Main content only** on, menus and sidebars are removed on purpose. Turn it off for the whole page.

**What happens when a URL fails?** You get a row with `status` and `error`, and you are not charged for it.

**Can it convert a whole site?** Yes: switch on the sitemap option or link following, set **Max pages** and optionally glob filters.

**Which languages work?** Any. The language comes from the page and the character set is decoded from the response headers or the page.

**Is the Markdown GitHub compatible?** Yes. Headings use `#`, code uses fenced blocks with a language when the page gives one, and tables use pipe syntax.

### Support

Found a page that converts badly? Open an issue on the Issues tab with the run URL and the page link.

### Related actors

- [Sitemap Link Inventory](https://apify.com/transparent_meteorite/sitemap-link-inventory) to list every URL of a site before you convert it.
- [Website Email Extractor](https://apify.com/transparent_meteorite/website-contact-extractor) for emails and contact details from the same pages.
- [Technical SEO Page Audit](https://apify.com/transparent_meteorite/technical-seo-page-audit) to check titles, headings and meta tags on the pages you convert.

### Changelog

- **0.1**: First private build. URL to Markdown, main-content extraction, headings, links, images, chunks, sitemap and same-domain crawl.

# Actor input Schema

## `urls` (type: `array`):

Pages to convert, one per line. Full links such as https://example.com/docs. With crawling on, these are also the start pages.

## `mainContentOnly` (type: `boolean`):

Remove menus, headers, footers, cookie banners and sidebars and keep the article or documentation body. Turn off to get the whole page.

## `includeLinks` (type: `boolean`):

Add a links array (absolute url, link text, internal flag) for the links inside the content.

## `includeImages` (type: `boolean`):

Add an images array (absolute url and alt text) for the images inside the content.

## `chunkTokens` (type: `integer`):

0 = no chunks. Set 300 to 1000 to also get a chunks array split by heading and size, each chunk with its heading path and token count, ready for a vector store.

## `frontMatter` (type: `boolean`):

Start each markdown with a title, source, author and date block, the format most static site and note tools expect.

## `removeSelectors` (type: `array`):

Optional. CSS selectors removed before conversion, for example .ads or #newsletter-popup.

## `crawlSameDomain` (type: `boolean`):

Also convert pages linked from your start pages, on the same domain only.

## `useSitemap` (type: `boolean`):

Reads robots.txt and sitemap.xml (including sitemap indexes) for each start site and adds those pages.

## `maxPages` (type: `integer`):

Stops after this many pages are converted and charged, including pages found by crawling.

## `maxCrawlDepth` (type: `integer`):

Link hops from a start page when following links is on. 0 = start pages only.

## `includeGlobs` (type: `array`):

Optional glob patterns for crawled and sitemap URLs, for example https://example.com/docs/\*\*. Empty means everything on the site.

## `excludeGlobs` (type: `array`):

Optional glob patterns to skip, for example **/tag/** or \*\*/\*.pdf.

## `concurrency` (type: `integer`):

How many pages are fetched at once. Lower it for small sites that rate limit.

## `requestTimeoutSecs` (type: `integer`):

Give up on a slow page after this long.

## `extraHeaders` (type: `array`):

Optional, one per line as Name: value, for example Cookie: session=abc or Accept-Language: de.

## Actor input object example

```json
{
  "urls": [
    "https://docs.python.org/3/library/index.html"
  ],
  "mainContentOnly": true,
  "includeLinks": false,
  "includeImages": false,
  "chunkTokens": 500,
  "frontMatter": false,
  "crawlSameDomain": true,
  "useSitemap": false,
  "maxPages": 30,
  "maxCrawlDepth": 1,
  "includeGlobs": [
    "https://docs.python.org/3/library/**"
  ],
  "concurrency": 16,
  "requestTimeoutSecs": 20
}
```

# Actor output Schema

## `pages` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "urls": [
        "https://docs.python.org/3/library/index.html"
    ],
    "chunkTokens": 500,
    "crawlSameDomain": true,
    "maxPages": 30,
    "maxCrawlDepth": 1,
    "includeGlobs": [
        "https://docs.python.org/3/library/**"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("transparent_meteorite/website-to-markdown").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "urls": ["https://docs.python.org/3/library/index.html"],
    "chunkTokens": 500,
    "crawlSameDomain": True,
    "maxPages": 30,
    "maxCrawlDepth": 1,
    "includeGlobs": ["https://docs.python.org/3/library/**"],
}

# Run the Actor and wait for it to finish
run = client.actor("transparent_meteorite/website-to-markdown").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "urls": [
    "https://docs.python.org/3/library/index.html"
  ],
  "chunkTokens": 500,
  "crawlSameDomain": true,
  "maxPages": 30,
  "maxCrawlDepth": 1,
  "includeGlobs": [
    "https://docs.python.org/3/library/**"
  ]
}' |
apify call transparent_meteorite/website-to-markdown --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,transparent_meteorite/website-to-markdown"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/gK60K9xT3SC84Yto6/builds/3hWw2C4SfnpcsdgNS/openapi.json
