# Website to Markdown for LLM & RAG: Pages or Whole Sites (`torix33/website-to-markdown`) Actor

Convert web pages or entire websites into clean Markdown for ChatGPT, Claude, RAG and vector databases: main content only (Readability), GFM tables, absolute links, title, author, dates and word count. Crawl a site or use its sitemap. Fast HTTP, no browser. Pay per page.

- **URL**: https://apify.com/torix33/website-to-markdown.md
- **Developed by:** [Torix](https://apify.com/torix33) (community)
- **Categories:** AI, Developer tools, Automation
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $2.10 / 1,000 page scrapeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Website to Markdown for LLM & RAG — Pages or Whole Sites

Convert any web page — or a **whole website, blog or docs portal** — into **clean Markdown** ready for ChatGPT, Claude, LangChain, LlamaIndex and vector databases.

Only the **main content** is kept (Mozilla Readability): no menus, cookie banners, footers or ads. Tables become GFM tables, links and images become absolute URLs, and each page comes with **title, description, author, publish/modify dates, language, canonical URL and word count**.

### How to run it in 3 steps

1. **Add your input.** In **URLs** paste page or site URLs (turn on **Crawl** for whole sites) — e.g. `https://blog.apify.com/what-is-web-scraping/`. The form is already filled with a working example, so you can skip this step for a first try.
2. **Press Start.** The prefilled run returns 1 page in about 2 s and costs about **$0.003**. Set **Max results (cost cap)** to cap the cost of bigger runs; you only pay for pages actually delivered.
3. **Get the data.** Open the **Output** tab and export JSON, CSV or Excel — or run it from the API, Make, Zapier, n8n or an AI agent (see below).

### Features

- **HTTP only, no browser:** plain HTTP with an in-memory frontier — no browser, no request-queue writes. 40 docs pages in ~6 s.
- **Crawl or list:** convert a list of URLs, or crawl a site breadth-first with depth / per-site caps, regex include/exclude, and optional `sitemap.xml` discovery.
- **Two extraction modes:** *Article* (Readability, best for blogs/news/docs) or *Full page* (whole body minus nav/header/footer/sidebars).
- **RAG-friendly output:** one row per page with `markdown` + metadata, so you can chunk and embed straight away.
- **Pay per page:** $3 per 1,000 converted pages. Failed pages, non-HTML files and pages below `minWordCount` are free.

### Use cases

- Build a **knowledge base / RAG index** from documentation or a help center.
- Feed **competitor blogs** or news to an LLM for summarization.
- Create **fine-tuning or evaluation datasets** from web content.
- Archive articles as Markdown for Obsidian / Notion.

### Input

```json
{
  "startUrls": ["https://crawlee.dev/js/docs/quick-start"],
  "crawl": true,
  "maxDepth": 2,
  "maxPagesPerSite": 100,
  "includeUrlPattern": "/docs/",
  "extractionMode": "article",
  "minWordCount": 50
}
```

### Output

#### Real output (default input)

First 2 of 1 records from a run with the prefilled input (2.5 s on a test machine; long values shortened here):

```json
[
  {
    "url": "https://blog.apify.com/what-is-web-scraping/",
    "title": "What is web scraping?",
    "description": "The basics of web scraping: what it is, how it works, real-world use cases, and how to get started.",
    "author": "David Barton",
    "publishedAt": "2024-10-15T14:32:00.000Z",
    "modifiedAt": "2026-10-09T17:13:43.000Z",
    "siteName": "Apify Blog",
    "language": "en",
    "canonicalUrl": "https://blog.apify.com/what-is-web-scraping/",
    "image": "https://storage.ghost.io/c/f2/6e/f26ec999-9a90-4aee-a0d4-9b3ca2bb668f/content/images/2024/04/what-is-web-scraping-complete-guide.png",
    "markdown": "Web scraping is the process of automatically extracting data from a website. You use a program called a web scraper to access a web page, in…",
    "wordCount": 1428,
    "extraction": "article",
    "depth": 0,
    "httpStatus": 200,
    "scrapedAt": "2026-10-15T12:09:14.663Z"
  }
]
```

```json
{
  "url": "https://crawlee.dev/js/docs/introduction/setting-up",
  "title": "Setting up | Crawlee for JavaScript",
  "description": "…",
  "author": null,
  "publishedAt": null,
  "language": "en",
  "markdown": "To run Crawlee on your own computer, you need to meet the following pre-requisites first:\n\n1. Have **Node.js version 16.0** …",
  "wordCount": 407,
  "extraction": "article",
  "depth": 1,
  "httpStatus": 200,
  "scrapedAt": "2026-10-14T12:00:00.000Z"
}
```

### Pricing

- **$3.00 per 1,000 pages** ($0.003 per page).
- The prefilled input returns 1 pages → **$0.003**.
- Example: 1,000 pages → $3.00; 10,000 → $30.00. Set **Max results (cost cap)** to limit any run.

* Failed pages, non-HTML files and pages below `minWordCount` are free.
* Your Apify spending limit is respected exactly: the actor stops charging and finishes cleanly when it is reached. Apify platform usage (compute, proxy) is included in the price.

### Use with AI agents (MCP), LangChain, RAG & the API

This actor is a ready-made **tool for AI agents**: an LLM can call it, wait for the run and read clean JSON back.

**MCP server (Claude Desktop, Claude Code, Cursor, VS Code, any MCP client)** — add Apify's hosted MCP server with this actor enabled as a tool:

```json
{
  "mcpServers": {
    "apify": {
      "url": "https://mcp.apify.com/?tools=torix33/website-to-markdown",
      "headers": { "Authorization": "Bearer <APIFY_TOKEN>" }
    }
  }
}
```

**LangChain / LangGraph** (`pip install langchain-apify`, set `APIFY_API_TOKEN`):

```python
from langchain_apify import ApifyActorsTool, ApifyWrapper
from langchain_core.documents import Document

## 1) give the actor to an agent as a tool
tool = ApifyActorsTool("torix33/website-to-markdown")

## 2) or load results as Documents for a RAG index
loader = ApifyWrapper().call_actor(
    actor_id="torix33/website-to-markdown",
    run_input={"startUrls": ["https://crawlee.dev/js/docs/quick-start"], "crawl": True, "maxDepth": 2, "maxPagesPerSite": 100, "includeUrlPattern": "/docs/", "extractionMode": "article", "minWordCount": 50},
    dataset_mapping_function=lambda item: Document(page_content=item["markdown"], metadata={"url": item.get("url")}),
)
docs = loader.load()
```

**REST API — one synchronous call, dataset items in the response:**

```bash
curl -X POST "https://api.apify.com/v2/acts/torix33~website-to-markdown/run-sync-get-dataset-items?token=<APIFY_TOKEN>" \
  -H "Content-Type: application/json" \
  -d '{"startUrls": ["https://crawlee.dev/js/docs/quick-start"], "crawl": true, "maxDepth": 2, "maxPagesPerSite": 100, "includeUrlPattern": "/docs/", "extractionMode": "article", "minWordCount": 50}'
```

Also works with the Apify Python/JS clients, **Make, Zapier, n8n**, Google Sheets, webhooks and schedules. Export as JSON, CSV, Excel, XML or RSS.

**RAG tip:** Each row is one page with `markdown`, `title` and `url` — split `markdown` by headings, embed, and store `url` for citations. Need ready-made chunks? Use **Website to RAG Chunks**.

### Tips

- Use `maxResults` as a hard cost cap.
- Sites that block cloud servers: enable Apify Proxy in *Proxy configuration*.
- JavaScript-only single-page apps (content rendered in the browser) may return little text — the actor is HTTP-only for speed and price.

### Limitations

- HTTP only: single-page apps that render content in the browser may return little or no text.
- Login-protected pages are not supported; PDFs and other non-HTML files are skipped (free).
- A crawl stops at `maxPagesPerSite` / `maxDepth`; very large sites need several runs or a URL pattern.

### FAQ

**Is it a Firecrawl or Jina Reader alternative?**
Yes, for static and server-rendered sites: it crawls and returns clean Markdown per page at $3 per 1,000 pages, without a browser.

**Can it crawl a whole documentation site?**
Yes. Set `crawl: true`, `maxDepth`, `maxPagesPerSite`, `includeUrlPattern`, and optionally use `sitemap.xml`.

**Does it work on JavaScript-only sites?**
Single-page apps that render everything in the browser may return little text; the actor is HTTP-only for speed and price.

# Actor input Schema

## `startUrls` (type: `array`):

Pages to convert, or start pages of sites to crawl. One per line.

## `crawl` (type: `boolean`):

Follow links on the same site from each start URL (breadth-first).

## `maxDepth` (type: `integer`):

0 = start pages only, 1 = pages linked from them, etc.

## `maxPagesPerSite` (type: `integer`):

Cap per website when crawling.

## `useSitemap` (type: `boolean`):

Add the site's sitemap URLs to the crawl (fast way to cover a whole blog or docs site).

## `includeUrlPattern` (type: `string`):

Only crawl URLs matching this regular expression, e.g. <code>/blog/</code>.

## `excludeUrlPattern` (type: `string`):

Skip URLs matching this regular expression, e.g. <code>/tag/|/page/\d+</code>.

## `extractionMode` (type: `string`):

<b>Article</b>: main content only via Mozilla Readability (best for blogs, news, docs). <b>Full page</b>: whole body minus navigation, header, footer and sidebars.

## `includeImages` (type: `boolean`):

Keep images that have alt text as Markdown images.

## `includeLinks` (type: `boolean`):

Add a <code>links</code> array with every absolute link found on the page.

## `minWordCount` (type: `integer`):

Skip (and don't charge for) pages with fewer words, e.g. tag or listing pages.

## `maxResults` (type: `integer`):

Stop after this many pages in total. <code>0</code> = no limit.

## `maxConcurrency` (type: `integer`):

Pages fetched in parallel.

## `requestTimeoutSecs` (type: `integer`):

Give up on a page after this long.

## `proxyConfiguration` (type: `object`):

Not needed for most sites. Enable Apify Proxy for sites that block cloud servers.

## Actor input object example

```json
{
  "startUrls": [
    "https://blog.apify.com/what-is-web-scraping/"
  ],
  "crawl": false,
  "maxDepth": 2,
  "maxPagesPerSite": 50,
  "useSitemap": false,
  "extractionMode": "article",
  "includeImages": true,
  "includeLinks": false,
  "minWordCount": 0,
  "maxResults": 0,
  "maxConcurrency": 20,
  "requestTimeoutSecs": 30,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}
```

# Actor output Schema

## `pages` (type: `string`):

No description

## `runStats` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        "https://blog.apify.com/what-is-web-scraping/"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("torix33/website-to-markdown").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "startUrls": ["https://blog.apify.com/what-is-web-scraping/"] }

# Run the Actor and wait for it to finish
run = client.actor("torix33/website-to-markdown").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    "https://blog.apify.com/what-is-web-scraping/"
  ]
}' |
apify call torix33/website-to-markdown --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,torix33/website-to-markdown"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/zV4auKoh2gHJlb6jB/builds/wDsJFREQGMjrcCTPl/openapi.json