# Website to Markdown - Content Extractor for LLM & RAG (`teddythebuilder/website-to-markdown`) Actor

Convert web pages or whole sites to clean Markdown and text for LLMs, RAG and AI agents. Main content without menus or ads, plus title, headings, language, date and optional RAG chunks. Fast HTTP, no browser, respects robots.txt. Export JSON/CSV or call via API/MCP.

- **URL**: https://apify.com/teddythebuilder/website-to-markdown.md
- **Developed by:** [Teddy Wasserman](https://apify.com/teddythebuilder) (community)
- **Categories:** AI, Developer tools, SEO tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.83 / 1,000 page extracteds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

### What does Website to Markdown do?

Website to Markdown is a fast **web page to Markdown converter and website content extractor** for LLMs, RAG pipelines and AI agents. Give it a list of URLs, or one start URL to crawl, and it returns the **main content of every page as clean Markdown** (and optionally plain text): no menus, cookie banners, footers or ads. Each page also comes with its title, meta description, site name, language, published date, H1–H3 headings, word count and, if you want, ready-made **RAG chunks**.

It uses plain HTTP requests and the open-source trafilatura extraction engine, with no headless browser. That makes it cheap and quick: a 10-page docs crawl finishes in about 20 seconds. robots.txt is always respected.

### Why use Website to Markdown?

- **LLM-ready output.** Markdown keeps headings, lists, tables and code blocks, which is the format language models and embedding pipelines handle best. Boilerplate is removed so you don't pay tokens for navigation menus.
- **Bulk URL lists or crawling.** Paste 5,000 URLs with crawl depth 0, or give one docs or blog URL and let it follow links within the same folder.
- **RAG chunks built in.** Set `chunkSize` (e.g. 1,500 characters) and `chunkOverlap` and every page gets a `chunks` array split on paragraph boundaries, ready for your vector database.
- **Cheap and fast.** No browser means low compute cost and a low price per page. You pay only for pages that returned content. Errors, 404s, robots-blocked and empty pages are free.
- **Polite by default.** robots.txt rules and crawl-delay are honoured, concurrency is capped, and the user agent identifies the actor.
- **Stable schema.** The same fields for every site, ready for JSON, CSV, Excel, a webhook or the Apify API.

### What data can you extract?

| Field | Example | Description |
|---|---|---|
| `url` | `https://docs.python.org/3/tutorial/classes.html` | The requested URL |
| `loadedUrl` | `https://docs.python.org/3/tutorial/classes.html` | Final URL after redirects |
| `statusCode` | `200` | HTTP status |
| `crawlDepth` | `1` | 0 = start URL, 1 = linked from a start URL, … |
| `title` | `9. Classes` | Page title |
| `description` | `Classes provide a means of bundling data…` | Meta / Open Graph description |
| `siteName` | `Python documentation` | Site name from metadata |
| `language` | `en` | `<html lang>` value |
| `publishedDate` | `2024-10-15` | Publication or last-modified date when the page exposes one |
| `headings` | `[{"level": 1, "text": "9. Classes"}, …]` | H1–H3 outline (navigation and footer headings removed) |
| `markdown` | `# 9. Classes\n\nClasses provide…` | Main content as Markdown |
| `text` | `9. Classes Classes provide…` | Plain text (optional, `includePlainText`) |
| `wordCount` | `5456` | Words in the extracted Markdown |
| `chunks` | `[{"index": 0, "text": "…", "chars": 1480}, …]` | RAG chunks (optional, `chunkSize`) |
| `links` | `["https://docs.python.org/3/library/…", …]` | All absolute links on the page (optional, `includeLinks`) |
| `warning` | `Very little text: the page may be rendered by JavaScript` | Hints about thin pages |
| `error` | `http_404`, `blocked_by_robots_txt`, `timeout` | Why a page returned no content (free row) |
| `scrapedAt` | `2026-10-21T19:30:00Z` | Extraction time (UTC) |

### How to use Website to Markdown (step by step)

1. Open the actor and click **Try for free**.
2. In **Start URLs**, paste the pages you want, one per line. You can also upload a text file or a Google Sheet of URLs.
3. Choose **Crawl depth**: `0` converts only the URLs you gave (bulk mode); `1` or more also follows links found on them, staying within the same site and folder.
4. Set **Max pages** to cap the run (and the cost).
5. Optional: set **RAG chunk size**, switch on **plain text**, **links** or **images**, or add include/exclude URL patterns.
6. Click **Start**. When the run finishes, open the **Pages** view, or download the dataset as JSON, CSV, Excel or HTML.

### Input example

```json
{
  "startUrls": [
    { "url": "https://docs.python.org/3/tutorial/index.html" }
  ],
  "crawlDepth": 1,
  "maxPages": 50,
  "contentMode": "main",
  "chunkSize": 1500,
  "chunkOverlap": 150,
  "excludeUrlPatterns": ["/genindex", "?highlight="]
}
```

Bulk mode for a URL list (no crawling):

```json
{
  "startUrls": [
    { "url": "https://blog.apify.com/what-is-web-scraping/" },
    { "url": "https://www.gov.uk/browse/tax" }
  ],
  "crawlDepth": 0,
  "maxPages": 1000,
  "includePlainText": true
}
```

### Output example

```json
[
  {
    "url": "https://blog.apify.com/what-is-web-scraping/",
    "loadedUrl": "https://blog.apify.com/what-is-web-scraping/",
    "statusCode": 200,
    "crawlDepth": 0,
    "title": "What is web scraping? Complete guide to web data extraction",
    "description": "Learn what web scraping is, how it works and what it is used for.",
    "siteName": "Apify Blog",
    "language": "en",
    "publishedDate": "2024-10-15",
    "headings": [
      { "level": 1, "text": "What is web scraping?" },
      { "level": 2, "text": "How does web scraping work?" }
    ],
    "markdown": "Web scraping is the process of automatically extracting data from a website. You use a program called a web scraper…",
    "wordCount": 1455,
    "scrapedAt": "2026-10-21T19:31:02Z"
  },
  {
    "url": "https://www.python.org/nonexistent-page",
    "loadedUrl": null,
    "statusCode": 404,
    "crawlDepth": 0,
    "error": "http_404",
    "scrapedAt": "2026-10-21T19:31:01Z"
  }
]
```

The second row is an error row. It is **not charged**. Every dataset can be exported to CSV, Excel, JSON, XML or HTML from the run page or through the API.

### How much does it cost?

Website to Markdown uses pay-per-event pricing: you pay only for pages that returned content.

| Event | Price (Free plan) | When it is charged |
|---|---|---|
| Actor start | $0.0005 | Once per run |
| Page extracted | $0.0015 ($1.50 per 1,000) | One page converted to Markdown |

Failed pages (404, 5xx, timeouts, DNS errors), pages blocked by robots.txt, non-HTML files and pages without any text are **free**. Paid Apify plans get Store discounts on the per-page price (Bronze −15%, Silver −30%, Gold and up −45%).

Worked examples (Free plan prices):

- **A 200-page documentation site** for a RAG chatbot: 200 × $0.0015 + $0.0005 ≈ **$0.30**.
- **10,000 URLs from a spreadsheet** (bulk mode, depth 0): ≈ **$15**.
- **A weekly refresh of a 500-page knowledge base**: ≈ $0.75 per run, **about $3 per month**.
- **An AI agent fetching 3 pages per question**, 100 questions a day: ≈ $0.45 per day.

The Apify Free plan includes monthly platform credit, which covers many runs like these.

### Use cases

- **RAG and chatbots.** Turn product docs, help centres or policy pages into chunks for a vector database (Pinecone, Qdrant, Weaviate, pgvector) and keep them fresh with a scheduled run.
- **LLM context for AI agents.** Give an agent a cheap tool that returns the readable content of any URL instead of raw HTML full of scripts.
- **Content analysis and SEO.** Pull the text, headings and word counts of competitor pages or your own blog to compare topics, length and structure.
- **Dataset building.** Collect clean article text for classification, summarisation or fine-tuning datasets (check each site's terms for reuse).
- **Monitoring.** Schedule a daily run on a set of pages and compare the Markdown between runs to spot changes in prices, terms or documentation.
- **Migration.** Export the content of an old website as Markdown before moving it to a new CMS or static site generator.

### Use with AI agents, MCP and the API

Run it from any language with one HTTP call (replace `<TOKEN>` with your Apify API token):

```bash
curl -X POST "https://api.apify.com/v2/acts/teddythebuilder~website-to-markdown/run-sync-get-dataset-items?token=<TOKEN>" \
  -H "Content-Type: application/json" \
  -d '{"startUrls":[{"url":"https://docs.python.org/3/tutorial/classes.html"}],"crawlDepth":0}'
```

Python with the `apify-client` package:

```python
from apify_client import ApifyClient

client = ApifyClient("<TOKEN>")
run = client.actor("teddythebuilder/website-to-markdown").call(run_input={
    "startUrls": [{"url": "https://docs.python.org/3/tutorial/"}],
    "crawlDepth": 1,
    "maxPages": 30,
    "chunkSize": 1500,
})
for page in client.dataset(run["defaultDatasetId"]).iterate_items():
    for chunk in page.get("chunks", []):
        print(page["url"], chunk["index"], chunk["chars"])
```

**MCP:** the actor works as a tool in the Apify MCP server (`https://mcp.apify.com`), so Claude, ChatGPT, Cursor and other MCP clients can call it to read web pages as Markdown. Agents get short, clean content, which saves tokens compared with raw HTML.

### Integrations

Connect results to Make, Zapier, n8n, LangChain, LlamaIndex, Google Sheets, Airtable, Slack or your own webhook. Apify's integrations run when a run finishes, so a scheduled crawl can refresh a vector database or a spreadsheet automatically.

### Data source and legality

The actor downloads public web pages over HTTP, like a browser without JavaScript, and extracts the readable content with trafilatura (Apache-2.0 licence). It reads **robots.txt before fetching any page** and skips URLs that the site disallows; crawl-delay rules are followed. It does not log in, does not bypass paywalls, CAPTCHAs or bot protection, and does not use proxies to hide.

You are responsible for how you use the content: copyright and site terms still apply to the text you extract, and pages can contain personal data. Use it for analysis, search and AI context on content you are allowed to process.

### FAQ

**Does it render JavaScript?**
No. It is a fast HTTP extractor. Server-rendered sites, blogs, news, documentation, e-commerce product pages and most CMS sites work well. Pure single-page apps that show "enable JavaScript" return little text; those rows carry a `warning`, and pages with no text at all are free.

**How is "main content" chosen?**
trafilatura scores the HTML blocks of the page and keeps the article body, dropping navigation, sidebars, footers, comments and cookie notices. Use `contentMode: "fullPage"` to keep more (lists, sidebars) or `"precise"` for the least boilerplate.

**Which pages does the crawler follow?**
Only links on the same site (www and non-www count as the same), and by default only under the start URL's folder. Add `includeUrlPatterns` / `excludeUrlPatterns` to fine-tune, or `useSitemap` to queue URLs from sitemap.xml.

**How big can a run be?**
Set `maxPages` up to 100,000. The crawler stops before the run timeout and keeps everything it extracted. Raise the timeout and memory for very large sites.

**Do I pay for errors?**
No. 404s, timeouts, robots-blocked URLs, PDFs and other non-HTML files, and pages without text are returned as free rows with an `error` field.

**Can it read PDFs?**
Not yet; PDF links are skipped. A dedicated PDF text extractor is planned.

**Can I get the raw HTML?**
No. The actor is built for clean text. If you need HTML, use a general-purpose crawler.

**I found a page that extracts badly.**
Open an issue with the URL. We reply within 24 hours. If the actor works well for you, a rating on the Store page helps other users find it.

### Related actors

- [Website SEO Audit Crawler & Broken Link Checker](https://apify.com/teddythebuilder/website-seo-audit): titles, meta, canonical, noindex, broken links and sitemap coverage for a whole site.
- [Tech Stack Detector](https://apify.com/teddythebuilder/tech-stack-detector): which CMS, analytics and e-commerce tools a website uses.
- [Domain WHOIS & DNS Lookup API](https://apify.com/teddythebuilder/domain-whois-dns-api): registration, DNS, email security and SSL data for domains.
- [Hacker News Scraper](https://apify.com/teddythebuilder/hacker-news-scraper): HN stories by keyword and "Who is hiring?" jobs.

### Changelog and support

- **1.0 (October 2026):** first release. Bulk URL and crawl modes, main-content Markdown and text, metadata and headings, RAG chunks, sitemap seeding, robots.txt and crawl-delay support, free error rows.

Questions or feature requests? Open an issue on the actor page. We reply within 24 hours.

# Actor input Schema

## `startUrls` (type: `array`):

Web pages to convert, e.g. https://docs.python.org/3/tutorial/. With crawl depth 0 only these pages are returned; with depth >= 1 links found on them are followed.

## `crawlDepth` (type: `integer`):

How many link levels to follow from each start URL. 0 = only the start URLs (bulk URL list mode), 1 = also the pages they link to.

## `maxPages` (type: `integer`):

Stop after this many pages have been extracted (you pay only for extracted pages).

## `contentMode` (type: `string`):

main = article/main content without menus, footers and cookie banners (best for LLMs); precise = stricter, less boilerplate; fullPage = keep more of the page (lists, sidebars).

## `chunkSize` (type: `integer`):

If > 0, each page also gets a 'chunks' array split on paragraph boundaries, e.g. 1500 for embeddings. 0 = no chunks.

## `chunkOverlap` (type: `integer`):

Characters carried over from the previous chunk, e.g. 150. At most half the chunk size.

## `useSitemap` (type: `boolean`):

Also queue the site's sitemap URLs (within the start URL's path), useful to convert a whole docs site.

## `stayWithinPath` (type: `boolean`):

Follow only links under the start URL's folder, e.g. /docs/... for https://example.com/docs/intro. Links to other domains are never followed.

## `includeUrlPatterns` (type: `array`):

Follow/return only URLs that contain one of these strings or match a glob, e.g. /blog/ or https://example.com/docs/\*.

## `excludeUrlPatterns` (type: `array`):

Skip URLs that contain one of these strings or match a glob, e.g. /tag/ or ?page=.

## `maxConcurrency` (type: `integer`):

Parallel page downloads (robots.txt crawl-delay is always honoured).

## `includePlainText` (type: `boolean`):

Add a 'text' field without Markdown formatting.

## `keepLinksInMarkdown` (type: `boolean`):

Write links as [text](url) in the Markdown instead of plain text.

## `keepImages` (type: `boolean`):

Keep images as ![alt](src) in the Markdown.

## `includeLinks` (type: `boolean`):

Add a 'links' array with every absolute link found on the page (max 1,000).

## `minWordCount` (type: `integer`):

Pages with fewer extracted words are returned as free 'no_main_content' rows instead of charged results, e.g. 50.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://docs.python.org/3/tutorial/index.html"
    }
  ],
  "crawlDepth": 1,
  "maxPages": 10,
  "contentMode": "main",
  "chunkSize": 0,
  "chunkOverlap": 0,
  "useSitemap": false,
  "stayWithinPath": true,
  "includeUrlPatterns": [],
  "excludeUrlPatterns": [],
  "maxConcurrency": 5,
  "includePlainText": false,
  "keepLinksInMarkdown": false,
  "keepImages": false,
  "includeLinks": false,
  "minWordCount": 0
}
```

# Actor output Schema

## `results` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://docs.python.org/3/tutorial/index.html"
        }
    ],
    "crawlDepth": 1,
    "maxPages": 10
};

// Run the Actor and wait for it to finish
const run = await client.actor("teddythebuilder/website-to-markdown").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": [{ "url": "https://docs.python.org/3/tutorial/index.html" }],
    "crawlDepth": 1,
    "maxPages": 10,
}

# Run the Actor and wait for it to finish
run = client.actor("teddythebuilder/website-to-markdown").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://docs.python.org/3/tutorial/index.html"
    }
  ],
  "crawlDepth": 1,
  "maxPages": 10
}' |
apify call teddythebuilder/website-to-markdown --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,teddythebuilder/website-to-markdown"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/kV5oAbD3r3HGs9pzE/builds/DihZxMfahxLgYEeca/openapi.json