# Website to Markdown Crawler (`juliob.rangel/website-to-markdown-crawler`) Actor

Crawl websites and turn every page into clean Markdown for LLMs, RAG and AI agents. Removes menus, footers, cookie banners and images. Fast HTTP crawler, pay only per page.

- **URL**: https://apify.com/juliob.rangel/website-to-markdown-crawler.md
- **Developed by:** [Julio Barcellos Rangel](https://apify.com/juliob.rangel) (community)
- **Categories:** AI, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.70 / 1,000 page crawleds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Website to Markdown Crawler (for AI, RAG and LLMs)

Website to Markdown Crawler crawls any website and returns clean Markdown of every page, without menus, footers, cookie banners or scripts, plus the title, description, language and word count. It is a website to Markdown converter for developers and AI teams who feed ChatGPT, Claude, RAG pipelines, vector databases and AI agents. Use it to turn a docs site, help center or blog into LLM-ready text, or to convert a whole website to Markdown in one run. Fast: plain HTTP, no browser.

### What does Website to Markdown Crawler do?

You give it one or more start URLs. The crawler downloads each page, follows its links up to the depth and page limits you set, cleans the HTML and converts the main content to Markdown. Every page becomes one row with its URL, metadata and Markdown.

Start URLs can be:

| Start URL | What gets crawled |
|---|---|
| A home page, e.g. `https://example.com` | The whole website, up to your limits |
| A section, e.g. `https://docs.apify.com/academy/` | That section and the pages it links to (turn on "Stay under the start path" to keep it inside the section) |
| A single page with `maxDepth` 0 | Only that page |

URLs without `https://` get it added. Several websites in one run are fine, and each one gets its own page limit.

What the cleaning does:

- **Removed:** scripts, styles, iframes, SVG and canvas, forms and buttons, `nav`, `header`, `footer` and `aside` blocks, navigation, banner and footer roles, hidden elements, common cookie banner and newsletter boxes, screen-reader-only text, and all images, video and audio.
- **Main content first:** if the page has a `main`, `article`, `[role="main"]`, `#content` or `.content` element with more than 200 characters of text, only that part is converted. Otherwise the cleaned page body is used.
- **Kept:** headings (`#` style), paragraphs, lists (`-` bullets), bold and italic text, fenced code blocks, data tables (tables with header cells become Markdown tables) and links, which are made absolute. Layout tables without header cells are flattened into plain text blocks.

### What data can you extract?

| Field | What it means | Example |
|---|---|---|
| `url` | Final page URL (after redirects) | `https://docs.apify.com/academy` |
| `title` | Page `<title>`, or the first H1 when there is no title | `Apify Academy \| Academy \| Apify Documentation` |
| `description` | Meta description (or `og:description`) | `Learn everything about web scraping and automation…` |
| `language` | Language from the page's `lang` attribute | `en` |
| `canonicalUrl` | Canonical link as written in the page | `https://docs.apify.com/academy` |
| `depth` | Link depth from the start URL (0 = start page) | `0` |
| `statusCode` | HTTP status code of the page | `200` |
| `wordCount` | Number of words in the Markdown | `204` |
| `markdown` | Clean Markdown of the page's main content | `# Apify Academy\n\nLearn everything about…` |
| `crawledAt` | When the page was crawled | `2026-10-11T20:23:30.322Z` |
| `error` | Only on the error row when no page at all could be loaded | `No pages could be loaded` |

Fields that a page does not have (for example no meta description) come as `null`.

### How to use Website to Markdown Crawler

1. Open Website to Markdown Crawler in Apify Console (sign in or create an Apify account).
2. In **Start URLs**, add one or more websites or pages, one per line.
3. Set **Max pages per website** and **Max link depth**. Turn on **Stay under the start path** to crawl only one section, such as `/docs/`.
4. Optionally add CSS selectors in **Extra elements to remove**, for example `.sidebar, #comments`.
5. Click **Start**. The log shows how many pages were saved.
6. Open the **Output** tab and click **Export** to download JSON, CSV, Excel or HTML, or load the dataset from the API straight into your RAG pipeline.

### Input

| Field | Type | Default | Description |
|---|---|---|---|
| `startUrls` | array of strings (required) | `["https://docs.apify.com/academy"]` | Websites or pages to start from. |
| `maxPages` | integer, 1 to 100,000 | `10` | Pages returned for each start website, counted by domain. |
| `maxDepth` | integer, 0 to 20 | `3` | 0 = only the start pages, 1 = also the pages they link to, and so on. |
| `sameDomainOnly` | boolean | `true` | Only follow links to the start URLs' domains. |
| `stayUnderStartPath` | boolean | `false` | Only follow links that start with the start URL's folder (for example only `/docs/...`). |
| `removeSelectors` | string | empty | Extra CSS selectors to remove before converting, comma-separated. |
| `proxy` | object | `{"useApifyProxy": true}` | Apify Proxy settings. |

Notes on the settings:

- **Websites are counted by host name**, ignoring `www.`. Subdomains are separate websites: `docs.example.com` and `example.com` each have their own `maxPages`. If a start URL redirects to another host (for example to a `docs.` subdomain), that host is crawled as the start website.
- **"Start path" means the folder up to the last slash.** `https://example.com/docs/` keeps the crawl inside `/docs/`, but `https://example.com/docs` (no trailing slash) only keeps it under `https://example.com/`. End your start URL with a slash when you use this option.
- **With "Stay on the same website" off**, external links are followed too, and each new domain gets its own `maxPages` limit.

Example input:

```json
{
  "startUrls": [
    "https://docs.apify.com/academy/"
  ],
  "maxPages": 200,
  "maxDepth": 3,
  "sameDomainOnly": true,
  "stayUnderStartPath": true,
  "removeSelectors": ".sidebar, #comments",
  "proxy": {
    "useApifyProxy": true
  }
}
```

### Output example

A real row from the latest run (the Markdown is shortened; the full page had 204 words):

```json
{
  "url": "https://docs.apify.com/academy",
  "title": "Apify Academy | Academy | Apify Documentation",
  "description": "Learn everything about web scraping and automation with our free courses that will turn you into an expert scraper developer.",
  "language": "en",
  "canonicalUrl": "https://docs.apify.com/academy",
  "depth": 0,
  "statusCode": 200,
  "wordCount": 204,
  "markdown": "# Apify Academy\n\nCopy for LLM\n\nLearn everything about web scraping and automation with our free courses that will turn you into an expert scraper developer.\n\n## Beginner courses\n\n[\n\n#### Web scraping basics with JS\n\nLearn how to use JavaScript to extract information from websites in this practical course, starting from the absolute basics.\n\n](https://docs.apify.com/academy/scraping-basics-javascript)\n\n…",
  "crawledAt": "2026-10-11T20:23:30.322Z"
}
```

### How much does it cost?

Website to Markdown Crawler uses pay per event pricing. You pay **$1.00 per 1,000 pages** returned ($0.001 per page) on the Free plan. Platform usage (compute and proxy) is included in that price.

| Apify plan | Price per 1,000 pages |
|---|---|
| Free | $1.00 |
| Bronze (10% off) | $0.90 |
| Silver (20% off) | $0.80 |
| Gold and higher (30% off) | $0.70 |

Worked examples on the Free plan:

- A default run (10 pages) = **$0.01**
- A help center with 1,500 pages = **$1.50**
- 20 company websites with 500 pages each = 10,000 pages = **$10.00** ($7.00 on Gold)

What you are **not** charged for:

- Pages that fail to load after retries.
- Pages whose Markdown is shorter than 20 characters (for example empty pages). They are skipped, not saved.
- Links to files that are never downloaded (PDF, images, ZIP, Office documents and similar).
- The error row "No pages could be loaded".

### Use cases

- **RAG knowledge bases.** Crawl your docs or help center, split the Markdown into chunks and load them into a vector database for a support chatbot.
- **Custom GPTs and Claude projects.** Export a site section as Markdown and upload it as knowledge files.
- **AI agents.** Give an agent the content of a competitor's website, a product catalog or a pricing page as clean text.
- **Summaries and analysis.** Feed blog posts or product pages into an LLM for summaries, classification or sentiment.
- **Content audits.** Use `wordCount`, `title`, `description` and `language` to find thin pages, missing meta descriptions or pages in the wrong language.
- **Site migration and archiving.** Save a website's text as Markdown before a redesign or CMS move.
- **Documentation monitoring.** Re-crawl docs on a schedule and compare the Markdown to spot changes.

### Tips and limits

- **No JavaScript rendering.** The crawler uses plain HTTP and does not run a browser. It works best on server-rendered sites (most docs, blogs, marketing and shop pages). Pages that only render with JavaScript may return little text or be skipped.
- **No images.** Images, video and audio are removed, so the Markdown contains text and links only.
- **Files are not converted.** Links to PDF, images, ZIP, audio and video, Word, Excel and PowerPoint files, TXT, MD, YAML, JSON, XML, RSS, CSS and JS are not followed.
- **Pages are found by following links.** The crawler does not read `sitemap.xml`. Pages that nothing links to are only crawled if you add them as start URLs.
- **Same page, different anchors.** The `#section` part of links is removed, so a page is not crawled twice for different anchors.
- **Budget per website.** The crawler queues at most 2 × `maxPages` links per website and makes at most 4 × `maxPages` × (number of start websites) requests in total. On sites where many pages fail or are skipped you can get fewer than `maxPages` rows.
- **Speed and retries.** Up to 10 pages load at the same time and each page gets 2 retries. With `sameDomainOnly` on, the run stops as soon as every start website has reached `maxPages`.
- **Remove what is left.** If a site's sidebar or "related posts" box still ends up in the Markdown, add its CSS selector in `removeSelectors`.
- **Error row.** When not a single page could be loaded (bad URL, site down or blocking), the run saves one row with `error: "No pages could be loaded"`.

### Use it from the API, Python or JavaScript

JavaScript (Node.js) with the official `apify-client` package:

```javascript
const { ApifyClient } = require('apify-client');

const client = new ApifyClient({ token: 'YOUR_TOKEN' });
const input = { startUrls: ['https://docs.apify.com/academy/'], maxPages: 50, stayUnderStartPath: true };

(async () => {
    const run = await client.actor('juliob.rangel/website-to-markdown-crawler').call(input);
    const { items } = await client.dataset(run.defaultDatasetId).listItems();
    for (const page of items) console.log(page.url, page.wordCount);
})();
```

Python with the official `apify-client` package, saving each page as a `.md` file:

```python
from apify_client import ApifyClient

client = ApifyClient('YOUR_TOKEN')
run_input = {'startUrls': ['https://docs.apify.com/academy/'], 'maxPages': 50, 'stayUnderStartPath': True}

run = client.actor('juliob.rangel/website-to-markdown-crawler').call(run_input=run_input)
items = client.dataset(run['defaultDatasetId']).list_items().items
for i, page in enumerate(items):
    if page.get('markdown'):
        with open(f'page_{i}.md', 'w', encoding='utf-8') as f:
            f.write(page['markdown'])
```

Plain HTTP, running the crawler and getting the rows in one call:

```bash
curl -X POST "https://api.apify.com/v2/acts/juliob.rangel~website-to-markdown-crawler/run-sync-get-dataset-items?token=YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"startUrls": ["https://docs.apify.com/academy/"], "maxPages": 20}'
```

The crawler also works with Make, Zapier and n8n, can export results to Google Sheets, can call your webhook when a run finishes, and can be used by AI agents through the Apify MCP server.

### FAQ

#### Does it render JavaScript?

No. It downloads pages over plain HTTP, which makes it fast and cheap. Sites that build their content only with JavaScript may return little text.

#### Do I need a login or cookies?

No, and it cannot crawl pages that are behind a login. It reads the public pages any visitor can open.

#### Can I export the Markdown to CSV, Excel or JSON?

Yes. The Markdown is one column of the dataset, so you can export as JSON, CSV, Excel or HTML. For RAG pipelines, JSON or the API keeps the line breaks intact.

#### How many pages can I crawl?

Up to 100,000 pages per website (`maxPages`), on as many websites as you add as start URLs.

#### How do I crawl only one section of a site?

Start from that section with a trailing slash, for example `https://example.com/docs/`, and turn on `stayUnderStartPath`.

#### Is it legal to crawl websites?

The crawler only collects publicly available pages. You are responsible for complying with each website's terms and with privacy and copyright rules, including laws such as GDPR and LGPD when pages contain personal data. Personal data needs a legitimate reason to be collected and processed.

#### Why did I get fewer pages than `maxPages`, or an error row?

The site may have fewer reachable pages within `maxDepth`, many pages may be files or JavaScript-only, or some pages may have failed. If not a single page loaded, you get the error row "No pages could be loaded". Check the URL and try a higher `maxDepth`.

### Related scrapers

- [Website Contact Details Scraper (Emails, Phones, Socials)](https://apify.com/juliob.rangel/website-contact-details-scraper)
- [YouTube Transcript Scraper (Fast & Cheap)](https://apify.com/juliob.rangel/youtube-transcript-fast)
- [Substack Scraper (Posts, Likes, Comments, Full Text)](https://apify.com/juliob.rangel/substack-scraper-fast)
- [Google News Scraper (Real Article URLs)](https://apify.com/juliob.rangel/google-news-scraper-fast)

***

## Sites em Markdown para IA (PT-BR)

Rastreie qualquer site e receba o texto limpo de cada página em Markdown, sem menus, rodapés ou banners de cookies, pronto para ChatGPT, Claude e bases de conhecimento (RAG). Você paga só por página retornada.

# Actor input Schema

## `startUrls` (type: `array`):

Websites or pages to start from, e.g. https://docs.apify.com/academy

## `maxPages` (type: `integer`):

Pages returned for each start website (counted by domain).

## `maxDepth` (type: `integer`):

0 = only the start pages, 1 = also pages they link to, and so on.

## `sameDomainOnly` (type: `boolean`):

Only follow links to the start URLs' domains.

## `stayUnderStartPath` (type: `boolean`):

Only follow links that start with the start URL's folder (e.g. only /docs/...).

## `removeSelectors` (type: `string`):

Optional CSS selectors to remove before converting, e.g. .sidebar, #comments

## `proxy` (type: `object`):

Apify Proxy settings.

## Actor input object example

```json
{
  "startUrls": [
    "https://docs.apify.com/academy"
  ],
  "maxPages": 10,
  "maxDepth": 3,
  "sameDomainOnly": true,
  "stayUnderStartPath": false,
  "proxy": {
    "useApifyProxy": true
  }
}
```

# Actor output Schema

## `results` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        "https://docs.apify.com/academy"
    ],
    "maxPages": 10
};

// Run the Actor and wait for it to finish
const run = await client.actor("juliob.rangel/website-to-markdown-crawler").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": ["https://docs.apify.com/academy"],
    "maxPages": 10,
}

# Run the Actor and wait for it to finish
run = client.actor("juliob.rangel/website-to-markdown-crawler").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    "https://docs.apify.com/academy"
  ],
  "maxPages": 10
}' |
apify call juliob.rangel/website-to-markdown-crawler --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,juliob.rangel/website-to-markdown-crawler"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/dev7u2yadu2gAby3w/builds/xgmnUfZCia77qSg1g/openapi.json