# Website to Markdown for LLM & RAG + llms.txt Generator (`integriapps/web-to-markdown`) Actor

Convert any web page or whole site (sitemap or crawl) to clean Markdown with metadata and token-sized chunks ready for RAG, embeddings and AI agents. Generates llms.txt and llms-full.txt. Respects robots.txt. Flat price per page. By IntegriApps.

- **URL**: https://apify.com/integriapps/web-to-markdown.md
- **Developed by:** [IntegriApps .com](https://apify.com/integriapps) (community)
- **Categories:** AI, Developer tools
- **Features:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

$1.00 / 1,000 page converted to markdowns

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Website to Markdown for LLM & RAG + llms.txt Generator

Turn any web page — or a whole website via its **sitemap.xml** or a same-site **crawl** — into **clean Markdown** ready for **LLMs, RAG pipelines, embeddings, vector databases and AI agents**. Every page comes with metadata (title, description, language, canonical URL, dates), a token estimate and **token-sized chunks with overlap**. With one switch it also generates **`llms.txt` and `llms-full.txt`** for your site.

Part of [IntegriApps MicroApps](https://integriapps.com/microapps/). Support Sunday to Friday.

### What it does

- **Clean Markdown:** removes navigation, headers, footers, sidebars, cookie banners, scripts and ads-like boilerplate; keeps headings, lists, links (made absolute), code blocks and **tables as GitHub-flavored Markdown**.
- **Three modes:** `urls` (only the pages you give), `sitemap` (reads sitemap.xml from robots.txt or `/sitemap.xml`, including sitemap indexes), `crawl` (follows same-site links up to a depth).
- **RAG-ready chunks:** each page is split into chunks of ~N tokens (1 token ≈ 4 characters) on heading/paragraph boundaries, with configurable overlap and the nearest heading.
- **llms.txt generator:** builds an [llms.txt](https://llmstxt.org/) index and an `llms-full.txt` with all the Markdown, saved in the run's key-value store.
- **Native Markdown:** if a site already serves Markdown (`text/markdown`), it is used as is.
- **Deduplication:** pages with identical content are skipped and not charged.
- **Polite by design:** always respects **robots.txt** (including `Crawl-delay`), uses plain HTTP requests (no browser, no proxies), and does **not** bypass logins, paywalls or CAPTCHAs.

### Use cases

- Feed documentation, help centers or blogs into a **RAG chatbot** or **vector database** (Pinecone, Qdrant, pgvector, Weaviate…).
- Give an **AI agent / MCP client** a tool that reads a URL and returns Markdown.
- Publish an **llms.txt** for your own website so AI assistants understand it.
- Archive or diff the text content of your pages.

### Input

| Field | Description | Default |
|---|---|---|
| `startUrls` | Pages (or site start points) to convert | — (empty = free demo with example.com) |
| `mode` | `urls`, `sitemap` or `crawl` | `urls` |
| `maxPages` | Max pages converted in the run (1–10,000) | 100 |
| `maxDepth` | Link depth in `crawl` mode | 2 |
| `includeUrlPattern` / `excludeUrlPattern` | Regular expressions to filter discovered URLs | — |
| `chunkSize` / `chunkOverlap` | Chunk size and overlap in tokens (0 = no chunks) | 800 / 100 |
| `onlyMainContent` | Keep only the main content | true |
| `keepImages` | Keep image links | false |
| `removeSelectors` | Extra CSS selectors to delete | — |
| `generateLlmsTxt`, `siteName`, `siteDescription` | Build llms.txt / llms-full.txt | false |
| `maxConcurrency` | Parallel requests (1–20) | 5 |

Example — whole docs section from the sitemap, with llms.txt:

```json
{
  "startUrls": [{ "url": "https://crawlee.dev/" }],
  "mode": "sitemap",
  "includeUrlPattern": "/js/docs/",
  "maxPages": 25,
  "generateLlmsTxt": true
}
```

### Output

One dataset item per URL (real output from a `crawl` run with `chunkSize: 300`, shortened):

```json
{
  "url": "https://crawlee.dev/js/docs/quick-start",
  "status": "ok",
  "httpStatus": 200,
  "metadata": { "title": "Quick Start", "description": null, "language": null, "canonicalUrl": null, "servedAsMarkdown": true },
  "markdown": "# Quick Start\n\nWith this short tutorial you can start scraping with Crawlee...",
  "wordCount": 1301,
  "tokenEstimate": 4552,
  "contentHash": "…",
  "chunks": [{ "index": 0, "text": "# Quick Start\n\n…", "tokens": 293, "heading": "Quick Start" }]
}
```

`status` is one of `ok`, `blocked_by_robots_txt`, `error`, `unsupported_content_type` (e.g. PDF, images) or `duplicate_content`. Only `ok` is charged.

Key-value store: `llms.txt`, `llms-full.txt` (when enabled) and `RUN_SUMMARY` (counts per status and pages charged). Export the dataset as JSON, JSONL, CSV or Excel from Apify Console or the API.

### Pricing

**Pay per event: USD 0.001 per page converted** (USD 1 per 1,000 pages), flat — no extra charge for chunks, metadata or llms.txt. Pages blocked by robots.txt, failed requests, unsupported files and duplicates are **free**. You can cap spending with the run's maximum charge setting; the Actor stops cleanly when the limit is reached.

### Use it from your code

```bash
curl -X POST "https://api.apify.com/v2/acts/integriapps~web-to-markdown/run-sync-get-dataset-items?token=YOUR_APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"startUrls":[{"url":"https://example.com/"}],"chunkSize":0}'
```

It also works from the Apify API clients (JavaScript/Python), Make, Zapier, n8n, LangChain/LlamaIndex Apify integrations and the Apify MCP server.

### FAQ

**Does it render JavaScript?** No. It uses fast HTTP requests, which works for most documentation sites, blogs and server-rendered pages. Pages that need a browser to show content may come out empty.

**Can it read pages behind a login or a paywall?** No, and it never tries to bypass logins, paywalls or CAPTCHAs.

**Why was a page skipped?** Check its `status`. `blocked_by_robots_txt` means the site's robots.txt does not allow it (or robots.txt was unreachable, in which case we stay on the safe side).

**Is it legal to use?** You are responsible for having the right to use the content you convert (your own site, public documentation, content whose terms allow it). The Actor respects robots.txt but does not read each site's terms of service for you.

**How are tokens counted?** As an estimate of 4 characters per token, which is close to common LLM tokenizers for English text.

**Can it convert PDFs?** Not in this version; PDFs are returned as `unsupported_content_type` and are not charged.

### Changelog

See [CHANGELOG.md](./CHANGELOG.md).

Part of [IntegriApps MicroApps](https://integriapps.com/microapps/) — small tools by IntegriApps, a SaaS for SMBs. Support Sunday to Friday.

# Changelog

This Actor's version history is a separate document: https://apify.com/integriapps/web-to-markdown/changelog.md

# Actor input Schema

## `startUrls` (type: `array`):

Pages to convert. In 'sitemap' or 'crawl' mode, each URL is the starting point of a site. If empty, https://example.com/ is converted for free as a demo.

## `mode` (type: `string`):

'urls' converts only the given pages. 'sitemap' reads the site's sitemap.xml (from robots.txt or /sitemap.xml) and converts its pages. 'crawl' follows same-site links up to Max depth.

## `maxPages` (type: `integer`):

Maximum number of pages to convert in this run (1–10,000). Each converted page is one charged event.

## `maxDepth` (type: `integer`):

Only for 'crawl' mode: how many link levels to follow from the start URL.

## `includeUrlPattern` (type: `string`):

Optional. In sitemap/crawl mode only discovered URLs matching this regular expression are converted, e.g. /docs/

## `excludeUrlPattern` (type: `string`):

Optional. URLs matching this regular expression are skipped, e.g. /(tag|author)/

## `chunkSize` (type: `integer`):

Split each page into chunks of about this many tokens (1 token ≈ 4 characters) for RAG/embeddings. 0 = no chunks.

## `chunkOverlap` (type: `integer`):

Tokens repeated between consecutive chunks to keep context.

## `onlyMainContent` (type: `boolean`):

Remove navigation, header, footer, sidebars and cookie banners and keep the main article/content.

## `keepImages` (type: `boolean`):

Keep image links in the Markdown.

## `removeSelectors` (type: `string`):

Optional comma-separated CSS selectors to delete before converting, e.g. .ads, #newsletter

## `generateLlmsTxt` (type: `boolean`):

Build llms.txt (index of pages, llmstxt.org format) and llms-full.txt (all Markdown) and save them in the run's key-value store. Best with 'sitemap' mode on your own site.

## `siteName` (type: `string`):

Optional H1 title of llms.txt. Defaults to the first page's title.

## `siteDescription` (type: `string`):

Optional one-line summary for llms.txt. Defaults to the first page's meta description.

## `maxConcurrency` (type: `integer`):

Parallel requests (1–20). robots.txt Crawl-delay is always honored.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://docs.apify.com/platform/actors"
    }
  ],
  "mode": "urls",
  "maxPages": 100,
  "maxDepth": 2,
  "chunkSize": 800,
  "chunkOverlap": 100,
  "onlyMainContent": true,
  "keepImages": false,
  "generateLlmsTxt": false,
  "maxConcurrency": 5
}
```

# Actor output Schema

## `pages` (type: `string`):

No description

## `llmsTxt` (type: `string`):

No description

## `llmsFullTxt` (type: `string`):

No description

## `summary` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://docs.apify.com/platform/actors"
        }
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("integriapps/web-to-markdown").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "startUrls": [{ "url": "https://docs.apify.com/platform/actors" }] }

# Run the Actor and wait for it to finish
run = client.actor("integriapps/web-to-markdown").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://docs.apify.com/platform/actors"
    }
  ]
}' |
apify call integriapps/web-to-markdown --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,integriapps/web-to-markdown"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/NpVvW2YF9vhEYRKEq/builds/a47q0ZXPegFMATJSw/openapi.json
