# Website Content to Markdown & llms.txt Crawler - AI & RAG Ready (`abi4/website-content-llmstxt-crawler`) Actor

High-performance, lightweight asynchronous crawler converting web pages into clean Markdown, computing token estimates, and generating standard /llms.txt manifests for AI and RAG ingestion.

- **URL**: https://apify.com/abi4/website-content-llmstxt-crawler.md
- **Developed by:** [Abi](https://apify.com/abi4) (community)
- **Categories:**
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.70 / 1,000 scraped results

This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Website Content to Markdown & llms.txt Crawler - AI & RAG Ready

A lightweight, high-velocity web scraper built on **Crawlee for Python** that transforms websites into clean, token-efficient Markdown and automatically generates standards-compliant `/llms.txt` and `/llms-full.txt` manifests directly in your Apify Key-Value Store.

***

### Why Choose This Actor?

Traditional website crawlers rely on heavy headless browsers (Playwright/Puppeteer with Chromium) that allocate **4,096 MB to 8,192 MB of RAM**, costing $3.20+ per compute hour and suffering frequent memory leaks and navigation timeouts during deep crawls.

**Website Content to Markdown & llms.txt Crawler** solves this with an asynchronous HTTP and BeautifulSoup architecture:

- **90% Compute Cost Reduction**: Runs efficiently within a **512 MB - 1,024 MB RAM** footprint.
- **10x - 20x Throughput Advantage**: Crawls 15 to 25 pages per second on standard broadband connections.
- **Automated `/llms.txt` & `/llms-full.txt` Generation**: Built according to the [llmstxt.org](https://llmstxt.org) standard for zero-friction AI ingestion.
- **Intelligent Boilerplate Pruning**: Strips navigation menus, cookie banners, tracking scripts, and footers, keeping vector databases clean.
- **Token Accounting**: Calculates accurate character, word, and estimated LLM token counts per page for RAG context budgeting.
- **100% Windows & Linux Compatible**: Defensive UTF-8 handling and pure ASCII logging ensure zero encoding crashes.

***

### Performance & Capability Comparison

| Metric / Capability | Legacy Headless Browser Crawlers | Basic Web Scrapers | **Website Content to llms.txt Crawler (This Actor)** |
|---------------------|--------------------------------|------------------------|--------------------------------------------------|
| **Default RAM Allocation** | 8,192 MB (Heavy Browser) | 1,024 MB | **512 MB - 1,024 MB** |
| **Throughput (pages/sec)** | 1 - 2 pages/sec | 5 - 10 pages/sec | **15 - 25 pages/sec** |
| **Pricing Model** | Platform CUs (High) | $3.00 / 1,000 pages PPE | **Standard Platform CUs (Ultra-Low)** |
| **`/llms.txt` Generation** | No (Read-only seed) | No | **Yes (Automatic in Key-Value Store)** |
| **`/llms-full.txt` Single File** | No | No | **Yes (Concatenated RAG context)** |
| **Boilerplate Stripping** | Manual regex/selectors | Minimal | **Automated + Configurable Selectors** |
| **LLM Token Estimation** | No | No | **Yes (~4 chars/token heuristic)** |
| **Headings Hierarchy Tree** | Flat list | None | **Structured AST (Levels 1 - 6)** |

***

### Input Parameters

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `startUrls` | Array | `[{"url": "https://crawlee.dev"}]` | Seed URLs from which crawling commences. |
| `maxPages` | Integer | `20` | Maximum number of pages to crawl (1 - 1,000). |
| `maxDepth` | Integer | `3` | Maximum link depth from start URLs (0 - 10). |
| `includeLlmsTxtGeneration` | Boolean | `true` | Synthesize `/llms.txt` and `/llms-full.txt` in Key-Value Store. |
| `removeElementsCssSelector` | String | Standard boilerplate selector | Elements stripped from DOM before Markdown conversion. |
| `proxyConfiguration` | Object | `{ "useApifyProxy": true }` | Proxy settings to prevent IP bans and rate limiting. |

***

### Outputs

#### 1. Default Dataset

Each crawled page emits a record with:

````json
{
  "url": "https://crawlee.dev/python/docs/quickstart",
  "title": "Quick Start | Crawlee for Python",
  "description": "Learn how to build your first scraper using Crawlee in Python.",
  "markdown": "# Quick Start\n\nInstall Crawlee using pip:\n\n```bash\npip install crawlee[beautifulsoup]\n```\n\n## First Steps\n\n...",
  "tokenCountEstimate": 420,
  "wordCount": 315,
  "characterCount": 1680,
  "headings": [
    { "level": 1, "text": "Quick Start" },
    { "level": 2, "text": "First Steps" }
  ],
  "metadata": {
    "canonical": "https://crawlee.dev/python/docs/quickstart",
    "author": "Apify",
    "language": "en",
    "ogTitle": "Quick Start | Crawlee for Python",
    "ogDescription": "Learn how to build your first scraper using Crawlee in Python.",
    "ogImage": "https://crawlee.dev/img/og-image.png",
    "ogType": "article"
  },
  "crawledAt": "2026-09-07T11:00:00.000Z"
}
````

#### 2. Key-Value Store Artifacts

- **`LLMS_TXT`**: A standard `/llms.txt` file linking to the key documentation and article pages with concise summaries.
- **`LLMS_FULL_TXT`**: A consolidated single-file Markdown document containing all crawled pages clearly delimited, ready for direct upload into:
  - Cursor AI (`.cursorrules` or context)
  - Claude Projects
  - OpenAI Custom GPTs
  - Google NotebookLM
  - LangChain / LlamaIndex Vector Stores

***

### ⚡ How to Use in 3 Simple Steps

1. **Set Seed URLs**: Enter your target website or documentation URL (e.g., `https://crawlee.dev`).
2. **Configure Scope**: Choose `maxPages` (e.g., `20` for a quick scan, or up to `1000` for an entire documentation site) and keep `includeLlmsTxtGeneration` checked.
3. **Run & Ingest**: Click **Start**. When complete, download your clean Markdown dataset or copy `LLMS_TXT` and `LLMS_FULL_TXT` directly from the **Key-Value Store** tab for your AI workflows.

***

### 💡 Use Cases & AI Integrations

- **Cursor & AI Code Editors**: Drop `LLMS_FULL_TXT` directly into `.cursorrules` or editor context for instant documentation retrieval.
- **Claude Projects & OpenAI Custom GPTs**: Upload the clean, consolidated Markdown artifact directly as a persistent knowledge base file.
- **RAG & Vector Pipelines**: Ingest boilerplate-free Markdown into LangChain, LlamaIndex, Pinecone, or Weaviate with pre-calculated token counts.
- **Google NotebookLM**: Feed clean, noise-free website content into NotebookLM for instant Q\&A and podcast generation.

***

### ❓ FAQ & Support

#### Does this crawler require a heavy headless browser?

No. By using an asynchronous HTTP and BeautifulSoup architecture, this Actor achieves 15–25 pages per second with only 512 MB–1,024 MB RAM, cutting compute costs by up to 90% compared to heavy browser crawlers.

#### Where can I find the generated `/llms.txt` manifests?

Open the **Storage** > **Key-Value Store** tab of your run:

- **`LLMS_TXT`**: Standard summary manifest conforming to the [llmstxt.org](https://llmstxt.org) standard.
- **`LLMS_FULL_TXT`**: Complete, concatenated full-text Markdown document ready for direct LLM context upload.

#### Need custom scrapers or feature enhancements?

Submit an inquiry or bug report on the **Issues** tab in the Apify Console. We respond promptly and actively maintain this Actor.

# Actor input Schema

## `startUrls` (type: `array`):

List of URLs from which the crawl begins. Enter one or more URLs.

## `maxPages` (type: `integer`):

Maximum number of pages to crawl before completing the run.

## `maxDepth` (type: `integer`):

Maximum distance in clicks from start URLs (0 = start URLs only, 1 = 1 click away, etc.).

## `includeLlmsTxtGeneration` (type: `boolean`):

Automatically synthesize standard /llms.txt and /llms-full.txt files and save them in the Key-Value Store.

## `removeElementsCssSelector` (type: `string`):

CSS selectors of elements to strip from the HTML DOM before converting to Markdown.

## `enableMcpMode` (type: `boolean`):

When enabled, executes the Actor as a Model Context Protocol (MCP) server for direct integration with Cursor, Claude Desktop, and Windsurf.

## `mcpTransport` (type: `string`):

Transport protocol for MCP tool execution. Use stdio for local CLI/IDE execution and sse for Apify Standby web integration.

## `renderJs` (type: `boolean`):

Enable Playwright headless browser rendering for client-rendered SPAs (React, Vue, Angular). Note: If enabled, ensure your Actor run memory is set to at least 1,024 MB.

## `waitForSelector` (type: `string`):

Optional CSS selector to wait for when renderJs is enabled before extracting content (e.g. 'article, .docs-content').

## `contentSelector` (type: `string`):

Scope Markdown extraction exclusively to matching element(s) (e.g. 'main, article, .markdown-body') before pruning boilerplate.

## `customExtractionSelectors` (type: `object`):

Optional JSON map of field names to CSS selectors to extract specific elements into structured output (e.g. {"author": ".author-name", "published": "time"}).

## `proxyConfiguration` (type: `object`):

Proxy settings for the crawler. Select Apify Proxy to bypass geo-restrictions and rate limits.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://crawlee.dev"
    }
  ],
  "maxPages": 20,
  "maxDepth": 3,
  "includeLlmsTxtGeneration": true,
  "removeElementsCssSelector": "nav, footer, script, style, noscript, .cookie-banner, .advertisement, [role=navigation], [role=banner], [role=contentinfo]",
  "enableMcpMode": false,
  "mcpTransport": "stdio",
  "renderJs": false,
  "waitForSelector": "",
  "contentSelector": "",
  "customExtractionSelectors": {},
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}
```

# Actor output Schema

## `dataset` (type: `string`):

No description

## `llmsTxt` (type: `string`):

No description

## `llmsFullTxt` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://crawlee.dev"
        }
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("abi4/website-content-llmstxt-crawler").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "startUrls": [{ "url": "https://crawlee.dev" }] }

# Run the Actor and wait for it to finish
run = client.actor("abi4/website-content-llmstxt-crawler").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://crawlee.dev"
    }
  ]
}' |
apify call abi4/website-content-llmstxt-crawler --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,abi4/website-content-llmstxt-crawler"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/XMozJiTbJGvGg1aEL/builds/GL16NJmqDxctJNo6v/openapi.json