# llms.txt Generator + Docs Site to Markdown (RAG-ready) (`wintersworks/docs-to-markdown-llms-txt`) Actor

Input: a documentation site URL. Output: a generated llms.txt and llms-full.txt, plus clean Markdown per page with token counts and RAG chunks. $1 per 1,000 pages. Sitemap-first, respects robots.txt.

- **URL**: https://apify.com/wintersworks/docs-to-markdown-llms-txt.md
- **Developed by:** [Winters Works](https://apify.com/wintersworks) (community)
- **Categories:** Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

$1.00 / 1,000 page converteds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## llms.txt Generator + Docs Site to Markdown (RAG-ready)

Turn any documentation site into clean Markdown for **LLM context, RAG pipelines and AI agents**, and get a ready-made **llms.txt** and **llms-full.txt** for the site. Point it at a docs URL, get one dataset row per page with the main content converted to Markdown, plus optional token-sized chunks for vector databases.

Typical uses: feed a framework's docs to Cursor, Claude, ChatGPT or a custom agent; build a RAG index of a product's documentation; keep a local Markdown mirror of docs; generate an llms.txt for your own docs site.

### How it works

- **HTTP only** (no browser): fetches pages and converts them with cheerio + Turndown. Fast and cheap. Sites that render content only with JavaScript in the browser are not supported (see Limitations).
- **Discovery:** reads robots.txt and `sitemap.xml` (including sitemap indexes) first; if the sitemap has nothing in scope it falls back to following links within the start URL's path prefix.
- **Polite:** identifies itself as `WintersWorksDocsToLLMs/1.0 (+wintersworks.shop@gmail.com)`, respects robots.txt (disallowed pages are skipped), limits request rate per host.
- **Clean content:** strips navigation, sidebars, footers, breadcrumbs, table of contents and "edit this page" links, keeping the main article (`main`, `article`, `.markdown`, `.theme-doc-markdown`, etc., or your own CSS selector).

### Input

| Field | Description |
|---|---|
| `startUrls` | Docs URLs, e.g. `https://docs.python.org/3/tutorial/`. Pages under the same path prefix are converted. |
| `maxPages` | Cap on pages converted (and charged). Default 50. |
| `includeGlobs` / `excludeGlobs` | URL globs (`*` within a segment, `**` anywhere). |
| `sameDomainOnly`, `pathPrefixOnly` | Scope controls (both default on). |
| `useSitemap` | Try sitemap.xml first. Default on. |
| `contentSelector` | Optional CSS selector for the main content. |
| `includeMarkdown` | Put the Markdown in each dataset row. Default on. |
| `chunkSize`, `chunkOverlap` | If `chunkSize` > 0, also output RAG chunks (approx tokens, chars/4), split on paragraphs, each with its nearest heading. |
| `maxConcurrency` | Parallel requests (default 4, max 10). |
| `respectRobotsTxt` | Default on. |

Example:

```json
{ "startUrls": ["https://docs.apify.com/sdk/js/docs"], "maxPages": 15, "chunkSize": 500, "chunkOverlap": 50 }
```

### Output

One dataset item per page: `url`, `title`, `description`, `headings` (outline of h1-h4), `markdown`, `wordCount`, `approxTokens` (characters / 4, an estimate, not a real tokenizer count), `crawledAt`, and `chunks` (`index`, `heading`, `tokens`, `text`) when enabled.

```json
{ "url": "https://docs.apify.com/sdk/js/docs/overview", "title": "Apify SDK for JavaScript",
  "headings": [{"level":1,"text":"Apify SDK for JavaScript"},{"level":2,"text":"Quick start"}],
  "wordCount": 242, "approxTokens": 410, "markdown": "# Apify SDK for JavaScript\n\n..." }
```

The run also saves **`llms.txt`** (site title, summary, list of page links with one-line descriptions) and **`llms-full.txt`** (all pages' Markdown concatenated, each prefixed with its source URL) to the key-value store. Their URLs are logged and listed in the `OUTPUT` record. With several start URLs you get one pair of files per start URL.

### Pricing

Pay per event: **$0.001 per page converted** ($1 per 1,000 pages). Pages that fail, are blocked by robots.txt, or have no content are not charged. llms.txt and llms-full.txt are free. The run stops cleanly when your spending limit is reached.

### FAQ

**Does it work for GitBook, Docusaurus, MkDocs, Sphinx, ReadTheDocs?** These render server-side HTML and are the main target; tested on a Docusaurus site and the Sphinx-built Python docs. Results vary by theme; use `contentSelector` if the default misses.
**Can I get only part of a site?** Use a deeper start URL, or include/exclude globs.
**Are the token counts exact?** No: characters / 4, an estimate.
**Is the llms.txt official?** It follows the llms.txt proposal format (title, summary, link list) but descriptions come from each page's meta description.

### Limitations

- No JavaScript rendering: pages whose content is built client-side (some SPA docs) will come out empty and are skipped (not charged).
- No login or paywalled docs. Respect each site's terms of use and copyright; you are responsible for how you use the converted content.
- Images are kept as absolute links, not downloaded. Very large tables and unusual widgets may convert imperfectly.
- Sitemaps listing multiple doc versions can include old versions within your path prefix; narrow the start URL or use `excludeGlobs`.

### Is it legal? (public data only)

Yes, in the sense that it only reads public or official data: public documentation pages you point it at (respecting robots.txt). It does not log in, bypass paywalls or access private data. You remain responsible for how you use the results, including privacy, copyright and marketing laws. Not affiliated with or endorsed by the data sources.

### Quick start and cost

Run it with the prefilled input to see sample output in seconds; the default run costs only a few cents at most. Then adjust the input and schedule it.

### Use from code and AI agents

Call it from any language with the Apify API, or let an AI agent find and run it through the [Apify MCP server](https://mcp.apify.com).

```python
from apify_client import ApifyClient
run = ApifyClient("<YOUR_APIFY_TOKEN>").actor("wintersworks/docs-to-markdown-llms-txt").call(run_input={"startUrls": ["https://docs.apify.com/sdk/js/docs"], "maxPages": 10})
items = ApifyClient("<YOUR_APIFY_TOKEN>").dataset(run["defaultDatasetId"]).list_items().items
```

Results land in the run's dataset (JSON, CSV, Excel). Set a maximum charge per run in the run options to cap spend.

### About

Built and maintained by Winters Works. Questions or feature requests: open an issue on the Actor page or email wintersworks.shop@gmail.com.

# Actor input Schema

## `startUrls` (type: `array`):

One or more docs URLs. Pages under each URL's path prefix are converted (e.g. https://docs.python.org/3/tutorial/).

## `maxPages` (type: `integer`):

Maximum pages converted (and charged) per run, across all start URLs.

## `includeGlobs` (type: `array`):

Only convert URLs matching one of these globs (\* = within a path segment, \*\* = anything). Empty = all.

## `excludeGlobs` (type: `array`):

Skip URLs matching any of these globs.

## `sameDomainOnly` (type: `boolean`):

Only follow links on the start URL's host.

## `pathPrefixOnly` (type: `boolean`):

Only convert pages whose path starts with the start URL's path.

## `useSitemap` (type: `boolean`):

Try sitemap.xml first; fall back to link crawling if none is found.

## `contentSelector` (type: `string`):

Optional CSS selector for the main content. If empty, common selectors (main, article, .markdown, ...) are tried.

## `includeMarkdown` (type: `boolean`):

Include markdown in dataset

## `chunkSize` (type: `integer`):

If > 0, also output RAG chunks of about this many tokens (approx chars/4), split on headings/paragraphs.

## `chunkOverlap` (type: `integer`):

Chunk overlap (tokens)

## `maxConcurrency` (type: `integer`):

Parallel requests per run (kept polite).

## `respectRobotsTxt` (type: `boolean`):

Respect robots.txt

## Actor input object example

```json
{
  "startUrls": [
    "https://docs.apify.com/sdk/js/docs"
  ],
  "maxPages": 10,
  "includeGlobs": [],
  "excludeGlobs": [],
  "sameDomainOnly": true,
  "pathPrefixOnly": true,
  "useSitemap": true,
  "includeMarkdown": true,
  "chunkSize": 0,
  "chunkOverlap": 50,
  "maxConcurrency": 4,
  "respectRobotsTxt": true
}
```

# Actor output Schema

## `results` (type: `string`):

All result items from the run's default dataset.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        "https://docs.apify.com/sdk/js/docs"
    ],
    "maxPages": 10
};

// Run the Actor and wait for it to finish
const run = await client.actor("wintersworks/docs-to-markdown-llms-txt").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": ["https://docs.apify.com/sdk/js/docs"],
    "maxPages": 10,
}

# Run the Actor and wait for it to finish
run = client.actor("wintersworks/docs-to-markdown-llms-txt").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    "https://docs.apify.com/sdk/js/docs"
  ],
  "maxPages": 10
}' |
apify call wintersworks/docs-to-markdown-llms-txt --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,wintersworks/docs-to-markdown-llms-txt"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/Ey5Xqu8gBOWRU5FGb/builds/y1lvWneYBnGbHB9Ze/openapi.json