# Suchi.Anantasutra: llms.txt & llms-full.txt Generator (`motivating_bracelet/suchi-anantasutra`) Actor

Generate a spec-compliant llms.txt and llms-full.txt for any website: robots.txt-respecting crawl, sitemap-first discovery and clean Markdown per page. Pay per page, no subscription.

- **URL**: https://apify.com/motivating_bracelet/suchi-anantasutra.md
- **Developed by:** [Bhavy Duneja](https://apify.com/motivating_bracelet) (community)
- **Categories:** AI, SEO tools, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.00 / 1,000 page processeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Suchi.Anantasutra: llms.txt & llms-full.txt Generator

*Suchi* (सूची) is Sanskrit for "index" or "list". This Actor crawls your website politely and writes the two files
AI agents look for: **`llms.txt`**, a compact Markdown index of your key pages ([llmstxt.org](https://llmstxt.org)
format), and **`llms-full.txt`**, the clean Markdown text of those pages in one document. It is part of the
**Anantasutra** family of tools.

### Who it is for

- **Site and shop owners** who want AI assistants (ChatGPT, Claude, Perplexity, Cursor ...) to find and quote the
  right pages, without writing llms.txt by hand.
- **Documentation teams** who want a section-level file such as `/docs/llms.txt` for their docs only.
- **Agencies and developers** who need clean Markdown per page for RAG pipelines (every page is also a dataset row).

### How it works

1. Reads your robots.txt and obeys it for its own user agent (`LlmsTxtGenerator`) or `*`, including Crawl-delay
   (up to 10 seconds, one request at a time).
2. Fetches the start page first; its links are treated as your main navigation.
3. Reads your sitemap (declared in robots.txt, or `/sitemap.xml`), then follows same-site links breadth first.
   Blog, tag, archive and old-version pages are crawled last.
4. Extracts the main content of each page as clean Markdown (navigation, footers, cookie banners and scripts
   removed) and groups pages into sections by their first path segment (Docs, Guides, API ...).
5. Saves `llms.txt` and `llms-full.txt` to the run's key-value store, checks `llms.txt` against the spec, and
   pushes one dataset row per page considered.

Pages are skipped (and not charged) when robots.txt disallows them, they are marked `noindex`, they have little text
without JavaScript, they redirect to another site, they are not HTML, or they are near-duplicates of a page already
included (see "Crawl traps" below). Login, search, cart and checkout paths are never included.

### Input

```json
{
  "startUrl": "https://docs.example.com/",
  "maxPages": 50,
  "crawlScope": "path",
  "optionalGlobs": ["/blog/**"],
  "excludeGlobs": ["/tag/*"]
}
```

- `startUrl`: home page or docs root. By default only pages under this path are included; set `crawlScope` to
  `site` for the whole website.
- `maxPages`: 1-500. Each included page is billed once.
- `includeGlobs` / `excludeGlobs` / `optionalGlobs`: path filters; `optionalGlobs` pages go under `## Optional`.
- `siteName` / `siteSummary`: your own H1 and summary line instead of the detected ones.
- `maxCharsPerPage` (1,000-200,000): longer pages are cut in `llms-full.txt` and the dataset.

### Output

Key-value store record `llms.txt` (abridged):

```markdown
## Example Docs

> Example Docs explains how to install, configure and deploy the Example widget toolkit.

### Docs

- [Getting started](https://docs.example.com/docs/getting-started): Create your first widget project in five minutes.
- [Install](https://docs.example.com/docs/install.md): System requirements and installation options for the Example CLI.

### Optional

- [We launched Example 2.0](https://docs.example.com/blog/launch): Release notes for Example 2.0.
```

Dataset row per page (Markdown omitted):

```json
{
  "url": "https://docs.example.com/docs/getting-started",
  "status": "ok",
  "title": "Getting started",
  "description": "Create your first widget project in five minutes.",
  "section": "Docs",
  "inLlmsTxt": true,
  "wordCount": 105,
  "truncated": false,
  "promptInjectionSuspected": false,
  "reason": null
}
```

Upload `llms.txt` (and optionally `llms-full.txt`) to your site root, or to the section path you crawled.

### Crawled text is untrusted: review before publishing

These files are written for AI agents, so text copied from your pages matters. Suchi protects the prominent lines:

- The H1 (max 80 characters), the summary blockquote (a sentence or two, max 300 characters), link titles (max 120)
  and link notes (max 200) are cleaned of markup and invisible characters and kept short, so one huge paragraph
  cannot bloat the index.
- Text that reads like instructions aimed at AI agents ("ignore all previous instructions", "\[\[SYSTEM]]",
  "instruction to any AI reading this", "reveal your system prompt" ...) is never used for those lines. The host
  name and a neutral summary are used instead.
- Pages whose text (outside code blocks) contains such phrases are flagged with `promptInjectionSuspected: true`
  and listed in the run log and status. The check is a deliberately narrow heuristic: it avoids flagging ordinary
  documentation, so it cannot catch every possible injection.

Page bodies are copied as they are into `llms-full.txt` (markup is escaped, so `<script>` stays text). If your site
has comments, reviews, forums or other user-generated content, read the flagged pages before you publish, or exclude
those paths with `excludeGlobs`. Set `siteName` and `siteSummary` to control the top of the file yourself.

### Crawl traps

Session IDs, sort orders and calendar links can turn one page into thousands of URLs. Well-known session parameters
(`PHPSESSID`, `jsessionid`, `sessionid`, `cfid` ...) and ASP.NET cookieless session segments (`/(S(...))/`) are
removed from URLs. When pages that differ only by URL parameters have near-identical text, the later ones are skipped
as near-duplicates and not charged; after a few such duplicates the remaining parameter variants of that path are not
fetched at all. The same applies to session IDs inside the path (`/s/<id>/booking`): a page that is a near-copy of one
already included and links mostly to new copies of itself is skipped and not charged, and after a few of those the
whole pattern (`/s/*/booking`) is not fetched any more. The log names each detected pattern. Similar pages that are
separate URLs on your site, such as per-platform install guides or placeholder pages sharing one menu, are kept.

### Pricing

Pay per event, no subscription.

| Event | Price | When |
|---|---|---|
| Page processed | $0.001 | Per page included in llms.txt and llms-full.txt |
| llms.txt files generated | $0.01 | Once per run, after both files are saved |
| Actor start | $0.00005 | Charged automatically by Apify at the start of each run (per GB of memory) |

A 100-page site costs about $0.11 ($0.10 for pages + $0.01 for the files). Skipped and failed pages are never
charged. If robots.txt is unreachable or disallows the site, nothing is crawled and only the Actor start is charged.
Set "Maximum cost per run" to cap spending: the crawl is sized so the one-time file fee always fits.

### Use it from Claude, ChatGPT or Cursor (MCP)

The [Apify MCP server](https://mcp.apify.com) exposes this Actor as a tool to AI assistants.

- **Claude Desktop / Claude Code, Cursor and other MCP clients:** add the hosted server
  `https://mcp.apify.com/?actors=motivating_bracelet/suchi-anantasutra` and sign in with your Apify account, or run it
  locally with `npx -y @apify/actors-mcp-server --actors motivating_bracelet/suchi-anantasutra` and your Apify API token
  in the `APIFY_TOKEN` environment variable.
- **ChatGPT:** add `https://mcp.apify.com` as a custom connector (MCP) where your plan allows it.

Then ask, for example: "Generate an llms.txt for docs.example.com with Suchi, at most 30 pages." The assistant
passes `startUrl` and `maxPages` and can read the `llms.txt` record from the run's key-value store.

### Limits

- Up to 500 pages per run and 5 parallel requests; one request at a time when robots.txt sets a Crawl-delay.
- Only public websites over http or https. Private, loopback, link-local and other reserved addresses are refused,
  including through DNS names, redirects and sitemap links.
- Raw HTML only (no browser): pages that need JavaScript to show their text are skipped as "little text".
- Page downloads are capped at 3 MB, `llms-full.txt` at 4,000,000 characters, robots.txt at 500 KiB.
- The crawl stops shortly before the run timeout and saves what it has.

### FAQ

**Why are some pages missing?** Look at the dataset rows with `status: skipped` or `error`: the `reason` column says
why (robots.txt, noindex, little text, near-duplicate, redirect to another site, page limit).

**Why does my llms.txt start with the host name instead of my site name?** The detected title was missing or looked
like instructions aimed at AI agents. Set `siteName` (and `siteSummary`) in the input.

**Is it allowed to crawl my site?** It only crawls what your robots.txt allows for `LlmsTxtGenerator` or `*`, and
identifies itself honestly. Run it on sites you own or are allowed to index.

**Do I pay for pages that were skipped?** No. Only pages that end up in the files are charged, plus the one-time file
fee when the files are saved.

**How can I check the result?** Audit your site with the sibling Actor **Drishti.Anantasutra**; it validates
`llms.txt` and shows which AI crawlers can reach your pages.

# Actor input Schema

## `startUrl` (type: `string`):

Homepage or docs root, e.g. https://docs.example.com/ or https://example.com/docs/. By default only pages under this path are included, which also lets you build a section-level llms.txt such as /docs/llms.txt.

## `maxPages` (type: `integer`):

How many pages to include. Each processed page is billed once; skipped pages (robots.txt, noindex, little text, near-duplicate copies under session IDs or crawl-trap URLs) are not. llms.txt lists every processed page; llms-full.txt contains their full text.

## `crawlScope` (type: `string`):

Only pages under the start URL's path (default), or the whole site.

## `includeGlobs` (type: `array`):

Optional path globs to include, e.g. /docs/\*\* or /guides/\*. \*\* matches any depth, \* one path segment. Empty = everything in scope.

## `excludeGlobs` (type: `array`):

Path globs to skip, e.g. /blog/\*\* or /tag/\*.

## `optionalGlobs` (type: `array`):

Pages matching these globs are listed under "## Optional" in llms.txt (secondary links agents may skip), e.g. /blog/\*\* or /changelog/\*\*.

## `sectionMode` (type: `string`):

Group links by their first path segment (Docs, Guides, API ...) or list everything in one section.

## `siteName` (type: `string`):

Overrides the detected site name used as the llms.txt H1.

## `siteSummary` (type: `string`):

Overrides the detected one-paragraph summary shown under the H1.

## `useSitemap` (type: `boolean`):

Discover pages from the sitemaps declared in robots.txt (or /sitemap.xml) before following links.

## `preferMarkdownLinks` (type: `boolean`):

If a page declares <link rel="alternate" type="text/markdown">, llms.txt links to that Markdown URL, as the llmstxt.org proposal recommends.

## `skipNoindex` (type: `boolean`):

Do not include pages marked with a robots noindex meta tag.

## `includeMarkdownInDataset` (type: `boolean`):

Store each page's Markdown in its dataset row (useful for RAG pipelines). Turn off for smaller datasets.

## `maxCharsPerPage` (type: `integer`):

Longer pages are truncated in llms-full.txt and the dataset.

## `maxConcurrency` (type: `integer`):

Pages fetched at the same time (1-5). A Crawl-delay in robots.txt forces 1.

## `requestTimeoutSecs` (type: `integer`):

Timeout for each HTTP request.

## Actor input object example

```json
{
  "startUrl": "https://docs.example.com/",
  "maxPages": 10,
  "crawlScope": "path",
  "includeGlobs": [],
  "excludeGlobs": [],
  "optionalGlobs": [],
  "sectionMode": "path",
  "useSitemap": true,
  "preferMarkdownLinks": true,
  "skipNoindex": true,
  "includeMarkdownInDataset": true,
  "maxCharsPerPage": 30000,
  "maxConcurrency": 3,
  "requestTimeoutSecs": 20
}
```

# Actor output Schema

## `llmsTxt` (type: `string`):

The generated llms.txt (text/plain). Upload it to your site root or section path.

## `llmsFullTxt` (type: `string`):

All processed pages as one Markdown document (text/plain).

## `pages` (type: `string`):

Dataset with one row per page: status, section, title, description and Markdown.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrl": "https://crawlee.dev",
    "maxPages": 10
};

// Run the Actor and wait for it to finish
const run = await client.actor("motivating_bracelet/suchi-anantasutra").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrl": "https://crawlee.dev",
    "maxPages": 10,
}

# Run the Actor and wait for it to finish
run = client.actor("motivating_bracelet/suchi-anantasutra").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrl": "https://crawlee.dev",
  "maxPages": 10
}' |
apify call motivating_bracelet/suchi-anantasutra --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,motivating_bracelet/suchi-anantasutra"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/cDPRGCEqt0fp5BeEF/builds/8mssKZZljoxpZhfVv/openapi.json