Suchi.Anantasutra: llms.txt & llms-full.txt Generator

Suchi (सूची) is Sanskrit for "index" or "list". This Actor crawls your website politely and writes the two files AI agents look for: llms.txt, a compact Markdown index of your key pages (llmstxt.org format), and llms-full.txt, the clean Markdown text of those pages in one document. It is part of the Anantasutra family of tools.

Who it is for

• Site and shop owners who want AI assistants (ChatGPT, Claude, Perplexity, Cursor ...) to find and quote the right pages, without writing llms.txt by hand.
• Documentation teams who want a section-level file such as /docs/llms.txt for their docs only.
• Agencies and developers who need clean Markdown per page for RAG pipelines (every page is also a dataset row).

How it works

1. Reads your robots.txt and obeys it for its own user agent (LlmsTxtGenerator) or *, including Crawl-delay (up to 10 seconds, one request at a time).
2. Fetches the start page first; its links are treated as your main navigation.
3. Reads your sitemap (declared in robots.txt, or /sitemap.xml), then follows same-site links breadth first. Blog, tag, archive and old-version pages are crawled last.
4. Extracts the main content of each page as clean Markdown (navigation, footers, cookie banners and scripts removed) and groups pages into sections by their first path segment (Docs, Guides, API ...).
5. Saves llms.txt and llms-full.txt to the run's key-value store, checks llms.txt against the spec, and pushes one dataset row per page considered.

Pages are skipped (and not charged) when robots.txt disallows them, they are marked noindex, they have little text without JavaScript, they redirect to another site, they are not HTML, or they are near-duplicates of a page already included (see "Crawl traps" below). Login, search, cart and checkout paths are never included.

Input

{
  "startUrl": "https://docs.example.com/",
  "maxPages": 50,
  "crawlScope": "path",
  "optionalGlobs": ["/blog/**"],
  "excludeGlobs": ["/tag/*"]
}

• startUrl: home page or docs root. By default only pages under this path are included; set crawlScope to site for the whole website.
• maxPages: 1-500. Each included page is billed once.
• includeGlobs / excludeGlobs / optionalGlobs: path filters; optionalGlobs pages go under ## Optional.
• siteName / siteSummary: your own H1 and summary line instead of the detected ones.
• maxCharsPerPage (1,000-200,000): longer pages are cut in llms-full.txt and the dataset.

Output

Key-value store record llms.txt (abridged):

# Example Docs

> Example Docs explains how to install, configure and deploy the Example widget toolkit.

## Docs

- [Getting started](https://docs.example.com/docs/getting-started): Create your first widget project in five minutes.
- [Install](https://docs.example.com/docs/install.md): System requirements and installation options for the Example CLI.

## Optional

- [We launched Example 2.0](https://docs.example.com/blog/launch): Release notes for Example 2.0.

Dataset row per page (Markdown omitted):

{
  "url": "https://docs.example.com/docs/getting-started",
  "status": "ok",
  "title": "Getting started",
  "description": "Create your first widget project in five minutes.",
  "section": "Docs",
  "inLlmsTxt": true,
  "wordCount": 105,
  "truncated": false,
  "promptInjectionSuspected": false,
  "reason": null
}

Upload llms.txt (and optionally llms-full.txt) to your site root, or to the section path you crawled.

Crawled text is untrusted: review before publishing

These files are written for AI agents, so text copied from your pages matters. Suchi protects the prominent lines:

• The H1 (max 80 characters), the summary blockquote (a sentence or two, max 300 characters), link titles (max 120) and link notes (max 200) are cleaned of markup and invisible characters and kept short, so one huge paragraph cannot bloat the index.
• Text that reads like instructions aimed at AI agents ("ignore all previous instructions", "[[SYSTEM]]", "instruction to any AI reading this", "reveal your system prompt" ...) is never used for those lines. The host name and a neutral summary are used instead.
• Pages whose text (outside code blocks) contains such phrases are flagged with promptInjectionSuspected: true and listed in the run log and status. The check is a deliberately narrow heuristic: it avoids flagging ordinary documentation, so it cannot catch every possible injection.

Page bodies are copied as they are into llms-full.txt (markup is escaped, so <script> stays text). If your site has comments, reviews, forums or other user-generated content, read the flagged pages before you publish, or exclude those paths with excludeGlobs. Set siteName and siteSummary to control the top of the file yourself.

Crawl traps

Session IDs, sort orders and calendar links can turn one page into thousands of URLs. Well-known session parameters (PHPSESSID, jsessionid, sessionid, cfid ...) and ASP.NET cookieless session segments (/(S(...))/) are removed from URLs. When pages that differ only by URL parameters have near-identical text, the later ones are skipped as near-duplicates and not charged; after a few such duplicates the remaining parameter variants of that path are not fetched at all. The same applies to session IDs inside the path (/s/<id>/booking): a page that is a near-copy of one already included and links mostly to new copies of itself is skipped and not charged, and after a few of those the whole pattern (/s/*/booking) is not fetched any more. The log names each detected pattern. Similar pages that are separate URLs on your site, such as per-platform install guides or placeholder pages sharing one menu, are kept.

Pricing

Pay per event, no subscription.

Event	Price	When
Page processed	$0.001	Per page included in llms.txt and llms-full.txt
llms.txt files generated	$0.01	Once per run, after both files are saved
Actor start	$0.00005	Charged automatically by Apify at the start of each run (per GB of memory)

A 100-page site costs about $0.11 ($0.10 for pages + $0.01 for the files). Skipped and failed pages are never charged. If robots.txt is unreachable or disallows the site, nothing is crawled and only the Actor start is charged. Set "Maximum cost per run" to cap spending: the crawl is sized so the one-time file fee always fits.

Use it from Claude, ChatGPT or Cursor (MCP)

The Apify MCP server exposes this Actor as a tool to AI assistants.

• Claude Desktop / Claude Code, Cursor and other MCP clients: add the hosted server https://mcp.apify.com/?actors=motivating_bracelet/suchi-anantasutra and sign in with your Apify account, or run it locally with npx -y @apify/actors-mcp-server --actors motivating_bracelet/suchi-anantasutra and your Apify API token in the APIFY_TOKEN environment variable.
• ChatGPT: add https://mcp.apify.com as a custom connector (MCP) where your plan allows it.

Then ask, for example: "Generate an llms.txt for docs.example.com with Suchi, at most 30 pages." The assistant passes startUrl and maxPages and can read the llms.txt record from the run's key-value store.

Limits

• Up to 500 pages per run and 5 parallel requests; one request at a time when robots.txt sets a Crawl-delay.
• Only public websites over http or https. Private, loopback, link-local and other reserved addresses are refused, including through DNS names, redirects and sitemap links.
• Raw HTML only (no browser): pages that need JavaScript to show their text are skipped as "little text".
• Page downloads are capped at 3 MB, llms-full.txt at 4,000,000 characters, robots.txt at 500 KiB.
• The crawl stops shortly before the run timeout and saves what it has.

FAQ

Why are some pages missing? Look at the dataset rows with status: skipped or error: the reason column says why (robots.txt, noindex, little text, near-duplicate, redirect to another site, page limit).

Why does my llms.txt start with the host name instead of my site name? The detected title was missing or looked like instructions aimed at AI agents. Set siteName (and siteSummary) in the input.

Is it allowed to crawl my site? It only crawls what your robots.txt allows for LlmsTxtGenerator or *, and identifies itself honestly. Run it on sites you own or are allowed to index.

Do I pay for pages that were skipped? No. Only pages that end up in the files are charged, plus the one-time file fee when the files are saved.

How can I check the result? Audit your site with the sibling Actor Drishti.Anantasutra; it validates llms.txt and shows which AI crawlers can reach your pages.