Pricing

Pay per usage

llms.txt Generator — Make Any Website AI-Readable

Crawl a website and generate llms.txt + llms-full.txt for AI/LLM discoverability and RAG pipelines. Batch, API-callable, pay-per-page.

Pricing

Pay per usage

Rating

0.0

(0)

Developer

Cenk KURTOĞLU

Actor stats

Bookmarked

Total users

Monthly active users

2 hours ago

Last modified

llms.txt Generator — Make Any Website AI-Readable (llms.txt + llms-full.txt)

Generate llms.txt and llms-full.txt for any website — the emerging standard that tells AI models (ChatGPT, Claude, Perplexity, Gemini) what your site is about and how to reference it. Think robots.txt, but for LLMs.

Point this Actor at a URL. It crawls the site, extracts titles, descriptions, and clean main content, and produces two downloadable files ready to drop at your domain root.

Why llms.txt matters in 2026

RAG is the default architecture for AI apps — llms.txt files are pre-structured, information-dense input made for retrieval pipelines.
Google added llms.txt to Chrome Lighthouse's "Agentic Browsing" audit (May 2026) as an AI-readiness check.
AI search visibility (GEO/AEO) increasingly depends on giving models a clean, structured map of your content.

What this Actor does

Crawls a website (same-domain links, up to your maxPages limit).
Extracts title, meta description, and clean main content (<article>/<main> aware, strips nav/footer/scripts).
Builds llms.txt — a concise, linked index of your pages.
Builds llms-full.txt — the full text of every page for complete LLM/RAG ingestion.
Pushes a per-page dataset (URL, title, description) you can export as JSON/CSV.

Built for developers & scale (not just one page)

Unlike one-off web UI tools, this Actor is API-callable, batch-friendly, and pipeline-ready:

Generate llms.txt for hundreds of client sites programmatically.
Wire it into a content/RAG pipeline (n8n, cron, CI) via the Apify API.
Pay only for pages processed.

Input

Field	Type	Default	Description
`websiteUrl`	string	—	Site to crawl (required).
`maxPages`	integer	50	Max pages processed (controls cost/runtime).
`includeFullText`	boolean	true	Also generate `llms-full.txt`.
`maxContentCharsPerPage`	integer	12000	Truncate very long pages in the full file.

Output

Key-value store: llms.txt and llms-full.txt (download from the run's Storage tab).
Dataset: one record per page (url, title, description, contentChars).

Example

{
  "websiteUrl": "https://example.com",
  "maxPages": 100,
  "includeFullText": true
}

Then place the resulting llms.txt at https://example.com/llms.txt (root, like robots.txt).

Use cases

AI SEO / GEO: make your site discoverable and quotable by AI assistants.
RAG ingestion: turn any docs/marketing site into clean, chunk-ready text.
Agencies: batch-generate llms.txt for every client site.
AI agents: give agents a structured, low-noise view of a domain.

Tip: run on your documentation or marketing site first, review llms.txt, then publish it at your domain root.

llms.txt Generator - Make Your Site AI-Discoverable

dltik/llms-txt-generator

Generate a standards-compliant llms.txt file for any website. Crawls your sitemap, extracts page titles & descriptions, groups them into sections, and outputs a clean llms.txt that helps ChatGPT, Claude, Gemini & Perplexity understand your site. Optional AI enrichment.

Walid

📄 llms.txt Generator & Validator — AI Search Visibility

upstanding_biobot/llms-txt-generator

Generate llms.txt files for AI search visibility. Fetches sitemap, validates structure, checks AI bot access (OAI-SearchBot, PerplexityBot, ClaudeBot, Google-Extended). 2 blue ocean competitors only.

Alexander Maksimchuk

Ai Visibility Suite - Dark Visitors Alternative

alizarin_refrigerator-owner/ai-visibility-suite---dark-visitors-alternative

Comprehensive AI bot monitoring, robots.txt analysis, LLMs.txt generation & AI shopping optimization. Monitor AI crawlers visits, check AI compliance, generate AI-friendly configurations, and optimize for AI shopping agents. AI Bot Directory Robots.txt LLMs.txt AI Shopping Competitor AI Audit

The Howlers

Site to llms.txt Generator

devsef/site-to-llms-txt

Generate a complete llms.txt file for any website in one run. Crawls up to 200 same-origin pages, extracts titles and meta descriptions, and outputs a clean, spec-compliant llms.txt that makes your site readable for AI assistants and agents.

Steffano van Hoven

Html to Markdown Converter

antonio_espresso/html-to-markdown-converter

Crawl a target URL and convert its HTML content into clean, structured Markdown with optional heading-based chunking.

Antonio Blago

Website Metadata Extractor(sitemap, socialLinks, robotsTxt)

codescraper/website-metadata-extractor

A very fast metadata extractor to get all meta tags, robots.txt, sitemaps, social links, H1s, word count, and JSON-LD data. Also provides technology detection for a full analysis. Get your data fast for just $3/month.

CodeScraper

101

5.0

(2)

Apify Actor Discoverability & SEO Audit

shelvick/apify-discoverability-audit

Audit an Apify Actor for discoverability by LLM agents and Apify Store ranking. Scores description quality, schema, disambiguation, agentic-pay, maintenance, and live agent-search retrieval across 10 dimensions, returning copy-paste-ready fixes so agents can find and call it as a tool.

Scott Helvick

Company Deep Research — SEC, GitHub, DNS & Social

ryanclinton/company-deep-research

Generate comprehensive company research reports from 7+ sources: SEC filings, stock data, Wikipedia, GitHub, Trustpilot reviews, DNS records, and social media verification. One domain in, full intelligence report out.

Ryan Clinton

Firecrawl MCP

red.cars/firecrawl-mcp

AI agents that need web data without anti-bot headaches. 20 tools for API-based web scraping, crawl, search, and extract — no proxy rotation, no stealth needed.

AutomateLab

LLMs.txt Generator

onescales/the-llms-txt-generator

The most powerful LLMs.txt Generator tool online. Generates LLMs.txt , llms-full.txt and markdown .md files within seconds! Get your website discovered, and recommended by ChatGPT, Claude, Google Gemini, Perplexity, Grok, and every AI. (Great for AEO, AIO, GEO, AI SEO) Made by Hi LLMs

One Scales

118

5.0

(2)

LLMs.txt File Generator

justa/llms-txt-file-generator

Generate an llms.txt file from a website sitemap. Crawls all URLs, extracts titles and meta descriptions, and creates a Markdown-formatted file following the llms.txt specification. Upload then the output of your file directly on your website (Webflow, Wordpress etc.)