Pricing

from $1.00 / 1,000 page processeds

Try for free

Go to Apify Store

llms.txt Generator: crawl a site, write llms.txt and full text

Try for free

Crawls each site up to a page limit, respecting robots.txt, and writes an llms.txt file in the format of the llms.txt proposal, with an optional llms-full.txt holding the main text of every page as Markdown.

Pricing

from $1.00 / 1,000 page processeds

Rating

0.0

(0)

Developer

Ed Rayyis

Actor stats

Bookmarked

Total users

Monthly active users

9 days ago

Last modified

Categories

SEO tools

Security

You can access the llms.txt Generator: crawl a site, write llms.txt and full text programmatically from your own applications by using the Apify API. You can also choose the language preference from below. To use the Apify API, you’ll need an Apify account and your API token, found in API & Integrations in Apify Console.

Python

JavaScript

CLI

OpenAPI

HTTP

MCP

1from apify_client import ApifyClient
2
3# Initialize the ApifyClient with your Apify API token
4# Replace '<YOUR_API_TOKEN>' with your token.
5client = ApifyClient("<YOUR_API_TOKEN>")
6
7# Prepare the Actor input
8run_input = { "startUrls": [{ "url": "https://docs.apify.com/" }] }
9
10# Run the Actor and wait for it to finish
11run = client.actor("examsens.ai/llms-txt-generator").call(run_input=run_input)
12
13# Fetch and print Actor results from the run's dataset (if there are any)
14print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
15for item in client.dataset(run.default_dataset_id).iterate_items():
16    print(item)
17
18# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

llms.txt Generator: Crawl a Site, Write llms.txt API in Python

The Apify API client for Python is the official library that allows you to use llms.txt Generator: crawl a site, write llms.txt and full text API in Python, providing convenience functions and automatic retries on errors.

Install the apify-client

$pip install apify-client

Other API clients include:

llms.txt Generator: crawl a site, write llms.txt and full text API in JavaScript

llms.txt Generator: crawl a site, write llms.txt and full text API through CLI

llms.txt Generator: crawl a site, write llms.txt and full text OpenAPI definition

llms.txt Generator: crawl a site, write llms.txt and full text API

LLMs.txt File Generator

justa/llms-txt-file-generator

Generate an llms.txt file from a website sitemap. Crawls all URLs, extracts titles and meta descriptions, and creates a Markdown-formatted file following the llms.txt specification. Upload then the output of your file directly on your website (Webflow, Wordpress etc.)

Benoit Eveillard

llms.txt Generator: llms-full.txt for Any Website (AI SEO)

fayoussef/llms-txt-generator

Generate a spec-compliant llms.txt and llms-full.txt for any website in one run. Crawls your sitemap or internal links, extracts every page's title, meta description and full Markdown content, and outputs two ready-to-upload files for ChatGPT, Claude, Perplexity and Gemini.

Youssef Farhan

LLMs.txt Checker - AI Readiness Scanner

alizarin_refrigerator-owner/llms-txt-checker

Batch check websites for llms.txt files & identify AI optimization opportunities. A emerging standard that websites use to communicate with AI systems. It helps LLMs understand, What the company does, How to present information about the business, Contact information & key pages Brand guidelines

The Howlers

llms.txt Generator: Make Any Website LLM-Ready

f0rty7even/llms-txt-generator

Crawl a website and generate an llms.txt file (the emerging standard that helps LLMs and AI agents understand your site). Optionally build llms-full.txt with full page content. Output saved to the key-value store.

Michael Yousrie

llms.txt Generator

exalted_mud_vjz/llms-txt-generator

Generate a ready to publish llms.txt file for any website, the emerging standard that tells AI assistants what your site is about and which pages matter.

Asish Singh

LLM-Ready Web Scraper

devoted_helix/llm-web-scraper

Convert web pages to clean, LLM-friendly text. Perfect for RAG pipelines, AI chatbot training, and fine-tuning datasets. Removes ads,menus, and clutter automatically.

batuhan senavci

Website Content Crawler — Text, Markdown & HTML for AI/LLM

hichemdev/website-content-crawler

Crawl any website and extract clean text, Markdown, and HTML from every page — ready for LLM, RAG, and AI ingestion.

Hichem Ben Moussa

llms.txt Checker: AI Search Readiness Audit

apifmcpfactory/llms-txt-checker

Audits whether a website is readable by AI assistants (llms.txt, robots.txt AI-crawler access, sitemap) and returns a 0-100 score with concrete fixes - use when asked whether a site is visible to ChatGPT/Claude or how to improve that; $0.005 per site audited.

Howth Technology Factory

llms.txt + llms-full.txt Generator for Any Docs Site

ianymu/llms-txt-converter

llms-txt-converter is an Apify Actor that crawls any documentation site and outputs the two GEO-standard files (llms.txt and llms-full.txt) used by ChatGPT, Claude, Perplexity, Gemini, and Bing Copilot to find and cite your docs.

Yanlong Mu

📄 llms.txt Generator & Validator — AI Search Visibility

upstanding_biobot/llms-txt-generator

Generate llms.txt files for AI search visibility. Fetches sitemap, validates structure, checks AI bot access (OAI-SearchBot, PerplexityBot, ClaudeBot, Google-Extended). 2 blue ocean competitors only.