llms.txt Generator (Website to llms.txt & llms-full.txt) avatar

llms.txt Generator (Website to llms.txt & llms-full.txt)

Pricing

from $0.60 / 1,000 pages

Try for free
Go to Apify Store
llms.txt Generator (Website to llms.txt & llms-full.txt)

llms.txt Generator (Website to llms.txt & llms-full.txt)

Try for free

Turn any website or docs site into an llms.txt (sectioned links with one-line descriptions) and llms-full.txt (every page as clean Markdown) for AI assistants, RAG and AI search. Plain HTTP, sitemap-aware, robots.txt-friendly.

Pricing

from $0.60 / 1,000 pages

Rating

0.0

(0)

Developer

Murat Uzun

Murat Uzun

Maintained by Community

Actor stats

0

Bookmarked

2

Total users

1

Monthly active users

2 days ago

Last modified

Categories

AI

Developer tools

SEO tools

Share

What is llms.txt Generator?

llms.txt Generator turns any website or documentation site into the two files AI assistants and AI search engines read: llms.txt (the site's name, a one-line summary and a sectioned list of its pages with one-line descriptions, in the llmstxt.org format) and llms-full.txt (every page as clean Markdown in one file). Give it a start URL, and it reads the site's sitemap and links with plain HTTP, strips navigation and boilerplate, and writes both files plus one dataset row per page with its title, description, section and token count. $1 per 1,000 pages.

Use it to publish an llms.txt for your own site or product docs, to feed a competitor's or a library's documentation into ChatGPT, Claude or Cursor, to build a RAG knowledge base from a docs site, or to check how an AI will see your content.

What does llms.txt Generator produce?

Files (in the run's key-value store, linked from the Output tab):

FileContent
llms.txt# Site name, > summary, then ## Section headings with - [Page title](url): description lines; blog, changelog, legal and similar sections go under ## Optional
llms-full.txtThe same header, then each page's full Markdown under its title and source URL
SUMMARYPer site: pages, sections, token counts of both files, download links, and whether the site already publishes its own /llms.txt

With several websites in one run, each file is prefixed with its domain (docs.example.com-llms.txt).

Dataset (one row per page, the billed unit):

FieldDescription
site, urlWebsite and page
title, descriptionClean page title (site-name suffix removed) and one-line description
sectionllms.txt section the page is listed under
wordCount, tokensSize of the page content (tokens ≈ characters / 4)
markdownThe page as Markdown (turn off Markdown in the dataset to leave it out)

How to use llms.txt Generator

  1. Enter one or more Start URLs, e.g. https://crawlee.dev/js/docs/quick-start or your home page.
  2. Set Max pages per website (default 100).
  3. Run, then open Output and download llms.txt and llms-full.txt.
  4. To publish it on your site, upload llms.txt to your web root so it is served at https://yoursite.com/llms.txt.

A start URL inside a folder keeps the crawl in that folder: https://example.com/docs/intro reads only /docs pages, a home-page start URL reads the whole site. Set Include path prefixes to choose the folders yourself.

Example input

{
    "startUrls": [{ "url": "https://crawlee.dev/js/docs/quick-start" }],
    "maxPages": 200
}

Example output (llms.txt)

# Crawlee for JavaScript


> With this short tutorial you can start scraping with Crawlee in a minute or two. To learn more, read the Introduction.


## Docs


- [Quick Start](https://crawlee.dev/js/docs/quick-start): With this short tutorial you can start scraping with Crawlee in a minute or two. To learn more, read the Introduction.
- [Deployment guides](https://crawlee.dev/js/docs/3.10/deployment): Here you can find guides on how to deploy your crawlers to various cloud providers.
- [Add data to dataset](https://crawlee.dev/js/docs/3.10/examples/add-data-to-dataset): This example saves data to the default dataset. If the dataset doesn't exist, it will be created.

How much does it cost?

Pay per result: $0.001 per page read ($1 per 1,000), with volume discounts on paid Apify plans. Pages that fail to load are listed in the ERRORS record and cost nothing; the llms.txt files themselves are free. Set Maximum cost per run to cap spend.

Limits

  • Plain HTTP only: pages that render their text with JavaScript after loading come out short. Sites that serve their text in the HTML work (tested on the Docusaurus-based Crawlee and Apify docs).
  • Descriptions come from each page's meta description, or its first paragraph when the meta description is missing or the same site-wide text.
  • Tag, author, archive, search, login and cart pages are skipped because they add noise to llms.txt.
  • It respects robots.txt by default.

Using llms.txt Generator with AI agents

The Actor is pay-per-event with limited permissions, so AI agents can call it through Apify's MCP server (mcp.apify.com), for example with {"startUrls": [{"url": "https://docs.example.com"}], "maxPages": 50}, then read llms-full.txt from the run's key-value store. It also accepts url, urls, website, websites, and inputDatasetId + inputField to take URLs from another run.

FAQ

What is llms.txt? A proposed standard (llmstxt.org) for a Markdown file at /llms.txt that tells language models what a site contains and where its key pages are, the way robots.txt and sitemap.xml do for crawlers.

The site already has an llms.txt. Why generate one? SUMMARY tells you when it does. A generated one is still useful to compare coverage, or when you need llms-full.txt and the site does not publish it.

Part of the webdatatools web-intelligence suite — every Actor is pay-per-event, reads public data without a login, and returns one clean row per entity:

Browse the whole suite at webdatatools, or call ten of these Actors straight from Claude, Cursor or Cline with the webdatatools MCP server.

Website & domain intelligence

Content for AI, LLMs and RAG

Search, video and social

Leads, jobs and company data

Developer, app and research data

You might also like

Instagram Scraper avatar

Instagram Scraper

apify/instagram-scraper

Extract Instagram posts, reels, profiles, places, hashtags, carousels, and comments. Get data from Instagram using one or more Instagram URLs or search queries: content, context, metrics, metadata. Export scraped data, run the scraper via API, schedule and monitor runs or integrate with other tools.

User avatar

Apify

410K

4.7

(600)

Google Maps Scraper avatar

Google Maps Scraper

compass/crawler-google-places

Extract data from thousands of Google Maps locations and businesses, including reviews, reviewer details, images, contact info, including full name, email, and job title, opening hours, prices & more. Export data, run via API, schedule and monitor runs, or integrate with other tools.

User avatar

Compass

625K

4.7

(1,816)

TikTok Scraper avatar

TikTok Scraper

clockworks/tiktok-scraper

Extract data from TikTok videos, hashtags, and users. Use URLs or search queries to scrape TikTok profiles, hashtags, posts, URLs, shares, followers, hearts, names, video, and music-related data. Export scraped data, run the scraper via API, schedule and monitor runs or integrate with other tools.

User avatar

Clockworks

296K

4.8

(370)

Instagram Profile Scraper avatar

Instagram Profile Scraper

apify/instagram-profile-scraper

Scrape all Instagram profile info. Just add Instagram usernames, IDs or URLs and extract name, join date, number of followers, location, bio, website, related profiles, video&post count, latest posts. Export scraped data, schedule scraper via API, and integrate with other tools or AI workflows.

User avatar

Apify

231K

4.7

(175)

Instagram Post Scraper avatar

Instagram Post Scraper

apify/instagram-post-scraper

Scrape Instagram posts. Just add one or more Instagram usernames and get your data in seconds including caption, metrics, images, mentions, coauthors, recent comments, sponsored status, video duration, views. Export scraped data, schedule scraper via API, integrate with other tools or AI workflows.

User avatar

Apify

133K

4.3

(139)

Google Search Results Scraper avatar

Google Search Results Scraper

apify/google-search-scraper

Scrape Google Search Engine Results Pages (SERPs). Select the country or language and extract organic and paid results, AI Mode, AI overviews, ads, queries, People Also Ask, prices, reviews, like a Google SERP API. Export data, run the scraper via API, schedule runs, or integrate with other tools.

User avatar

Apify

193K

4.6

(218)

LinkedIn Profile Scraper + Email ✅ No Cookies avatar

LinkedIn Profile Scraper + Email ✅ No Cookies

harvestapi/linkedin-profile-scraper

Extract detailed information from LinkedIn profiles in bulk, including complete work experience, education history, skills and more. No cookies or account required.

User avatar

HarvestAPI

70K

4.6

(92)

Facebook Posts Scraper avatar

Facebook Posts Scraper

apify/facebook-posts-scraper

Extract posts, videos, and engagement metrics from Facebook pages. Get text captions, reactions, video transcripts, images, external links, collaborators, and more from Facebook pages and profiles. Export ad data, schedule runs via API, and integrate with other tools or AI workflows.

User avatar

Apify

115K

4.6

(234)

Instagram Reel Scraper avatar

Instagram Reel Scraper

apify/instagram-reel-scraper

Scrape or download Instagram reels. Just add Instagram profile or reel URL, and get caption, timestamp, transcript, hashtags, mentions, tagged users, comments, likes, shares, views, duration, and downloaded video. Export scraped data, schedule via API, and integrate with other tools or AI workflows.

User avatar

Apify

153K

4.4

(103)

YouTube Scraper avatar

YouTube Scraper

streamers/youtube-scraper

YouTube crawler and video scraper. Alternative YouTube API with no limits or quotas. Extract and download channel name, likes, number of views, and number of subscribers.

User avatar

Streamers

128K

4.8

(196)