llms.txt Generator (Website to llms.txt & llms-full.txt)

Created by

Murat Uzun

Actor

llms.txt Generator (Website to llms.txt & llms-full.txt)

Turn any website or docs site into an llms.txt (sectioned links with one-line descriptions) and llms-full.txt (every page as clean Markdown) for AI assistants, RAG and AI search. Plain HTTP, sitemap-aware, robots.txt-friendly.

Try for free

llms.txt Generator (Website to llms.txt & llms-full.txt)webdatatools/llms-txt-generator

Site

Section

Title

Description

+2 fields

Input

Start URLs

url:https://crawlee.dev/js/docs/quick-start

Max pages per website:50

Max link depth:3

Include path prefixes

Exclude URL patterns:\.(png|jpe?g|gif|svg|webp|pdf|zip|mp4|css|js)$+3

Seed from sitemap.xml:true

Markdown in the dataset:true

Extra CSS selectors to remove

Max concurrency:10

Respect robots.txt:true

Proxy:{1 field}

Output fields

Site

Section

Title

Description

Tokens

Link

How it works

Sign up on Apify or log in

The llms.txt Generator (Website to llms.txt & llms-full.txt) Actor is ready to run with the inputs already set for "llms.txt Generator (Website to llms.txt & llms-full.txt)". Sign up or log in and you'll land in Apify Console with a copy of this exact task.

Start the "llms.txt Generator (Website to llms.txt & llms-full.txt)" task

The llms.txt Generator (Website to llms.txt & llms-full.txt) Actor will start running based on the input automatically. You can also change all the input fields, like start urls, max pages per website, and more, to match your specific use case.

Receive the "llms.txt Generator (Website to llms.txt & llms-full.txt)" output

Monitor the progress in real time. You will be notified as soon as your dataset is complete and ready for review. Each result includes data such as site, section, title, and more.