llms.txt Generator — Make Any Website AI-Readable
Pricing
Pay per usage
llms.txt Generator — Make Any Website AI-Readable
Crawl any website and generate llms.txt and llms-full.txt files following the llms.txt standard. BFS crawler with URL filtering and markdown/plaintext output. For AI readiness, SEO for AI, documentation portals, and archiving.
Pricing
Pay per usage
Rating
0.0
(0)
Developer
Oaida Adrian
Maintained by CommunityActor stats
0
Bookmarked
1
Total users
0
Monthly active users
3 hours ago
Last modified
Share
Crawl any website and generate llms.txt and llms-full.txt files following the llms.txt standard.
What is llms.txt?
The llms.txt convention is a proposed standard for providing AI models and LLMs with structured information about a website. Similar to robots.txt, but for AI readability.
Features
- Full website crawl — BFS crawler with configurable depth and page limits
- Content extraction — Uses trafilatura for precise main content extraction
- Dual output — Generates both
llms.txt(summary) andllms-full.txt(complete content) - URL filtering — Include/exclude URL patterns with glob syntax
- Markdown or plaintext — Choose output format for compatibility
- Structured output — Page-level dataset with titles, descriptions, and word counts
Input Parameters
| Parameter | Type | Default | Description |
|---|---|---|---|
startUrls | array | required | URLs to start crawling from |
maxPages | integer | 50 | Maximum pages to crawl |
globs | array | [] | Include URL patterns (e.g. /docs/**) |
excludes | array | [] | Exclude URL patterns (e.g. /blog/**) |
outputFormat | string | "markdown" | Content format: markdown or plaintext |
proxyConfiguration | object | Apify proxy | Proxy settings |
Output
| Output | Description |
|---|---|
| llms.txt (KV store) | Summary file with page titles, descriptions, and links |
| llms-full.txt (KV store) | Full content file with all extracted text |
| Dataset | Per-page records with URL, title, description, content, word count |
Use Cases
- AI/LLM readiness — Make your website content available to AI assistants
- SEO for AI — Ensure AI models can understand and reference your content
- Documentation portals — Generate AI-readable docs for your product
- Content archiving — Create structured snapshots of website content
Pricing
This actor uses pay-per-event pricing. You are charged per page successfully crawled and processed.
