Pricing

Pay per usage

Go to Store

/llms.txt Generator

Try for free

Developed by

Jakub Kopecký

The /llms.txt Generator 🕸️📄 extracts website content to create an llms.txt file for AI apps 🤖✨ like LLM fine-tuning and indexing. Output is available 📥 in the Key-Value Store for easy download and integration into workflows. 🚀

2.7 (2)

Pricing

Pay per usage

Total users

248

Monthly users

Runs succeeded

83%

Issues response

1.8 days

Last modified

6 days ago

Automation

Open source

You can access the /llms.txt Generator programmatically from your own applications by using the Apify API. You can also choose the language preference from below. To use the Apify API, you’ll need an Apify account and your API token, found in Integrations settings in Apify Console.

Python

JavaScript

CLI

OpenAPI

HTTP

MCP

$# Start Server-Sent Events (SSE) session and keep it running
<curl "https://actors-mcp-server.apify.actor/sse?token=<YOUR_API_TOKEN>&actors=jakub.kopecky/llmstxt-generator"

# Session id example output:
# event: endpoint
# data: /message?sessionId=9d820491-38d4-4c7d-bb6a-3b7dc542f1fa

Using /llms.txt Generator via Model Context Protocol (MCP) server

MCP server lets you use /llms.txt Generator within your AI workflows. Send API requests to trigger actions and receive real-time results. Take the received sessionId and use it to communicate with the MCP server. The message starts the /llms.txt Generator Actor with the provided input.

$curl -X POST "https://actors-mcp-server.apify.actor/message?token=<YOUR_API_TOKEN>&session_id=<SESSION_ID>" -H "Content-Type: application/json" -d '{
$  "jsonrpc": "2.0",
$  "id": 1,
$  "method": "tools/call",
$  "params": {
$    "arguments": {
$      "startUrl": "https://docs.apify.com/cli/docs"
$},
$    "name": "jakub.kopecky/llmstxt-generator"
$  }
$}'

The response should be: Accepted. You should received response via SSE (JSON) as:

$event: message
$data: {
$  "result": {
$    "content": [
$      {
$        "type": "text",
$        "text": "ACTOR_RESPONSE"
$      }
$    ]
$  }
$}

Configure local MCP Server via standard input/output for /llms.txt Generator

You can connect to the MCP Server using clients like ClaudeDesktop and LibreChat or build your own. The server can run both locally and remotely, giving you full flexibility. Set up the server in the client configuration as follows:

{
  "mcpServers": {
    "actors-mcp-server": {
      "command": "npx",
      "args": [
        "-y",
        "@apify/actors-mcp-server",
        "--actors",
        "jakub.kopecky/llmstxt-generator"
      ],
      "env": {
        "APIFY_TOKEN": "<YOUR_API_TOKEN>"
      }
    }
  }
}

You can further access the MCP client through the Tester MCP Client, a chat user interface to interact with the server.

To get started, check out the documentation and example clients. If you are interested in learning more about our MCP server, check out our blog post.

Real-time knowledge for LLMs

ai-developer/real-time-knowledge-for-llms

Unlock the Full Potential of Your LLM with Real-Time Web Knowledge! Say goodbye to outdated responses, misinformation, and hallucinations. Now you can ground your Language Model with the freshest information from the web. NO API NEEDED!

aideveloper

Website Content to Markdown for LLM Training

easyapi/website-content-to-markdown-for-llm-training

🚀 Transform web content into clean, LLM-ready Markdown! 📘 Scrape multiple pages, extract main content, and convert to Markdown format. Perfect for AI researchers, data scientists, and LLM developers. Fast, efficient, and customizable. Supercharge your AI training data today! 🌐📝🧠

EasyApi

5.0

AI Web Scraper - Powered by Crawl4AI

raizen/ai-web-scraper

A blazing-fast AI web scraper powered by Crawl4AI. Perfect for LLMs, AI agents, AI automation, model training, sentiment analysis, and content generation. Supports deep crawling, multiple extraction strategies and flexible output (Markdown/JSON). Seamlessly integrates with Make.com, n8n, and Zapier.

Raizen Technology

1.0

Extract-any-webpage-content-for-llm

ai-developer/extract-any-webpage-content-for-llm

Fast and easy way to extract data from any webpage and are LLM friendly. The tool lets you easily extract content from any website. Ideal for researchers, marketers, and developers.

aideveloper

450

AI Website Content Markdown Scraper

quaking_pail/ai-website-content-markdown-scraper

This Apify Actor, "Website Content Crawler with Markdown Extraction," is designed to perform a comprehensive crawl of specified websites, extract their text content, convert it into Markdown format, and store it in a structured dataset. The extracted content is suitable for feeding LLMs.

AI_Builder

553

3.8

Crawl4AI

janbuchar/crawl4ai

Wraps the Crawl4AI open-source library for retrieving text content from websites.

Jan Buchar

385

5.0

Auto GPT

lukaskrivka/auto-gpt

Run Auto GPT sessions directly on Apify. No OpenAI account or API token is required! Store parsed thoughts into datasets for later analysis.

Lukáš Křivka

196

Gemini AI Scraper

jupri/google-bard

Interact with Gemini AI formerly (Google Bard) and save conversation to dataset

cat

🔥 FireScrape AI Website Content Markdown Scraper

mohamedgb00714/fireScraper-AI-Website-Content-Markdown-Scraper

Advanced web scraper powered by Crawlee and Puppeteer — extracts website content, converts it to Markdown, and structures it for LLM training datasets.

mohamed el hadi msaid

5.0

Ui Generator

tinkerbrains/ui-generator

Get UI designs for your app ideas. Get HTML + TailwindCSS code and a full screenshot of the design in output data. You will need a Anthropic Claude API Key to add in environment variables.