Pricing

Pay per usage

Go to Apify Store

/llms.txt Generator

Try for free

Developed by

Jakub Kopecký

The /llms.txt Generator 🕸️📄 extracts website content to create an llms.txt file for AI apps 🤖✨ like LLM fine-tuning and indexing. Output is available 📥 in the Key-Value Store for easy download and integration into workflows. 🚀

3.1 (2)

Pricing

Pay per usage

Issues response

1.1 hours

Last modified

3 months ago

Automation

Open source

Start URL

startUrlstringRequired

The URL from which the crawler will start to generate the /llms.txt file.

Max crawl depth

maxCrawlDepthintegerOptional

The maximum depth of the crawl. Default is 1.

Default value of this property is 1

Max crawl pages

maxCrawlPagesintegerOptional

The maximum number of pages to crawl. Default is 50.

Default value of this property is 50

LLMs.txt generator

antonio_espresso/llms-txt-generator

Generates a clean and structured markdown format for your AI Agents LLMs.txt file. Helps define AI agent interactions, improve SEO, and ensure compatibility across platforms. Ready to integrate into your website for better transparency and control.

Antonio Blago

🔥fireScraper AI Prompt Website Content Markdown Scraper

mohamedgb00714/fireScraper-AI-prompt-Website-Content-Markdown-Scraper

fireScrape AI is an advanced web scraper built with Crawlee and Puppeteer. It crawls websites, extracts meaningful content, converts it into Markdown, then runs your custom prompt on the extracted text—ideal for generating enriched datasets, summaries or analyses for LLMs and AI pipelines

mohamed el hadi msaid

5.0

Website Content to Markdown for LLM Training

easyapi/website-content-to-markdown-for-llm-training

🚀 Transform web content into clean, LLM-ready Markdown! 📘 Scrape multiple pages, extract main content, and convert to Markdown format. Perfect for AI researchers, data scientists, and LLM developers. Fast, efficient, and customizable. Supercharge your AI training data today! 🌐📝🧠

EasyApi

100

5.0

AI Website Content Markdown Scraper

quaking_pail/ai-website-content-markdown-scraper

This Apify Actor, "Website Content Crawler with Markdown Extraction," is designed to perform a comprehensive crawl of specified websites, extract their text content, convert it into Markdown format, and store it in a structured dataset. The extracted content is suitable for feeding LLMs.

AI_Builder

660

4.6

Parsera

parsera-labs/parsera

Extract data from any website using just a URL and column descriptions

Parsera Labs

377

4.0

Firecrawl MCP Server

mcp-servers/firecrawl-mcp-server

A Model Context Protocol (MCP) server implementation that integrates with Firecrawl MCP for web scraping capabilities

MCP servers

🔥 FireScrape AI Website Content Markdown Scraper

mohamedgb00714/fireScraper-AI-Website-Content-Markdown-Scraper

Advanced web scraper powered by Crawlee and Puppeteer — extracts website content, converts it to Markdown, and structures it for LLM training datasets.

mohamed el hadi msaid

118

3.8

🔥fireSummarize AI Summarize any Website Content

mohamedgb00714/fireScraper-AI-sammarize-Website-Content

fireSummarize is an AI-powered tool that scrapes any website using Crawlee and Puppeteer, extracts and converts content into Markdown, and then summarizes it using a custom prompt — perfect for generating clean, structured insights from any webpage.

mohamed el hadi msaid

5.0

Ai Ready Web Page To Markdown Converter

mustafa.irshaid.113/ai-ready-web-page-to-markdown-converter

Convert any webpage into structured Markdown and HTML using just a URL. Get the page title, link, and content—perfect for SEO, devs, and AI crawlers. Fast, clean, and ideal for repurposing or analysis. Start turning websites into Markdown instantly.