Pricing

Pay per usage

Go to Store

/llms.txt Generator

Try for free

Developed by

Jakub Kopecký

The /llms.txt Generator 🕸️📄 extracts website content to create an llms.txt file for AI apps 🤖✨ like LLM fine-tuning and indexing. Output is available 📥 in the Key-Value Store for easy download and integration into workflows. 🚀

2.7 (2)

Pricing

Pay per usage

Total users

348

Monthly users

128

Runs succeeded

88%

Issues response

1.1 hours

Last modified

a month ago

Automation

Open source

Start URL

startUrlstringRequired

The URL from which the crawler will start to generate the /llms.txt file.

Max crawl depth

maxCrawlDepthintegerOptional

The maximum depth of the crawl. Default is 1.

Default value of this property is 1

Max crawl pages

maxCrawlPagesintegerOptional

The maximum number of pages to crawl. Default is 50.

Default value of this property is 50

AI Website Content Markdown Scraper

quaking_pail/ai-website-content-markdown-scraper

This Apify Actor, "Website Content Crawler with Markdown Extraction," is designed to perform a comprehensive crawl of specified websites, extract their text content, convert it into Markdown format, and store it in a structured dataset. The extracted content is suitable for feeding LLMs.

AI_Builder

579

3.8

Website Content to Markdown for LLM Training

easyapi/website-content-to-markdown-for-llm-training

🚀 Transform web content into clean, LLM-ready Markdown! 📘 Scrape multiple pages, extract main content, and convert to Markdown format. Perfect for AI researchers, data scientists, and LLM developers. Fast, efficient, and customizable. Supercharge your AI training data today! 🌐📝🧠

EasyApi

5.0

AI Web Scraper - Powered by Crawl4AI

raizen/ai-web-scraper

A blazing-fast AI web scraper powered by Crawl4AI. Perfect for LLMs, AI agents, AI automation, model training, sentiment analysis, and content generation. Supports deep crawling, multiple extraction strategies and flexible output (Markdown/JSON). Seamlessly integrates with Make.com, n8n, and Zapier.

Raizen Technology

122

1.0

LLMs.txt generator

antonio_espresso/llms-txt-generator

Generates a clean and structured markdown format for your AI Agents LLMs.txt file. Helps define AI agent interactions, improve SEO, and ensure compatibility across platforms. Ready to integrate into your website for better transparency and control.

Antonio Blago

Real-time knowledge for LLMs

ai-developer/real-time-knowledge-for-llms

Unlock the Full Potential of Your LLM with Real-Time Web Knowledge! Say goodbye to outdated responses, misinformation, and hallucinations. Now you can ground your Language Model with the freshest information from the web. NO API NEEDED!

aideveloper

Browser Use Apify

lexis-solutions/browser-use-apify

Open-source AI-powered browser automation based on browser-use and hosted on Apify. Run any task using Apify's platform and LLMs like ChatGPT, Claude, etc. Easy, scalable, resilient, and hosted solution for web-enabled AI agents.

Lexis Solutions

151

5.0

Steam Store Scraper

m0uka/steam-store-scraper

Steam Store Scraper crawls the Steam front page and extracts data from individual Steam Store pages. The extracted data is stored in a dataset, from where you can extract it in various formats, such as JSON, XML, CSV, and Excel.

Lukas Moucka

296

Ai Web Scraper - Natural language and Vision scraper

eloquent_mountain/ai-universal-web-scraper-natural-language

Powerful AI Web Scraper using Google's Gemini Vision. Specify data extraction in natural language. Supports infinite scroll, above-the-fold analysis, automatic cookie consent, pay-per-event pricing, and screenshot storage for debugging.

Paco

GPT Search

tri_angle/gpt-search

Send queries to ChatGPT and retrieve structured answers with full source citations. Easily integrate into your tools or workflows for flexible, scalable AI-powered solutions.