Pricing

from $0.50 / 1,000 pages

Try for free

Go to Apify Store

llms.txt Generator + Docs to Markdown & RAG Chunks

Try for free

Generate llms.txt and llms-full.txt for a public documentation website, plus clean LLM-ready Markdown per page and heading-aware RAG chunks with token estimates, in one run. Polite docs crawler/scraper: same-origin, robots.txt, rate limits, no logins. Pay per page. Callable by AI agents via MCP.

Pricing

from $0.50 / 1,000 pages

Rating

0.0

(0)

Developer

Lathesmith

Actor stats

Bookmarked

Total users

Monthly active users

9 days ago

Last modified

Categories

Developer tools

SEO tools

You can access the llms.txt Generator + Docs to Markdown & RAG Chunks programmatically from your own applications by using the Apify API. You can also choose the language preference from below. To use the Apify API, you’ll need an Apify account and your API token, found in API & Integrations in Apify Console.

Python

JavaScript

CLI

OpenAPI

HTTP

MCP

1from apify_client import ApifyClient
2
3# Initialize the ApifyClient with your Apify API token
4# Replace '<YOUR_API_TOKEN>' with your token.
5client = ApifyClient("<YOUR_API_TOKEN>")
6
7# Prepare the Actor input
8run_input = {
9    "startUrls": [{ "url": "https://crawlee.dev/js/docs/quick-start" }],
10    "pathPrefix": "/js/docs",
11    "excludeUrlPatterns": [],
12    "proxyConfiguration": { "useApifyProxy": False },
13}
14
15# Run the Actor and wait for it to finish
16run = client.actor("lathesmith/docs-to-llms-markdown").call(run_input=run_input)
17
18# Fetch and print Actor results from the run's dataset (if there are any)
19print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
20for item in client.dataset(run.default_dataset_id).iterate_items():
21    print(item)
22
23# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

llms.txt Generator: Docs to Markdown & RAG Chunks API in Python

The Apify API client for Python is the official library that allows you to use llms.txt Generator + Docs to Markdown & RAG Chunks API in Python, providing convenience functions and automatic retries on errors.

Install the apify-client

$pip install apify-client

Other API clients include:

llms.txt Generator + Docs to Markdown & RAG Chunks API in JavaScript

llms.txt Generator + Docs to Markdown & RAG Chunks API through CLI

llms.txt Generator + Docs to Markdown & RAG Chunks OpenAPI definition

llms.txt Generator + Docs to Markdown & RAG Chunks API

Website To Markdown

smart_api/website-to-markdown

Convert any webpage into clean, LLM-ready Markdown in seconds — perfect for AI training data, RAG pipelines, and content archiving.

SmartApi

5.0

(1)

Website Content Crawler – Markdown for AI, Emails & Contacts

locomint/website-crawler-content-contacts

Website content crawler that turns a whole site into clean Markdown for LLM and RAG pipelines, with the role emails, social profiles and WhatsApp numbers found on each page. Seeded from the sitemap, honours robots.txt, plain HTTP with no browser.

Locomint

Knowledge Intelligence Engine — Website to Markdown for RAG

ryanclinton/website-content-to-markdown

Turn any website, documentation site or help centre into a retrieval-ready knowledge corpus for RAG and AI search. Clean Markdown plus chunks, change detection, deduplication, retrieval scoring, version awareness and a full corpus audit, in one run.

Ryan Clinton

Llm Ready Documentation Scraper

direct_duty/llm-ready-documentation-scraper

Developers and AI agents need to read documentation (e.g. Stripe Docs, Next.js Docs), but standard scrapers return noisy HTML that includes: navigation bars headers / footers ads / cookie banners This Actor must return pure content-only Markdown, suitable for vectorization and semantic search.

Sean

Universal Web to Markdown (Bulk & AI-Ready)

lentic_october/web-to-markdown-converter

Bulk convert any website URLs to clean Markdown for AI & LLMs. Universal scraper that removes ads, scripts, and clutter. Optimized for RAG, ChatGPT, Claude, and LangChain. Fast, async, and API-ready.

kalthireddy Abhishek

Markdown Maker: HTML to Markdown 📝

shahidirfan/Markdown-Maker

Instantly convert complex HTML into clean, structured Markdown. This lightweight actor is optimized to render web content into a format that is easily readable for AI LLMs, reducing token usage and improving context. Perfect for RAG pipelines and preparing data for training.

Shahid Irfan

5.0

(1)

Web Crawler & Semantic Schema-Enhanced Extractor

devil_port369-owner/web-crawler

Depth-controlled web crawler that transforms websites into structured analytics-ready data. Starting from one or more URLs, it crawls internal links up to a configurable depth and outputs detailed JSON records per page

DataFusionX

5.0

(8)

Website to Markdown for LLM & RAG — Crawl4AI URL to Clean

bikram07/web-to-markdown-crawl4ai

Convert any URL, sitemap, or whole website into clean, LLM-ready Markdown for RAG, vector databases, and AI agents. Hosted Crawl4AI in a real Chromium browser — renders JavaScript and SPAs, strips boilerplate, and exports JSON/CSV. Callable over MCP from Claude and Cursor.

Bikram

Doc2RAG AI - Documentation to LLM-Ready Markdown Converter

automation_studio/doc-to-rag-markdown-converter

Convert technical documentation websites (Stripe, Supabase, Next.js, Mintlify, Docusaurus) into clean, LLM-ready Markdown with preserved multi-language code blocks, tables, and RAG frontmatter. Exports JSON dataset, single llms-full.txt, and ZIP archive.

Automation Studio

HTML to Markdown Converter - Bulk Web Content to MD

santamaria-automations/html-to-markdown

Extract main article content from any website and convert to clean Markdown including headings, links, images, tables, and code blocks. Perfect for LLM training, AI pipelines, and documentation. Export data, run via API, schedule and monitor runs, or integrate with other tools.