Pricing

from $1.20 / 1,000 processed sitemap pages

Try for free

Go to Apify Store

Sitemap & Website to llms.txt, Markdown & RAG Dataset Builder

Try for free

Auto-discover XML sitemaps or internal pages from any domain, generate a ready-to-publish /llms.txt standard file, extract clean LLM Markdown, calculate token savings, and build RAG vector datasets in 6 languages (EN, ES, PT, FR, DE, IT).

Pricing

from $1.20 / 1,000 processed sitemap pages

Rating

0.0

(0)

Developer

Cristian Duarte

Actor stats

Bookmarked

Total users

Monthly active users

a day ago

Last modified

💠 Sitemap & Website to `llms.txt`, Clean Markdown & RAG Dataset Builder

Automatically crawl any website's sitemap.xml (or internal navigation), generate a ready-to-publish /llms.txt standard file (used by OpenAI, Anthropic Claude, and Cursor IDE), and convert core pages into clean LLM Markdown in seconds — with 6-language AI audits (EN, ES, PT, FR, DE, IT).

Built for AI Engineers, RAG Pipeline Builders, SEO/GEO (Generative Engine Optimization) Agencies, and n8n / Make / LangChain Developers.

🔥 Why AI & GEO Teams Need This Actor

Instant /llms.txt Generator: Creates a compliant llms.txt index file (# Title, > Summary, and categorized Markdown links with descriptions) so ChatGPT Search, Perplexity, and Claude index your site accurately.
Automatic sitemap.xml Discovery: Finds /sitemap.xml or /sitemap_index.xml automatically (or falls back to internal navigation links) and extracts clean Markdown from up to 50 pages per run.
Multi-Page RAG Dataset in 1 Run: Outputs pagesDataset containing clean Markdown and exact LLM token counts per page.
Multilingual AI Readiness Audit: Summarizes discovered URLs, token footprint, and llms.txt status in 6 languages (EN, ES, PT, FR, DE, IT).

💻 Quick Start Code Examples (Python & Node.js)

Python (`apify-client`)

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_API_TOKEN")
run = client.actor("duartelabs/sitemap-to-llms-txt-markdown-rag-builder").call(run_input={
    "websiteUrls": [{"url": "https://apify.com"}],
    "maxPagesPerSite": 5,
    "reportLanguage": "en"
})
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item["generatedLlmsTxt"])

🤖 1-Click Integration with Claude Desktop & Cursor (MCP)

{
  "mcpServers": {
    "apify": {
      "command": "npx",
      "args": ["-y", "@apify/actors-mcp-server", "--actors", "duartelabs/sitemap-to-llms-txt-markdown-rag-builder"],
      "env": { "APIFY_TOKEN": "YOUR_APIFY_API_TOKEN" }
    }
  }
}

🛠️ Complete 6-Actor Automation Suite by `duartelabs`

🟢 B2B Lead, WhatsApp, Email & Ad-Pixel Intelligence Scraper
📍 Google Maps Local Business Email, WhatsApp & Ad-Pixel Enricher
🟠 Shopify Competitor Price, Stock, Discount & Best-Sellers Spy
🟣 AI-Ready Website to Clean Markdown, Token Counter & RAG Chunker
🔵 Technical SEO, Meta, Schema.org & Broken Links Auditor

Website To Markdown

smart_api/website-to-markdown

Convert any webpage into clean, LLM-ready Markdown in seconds — perfect for AI training data, RAG pipelines, and content archiving.

SmartApi

5.0

(1)

Sitemap Detector

coder_zoro/sitemap-detector

Find sitemap URLs fast with our free Sitemap Finder tool. Instantly detect sitemaps from any website for SEO audits, indexing checks, and crawl planning. Improve visibility, site structure insights, and search engine performance in just seconds

Zoro

182

5.0

(3)

llms.txt Generator: llms-full.txt for Any Website (AI SEO)

fayoussef/llms-txt-generator

Generate a spec-compliant llms.txt and llms-full.txt for any website in one run. Crawls your sitemap or internal links, extracts every page's title, meta description and full Markdown content, and outputs two ready-to-upload files for ChatGPT, Claude, Perplexity and Gemini.

Youssef Farhan

Website to Markdown for LLM & RAG — Crawl4AI URL to Clean

bikram07/web-to-markdown-crawl4ai

Convert any URL, sitemap, or whole website into clean, LLM-ready Markdown for RAG, vector databases, and AI agents. Hosted Crawl4AI in a real Chromium browser — renders JavaScript and SPAs, strips boilerplate, and exports JSON/CSV. Callable over MCP from Claude and Cursor.

Bikram

Doc2RAG AI - Documentation to LLM-Ready Markdown Converter

automation_studio/doc-to-rag-markdown-converter

Convert technical documentation websites (Stripe, Supabase, Next.js, Mintlify, Docusaurus) into clean, LLM-ready Markdown with preserved multi-language code blocks, tables, and RAG frontmatter. Exports JSON dataset, single llms-full.txt, and ZIP archive.

Automation Studio

Rag Knowledge Graph Builder

cspnair/rag-knowledge-graph-builder

Transform websites into RAG-ready datasets. Crawls pages, chunks content into semantic segments (500-1000 tokens), and generates hypothetical questions for each chunk. No API key needed with native mode. Output: pre-indexed JSON optimized for AI retrieval with 3x better accuracy than raw text.

csp

132

5.0

(7)

Quick Website Content Scraper ( Extract Text for RAG & LLMs )

automateitplease/ai-web-content-scraper-extract-text-for-rag-llms

Extract clean text from any website for AI/LLM applications. Supports both static and JavaScript-rendered sites (React, Vue, Angular). Perfect for RAG systems, chatbot training, and content analysis.

AutomateItPlease Workflow And Automaton Ops

Web Page to Clean Markdown for RAG & LLM — 77% Fewer Tokens

craftbox/fast-markdown-for-rag

Convert web pages into clean Markdown for LLM and RAG pipelines, using 77% fewer tokens than a raw crawl while keeping every content section. Works in any language.

Craftbox

llms.txt Generator: Make Any Website LLM-Ready

f0rty7even/llms-txt-generator

Crawl a website and generate an llms.txt file (the emerging standard that helps LLMs and AI agents understand your site). Optionally build llms-full.txt with full page content. Output saved to the key-value store.

Michael Yousrie

Website to RAG Dataset

sebastian-actors/website-to-rag-dataset

Convert public websites, docs, blogs, and XML sitemaps into clean Markdown, structured metadata, and stable chunks for RAG pipelines and vector databases.