Pricing

from $1.20 / 1,000 processed sitemap pages

Try for free

Go to Apify Store

Sitemap & Website to llms.txt, Markdown & RAG Dataset Builder

Try for free

Auto-discover XML sitemaps or internal pages from any domain, generate a ready-to-publish /llms.txt standard file, extract clean LLM Markdown, calculate token savings, and build RAG vector datasets in 6 languages (EN, ES, PT, FR, DE, IT).

Pricing

from $1.20 / 1,000 processed sitemap pages

Rating

0.0

(0)

Developer

Cristian Duarte

Actor stats

Bookmarked

Total users

Monthly active users

a day ago

Last modified

Categories

You can access the Sitemap & Website to llms.txt, Markdown & RAG Dataset Builder programmatically from your own applications by using the Apify API. You can also choose the language preference from below. To use the Apify API, you’ll need an Apify account and your API token, found in API & Integrations in Apify Console.

Python

JavaScript

CLI

OpenAPI

HTTP

MCP

1from apify_client import ApifyClient
2
3# Initialize the ApifyClient with your Apify API token
4# Replace '<YOUR_API_TOKEN>' with your token.
5client = ApifyClient("<YOUR_API_TOKEN>")
6
7# Prepare the Actor input
8run_input = { "websiteUrls": [
9        { "url": "https://apify.com" },
10        { "url": "https://fastapi.tiangolo.com" },
11    ] }
12
13# Run the Actor and wait for it to finish
14run = client.actor("duartelabs/sitemap-to-llms-txt-markdown-rag-builder").call(run_input=run_input)
15
16# Fetch and print Actor results from the run's dataset (if there are any)
17print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
18for item in client.dataset(run.default_dataset_id).iterate_items():
19    print(item)
20
21# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

Sitemap & Website to llms.txt, Markdown & RAG Dataset Builder API in Python

The Apify API client for Python is the official library that allows you to use Sitemap & Website to llms.txt, Markdown & RAG Dataset Builder API in Python, providing convenience functions and automatic retries on errors.

Install the apify-client

$pip install apify-client

Other API clients include:

Sitemap & Website to llms.txt, Markdown & RAG Dataset Builder API in JavaScript

Sitemap & Website to llms.txt, Markdown & RAG Dataset Builder API through CLI

Sitemap & Website to llms.txt, Markdown & RAG Dataset Builder OpenAPI definition

Sitemap & Website to llms.txt, Markdown & RAG Dataset Builder API

Website To Markdown

smart_api/website-to-markdown

Convert any webpage into clean, LLM-ready Markdown in seconds — perfect for AI training data, RAG pipelines, and content archiving.

SmartApi

5.0

(1)

Sitemap Detector

coder_zoro/sitemap-detector

Find sitemap URLs fast with our free Sitemap Finder tool. Instantly detect sitemaps from any website for SEO audits, indexing checks, and crawl planning. Improve visibility, site structure insights, and search engine performance in just seconds

Zoro

182

5.0

(3)

llms.txt Generator: llms-full.txt for Any Website (AI SEO)

fayoussef/llms-txt-generator

Generate a spec-compliant llms.txt and llms-full.txt for any website in one run. Crawls your sitemap or internal links, extracts every page's title, meta description and full Markdown content, and outputs two ready-to-upload files for ChatGPT, Claude, Perplexity and Gemini.

Youssef Farhan

Website to Markdown for LLM & RAG — Crawl4AI URL to Clean

bikram07/web-to-markdown-crawl4ai

Convert any URL, sitemap, or whole website into clean, LLM-ready Markdown for RAG, vector databases, and AI agents. Hosted Crawl4AI in a real Chromium browser — renders JavaScript and SPAs, strips boilerplate, and exports JSON/CSV. Callable over MCP from Claude and Cursor.

Bikram

Doc2RAG AI - Documentation to LLM-Ready Markdown Converter

automation_studio/doc-to-rag-markdown-converter

Convert technical documentation websites (Stripe, Supabase, Next.js, Mintlify, Docusaurus) into clean, LLM-ready Markdown with preserved multi-language code blocks, tables, and RAG frontmatter. Exports JSON dataset, single llms-full.txt, and ZIP archive.

Automation Studio

Rag Knowledge Graph Builder

cspnair/rag-knowledge-graph-builder

Transform websites into RAG-ready datasets. Crawls pages, chunks content into semantic segments (500-1000 tokens), and generates hypothetical questions for each chunk. No API key needed with native mode. Output: pre-indexed JSON optimized for AI retrieval with 3x better accuracy than raw text.

csp

132

5.0

(7)

Quick Website Content Scraper ( Extract Text for RAG & LLMs )

automateitplease/ai-web-content-scraper-extract-text-for-rag-llms

Extract clean text from any website for AI/LLM applications. Supports both static and JavaScript-rendered sites (React, Vue, Angular). Perfect for RAG systems, chatbot training, and content analysis.

AutomateItPlease Workflow And Automaton Ops

Web Page to Clean Markdown for RAG & LLM — 77% Fewer Tokens

craftbox/fast-markdown-for-rag

Convert web pages into clean Markdown for LLM and RAG pipelines, using 77% fewer tokens than a raw crawl while keeping every content section. Works in any language.

Craftbox

llms.txt Generator: Make Any Website LLM-Ready

f0rty7even/llms-txt-generator

Crawl a website and generate an llms.txt file (the emerging standard that helps LLMs and AI agents understand your site). Optionally build llms-full.txt with full page content. Output saved to the key-value store.

Michael Yousrie

Website to RAG Dataset

sebastian-actors/website-to-rag-dataset

Convert public websites, docs, blogs, and XML sitemaps into clean Markdown, structured metadata, and stable chunks for RAG pipelines and vector databases.