Pricing

Pay per usage

Try for free

Go to Apify Store

Sitemap to Clean Markdown & RAG Chunks Crawler

Try for free

Feed an entire website via XML Sitemap (or domain) and automatically generate structured, clean Markdown documents and vector chunks for AI embeddings.

Pricing

Pay per usage

Rating

0.0

(0)

Developer

knightwwang Wang

Actor stats

Bookmarked

Total users

Monthly active users

5 days ago

Last modified

🗺️ Sitemap to Clean Markdown & RAG Chunks Crawler

Crawl an entire website's XML sitemap (or documentation site), convert all pages into clean, structured Markdown, and automatically split them into semantic, vector-ready RAG chunks with exact token counts.

⚡ Ideal For

Enterprise AI Knowledge Bases: Ingest entire product docs (e.g. docs.stripe.com/sitemap.xml) into Pinecone / Weaviate / Qdrant in 1 click.
Competitor Content Auditing: Extract thousands of clean blog posts from any competitor's sitemap without headers, footers, or cookie banners.
RAG Preprocessing: Eliminate token-counting bottlenecks and complex text-splitters in your Python backend.

📥 Key Inputs

sitemapUrls: e.g. https://crawlee.dev/sitemap.xml
urlPattern: Filter URLs using regex (e.g. /docs/ or /blog/)
maxPages: Max number of pages to crawl
maxChunkTokens: Token size per chunk (default: 800)
removeImages: Strip images to save embedding tokens

📤 Output Dataset

Returns structured records with full markdown and individual chunk arrays:

{
  "url": "https://crawlee.dev/docs/introduction",
  "title": "Introduction | Crawlee",
  "totalTokens": 1420,
  "chunksCount": 2,
  "chunks": [
    { "chunkIndex": 0, "tokens": 780, "text": "## Overview\nCrawlee is..." },
    { "chunkIndex": 1, "tokens": 640, "text": "## Quick Start\nTo install..." }
  ]
}

Website Content Extractor for RAG: Markdown, HTML, Text

nezha/website-content-crawler

Turn docs sites, help centers, blogs, and websites into clean markdown, text, or HTML for RAG, AI knowledge bases, and internal search. Crawl from start URLs or sitemaps and keep the crawl in scope.

nezha

Sitemap Generator - Crawl Website & Create XML Sitemap

scrappy_garden/sitemap-generator

Generate an XML sitemap for any website. Crawls internal pages from start URLs (with depth + page limits), deduplicates URLs, and stores a ready-to-submit sitemap.xml plus a structured dataset and summary for SEO audits.

Bikram Adhikari

Tech Docs to LLM-Ready Markdown

hedelka/tech-docs-scraper

Scrapes technical documentation sites (Docusaurus, GitBook, MkDocs, ReadTheDocs) and converts them to clean, structured Markdown for RAG pipelines, LLM training, and AI assistants. Automatically detects documentation framework and removes navigation elements.

Dmitry Goncharov

Web-to-Markdown Generator for AI & RAG Pipelines

profitstack/web-to-markdown-generator-for-ai-rag-pipelines

Convert any website into clean, heading-based chunking, LLM-ready Markdown for RAG and AI agents.

Manas Mantri

Website & PDF to RAG JSONL Crawler

orbiscribe/linked-pdf-website-rag-crawler

Paste webpage and PDF URLs and get Markdown, JSONL chunks, PDF inventory, source warnings, and RAG-ready records.

Orbiscribe Labs

Docs To Rag

gabrielaxy/docs-to-rag

Transform documentation websites into RAG-ready chunks with semantic understanding, quality scoring, and direct vector database integration.

Gabriel Antony Xaviour

E-commerce Product Scraper for AI & RAG

coolinbex/ecommerce-product-scraper-ai-rag

Scrape product catalogs into clean, chunked data for AI search, RAG, embeddings, and vector databases.

coolinbex

XML Sitemap Finder & Extractor API

andok/find-sitemap-from-url

Find and extract all XML sitemaps for any domain. Automatically parses robots.txt, scans HTML tags, and recursively follows indexes. Perfect for SEO & web scraping.

Andok

AI Training Data Scraper

blukaze/AI-Training-Data-Scraper

AI Training Data Scraper converts websites into clean, semantically-chunked, vector-ready data for LLMs, RAG pipelines, and AI search. Built for documentation, tutorials, and code-heavy content, with smart chunking and rich metadata.