Pricing

from $2.00 / 1,000 sitemap extracted pages

Try for free

Go to Apify Store

XML Sitemap Crawler: Clean LLM Q&A Chunks

Try for free

Parse XML sitemaps, discover newly published URLs, and extract normalized article text segmented into optimal token chunks for RAG embedding.

Pricing

from $2.00 / 1,000 sitemap extracted pages

Rating

0.0

(0)

Developer

Lakshay Papneja

Actor stats

Bookmarked

Total users

Monthly active users

3 days ago

Last modified

XML Sitemap Crawler: Clean LLM Q&A Chunks (v1.0)

Crawl entire website XML sitemaps, extract documentation pages, strip navigation boilerplate, and generate semantic token chunks for LLMs.

⚡ Key Capabilities

High-Throughput Streaming: Pushes results continuously to the default dataset.
Enterprise-Grade Sanitization: Cleans and normalizes fields with zero extraneous DOM clutter.
Pay-Per-Event Compatibility: Fully monetized and configured with Apify PPE pricing.
MCP Server Ready: Can be invoked directly by AI agents (Claude, Cursor, LangChain).

⚙️ Input Parameters

Field	Type	Description
`sitemapUrls`	`array`	URLs of sitemap.xml files.
`maxPages`	`integer`	Upper page limit.

📦 Sample Dataset Record

{
  "url": "sample_value",
  "title": "sample_value",
  "tokensCount": "sample_value",
  "lastmod": "sample_value",
  "scrapedAt": "2026-10-31T20:56:00.000Z"
}

Website to Markdown Converter & Crawler for LLMs and RAG

s3nafps/ai-ready-webpage-extractor

Website to Markdown Crawler is an Apify Actor that turns web pages, docs sites and blogs into clean Markdown and RAG chunks for LLMs. $3 per 1,000 pages.

mohamed senator

URL to markdown

apify/url-to-markdown

An Apify Actor that takes a URL as input and returns the content of the page in Markdown format.

Apify

5.0

(6)

AI-Powered Smart Web Scraper

cloud9_ai/ai-web-scraper

Intelligent content extraction from any website using Crawlee + AI. Auto-detects structure, adapts to layout changes, handles JavaScript rendering. No custom code needed. Extract articles, products, listings from 1000s of pages.

cloud9

Knowledge Intelligence Engine — Website to Markdown for RAG

ryanclinton/website-content-to-markdown

Turn any website, documentation site or help centre into a retrieval-ready knowledge corpus for RAG and AI search. Clean Markdown plus chunks, change detection, deduplication, retrieval scoring, version awareness and a full corpus audit, in one run.

Ryan Clinton

Website to Markdown Scraper for AI & LLMs

quaking_pail/ai-website-content-markdown-scraper

Crawl a website and get clean Markdown of each page's main content, with URL and title. Ready for LLMs, RAG, vector databases and AI agents. Headless Chrome, same-domain crawl, API access.

AI_Builder

946

3.5

(2)

Website Content Crawler Pro

datascoutapi/website-content-crawler-pro

Crawl websites and extract clean, structured content in Markdown, JSON, or plain text for AI models, LLMs, vector DBs, or RAG pipelines. Fast, reliable, and stealthy, with bulk processing, advanced metadata extraction, and seamless integration with LangChain, LlamaIndex, and AI workflows.

halam

575

3.7

(3)

AI Training Data Scraper

blukaze/AI-Training-Data-Scraper

AI Training Data Scraper converts websites into clean, semantically-chunked, vector-ready data for LLMs, RAG pipelines, and AI search. Built for documentation, tutorials, and code-heavy content, with smart chunking and rich metadata.

Blukaze Automations

Contextractor — clean web content extraction for LLMs

markdownee/contextractor

Crawl any website and extract clean main-content text as Markdown, plain text, JSON, or HTML — ready for LLMs, RAG pipelines, and vector databases. Built on the rs-trafilatura engine and an adaptive Crawlee + Playwright crawler.

Markdownee

LLM-Ready Web Scraper

devoted_helix/llm-web-scraper

Convert web pages to clean, LLM-friendly text. Perfect for RAG pipelines, AI chatbot training, and fine-tuning datasets. Removes ads,menus, and clutter automatically.