Fast Website to AI/RAG Clean Markdown & Sitemap Crawler

Created by

H H

Actor

Fast Website Content Crawler (AI/RAG Markdown & Sitemap)

Convert any documentation site, blog, or company website into clean LLM-ready Markdown and token-bounded vector RAG chunks at high speed over HTTP/2.

Try for free

Fast Website Content Crawler (AI/RAG Markdown & Sitemap)church_of_loujess/fast-website-content-crawler

Page URL

Title

Words

Est. Tokens

+4 fields

Input

Start URLs / Domains(required):https://crawlee.dev+1

Max Pages per Website (maxPagesPerSite):3

Discover Pages via Sitemap.xml:true

Generate Token-Bounded AI/RAG Chunks:true

RAG Chunk Size (Words):250

Output fields

Page URL

Title

Words

Est. Tokens

Meta Description

Clean AI/RAG Markdown

RAG Chunks

Headings Outline

How it works

Sign up on Apify or log in

The Fast Website Content Crawler (AI/RAG Markdown & Sitemap) Actor is ready to run with the inputs already set for "Fast Website to AI/RAG Clean Markdown & Sitemap Crawler". Sign up or log in and you'll land in Apify Console with a copy of this exact task.

Start the "Fast Website to AI/RAG Clean Markdown & Sitemap Crawler" task

The Fast Website Content Crawler (AI/RAG Markdown & Sitemap) Actor will start running based on the input automatically. You can also change all the input fields, like start urls / domains, max pages per website (maxpagespersite), and more, to match your specific use case.

Receive the "Fast Website to AI/RAG Clean Markdown & Sitemap Crawler" output

Monitor the progress in real time. You will be notified as soon as your dataset is complete and ready for review. Each result includes data such as page url, title, words, and more.