Turn a Documentation Site Into Markdown Chunks for RAG

Created by

JT Palms

Actor

Website to Markdown Crawler for RAG: Fast, No Browser

Crawls a documentation site from its sitemap and returns clean Markdown per page, split into chunks of about 500 tokens by heading, ready for a vector database. Replace the example docs with your own site.

Try for free

Website to Markdown Crawler for RAG: Fast, No Browserjtpalms/website-to-markdown-crawler

URL

Title

HTTP

Words

+5 fields

Input

Start URLs(required)

url:https://crawlee.dev/js/docs/introduction

Max pages:25

Chunk for RAG:true

Max tokens per chunk:500

Output fields

URL

Title

HTTP

Words

Tokens (est.)

Changed

Needs JavaScript

Skipped

Error

How it works

Sign up on Apify or log in

The Website to Markdown Crawler for RAG: Fast, No Browser Actor is ready to run with the inputs already set for "Turn a Documentation Site Into Markdown Chunks for RAG". Sign up or log in and you'll land in Apify Console with a copy of this exact task.

Start the "Turn a Documentation Site Into Markdown Chunks for RAG" task

The Website to Markdown Crawler for RAG: Fast, No Browser Actor will start running based on the input automatically. You can also change all the input fields, like start urls, max pages, and more, to match your specific use case.

Receive the "Turn a Documentation Site Into Markdown Chunks for RAG" output

Monitor the progress in real time. You will be notified as soon as your dataset is complete and ready for review. Each result includes data such as URL, title, HTTP, and more.