Crawl a website into Markdown using its sitemap and llms.txt

Created by

Hriday Rungta

Actor

Website to Markdown Crawler | LLM & RAG | $0.5 / 1K

Let the site's own sitemap.xml and llms.txt decide which pages to convert to Markdown, instead of guessing with link depth. Good for building an LLM knowledge base from a docs site.

Try for free

Website to Markdown Crawler | LLM & RAG | $0.5 / 1Khridayrungta/website-to-markdown-crawler

URL

Title

Words

Depth

+3 fields

Input

Max pages (billed):10

Max crawl depth:1

Crawl scope:same-hostname

Discover pages from sitemap.xml:true

Discover pages from llms.txt:true

Output fields

URL

Title

Words

Depth

HTTP

Status

Markdown

How it works

Sign up on Apify or log in

The Website to Markdown Crawler | LLM & RAG | $0.5 / 1K Actor is ready to run with the inputs already set for "Crawl a website into Markdown using its sitemap and llms.txt". Sign up or log in and you'll land in Apify Console with a copy of this exact task.

Start the "Crawl a website into Markdown using its sitemap and llms.txt" task

The Website to Markdown Crawler | LLM & RAG | $0.5 / 1K Actor will start running based on the input automatically. You can also change all the input fields, like max pages (billed), max crawl depth, and more, to match your specific use case.

Receive the "Crawl a website into Markdown using its sitemap and llms.txt" output

Monitor the progress in real time. You will be notified as soon as your dataset is complete and ready for review. Each result includes data such as URL, title, words, and more.