Website to Markdown Crawler for LLM & RAG (fast, no browser)

Created by

Murat Uzun

Actor

Website to Markdown — Content Crawler for LLM & RAG

Website to Markdown Crawler turns any site into clean Markdown per page for LLMs, RAG pipelines and vector DBs — no headless browser, $1 per 1,000 pages.

Try for free

Website to Markdown — Content Crawler for LLM & RAGwebdatatools/website-to-markdown

URL

Page title

Word count

Status code

+3 fields

Input

Start URLs

url:https://docs.apify.com/platform+1

Max pages:50

Max link depth:3

Same domain only:true

Include path prefixes

Exclude URL patterns:\.(png|jpe?g|gif|svg|webp|pdf|zip|mp4|css|js)$+3

Seed from sitemap.xml:true

Output format:markdown

Extra CSS selectors to remove

Max concurrency:10

Respect robots.txt:true

Proxy:{1 field}

Output fields

URL

Page title

Word count

Status code

Depth

Crawled at

Error

How it works

Sign up on Apify or log in

The Website to Markdown — Content Crawler for LLM & RAG Actor is ready to run with the inputs already set for "Website to Markdown Crawler for LLM & RAG (fast, no browser)". Sign up or log in and you'll land in Apify Console with a copy of this exact task.

Start the "Website to Markdown Crawler for LLM & RAG (fast, no browser)" task

The Website to Markdown — Content Crawler for LLM & RAG Actor will start running based on the input automatically. You can also change all the input fields, like start urls, max pages, and more, to match your specific use case.

Receive the "Website to Markdown Crawler for LLM & RAG (fast, no browser)" output

Monitor the progress in real time. You will be notified as soon as your dataset is complete and ready for review. Each result includes data such as URL, page title, word count, and more.