🕸️ Website Content Crawler — Clean Markdown for AI

Created by

hiper soft

Actor

Website Content Crawler Pro — Markdown for AI & RAG

No-code, cheap website crawler API. Scrape any site into clean Markdown for RAG, vector DBs & AI agents: main content, sitemaps & globs — cheapest per item.

Try for free

Website Content Crawler Pro — Markdown for AI & RAGhipersoft/web-content-crawler-pro

Url

Title

Description

Author

+3 fields

Input

Start URLs:https://docs.apify.com/academy/web-scraping-for-beginners

Crawler type:http

Follow links:false

Max pages:50

Max depth:3

Same domain only:true

Use sitemaps:false

Save Markdown:true

Save plain text:true

Save HTML:false

Save links:false

Content extraction:readability

Remove cookie banners:true

Dynamic content wait (s):0

Max chars per page:0

Concurrency:10

Proxy:{1 field}

Output fields

Url

Title

Description

Author

Word count

Markdown

Text

How it works

Sign up on Apify or log in

The Website Content Crawler Pro — Markdown for AI & RAG Actor is ready to run with the inputs already set for "🕸️ Website Content Crawler — Clean Markdown for AI". Sign up or log in and you'll land in Apify Console with a copy of this exact task.

Start the "🕸️ Website Content Crawler — Clean Markdown for AI" task

The Website Content Crawler Pro — Markdown for AI & RAG Actor will start running based on the input automatically. You can also change all the input fields, like start urls, crawler type, and more, to match your specific use case.

Receive the "🕸️ Website Content Crawler — Clean Markdown for AI" output

Monitor the progress in real time. You will be notified as soon as your dataset is complete and ready for review. Each result includes data such as url, title, description, and more.