🕸️ Website Content Crawler — Clean Markdown for AI
No-code, cheap website crawler API. Scrape any site into clean Markdown for RAG, vector DBs & AI agents: main content, sitemaps & globs — cheapest per item.
Website Content Crawler Pro — Markdown for AI & RAGhipersoft/web-content-crawler-pro
Url
Title
Description
Canonical
+6 fieldsTextNumberBooleanListObject
Input
Start URLs:https://docs.apify.com/academy/web-scraping-for-beginners
Crawler type:http
Follow links:false
Max pages:50
Max depth:3
Same domain only:true
Use sitemaps:false
Save Markdown:true
Save plain text:true
Save HTML:false
Save links:false
Content extraction:readability
Remove cookie banners:true
Dynamic content wait (s):0
Max chars per page:0
Concurrency:10
Proxy
Output fields
Url
Title
Description
Canonical
Lang
Author
Site Name
Word Count
Markdown
Text
Sign up on Apify01
Create your Apify account to access the Website Content Crawler Pro — Markdown for AI & RAG.
Start the run02
The Actor will start running based on the input automatically.
Receive the output03
Monitor the progress in real-time. You will be notified as soon as your dataset is complete and ready for review.
Integrate into your workflow04
The final output is delivered in JSON, CSV, or Excel format, ready to be plugged into your workflow.
