Convert a website to Markdown to train a RAG
Created by
Apify
Convert websites to Markdown for training AI models. Easily extract text content for LLM applications. This is the perfect scraper for your RAG pipeline.
Website Content Crawlerapify/website-content-crawler
URL
Markdown
TextNumberBooleanListObject
Input
Start URLs(required)
url:https://docs.apify.com/academy/scraping-basics-javascript
Crawler type:playwright:adaptive
Include URL patterns (globs)
Exclude URL patterns (globs)
Max crawling depth:20
Max pages:9999999
Load URLs from Sitemaps:false
Crawl /llms.txt and Markdown files:false
Respect the robots.txt file:true
URL #fragments identify unique pages:false
Ignore canonical URLs:false
Proxy configuration(required)
Custom cookies:***
Custom HTTP headers
Sign HTTP requests (experimental):false
Initial concurrency:0
Max concurrency:200
Page request timeout:60
Minimum file download speed:128
Maximum number of retries on network / server errors:3
Maximum number of session rotations:10
Ignore HTTPS errors:false
Wait for dynamic content:10
Maximum scroll height:5000
Remove cookie warnings:true
Block loading of images and videos:true
Expand iframe elements:true
Expand clickable elements:[aria-expanded="false"]
Remove HTML elements (CSS selector):nav, footer, script, style, noscript, svg, img[src^='data:'],
[role="alert"],
[role="banner"],
[role="dialog"],
[role="alertdialog"],
[role="region"][aria-label*="skip" i],
[aria-modal="true"]
HTML transformer:readableText
Readable text extractor character threshold:100
Remove duplicate text lines:false
Debug mode (stores output of all HTML transformers):false
Debug log:false
Store skipped URLs:false
Save HTML to dataset (deprecated):false
Save HTML to key-value store:false
Save Markdown:true
Save files:false
Save screenshots (headless browser only):false
Max results:9999999
(Adaptive crawling only) Minimum client-side content change percentage:15
(Adaptive crawling only) How often should the crawler attempt to detect page rendering type:10
Reuse stored detections results (experimental):false
Output fields
URL
Markdown
Sign up on Apify01
Create your Apify account to access the Website Content Crawler.
Start the run02
The Actor will start running based on the input automatically.
Receive the output03
Monitor the progress in real-time. You will be notified as soon as your dataset is complete and ready for review.
Integrate into your workflow04
The final output is delivered in JSON, CSV, or Excel format, ready to be plugged into your workflow.
