Convert a website to Markdown to train a RAG

Created by

Apify

Actor

Website Content Crawler

Convert websites to Markdown for training AI models. Easily extract text content for LLM applications. This is the perfect scraper for your RAG pipeline.

Try for free

Website Content Crawlerapify/website-content-crawler

URL

Markdown

Input

Start URLs(required)

url:https://docs.apify.com/academy/scraping-basics-javascript

Crawler type:playwright:adaptive

Include URL patterns (globs)

Exclude URL patterns (globs)

Max crawling depth:20

Max pages:9999999

Load URLs from Sitemaps:false

Crawl /llms.txt and Markdown files:false

Respect the robots.txt file:true

URL #fragments identify unique pages:false

Ignore canonical URLs:false

Proxy configuration(required):{1 field}

Custom cookies:***

Custom HTTP headers

Sign HTTP requests (experimental):false

Initial concurrency:0

Max concurrency:200

Page request timeout:60

Minimum file download speed:128

Maximum number of retries on network / server errors:3

Maximum number of session rotations:10

Ignore HTTPS errors:false

Wait for dynamic content:10

Maximum scroll height:5000

Remove cookie warnings:true

Block loading of images and videos:true

Expand iframe elements:true

Expand clickable elements:[aria-expanded="false"]

Remove HTML elements (CSS selector):nav, footer, script, style, noscript, svg, img[src^='data:'], [role="alert"], [role="banner"], [role="dialog"], [role="alertdialog"], [role="region"][aria-label*="skip" i], [aria-modal="true"]

HTML transformer:readableText

Readable text extractor character threshold:100

Remove duplicate text lines:false

Debug mode (stores output of all HTML transformers):false

Debug log:false

Store skipped URLs:false

Save HTML to dataset (deprecated):false

Save HTML to key-value store:false

Save Markdown:true

Save files:false

Save screenshots (headless browser only):false

Max results:9999999

(Adaptive crawling only) Minimum client-side content change percentage:15

(Adaptive crawling only) How often should the crawler attempt to detect page rendering type:10

Reuse stored detections results (experimental):false

Output fields

URL

Markdown

How it works

Sign up on Apify or log in

The Website Content Crawler Actor is ready to run with the inputs already set for "Convert a website to Markdown to train a RAG". Sign up or log in and you'll land in Apify Console with a copy of this exact task.

Start the "Convert a website to Markdown to train a RAG" task

The Website Content Crawler Actor will start running based on the input automatically. You can also change all the input fields, like start urls, crawler type, and more, to match your specific use case.

Receive the "Convert a website to Markdown to train a RAG" output

Monitor the progress in real time. You will be notified as soon as your dataset is complete and ready for review. Each result includes data such as URL and markdown.

Integrate into your workflow

The final output is delivered in JSON, CSV, or Excel format, ready to be plugged into your workflow. You can also run the "Convert a website to Markdown to train a RAG" task via API.