Extract Crawlee documentation for an LLM knowledge base

Created by

Automation Lab

Actor

Website Crawler Scraper

Turn Crawlee documentation pages into clean text, Markdown, metadata, content hashes, and source links for RAG ingestion.

Try for free

Website Crawler Scraperautomation-lab/website-content-crawler

URL

HTTP status

Depth

Title

+8 fields

Input

Start URLs(required)

url:https://crawlee.dev/js/docs/introduction

Maximum pages:5

Maximum link depth:1

Content CSS selector:main

Maximum concurrency:2

Output fields

URL

HTTP status

Depth

Title

Language

Word count

Content hash

Cleaned text

Markdown

Internal links

Fetched at

Error

How it works

Sign up on Apify or log in

The Website Crawler Scraper Actor is ready to run with the inputs already set for "Extract Crawlee documentation for an LLM knowledge base". Sign up or log in and you'll land in Apify Console with a copy of this exact task.

Start the "Extract Crawlee documentation for an LLM knowledge base" task

The Website Crawler Scraper Actor will start running based on the input automatically. You can also change all the input fields, like start urls, maximum pages, and more, to match your specific use case.

Receive the "Extract Crawlee documentation for an LLM knowledge base" output

Monitor the progress in real time. You will be notified as soon as your dataset is complete and ready for review. Each result includes data such as URL, HTTP status, depth, and more.