Build an LLM training dataset from a documentation site

Created by

Vamsi Krishna

Actor

AI Training Data Curator

Crawl a docs site into clean page text for LLM fine-tuning and RAG. Filter by language, remove duplicates, and export JSONL or CSV.

Try for free

AI Training Data Curatorvamsi-krishna/ai-training-data-curator

URL

Title

Text

Words

+3 fields

Input

Start URLs

url:https://docs.python.org/3/tutorial/+2

Sitemap URL:https://docs.example.com/sitemap.xml

Max pages:30

Minimum text length:100

Language filter:en

Remove duplicates:true

Crawl strategy:recurse

Stay within seed domains:true

Output fields

URL

Title

Text

Words

Language

Author

Published

How it works

Sign up on Apify or log in

The AI Training Data Curator Actor is ready to run with the inputs already set for "Build an LLM training dataset from a documentation site". Sign up or log in and you'll land in Apify Console with a copy of this exact task.

Start the "Build an LLM training dataset from a documentation site" task

The AI Training Data Curator Actor will start running based on the input automatically. You can also change all the input fields, like start urls, sitemap url, and more, to match your specific use case.

Receive the "Build an LLM training dataset from a documentation site" output

Monitor the progress in real time. You will be notified as soon as your dataset is complete and ready for review. Each result includes data such as URL, title, text, and more.