AI Training Data Collector — Clean Web Datasets for LLMs
Pricing
Pay per usage
AI Training Data Collector — Clean Web Datasets for LLMs
Crawl websites and extract structured, clean text datasets perfect for fine-tuning LLMs and RAG pipelines. Removes boilerplate, deduplicates, and scores content quality.
AI Training Data Collector
Crawls a website from the start URLs you give it and returns the main text of each page, cleaned for LLM fine-tuning, RAG indexing or text analysis. Navigation, headers, footers, cookie banners, edit links, footnote markers, reference lists and "See also" link lists are removed. HTML entities are decoded. Each row also carries the page language, word count, a rough quality score and a content hash for de-duplication.
How it works
- Each page is fetched over plain HTTP (no browser), so it is fast and cheap. Pages that only render their text with JavaScript will come back short and get skipped by
minWordCount. - The main content is picked from
article,main, the MediaWiki content area or similar containers. If none holds real text, the whole body is used after chrome is stripped. - Everything from a "References", "Notes", "Sources", "Further reading", "External links" or "See also" heading onward is dropped.
- The text is converted to Markdown (headings, lists, bold and italics kept; link targets and images removed) or to plain text.
- Links inside the main content are followed on the same hostname, up to
crawlDepth. Query-string URLs, files (PDF, images) and MediaWiki namespace pages such asSpecial:orTalk:are not followed. - Pages shorter than
minWordCountand pages whose text exactly matches an earlier page are skipped. The run stops oncemaxPagesrows are saved.
The default input (the Wikipedia "Artificial intelligence" article, depth 1, 5 pages) finishes in under a minute.
Input
| Field | Default | Notes |
|---|---|---|
urls | Wikipedia AI article | Start pages. |
crawlDepth | 1 | 0 = start pages only. Max 3. |
maxPages | 5 | Rows to return, at most 1000. |
outputFormat | markdown | markdown or plainText fill cleanText; json fills sections instead. |
excludePatterns | /tag/, /category/, /author/, /search/ | Path substrings to skip. |
minWordCount | 100 | Shorter pages are skipped. |
proxyConfiguration | none | Optional. |
Output
{"url": "https://en.wikipedia.org/wiki/Artificial_intelligence","title": "Artificial intelligence - Wikipedia","description": "Artificial intelligence (AI) is the capability of computational systems to perform tasks typically associated with human intelligence, such as learning, reasoning, problem-solving, perception, and decision-making…","language": "en","cleanText": "**Artificial intelligence** (**AI**) is the capability of computational systems to perform tasks ...\n\n## Goals\n\n...","wordCount": 13300,"qualityScore": 83,"sourceDomain": "en.wikipedia.org","crawlDepth": 0,"headingCount": 66,"paragraphCount": 230,"linkCount": 680,"imageCount": 0,"contentHash": "3f1c0e9b7a2d4c51","scrapedAt": "2026-10-29T13:30:00.000Z"}
description is the page's meta description. Pages without one (Wikipedia, for example) get the first 300 characters of the text instead.
language comes from the page's lang attribute. When a page does not declare one, the actor guesses from common words for English, German, French, Spanish, Italian, Portuguese and Dutch, and returns null if the text is too short or in another language.
Limits
- No JavaScript rendering. Single-page apps and sites behind bot protection will return little or nothing.
- Tables, infoboxes and image captions are removed along with the rest of the page chrome, so facts that only appear in those are not in the text.
qualityScoreis a heuristic for ranking pages against each other, not a measure of factual quality.- Respect each site's terms and robots rules, and the licence of the text you collect (Wikipedia text is CC BY-SA).