AI Training Data Collector — Clean Web Datasets for LLMs avatar

AI Training Data Collector — Clean Web Datasets for LLMs

Pricing

Pay per usage

Go to Apify Store
AI Training Data Collector — Clean Web Datasets for LLMs

AI Training Data Collector — Clean Web Datasets for LLMs

Crawl websites and extract structured, clean text datasets perfect for fine-tuning LLMs and RAG pipelines. Removes boilerplate, deduplicates, and scores content quality.

Pricing

Pay per usage

Rating

0.0

(0)

Developer

Avinash

Avinash

Maintained by Community

Actor stats

0

Bookmarked

2

Total users

1

Monthly active users

16 days ago

Last modified

Categories

Features

Limited permissions

Share

AI Training Data Collector

Crawls a website from the start URLs you give it and returns the main text of each page, cleaned for LLM fine-tuning, RAG indexing or text analysis. Navigation, headers, footers, cookie banners, edit links, footnote markers, reference lists and "See also" link lists are removed. HTML entities are decoded. Each row also carries the page language, word count, a rough quality score and a content hash for de-duplication.

How it works

  1. Each page is fetched over plain HTTP (no browser), so it is fast and cheap. Pages that only render their text with JavaScript will come back short and get skipped by minWordCount.
  2. The main content is picked from article, main, the MediaWiki content area or similar containers. If none holds real text, the whole body is used after chrome is stripped.
  3. Everything from a "References", "Notes", "Sources", "Further reading", "External links" or "See also" heading onward is dropped.
  4. The text is converted to Markdown (headings, lists, bold and italics kept; link targets and images removed) or to plain text.
  5. Links inside the main content are followed on the same hostname, up to crawlDepth. Query-string URLs, files (PDF, images) and MediaWiki namespace pages such as Special: or Talk: are not followed.
  6. Pages shorter than minWordCount and pages whose text exactly matches an earlier page are skipped. The run stops once maxPages rows are saved.

The default input (the Wikipedia "Artificial intelligence" article, depth 1, 5 pages) finishes in under a minute.

Input

FieldDefaultNotes
urlsWikipedia AI articleStart pages.
crawlDepth10 = start pages only. Max 3.
maxPages5Rows to return, at most 1000.
outputFormatmarkdownmarkdown or plainText fill cleanText; json fills sections instead.
excludePatterns/tag/, /category/, /author/, /search/Path substrings to skip.
minWordCount100Shorter pages are skipped.
proxyConfigurationnoneOptional.

Output

{
"url": "https://en.wikipedia.org/wiki/Artificial_intelligence",
"title": "Artificial intelligence - Wikipedia",
"description": "Artificial intelligence (AI) is the capability of computational systems to perform tasks typically associated with human intelligence, such as learning, reasoning, problem-solving, perception, and decision-making…",
"language": "en",
"cleanText": "**Artificial intelligence** (**AI**) is the capability of computational systems to perform tasks ...\n\n## Goals\n\n...",
"wordCount": 13300,
"qualityScore": 83,
"sourceDomain": "en.wikipedia.org",
"crawlDepth": 0,
"headingCount": 66,
"paragraphCount": 230,
"linkCount": 680,
"imageCount": 0,
"contentHash": "3f1c0e9b7a2d4c51",
"scrapedAt": "2026-10-29T13:30:00.000Z"
}

description is the page's meta description. Pages without one (Wikipedia, for example) get the first 300 characters of the text instead.

language comes from the page's lang attribute. When a page does not declare one, the actor guesses from common words for English, German, French, Spanish, Italian, Portuguese and Dutch, and returns null if the text is too short or in another language.

Limits

  • No JavaScript rendering. Single-page apps and sites behind bot protection will return little or nothing.
  • Tables, infoboxes and image captions are removed along with the rest of the page chrome, so facts that only appear in those are not in the text.
  • qualityScore is a heuristic for ranking pages against each other, not a measure of factual quality.
  • Respect each site's terms and robots rules, and the licence of the text you collect (Wikipedia text is CC BY-SA).