# AI Training Data Collector — Clean Web Datasets for LLMs (`avinashchby/ai-training-data-collector`) Actor Crawl websites and extract structured, clean text datasets perfect for fine-tuning LLMs and RAG pipelines. Removes boilerplate, deduplicates, and scores content quality. - **URL**: https://apify.com/avinashchby/ai-training-data-collector.md - **Developed by:** [Avinash](https://apify.com/avinashchby) (community) - **Categories:** AI, Marketing - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, NaN bookmarks - **User rating**: No ratings yet ## Pricing Pay per event This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## AI Training Data Collector — Clean Web Datasets for LLMs Crawl websites and extract structured, clean text datasets perfect for fine-tuning LLMs and RAG pipelines. Removes boilerplate, deduplicates content, and scores quality automatically. ### What it does 1. **Cleans HTML content** — strips nav, headers, footers, ads, scripts, cookies, comments, and sidebars automatically 2. **Finds the main content** — intelligently targets `article`, `main`, `[role="main"]`, `.content`, `.post-content`, `.entry-content`, `#content` before falling back to `body` 3. **Converts to structured formats** — markdown, plain text, or JSON output 4. **Scores content quality** — 0-100 score based on length, word diversity, sentence structure, and document formatting 5. **Deduplicates pages** — skips duplicate content using MD5 hashing of the first 2,000 characters 6. **Crawls to configurable depth** — follows same-origin internal links up to 3 levels deep --- ### Input | Field | Type | Default | Description | |-------|------|---------|-------------| | `urls` | array | `[Wikipedia AI]` | Starting URLs to crawl | | `crawlDepth` | integer | `1` | Link depth to follow (0-3). `0` = only start URLs | | `maxPages` | integer | `5` | Maximum pages to process per run (1-1000) | | `outputFormat` | string | `markdown` | `markdown`, `plainText`, or `json` | | `excludePatterns` | array | `[/tag/, /category/]` | URL path patterns to skip | | `minWordCount` | integer | `100` | Skip pages below this word count threshold | --- ### Output #### Dataset Schema | Field | Type | Description | |-------|------|-------------| | `url` | string | Source URL | | `title` | string | Page title or first H1 | | `cleanText` | string | Extracted clean content (markdown/plain text) | | `structuredContent` | object | `{title, body}` — only when `outputFormat: json` | | `wordCount` | integer | Total words extracted | | `qualityScore` | integer | 0-100 quality score | | `sourceDomain` | string | Domain name (www stripped) | | `language` | string | `en` or `unknown` | | `crawlDepth` | integer | Depth level where page was found | | `headingCount` | integer | Number of H1-H6 tags | | `paragraphCount` | integer | Number of `

` tags | | `linkCount` | integer | Number of `` tags | | `images` | integer | Number of `` tags (count only, not extracted) | | `contentHash` | string | MD5 hash of first 2,000 chars for deduplication | | `extractionMethod` | string | Always `cheerio-html2text` | | `scrapedAt` | string | ISO timestamp | #### Output Example ```json { "url": "https://en.wikipedia.org/wiki/Artificial_intelligence", "title": "Artificial intelligence - Wikipedia", "wordCount": 34968, "qualityScore": 100, "sourceDomain": "en.wikipedia.org", "language": "en", "headingCount": 71, "paragraphCount": 180, "linkCount": 5084, "images": 39, "contentHash": "603675ced43c", "extractionMethod": "cheerio-html2text", "cleanText": "## Artificial intelligence\n\nArtificial intelligence (AI) is...", "scrapedAt": "2026-05-19T04:40:16.841Z" } ```` *** ### How Quality Scoring Works The `qualityScore` (0-100) is computed from four dimensions: | Dimension | Weight | How it's calculated | |-----------|--------|---------------------| | **Length** | 0-35 | `min(35, wordCount / 25)` | | **Vocabulary diversity** | 0-25 | `min(25, uniqueWords / totalWords * 100)` | | **Sentence structure** | 0-20 | `min(20, sentenceCount * 1.5)` | | **Document structure** | 0-20 | `min(20, headingCount * 4 + paragraphCount * 0.5)` | **Example scores:** - Wikipedia AI article (34,968 words, 71 headings): **100/100** - A 500-word blog post with 5 headings and 20 paragraphs: ~60-70 - A 200-word page with no headings: ~25-30 (likely skipped by `minWordCount`) *** ### Use Cases - **LLM fine-tuning dataset**: Crawl 100 medical research articles to create a specialized healthcare training corpus - **RAG knowledge base**: Extract clean text from your company docs and blog posts for retrieval-augmented generation - **Content analysis**: Build a dataset of competitor blog posts with quality scores for content strategy - **Academic research**: Collect and deduplicate article text from journal websites *** ### Battle-Tested Results | Test Site | Words | Quality Score | Headings | Paragraphs | Links | Images | |-----------|-------|---------------|----------|------------|-------|--------| | Wikipedia — Artificial Intelligence | 34,968 | 100 | 71 | 180 | 5,084 | 39 | - Deduplication tested across 16 pages — correctly skipped 2 duplicate articles - Low-quality filtering tested at `minWordCount: 500` — correctly skipped navigation-heavy index pages *** ### Limits & Architecture Constraints #### Hard Limits | Limit | Value | Impact | |-------|-------|--------| | **Crawler engine** | Cheerio (no browser) | Cannot execute JavaScript or scrape SPAs | | **Max pages** | 1,000 | Hard ceiling per run | | **Max crawl depth** | 3 levels | Deep pagination truncated | | **Same-origin only** | Yes | External links are not followed | | **Deduplication window** | First 2,000 chars | Pages with identical intros but different bodies flagged as duplicates | #### Content Extraction Weaknesses - **Unusual DOM structures**: If a site doesn't use semantic HTML (`

`, `
`, `.content`), the actor falls back to `body` and may include more noise - **JavaScript-rendered content**: No Playwright = no JS execution. Content loaded via XHR/fetch is invisible - **Paywalls & login gates**: Cheerio sees raw HTML — paywall blurbs or login prompts may get extracted as "content" - **Dynamic lazy loading**: Images and content loaded on scroll are missed #### When It Works Best - ✅ Static blogs and documentation sites - ✅ Wikipedia and wiki-style pages - ✅ News articles with semantic HTML - ✅ Corporate knowledge bases and help centers - ✅ Content-rich pages with clear `article` or `main` tags #### When It Struggles - ❌ JavaScript-heavy SPAs (React, Vue, Angular without SSR) - ❌ Sites with aggressive anti-bot (Cloudflare challenges, CAPTCHA) - ❌ Pages where main content loads dynamically after page load - ❌ Heavily paginated tag/category pages (use `excludePatterns` to skip these) *** ### Pricing - **Free tier**: 5 pages per run - **Pay-per-result**: $0.005 per page processed - **Subscription**: $59/month for unlimited runs *** ### Support Found a bug or need a custom feature? Open an issue or email support. # Actor input Schema ## `urls` (type: `array`): List of URLs to crawl for training data ## `crawlDepth` (type: `integer`): How many levels deep to crawl from start URLs (0 = only start URLs) ## `maxPages` (type: `integer`): Maximum number of pages to process ## `outputFormat` (type: `string`): Format for extracted content ## `excludePatterns` (type: `array`): Regex patterns for URLs to skip (e.g., '/blog/tag/', '/admin/') ## `minWordCount` (type: `integer`): Skip pages with fewer words than this threshold ## Actor input object example ```json { "urls": [ { "url": "https://en.wikipedia.org/wiki/Artificial_intelligence" } ], "crawlDepth": 1, "maxPages": 5, "outputFormat": "markdown", "excludePatterns": [ "/tag/", "/category/", "/author/", "/search/" ], "minWordCount": 100 } ``` # Actor output Schema ## `results` (type: `string`): All processed pages with clean text and quality scores. # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "urls": [ { "url": "https://en.wikipedia.org/wiki/Artificial_intelligence" } ] }; // Run the Actor and wait for it to finish const run = await client.actor("avinashchby/ai-training-data-collector").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "urls": [{ "url": "https://en.wikipedia.org/wiki/Artificial_intelligence" }] } # Run the Actor and wait for it to finish run = client.actor("avinashchby/ai-training-data-collector").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "urls": [ { "url": "https://en.wikipedia.org/wiki/Artificial_intelligence" } ] }' | apify call avinashchby/ai-training-data-collector --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=avinashchby/ai-training-data-collector", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "AI Training Data Collector — Clean Web Datasets for LLMs", "description": "Crawl websites and extract structured, clean text datasets perfect for fine-tuning LLMs and RAG pipelines. Removes boilerplate, deduplicates, and scores content quality.", "version": "1.0", "x-build-id": "TJIumdj47Aq8TOGLV" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/avinashchby~ai-training-data-collector/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-avinashchby-ai-training-data-collector", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/avinashchby~ai-training-data-collector/runs": { "post": { "operationId": "runs-sync-avinashchby-ai-training-data-collector", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/avinashchby~ai-training-data-collector/run-sync": { "post": { "operationId": "run-sync-avinashchby-ai-training-data-collector", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "properties": { "urls": { "title": "Start URLs", "type": "array", "description": "List of URLs to crawl for training data", "items": { "type": "object", "required": [ "url" ], "properties": { "url": { "type": "string", "title": "URL of a web page", "format": "uri" } } } }, "crawlDepth": { "title": "Crawl Depth", "minimum": 0, "maximum": 3, "type": "integer", "description": "How many levels deep to crawl from start URLs (0 = only start URLs)", "default": 1 }, "maxPages": { "title": "Max Pages", "minimum": 1, "maximum": 1000, "type": "integer", "description": "Maximum number of pages to process", "default": 5 }, "outputFormat": { "title": "Output Format", "enum": [ "markdown", "plainText", "json" ], "type": "string", "description": "Format for extracted content", "default": "markdown" }, "excludePatterns": { "title": "Exclude URL Patterns", "type": "array", "description": "Regex patterns for URLs to skip (e.g., '/blog/tag/', '/admin/')", "default": [ "/tag/", "/category/", "/author/", "/search/" ], "items": { "type": "string" } }, "minWordCount": { "title": "Minimum Word Count", "minimum": 10, "type": "integer", "description": "Skip pages with fewer words than this threshold", "default": 100 } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```