# DomainForge LLM Dataset Builder (`actorify/domainforge-llm-dataset-builder`) Actor
Crawls websites and transforms web content into clean, structured datasets optimized for LLM fine-tuning, RAG applications, and knowledge base construction
- **URL**: https://apify.com/actorify/domainforge-llm-dataset-builder.md
- **Developed by:** [fanio zilla](https://apify.com/actorify) (community)
- **Categories:** AI, Automation
- **Stats:** 1 total users, 1 monthly users, 0.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet
## Pricing
from $0.01 / 1,000 results
This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event
## What's an Apify Actor?
Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".
## How to integrate an Actor?
If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.
In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md):
```bash
npm install apify-client
```
In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md):
```bash
pip install apify-client
```
In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md):
````bash
# MacOS / Linux
curl -fsSL https://apify.com/install-cli.sh | bash
# Windows
irm https://apify.com/install-cli.ps1 | iex
```bash
In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md).
If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md).
For usage examples, see the [API](#api) section below.
For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).
# README
## DomainForge LLM Dataset Builder π οΈ
Turn Any Website into Production-Ready LLM Training & RAG Data with One Click.
---
### π‘ Stop Scraping, Start Training
AI Engineers and researchers spend **70-80% of their time** building web scrapers, stripping boilerplate HTML, deduplicating content, and chunking files.
**DomainForge** does all of this for you automatically.
Whether you are fine-tuning a custom LLM, building a Retrieval-Augmented Generation (RAG) pipeline, or feeding a vector database, DomainForge crawls your target websites and processes them into clean, structured, and deduplicated datasets instantly.
---
### β¨ Superpowers
* **π·οΈ Smart Website Crawling**: Crawl recursively, target specific URL paths (using globs or regex), or ingest entire sites instantly using `sitemap.xml` files.
* **π§Ό High-Fidelity Noise Removal**: Powered by Mozilla's Readability engine. We strip headers, footers, sidebars, cookie banners, navigation menus, and adsβleaving only the pristine, valuable content.
* **𧩠LLM-Optimized Auto-Chunking**: Automatically splits long articles into clean, overlapping chunks (customizable chunk sizes and overlaps) optimized for vector store embeddings.
* **β‘ Exact Deduplication**: Removes duplicate pages and identical content blocks using SHA-256 hashing, collapsing republished or near-identical content into a single canonical record. *(Semantic, embedding-based near-duplicate deduplication is on the roadmap.)*
* **π Multi-Format Export**: Download your clean dataset as **JSONL (Hugging Face ready)**, **JSON**, **CSV**, or raw **Markdown files**.
---
### π How It Works
DomainForge operates as a state-of-the-art data refinery:
```mermaid
graph LR
A[Raw URL / Sitemap] --> B[Smart Crawler]
B --> C[Noise & Boilerplate Stripper]
C --> D[SHA-256 Deduplication]
D --> E[Semantic Chunking]
E --> F[LLM-Ready Dataset]
style A fill:#4dabf7,stroke:#228be6,stroke-width:2px,color:#fff
style F fill:#37b24d,stroke:#2b8a3e,stroke-width:2px,color:#fff
style B fill:#1c7ed6,stroke:#1971c2,stroke-width:1px,color:#fff
style C fill:#1c7ed6,stroke:#1971c2,stroke-width:1px,color:#fff
style D fill:#1c7ed6,stroke:#1971c2,stroke-width:1px,color:#fff
style E fill:#1c7ed6,stroke:#1971c2,stroke-width:1px,color:#fff
````
1. **Input a URL**: Just enter a start URL (e.g. `https://docs.yourcompany.com`) or a sitemap.
2. **Forge processes the data**: The actor extracts metadata (author, publish date, language, counts), cleans the markup, deduplicates content, and splits it into semantic chunks.
3. **Deploy**: Directly ingest the output into your vector databases (Pinecone, Chroma, Qdrant) or Hugging Face datasets.
***
### π οΈ Simple No-Code Input
DomainForge is built for both developers and non-technical builders. You can configure it with a simple JSON or run it through the Apify Console UI.
#### Simple Configuration (Just the basics)
```json
{
"startUrls": [{ "url": "https://docs.apify.com" }],
"maxCrawlPages": 100
}
```
#### Full Enterprise Settings (Complete control)
```json
{
"startUrls": [
{ "url": "https://example.com/blog" },
{ "url": "https://example.com/sitemap.xml" }
],
"maxCrawlPages": 1000,
"maxDepth": 3,
"includePatterns": ["*/blog/*", "*/docs/*"],
"excludePatterns": ["*/admin/*", "*/login*"],
"respectRobotsTxt": true,
"saveMarkdown": true,
"enableDeduplication": true,
"chunkSize": 1024,
"chunkOverlap": 200
}
```
***
### π Beautiful Structured Output
Each crawled page is returned as a structured item ready for database storage or model ingestion:
```json
{
"url": "https://example.com/blog/getting-started",
"title": "Getting Started with AI Datasets",
"markdown": "# Getting Started with AI Datasets\n\nHigh quality data is all you need...",
"text": "Getting Started with AI Datasets High quality data is all you need...",
"metadata": {
"author": "Jane Doe",
"publishDate": "2026-06-21T00:00:00.000Z",
"language": "en",
"wordCount": 745,
"tokenCountApprox": 980,
"crawledAt": "2026-06-21T12:00:00.000Z"
},
"chunks": [
{ "text": "High quality data is all you need to train custom LLMs...", "tokenCount": 512 }
],
"dedupHash": "8f3b2a9e..."
}
```
***
### π‘ Quick Tips for Best Results
- **Sitemaps are your friend**: Use the sitemap URL (e.g. `https://example.com/sitemap.xml`) to crawl an entire site instantly. Set `maxDepth` to `0` to only crawl pages listed in the sitemap.
- **Targeted Ingestion**: Use `includePatterns` (like `*/docs/*` or `*/help/*`) to prevent wasting compute on irrelevant pages like contacts or terms of service.
- **Hugging Face Friendly**: Download the dataset in `JSONL` format using the Apify API for native integration with Hugging Face pipelines.
***
### π Example Configurations
Pre-built input presets for common scenarios live in the [`examples/`](./examples) directory. Copy one into the Apify Console (or pass it as the Actor input) and adjust the URLs.
| Preset | Best for | Highlights |
| --- | --- | --- |
| [`blog-crawl.json`](./examples/blog-crawl.json) | Harvesting a blog | Path-scoped to `/blog/`, skips tag/author/pagination noise, exact-dedup on |
| [`documentation-sitemap.json`](./examples/documentation-sitemap.json) | Ingesting a whole docs site | Sitemap-driven, `maxDepth 0` crawls only listed pages β fast and complete |
| [`rag-chunking.json`](./examples/rag-chunking.json) | Building a RAG / vector corpus | Small 512-char chunks, tight overlap, markdown omitted for a lean dataset |
***
### π Try DomainForge Now
Get started in seconds! Run this actor directly on the **[Apify Console](https://apify.com/store)**:
```bash
apify call domainforge-llm-dataset-builder
```
*For custom builds, bug reports, or feature requests, feel free to check the project [License](LICENSE) and source code.*
# Actor input Schema
## `startUrls` (type: `array`):
One or more URLs to begin crawling. Each entry must be a valid HTTP or HTTPS URL.
## `maxCrawlPages` (type: `integer`):
Maximum number of pages to crawl during this run. Crawling stops when this limit is reached.
## `maxDepth` (type: `integer`):
Maximum number of link levels to follow from start URLs. Depth 0 means only the start URLs are crawled.
## `includePatterns` (type: `array`):
Glob or regex patterns for URLs to include. When non-empty, only matching URLs are crawled. Use /regex/ syntax or glob patterns like */blog/*.
## `excludePatterns` (type: `array`):
Glob or regex patterns for URLs to exclude. Exclude patterns take precedence over include patterns.
## `respectRobotsTxt` (type: `boolean`):
Follow robots.txt rules when crawling. Disallowed URLs are skipped and counted in crawl statistics.
## `requestDelay` (type: `integer`):
Delay between consecutive requests to the same domain. Helps avoid overloading target servers.
## `saveMarkdown` (type: `boolean`):
Include Markdown formatted content in the output dataset item.
## `saveHtml` (type: `boolean`):
Include original HTML content in the output dataset item.
## `enableDeduplication` (type: `boolean`):
Remove exact duplicate content (identical SHA-256 hash) from the dataset.
## `embeddingProvider` (type: `string`):
Provider for generating embeddings used in semantic deduplication. Set to 'none' to disable semantic deduplication.
## `semanticDeduplicationThreshold` (type: `number`):
Cosine similarity threshold above which two pages are considered semantically duplicate (0.0-1.0).
## `chunkSize` (type: `integer`):
Target size for content chunks in characters. Used by the Chunking Engine for RAG applications.
## `chunkOverlap` (type: `integer`):
Number of characters to overlap between consecutive chunks. Must be less than chunkSize.
## Actor input object example
```json
{
"startUrls": [
{
"url": "https://docs.apify.com/academy/web-scraping-for-beginners"
}
],
"maxCrawlPages": 10,
"maxDepth": 1,
"includePatterns": [],
"excludePatterns": [],
"respectRobotsTxt": true,
"requestDelay": 300,
"saveMarkdown": true,
"saveHtml": false,
"enableDeduplication": true,
"embeddingProvider": "none",
"semanticDeduplicationThreshold": 0.85,
"chunkSize": 1024,
"chunkOverlap": 200
}
```
# Actor output Schema
## `dataset` (type: `string`):
Cleaned, deduplicated, and chunked content items ready for LLM fine-tuning, RAG, or knowledge bases
# API
You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.
## JavaScript example
```javascript
import { ApifyClient } from 'apify-client';
// Initialize the ApifyClient with your Apify API token
// Replace the '' with your token
const client = new ApifyClient({
token: '',
});
// Prepare Actor input
const input = {
"startUrls": [
{
"url": "https://docs.apify.com/academy/web-scraping-for-beginners"
}
],
"maxCrawlPages": 10,
"maxDepth": 1,
"requestDelay": 300
};
// Run the Actor and wait for it to finish
const run = await client.actor("actorify/domainforge-llm-dataset-builder").call(input);
// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`πΎ Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
console.dir(item);
});
// π Want to learn more π? Go to β https://docs.apify.com/api/client/js/docs
```
## Python example
```python
from apify_client import ApifyClient
# Initialize the ApifyClient with your Apify API token
# Replace '' with your token.
client = ApifyClient("")
# Prepare the Actor input
run_input = {
"startUrls": [{ "url": "https://docs.apify.com/academy/web-scraping-for-beginners" }],
"maxCrawlPages": 10,
"maxDepth": 1,
"requestDelay": 300,
}
# Run the Actor and wait for it to finish
run = client.actor("actorify/domainforge-llm-dataset-builder").call(run_input=run_input)
# Fetch and print Actor results from the run's dataset (if there are any)
print("πΎ Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"])
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
print(item)
# π Want to learn more π? Go to β https://docs.apify.com/api/client/python/docs/quick-start
```
## CLI example
```bash
echo '{
"startUrls": [
{
"url": "https://docs.apify.com/academy/web-scraping-for-beginners"
}
],
"maxCrawlPages": 10,
"maxDepth": 1,
"requestDelay": 300
}' |
apify call actorify/domainforge-llm-dataset-builder --silent --output-dataset
```
## MCP server setup
```json
{
"mcpServers": {
"apify": {
"command": "npx",
"args": [
"mcp-remote",
"https://mcp.apify.com/?tools=actorify/domainforge-llm-dataset-builder",
"--header",
"Authorization: Bearer "
]
}
}
}
```
## OpenAPI specification
```json
{
"openapi": "3.0.1",
"info": {
"title": "DomainForge LLM Dataset Builder",
"description": "Crawls websites and transforms web content into clean, structured datasets optimized for LLM fine-tuning, RAG applications, and knowledge base construction",
"version": "1.0",
"x-build-id": "qHJGNmXEjri4zxB3O"
},
"servers": [
{
"url": "https://api.apify.com/v2"
}
],
"paths": {
"/acts/actorify~domainforge-llm-dataset-builder/run-sync-get-dataset-items": {
"post": {
"operationId": "run-sync-get-dataset-items-actorify-domainforge-llm-dataset-builder",
"x-openai-isConsequential": false,
"summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.",
"tags": [
"Run Actor"
],
"requestBody": {
"required": true,
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/inputSchema"
}
}
}
},
"parameters": [
{
"name": "token",
"in": "query",
"required": true,
"schema": {
"type": "string"
},
"description": "Enter your Apify token here"
}
],
"responses": {
"200": {
"description": "OK"
}
}
}
},
"/acts/actorify~domainforge-llm-dataset-builder/runs": {
"post": {
"operationId": "runs-sync-actorify-domainforge-llm-dataset-builder",
"x-openai-isConsequential": false,
"summary": "Executes an Actor and returns information about the initiated run in response.",
"tags": [
"Run Actor"
],
"requestBody": {
"required": true,
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/inputSchema"
}
}
}
},
"parameters": [
{
"name": "token",
"in": "query",
"required": true,
"schema": {
"type": "string"
},
"description": "Enter your Apify token here"
}
],
"responses": {
"200": {
"description": "OK",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/runsResponseSchema"
}
}
}
}
}
}
},
"/acts/actorify~domainforge-llm-dataset-builder/run-sync": {
"post": {
"operationId": "run-sync-actorify-domainforge-llm-dataset-builder",
"x-openai-isConsequential": false,
"summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.",
"tags": [
"Run Actor"
],
"requestBody": {
"required": true,
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/inputSchema"
}
}
}
},
"parameters": [
{
"name": "token",
"in": "query",
"required": true,
"schema": {
"type": "string"
},
"description": "Enter your Apify token here"
}
],
"responses": {
"200": {
"description": "OK"
}
}
}
}
},
"components": {
"schemas": {
"inputSchema": {
"type": "object",
"required": [
"startUrls"
],
"properties": {
"startUrls": {
"title": "Start URLs",
"minItems": 1,
"type": "array",
"description": "One or more URLs to begin crawling. Each entry must be a valid HTTP or HTTPS URL.",
"items": {
"type": "object",
"required": [
"url"
],
"properties": {
"url": {
"type": "string",
"title": "URL",
"description": "A valid HTTP or HTTPS URL to crawl.",
"pattern": "^https?://"
}
},
"additionalProperties": false
}
},
"maxCrawlPages": {
"title": "Maximum pages to crawl",
"minimum": 1,
"maximum": 1000000,
"type": "integer",
"description": "Maximum number of pages to crawl during this run. Crawling stops when this limit is reached.",
"default": 1000
},
"maxDepth": {
"title": "Maximum crawl depth",
"minimum": 0,
"maximum": 10,
"type": "integer",
"description": "Maximum number of link levels to follow from start URLs. Depth 0 means only the start URLs are crawled.",
"default": 3
},
"includePatterns": {
"title": "URL include patterns",
"type": "array",
"description": "Glob or regex patterns for URLs to include. When non-empty, only matching URLs are crawled. Use /regex/ syntax or glob patterns like */blog/*.",
"items": {
"type": "string",
"minLength": 1
},
"default": []
},
"excludePatterns": {
"title": "URL exclude patterns",
"type": "array",
"description": "Glob or regex patterns for URLs to exclude. Exclude patterns take precedence over include patterns.",
"items": {
"type": "string",
"minLength": 1
},
"default": []
},
"respectRobotsTxt": {
"title": "Respect robots.txt",
"type": "boolean",
"description": "Follow robots.txt rules when crawling. Disallowed URLs are skipped and counted in crawl statistics.",
"default": true
},
"requestDelay": {
"title": "Request delay (milliseconds)",
"minimum": 100,
"maximum": 60000,
"type": "integer",
"description": "Delay between consecutive requests to the same domain. Helps avoid overloading target servers.",
"default": 1000
},
"saveMarkdown": {
"title": "Save Markdown",
"type": "boolean",
"description": "Include Markdown formatted content in the output dataset item.",
"default": true
},
"saveHtml": {
"title": "Save HTML",
"type": "boolean",
"description": "Include original HTML content in the output dataset item.",
"default": false
},
"enableDeduplication": {
"title": "Enable deduplication",
"type": "boolean",
"description": "Remove exact duplicate content (identical SHA-256 hash) from the dataset.",
"default": true
},
"embeddingProvider": {
"title": "Embedding provider for semantic deduplication",
"enum": [
"none",
"bge",
"openai"
],
"type": "string",
"description": "Provider for generating embeddings used in semantic deduplication. Set to 'none' to disable semantic deduplication.",
"default": "none"
},
"semanticDeduplicationThreshold": {
"title": "Semantic deduplication threshold",
"minimum": 0,
"maximum": 1,
"type": "number",
"description": "Cosine similarity threshold above which two pages are considered semantically duplicate (0.0-1.0).",
"default": 0.85
},
"chunkSize": {
"title": "Chunk size (characters)",
"minimum": 100,
"maximum": 10000,
"type": "integer",
"description": "Target size for content chunks in characters. Used by the Chunking Engine for RAG applications.",
"default": 1024
},
"chunkOverlap": {
"title": "Chunk overlap (characters)",
"minimum": 0,
"maximum": 9999,
"type": "integer",
"description": "Number of characters to overlap between consecutive chunks. Must be less than chunkSize.",
"default": 200
}
}
},
"runsResponseSchema": {
"type": "object",
"properties": {
"data": {
"type": "object",
"properties": {
"id": {
"type": "string"
},
"actId": {
"type": "string"
},
"userId": {
"type": "string"
},
"startedAt": {
"type": "string",
"format": "date-time",
"example": "2025-01-08T00:00:00.000Z"
},
"finishedAt": {
"type": "string",
"format": "date-time",
"example": "2025-01-08T00:00:00.000Z"
},
"status": {
"type": "string",
"example": "READY"
},
"meta": {
"type": "object",
"properties": {
"origin": {
"type": "string",
"example": "API"
},
"userAgent": {
"type": "string"
}
}
},
"stats": {
"type": "object",
"properties": {
"inputBodyLen": {
"type": "integer",
"example": 2000
},
"rebootCount": {
"type": "integer",
"example": 0
},
"restartCount": {
"type": "integer",
"example": 0
},
"resurrectCount": {
"type": "integer",
"example": 0
},
"computeUnits": {
"type": "integer",
"example": 0
}
}
},
"options": {
"type": "object",
"properties": {
"build": {
"type": "string",
"example": "latest"
},
"timeoutSecs": {
"type": "integer",
"example": 300
},
"memoryMbytes": {
"type": "integer",
"example": 1024
},
"diskMbytes": {
"type": "integer",
"example": 2048
}
}
},
"buildId": {
"type": "string"
},
"defaultKeyValueStoreId": {
"type": "string"
},
"defaultDatasetId": {
"type": "string"
},
"defaultRequestQueueId": {
"type": "string"
},
"buildNumber": {
"type": "string",
"example": "1.0.0"
},
"containerUrl": {
"type": "string"
},
"usage": {
"type": "object",
"properties": {
"ACTOR_COMPUTE_UNITS": {
"type": "integer",
"example": 0
},
"DATASET_READS": {
"type": "integer",
"example": 0
},
"DATASET_WRITES": {
"type": "integer",
"example": 0
},
"KEY_VALUE_STORE_READS": {
"type": "integer",
"example": 0
},
"KEY_VALUE_STORE_WRITES": {
"type": "integer",
"example": 1
},
"KEY_VALUE_STORE_LISTS": {
"type": "integer",
"example": 0
},
"REQUEST_QUEUE_READS": {
"type": "integer",
"example": 0
},
"REQUEST_QUEUE_WRITES": {
"type": "integer",
"example": 0
},
"DATA_TRANSFER_INTERNAL_GBYTES": {
"type": "integer",
"example": 0
},
"DATA_TRANSFER_EXTERNAL_GBYTES": {
"type": "integer",
"example": 0
},
"PROXY_RESIDENTIAL_TRANSFER_GBYTES": {
"type": "integer",
"example": 0
},
"PROXY_SERPS": {
"type": "integer",
"example": 0
}
}
},
"usageTotalUsd": {
"type": "number",
"example": 0.00005
},
"usageUsd": {
"type": "object",
"properties": {
"ACTOR_COMPUTE_UNITS": {
"type": "integer",
"example": 0
},
"DATASET_READS": {
"type": "integer",
"example": 0
},
"DATASET_WRITES": {
"type": "integer",
"example": 0
},
"KEY_VALUE_STORE_READS": {
"type": "integer",
"example": 0
},
"KEY_VALUE_STORE_WRITES": {
"type": "number",
"example": 0.00005
},
"KEY_VALUE_STORE_LISTS": {
"type": "integer",
"example": 0
},
"REQUEST_QUEUE_READS": {
"type": "integer",
"example": 0
},
"REQUEST_QUEUE_WRITES": {
"type": "integer",
"example": 0
},
"DATA_TRANSFER_INTERNAL_GBYTES": {
"type": "integer",
"example": 0
},
"DATA_TRANSFER_EXTERNAL_GBYTES": {
"type": "integer",
"example": 0
},
"PROXY_RESIDENTIAL_TRANSFER_GBYTES": {
"type": "integer",
"example": 0
},
"PROXY_SERPS": {
"type": "integer",
"example": 0
}
}
}
}
}
}
}
}
}
}
```