# Website Content Crawler for AI Datasets (`boztek-ltd/website-content-crawler`) Actor

Crawl any website and turn its content into AI-ready data: clean Markdown,
token-aware RAG chunks, Q\&A pairs, and OpenAI fine-tuning JSONL. Removes
navigation and boilerplate, scores content quality, and deduplicates pages.
Works with 🦜🔗 LangChain, LlamaIndex, Pinecone, and Qdrant.

- **URL**: https://apify.com/boztek-ltd/website-content-crawler.md
- **Developed by:** [Boztek LTD](https://apify.com/boztek-ltd) (community)
- **Categories:** AI, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.008 / actor start

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## AI Dataset Converter — Website to AI Training Data

Convert any website into AI-ready datasets for **RAG pipelines**, **LLM fine-tuning**, and **Q\&A training**. Token-aware chunking, quality scoring, content deduplication — all without external API calls.

### What does AI Dataset Converter do?

AI Dataset Converter crawls websites and transforms their content into structured, token-aware datasets optimized for AI/ML workflows:

- **RAG Chunks** — Embedding-ready JSON with configurable chunk size and overlap
- **Fine-tuning JSONL** — OpenAI-compatible `messages[]` format
- **Q\&A Pairs** — Automatically extracted from FAQ pages and heading structures
- **Clean Markdown** — Boilerplate-free content with full page metadata

Every chunk includes the cl100k\_base (GPT-4 compatible) token count, a 0.0–1.0 quality score, source URL, language, and canonical URL — ready to ingest into Pinecone, Qdrant, Weaviate, LangChain, LlamaIndex, or any vector store.

### Why AI Dataset Converter?

| Feature | Website Content Crawler | **AI Dataset Converter** |
|---|---|---|
| Output | Raw Markdown / text | **Structured AI-ready formats** |
| Chunking | Manual | **Token-aware, configurable** |
| Token counting | — | **cl100k\_base (GPT-4)** |
| Q\&A extraction | — | **5 rule-based strategies** |
| Quality scoring | — | **0.0–1.0 per page** |
| Deduplication | URL-based | **Content fingerprinting** |
| Fine-tuning format | — | **OpenAI JSONL** |
| External LLM cost | None | **None** |

### How much does it cost?

AI Dataset Converter uses **pay-per-event** pricing at approximately **$0.002 per output item** (chunk, Q\&A pair, or page). Platform compute units are included.

| Use case | Pages | Output items | Estimated cost |
|---|---|---|---|
| Small docs site | 50 | ~250 chunks | ~$0.50 |
| Medium blog | 500 | ~2,500 chunks | ~$5.00 |
| Large docs + FAQ | 2,000 | ~12,000 items | ~$24.00 |

Apify's free plan provides $5 of platform credit per month — enough to test on small sites.

### Output formats

#### 1. RAG Chunks (`rag-chunks`)

One JSON item per chunk with embedding-ready text plus rich metadata:

```json
{
  "chunk_id": "550e8400-e29b-41d4-a716-446655440000",
  "source_url": "https://docs.example.com/getting-started",
  "canonical_url": "https://docs.example.com/getting-started",
  "text": "Getting started with Example SDK...",
  "markdown": "# Getting Started\n\nWelcome to...",
  "chunk_index": 0,
  "total_chunks": 3,
  "token_count": 487,
  "char_count": 1843,
  "page_title": "Getting Started",
  "page_description": "Quick start guide",
  "page_language": "en",
  "page_author": "Docs Team",
  "page_date": "2026-04-12T00:00:00.000Z",
  "quality_score": 0.85,
  "content_type": "documentation",
  "crawled_at": "2026-05-12T08:30:00.000Z",
  "actor_version": "1.0.0"
}
```

#### 2. Fine-tuning JSONL (`fine-tuning-jsonl`)

OpenAI-compatible `messages[]` format. Prompts are synthesized rule-based (no LLM):

```json
{
  "messages": [
    { "role": "system", "content": "You are a helpful assistant that provides information about Example Documentation." },
    { "role": "user",   "content": "What is the chunk size?" },
    { "role": "assistant", "content": "The chunk size is the target number of tokens per output chunk..." }
  ],
  "_metadata": {
    "source_url": "https://docs.example.com/chunking",
    "chunk_id": "...",
    "token_count": 412,
    "quality_score": 0.81
  }
}
```

#### 3. Q\&A Pairs (`qa-pairs`)

Extracted from FAQ pages using five rule-based strategies:

```json
{
  "question": "Can I cancel my subscription?",
  "answer": "Yes, you can cancel anytime from the billing settings page in your account.",
  "source_url": "https://example.com/help/faq",
  "extraction_method": "faq_html",
  "confidence": 0.95,
  "token_count": 28,
  "page_title": "FAQ"
}
```

Extraction strategies (in confidence order):

1. **`faq_schema`** — JSON-LD `FAQPage` schema (confidence 1.0)
2. **`faq_html`** — `<details><summary>` elements (0.95)
3. **`dt_dd`** — Definition lists `<dl>/<dt>/<dd>` (0.90)
4. **`accordion`** — `aria-controls` / `data-toggle` patterns (0.85)
5. **`heading_paragraph`** — `<h2>/<h3>` + following content (0.5–0.9)

#### 4. Clean Markdown (`markdown`)

Full-page Markdown with boilerplate removed and complete metadata.

### Input options

| Option | Type | Default | Description |
|---|---|---|---|
| `startUrls` | array | **required** | Initial URLs to crawl |
| `maxPages` | integer | 100 | Maximum number of pages (0 = unlimited) |
| `maxDepth` | integer | 5 | Link-follow depth from start URLs |
| `crawlerType` | string | `adaptive` | `adaptive` / `cheerio` / `playwright` |
| `includeGlobs` / `excludeGlobs` | array | `[]` | URL pattern filters |
| `outputFormat` | string | `rag-chunks` | `rag-chunks` / `fine-tuning-jsonl` / `qa-pairs` / `markdown` / `all` |
| `chunkSize` | integer | 512 | Target tokens per chunk |
| `chunkOverlap` | integer | 50 | Token overlap between chunks |
| `extractQAPairs` | boolean | `true` | Run Q\&A extraction strategies |
| `language` | string | `""` | ISO 639-1 code language filter |
| `minContentLength` | integer | 100 | Skip pages shorter than this (chars) |
| `minQualityScore` | number | 0.3 | Skip pages below this score (0.0–1.0) |
| `removeDuplicates` | boolean | `true` | Content-fingerprint deduplication |
| `removeBoilerplate` | boolean | `true` | Strip nav/footer/cookie banners |
| `proxyConfiguration` | object | Apify Proxy | Proxy settings |
| `maxConcurrency` | integer | 10 | Parallel page processing |

### Use cases

1. **Build RAG chatbots** — Crawl documentation → chunk → embed in Pinecone/Qdrant/Weaviate
2. **Fine-tune LLMs** — Convert knowledge bases to OpenAI training format
3. **Create Q\&A datasets** — Extract FAQ data for customer-support AI
4. **Feed AI agents** — Provide structured web knowledge to autonomous agents

### Integrations

Output is plain JSON / JSONL and works with LangChain, LlamaIndex, Pinecone, Qdrant, Weaviate, Milvus, MongoDB Atlas, OpenAI fine-tuning, and any tool that accepts JSON.

### Quality scoring (heuristic, no LLM)

Each page receives a 0.0–1.0 score computed from:

- **Content length** (25%) — Pages between 500 and 10000 chars score highest
- **Text density** (25%) — Ratio of extracted text to original HTML
- **Paragraph count** (15%) — ≥3 paragraphs preferred
- **Heading presence** (10%) — At least one `<h1>`–`<h6>`
- **Link density** (10%) — Low anchor-text ratio preferred
- **Repetition** (15%) — Unique-sentence ratio

Pages scoring below `minQualityScore` are filtered out before token usage.

### Token-aware chunking

Chunks are produced with a recursive splitter that respects natural boundaries:

1. Split by paragraph (`\n\n`)
2. If a paragraph exceeds `chunkSize`, split by sentence
3. If a sentence exceeds `chunkSize`, split by token
4. Apply `chunkOverlap` by prepending the last N tokens of the previous chunk

Token counts are computed with **js-tiktoken** using the `cl100k_base` encoding — identical to GPT-4 / `text-embedding-3-*`.

### Limitations

- No LLM-based extraction (by design — keeps cost predictable)
- Q\&A extraction works best on structured pages (FAQ, docs with headings)
- Login-protected content not supported without cookie injection
- JavaScript-heavy SPAs may need `crawlerType: "playwright"` for full rendering

# Actor input Schema

## `startUrls` (type: `array`):

Initial URLs to crawl. Sub-pages are automatically discovered and crawled based on depth and limit settings.

## `maxPages` (type: `integer`):

Maximum number of pages to crawl. Set to 0 for unlimited.

## `maxDepth` (type: `integer`):

How many link hops from a start URL the crawler will follow. 0 = only start URLs.

## `crawlerType` (type: `string`):

Crawler engine. Cheerio (default) is fast and cheap; switch to Playwright or Adaptive only for JavaScript-heavy single-page apps.

## `includeGlobs` (type: `array`):

Only URLs matching these glob patterns will be crawled. Leave empty to crawl all discovered URLs.

## `excludeGlobs` (type: `array`):

URLs matching these glob patterns will be skipped.

## `outputFormat` (type: `string`):

Primary output format. Choose 'All formats' to produce all four into separate named datasets.

## `chunkSize` (type: `integer`):

Target token count per chunk (cl100k\_base / GPT-4 compatible). Applies to RAG and fine-tuning formats.

## `chunkOverlap` (type: `integer`):

Token overlap between consecutive chunks. Helps preserve context across boundaries.

## `extractQAPairs` (type: `boolean`):

Automatically extract question-answer pairs from FAQ pages and heading+paragraph structures.

## `language` (type: `string`):

ISO 639-1 code (e.g. 'en', 'tr', 'de'). Only pages in this language are included. Leave empty to allow all.

## `minContentLength` (type: `integer`):

Pages with cleaned content shorter than this are skipped.

## `minQualityScore` (type: `number`):

Pages scoring below this heuristic threshold are excluded from output.

## `removeDuplicates` (type: `boolean`):

Detect and skip near-duplicate pages using content fingerprinting.

## `removeBoilerplate` (type: `boolean`):

Strip navigation, footers, cookie banners, and other non-content elements.

## `proxyConfiguration` (type: `object`):

Apify Proxy settings. Residential proxy recommended for geo-aware sites.

## `maxConcurrency` (type: `integer`):

Maximum number of pages processed in parallel. Browser crawler: 1-5; HTTP crawler: 10-50.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://docs.apify.com/"
    }
  ],
  "maxPages": 100,
  "maxDepth": 5,
  "crawlerType": "cheerio",
  "includeGlobs": [],
  "excludeGlobs": [
    {
      "glob": "**/*.pdf"
    },
    {
      "glob": "**/*.zip"
    },
    {
      "glob": "**/login*"
    }
  ],
  "outputFormat": "rag-chunks",
  "chunkSize": 512,
  "chunkOverlap": 50,
  "extractQAPairs": true,
  "language": "",
  "minContentLength": 100,
  "minQualityScore": 0.3,
  "removeDuplicates": true,
  "removeBoilerplate": true,
  "proxyConfiguration": {
    "useApifyProxy": true
  },
  "maxConcurrency": 10
}
```

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://docs.apify.com/"
        }
    ],
    "includeGlobs": [],
    "excludeGlobs": [
        {
            "glob": "**/*.pdf"
        },
        {
            "glob": "**/*.zip"
        },
        {
            "glob": "**/login*"
        }
    ],
    "proxyConfiguration": {
        "useApifyProxy": true
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("boztek-ltd/website-content-crawler").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": [{ "url": "https://docs.apify.com/" }],
    "includeGlobs": [],
    "excludeGlobs": [
        { "glob": "**/*.pdf" },
        { "glob": "**/*.zip" },
        { "glob": "**/login*" },
    ],
    "proxyConfiguration": { "useApifyProxy": True },
}

# Run the Actor and wait for it to finish
run = client.actor("boztek-ltd/website-content-crawler").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://docs.apify.com/"
    }
  ],
  "includeGlobs": [],
  "excludeGlobs": [
    {
      "glob": "**/*.pdf"
    },
    {
      "glob": "**/*.zip"
    },
    {
      "glob": "**/login*"
    }
  ],
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}' |
apify call boztek-ltd/website-content-crawler --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,boztek-ltd/website-content-crawler"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/cwZEchd2bI7JLSArx/builds/JCdi7bWCVJOC3Vluv/openapi.json