# Vertical Corpora Builder - Domain Datasets for LLM Fine-Tuning (`darknezz/vertical-corpus-builder`) Actor

Build domain-specific text corpora (legal, medical, financial) for LLM fine-tuning: crawl seed sources, strip boilerplate, dedupe, and emit token-aware chunks as JSONL with full provenance. Outputs a ready-to-train dataset.

- **URL**: https://apify.com/darknezz/vertical-corpus-builder.md
- **Developed by:** [Oaida Adrian](https://apify.com/darknezz) (community)
- **Categories:** AI, Developer tools, Automation
- **Stats:** 2 total users, 1 monthly users, 0.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.00 / 1,000 corpus chunk extracteds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Vertical Corpora Builder — Domain Datasets for LLM Fine-Tuning

Build **domain-specific text corpora** (legal, medical, financial) for LLM
fine-tuning, RAG evaluation, or model pre-training. Point the actor at seed
sources in your vertical; it crawls, strips boilerplate, removes duplicates,
and emits **token-aware chunks as JSONL with full provenance** — a
ready-to-train dataset, not a pile of HTML.

### Why this actor

Raw web pages are terrible training data: nav bars, cookie banners, "related
articles" and ads drown the signal, and near-identical syndicated text
pollutes the corpus. This actor applies the three cleaning steps that matter
for vertical corpora:

1. **Boilerplate strip** — removes `script`/`style`/`nav`/`footer`/`header`/
   `aside` and common utility blocks (breadcrumbs, shares, comments, menus),
   keeping the article body's paragraphs, list items and quotes.
2. **Dedupe (exact + near)** — exact duplicates are removed by normalised
   text hash; near-duplicates are collapsed by 6-gram Jaccard similarity
   (default threshold 0.95), so syndicated copies appear once.
3. **Token-aware chunking** — chunks are split at paragraph boundaries and
   hard-split on sentence/word boundaries only when a single paragraph
   exceeds the budget (default 512 tokens, ~4 chars/token). Each chunk stays
   coherent, which is what fine-tuning actually wants.

Every output record carries provenance: `source` (hostname), `url` (exact
page), `domain` (your vertical), `title` and `chunk_index` — so you can
filter, cite, or re-weight the corpus later.

### How it works

```
seed URLs ──▶ crawl (same-domain BFS) ──▶ clean (boilerplate strip)
   ──▶ dedupe (exact + near, configurable) ──▶ token-aware chunk
   ──▶ JSONL records {text, source, url, domain, chunk_index, title}
```

The crawl logic follows the house `ai-web-crawler` pattern: a browser
User-Agent, same-domain breadth-first link discovery (bounded by
`maxPagesPerSource`), and polite parallel fetching. Each seed URL is a
starting point; set `maxPagesPerSource: 1` to extract only the seed pages,
or raise it to crawl whole sites.

### Input

```json
{
  "domain": "legal",
  "sources": [
    "https://www.law.cornell.edu/supremecourt/text/19-1392"
  ],
  "outputFormat": "jsonl",
  "chunkSize": 512,
  "maxPagesPerSource": 1,
  "maxChunksPerPage": 0,
  "dedupeSimilarity": 0.95
}
```

| Field | Type | Default | Description |
|---|---|---|---|
| `domain` | string | `legal` | Vertical: `legal`, `medical` or `financial`. Written into every record's `domain` field. |
| `sources` | array\<string> | — | Seed URLs to crawl (required). |
| `outputFormat` | string | `jsonl` | `jsonl` also writes a downloadable `output.jsonl` to the key-value store. |
| `chunkSize` | integer | `512` | Target chunk size in tokens (64–4096). |
| `maxPagesPerSource` | integer | `1` | Pages crawled per seed (1–100). `1` = seed pages only. |
| `maxChunksPerPage` | integer | `0` | Cap on chunks kept per page (`0` = unlimited). |
| `dedupeSimilarity` | integer | `0.95` | Near-duplicate Jaccard threshold (0.5–1.0). `1.0` = exact dedupe only. |

### Output

One dataset record per chunk (JSONL export = one line per chunk):

```json
{
  "text": "The Court has long recognized ...",
  "source": "courtlistener.com",
  "url": "https://www.courtlistener.com/opinion/4855702/...",
  "domain": "legal",
  "chunk_index": 0,
  "title": "Dobbs v. Jackson Women's Health Organization"
}
```

With `outputFormat: "jsonl"`, the same records are also written to
`output.jsonl` in the actor's default key-value store (Content-Type
`application/x-ndjson`) for direct download. A `SUMMARY` record reports
`totalChunks`, `totalPages`, `rawChunks`, `duplicatesRemoved`,
`estimatedTokens`, and any `failedSources`.

### Example: legal corpus from public court opinions

```json
{
  "domain": "legal",
  "sources": [
    "https://www.law.cornell.edu/supremecourt/text/19-1392",
    "https://www.law.cornell.edu/supremecourt/text/20-843"
  ],
  "chunkSize": 512,
  "maxPagesPerSource": 1
}
```

Run from the API:

```bash
curl -X POST "https://api.apify.com/v2/acts/darknezz~vertical-corpus-builder/run-sync-get-dataset-items?token=YOUR_TOKEN&timeout=120" \
  -H "Content-Type: application/json" \
  -d '{"domain":"legal","sources":["https://www.law.cornell.edu/supremecourt/text/19-1392"],"chunkSize":512,"maxPagesPerSource":1}'
```

### Use cases

- **Fine-tuning** a domain-specialised LLM (legal reasoning, medical QA,
  financial analysis) on clean vertical text.
- **RAG evaluation** — provenance fields let you ground and cite every chunk.
- **Corpus research** — dedupe keeps token budgets honest; the `SUMMARY`
  record reports how many duplicates were removed.
- **Dataset curation** — filter by `source`/`domain`/`title` downstream to
  assemble a bespoke mix.

### Pricing

Pay-per-event: **$0.001 per chunk-extracted** (`chunk-extracted` is the
primary event). No charge for pages crawled with no extractable text.

### Limitations

- Seed URLs must serve server-rendered HTML. JavaScript-only SPAs and sites
  behind JS challenge walls (CourtListener's AWS WAF, Justia from datacenter
  IPs) need a browser-rendering actor instead; plain HTML opinion archives
  such as Cornell LII (`law.cornell.edu/supremecourt/text/`) work well.
- Chunk size is an estimate (~4 chars/token), not a byte-exact tokeniser
  count.
- Near-dedupe is O(n²) in chunk count; very large crawls
  (`maxPagesPerSource` high, many seeds) may be slow. Cap with
  `maxChunksPerPage` when crawling long documents.
- The actor follows same-domain links only; cross-domain citations are not
  crawled.

# Actor input Schema

## `domain` (type: `string`):

Vertical domain of the corpus. Controls the `domain` field in every output record and the crawl profile.

## `sources` (type: `array`):

Seed URLs to crawl. Each URL is treated as a starting point; same-domain links are followed up to maxPagesPerSource pages.

## `outputFormat` (type: `string`):

jsonl = dataset records are JSON Lines with one chunk per line (plus a downloadable output.jsonl in the key-value store). json = plain dataset records without the key-value file.

## `chunkSize` (type: `integer`):

Target chunk size in tokens (estimated at ~4 chars/token). Text is split at paragraph boundaries to stay under the budget.

## `maxPagesPerSource` (type: `integer`):

Maximum pages to crawl per seed URL (1 = the seed page only). Same-domain links are followed breadth-first up to this limit.

## `maxChunksPerPage` (type: `integer`):

Maximum chunks kept per page (0 = unlimited). Caps dataset size when crawling long documents.

## `dedupeSimilarity` (type: `number`):

Near-duplicate threshold: chunks whose similarity is >= this value are collapsed. 1.0 = exact-duplicate removal only.

## Actor input object example

```json
{
  "domain": "legal",
  "sources": [
    "https://www.law.cornell.edu/supremecourt/text/19-1392"
  ],
  "outputFormat": "jsonl",
  "chunkSize": 512,
  "maxPagesPerSource": 1,
  "maxChunksPerPage": 0,
  "dedupeSimilarity": 0.95
}
```

# Actor output Schema

## `results` (type: `string`):

No description

## `text` (type: `string`):

No description

## `source` (type: `string`):

No description

## `url` (type: `string`):

No description

## `domain` (type: `string`):

No description

## `chunk_index` (type: `string`):

No description

## `title` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "domain": "legal",
    "sources": [
        "https://www.law.cornell.edu/supremecourt/text/19-1392"
    ],
    "outputFormat": "jsonl"
};

// Run the Actor and wait for it to finish
const run = await client.actor("darknezz/vertical-corpus-builder").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "domain": "legal",
    "sources": ["https://www.law.cornell.edu/supremecourt/text/19-1392"],
    "outputFormat": "jsonl",
}

# Run the Actor and wait for it to finish
run = client.actor("darknezz/vertical-corpus-builder").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "domain": "legal",
  "sources": [
    "https://www.law.cornell.edu/supremecourt/text/19-1392"
  ],
  "outputFormat": "jsonl"
}' |
apify call darknezz/vertical-corpus-builder --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,darknezz/vertical-corpus-builder"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/rIsGOjagZinuYoirD/builds/wUdCOeDqYXTrU4RWR/openapi.json