Vertical Corpora Builder — Domain Datasets for LLM Fine-Tuning

Build domain-specific text corpora (legal, medical, financial) for LLM fine-tuning, RAG evaluation, or model pre-training. Point the actor at seed sources in your vertical; it crawls, strips boilerplate, removes duplicates, and emits token-aware chunks as JSONL with full provenance — a ready-to-train dataset, not a pile of HTML.

Why this actor

Raw web pages are terrible training data: nav bars, cookie banners, "related articles" and ads drown the signal, and near-identical syndicated text pollutes the corpus. This actor applies the three cleaning steps that matter for vertical corpora:

1. Boilerplate strip — removes script/style/nav/footer/header/ aside and common utility blocks (breadcrumbs, shares, comments, menus), keeping the article body's paragraphs, list items and quotes.
2. Dedupe (exact + near) — exact duplicates are removed by normalised text hash; near-duplicates are collapsed by 6-gram Jaccard similarity (default threshold 0.95), so syndicated copies appear once.
3. Token-aware chunking — chunks are split at paragraph boundaries and hard-split on sentence/word boundaries only when a single paragraph exceeds the budget (default 512 tokens, ~4 chars/token). Each chunk stays coherent, which is what fine-tuning actually wants.

Every output record carries provenance: source (hostname), url (exact page), domain (your vertical), title and chunk_index — so you can filter, cite, or re-weight the corpus later.

How it works

seed URLs ──▶ crawl (same-domain BFS) ──▶ clean (boilerplate strip)
   ──▶ dedupe (exact + near, configurable) ──▶ token-aware chunk
   ──▶ JSONL records {text, source, url, domain, chunk_index, title}

The crawl logic follows the house ai-web-crawler pattern: a browser User-Agent, same-domain breadth-first link discovery (bounded by maxPagesPerSource), and polite parallel fetching. Each seed URL is a starting point; set maxPagesPerSource: 1 to extract only the seed pages, or raise it to crawl whole sites.

Input

{
  "domain": "legal",
  "sources": [
    "https://www.law.cornell.edu/supremecourt/text/19-1392"
  ],
  "outputFormat": "jsonl",
  "chunkSize": 512,
  "maxPagesPerSource": 1,
  "maxChunksPerPage": 0,
  "dedupeSimilarity": 0.95
}

Field	Type	Default	Description
domain	string	legal	Vertical: legal, medical or financial. Written into every record's domain field.
sources	array<string>	—	Seed URLs to crawl (required).
outputFormat	string	jsonl	jsonl also writes a downloadable output.jsonl to the key-value store.
chunkSize	integer	512	Target chunk size in tokens (64–4096).
maxPagesPerSource	integer	1	Pages crawled per seed (1–100). 1 = seed pages only.
maxChunksPerPage	integer	0	Cap on chunks kept per page (0 = unlimited).
dedupeSimilarity	integer	0.95	Near-duplicate Jaccard threshold (0.5–1.0). 1.0 = exact dedupe only.

Output

One dataset record per chunk (JSONL export = one line per chunk):

{
  "text": "The Court has long recognized ...",
  "source": "courtlistener.com",
  "url": "https://www.courtlistener.com/opinion/4855702/...",
  "domain": "legal",
  "chunk_index": 0,
  "title": "Dobbs v. Jackson Women's Health Organization"
}

With outputFormat: "jsonl", the same records are also written to output.jsonl in the actor's default key-value store (Content-Type application/x-ndjson) for direct download. A SUMMARY record reports totalChunks, totalPages, rawChunks, duplicatesRemoved, estimatedTokens, and any failedSources.

Example: legal corpus from public court opinions

{
  "domain": "legal",
  "sources": [
    "https://www.law.cornell.edu/supremecourt/text/19-1392",
    "https://www.law.cornell.edu/supremecourt/text/20-843"
  ],
  "chunkSize": 512,
  "maxPagesPerSource": 1
}

Run from the API:

curl -X POST "https://api.apify.com/v2/acts/darknezz~vertical-corpus-builder/run-sync-get-dataset-items?token=YOUR_TOKEN&timeout=120" \
  -H "Content-Type: application/json" \
  -d '{"domain":"legal","sources":["https://www.law.cornell.edu/supremecourt/text/19-1392"],"chunkSize":512,"maxPagesPerSource":1}'

Run from your code (Python):

from apify_client import ApifyClient

client = ApifyClient("YOUR-APIFY-TOKEN")

run = client.actor("darknezz/vertical-corpus-builder").call(
    run_input={
        "domain": "legal",
        "sources": ["https://www.law.cornell.edu/supremecourt/text/19-1392"],
        "chunkSize": 512,
        "maxPagesPerSource": 1,
    }
)

chunks = client.dataset(run["defaultDatasetId"]).list_items().items
print(f"{len(chunks)} chunks extracted")
print(chunks[0]["title"], "|", chunks[0]["domain"], "|", chunks[0]["text"][:80])

Schedule it from the Apify console (or client.actor(...).start() in a cron) to keep a fresh corpus in sync with a site that publishes regularly.

What you get back (worked example)

A cloud run over two Cornell LII opinions (Dobbs 19-1392, Bruen 20-843, maxPagesPerSource: 1) produced 443 chunks: every record carries all six keys (text, source, url, domain, chunk_index, title), all text non-empty, domain=legal on every record, and zero duplicates after the exact+near dedupe pass. The SUMMARY record reported totalChunks 443, totalPages 2, estimatedTokens ~173,000, failedSources [].

title	chunk_index	domain	source
Dobbs v. Jackson Women's Health Organization	0	legal	law.cornell.edu
New York State Rifle & Pistol Assn., Inc. v. Bruen	0	legal	law.cornell.edu

FAQ

• How do I build a bigger corpus? Raise maxPagesPerSource (1–100) to crawl beyond the seed pages, add more seed URLs, and leave maxChunksPerPage: 0 (unlimited) when you want every chunk. For very large crawls, cap maxChunksPerPage — near-dedupe is O(n²) in chunk count.
• What is the difference between jsonl and the default output? With outputFormat: "jsonl" the actor also writes output.jsonl to the key-value store (one JSON object per line) for direct download — the format Hugging Face datasets and most trainers ingest natively. Without it, the records still land in the dataset as usual.
• How accurate is chunkSize? It is an estimate (~4 chars/token), not a tokeniser. 512 tokens ≈ 2,000 characters; paragraphs are kept intact unless a single paragraph exceeds the budget, in which case it is hard-split at sentence boundaries.

Use cases

• Fine-tuning a domain-specialised LLM (legal reasoning, medical QA, financial analysis) on clean vertical text.
• RAG evaluation — provenance fields let you ground and cite every chunk.
• Corpus research — dedupe keeps token budgets honest; the SUMMARY record reports how many duplicates were removed.
• Dataset curation — filter by source/domain/title downstream to assemble a bespoke mix.

Pricing

Pay-per-event: $0.001 per chunk-extracted (chunk-extracted is the primary event). No charge for pages crawled with no extractable text.

Limitations

• Seed URLs must serve server-rendered HTML. JavaScript-only SPAs and sites behind JS challenge walls (CourtListener's AWS WAF, Justia from datacenter IPs) need a browser-rendering actor instead; plain HTML opinion archives such as Cornell LII (law.cornell.edu/supremecourt/text/) work well.
• Chunk size is an estimate (~4 chars/token), not a byte-exact tokeniser count.
• Near-dedupe is O(n²) in chunk count; very large crawls (maxPagesPerSource high, many seeds) may be slow. Cap with maxChunksPerPage when crawling long documents.
• The actor follows same-domain links only; cross-domain citations are not crawled.