Pricing

from $3.00 / 1,000 saved training pages

AI Training Data Curator

Turn any public website into a clean LLM training dataset. Crawl docs, blogs, and help centers, extract readable text, filter by language, remove duplicates, and export JSON, JSONL, or CSV for fine-tuning, RAG, and AI workflows. No coding required.

Pricing

from $3.00 / 1,000 saved training pages

Rating

0.0

(0)

Developer

Vamsi Krishna

Actor stats

Bookmarked

Total users

Monthly active users

a day ago

Last modified

Categories

Developer tools

Other

Start URLs

startUrls

Optional

Seed URLs to begin crawling. Provide at least one start URL or a sitemapUrl.

Type:array

Example:

[
  {
    "url": "https://docs.example.com/getting-started"
  }
]

Sitemap URL

sitemapUrl

Optional

Optional sitemap.xml URL; URLs are merged with startUrls

Type:string

Example:https://docs.example.com/sitemap.xml

Max pages

maxPages

Optional

Maximum number of pages to crawl and export.

Type:integer

Minimum:1

Maximum:100000

Default:10

Example:100

Max sitemap URLs

maxSitemapUrls

Optional

Cap merged seed/sitemap URLs before crawling (defaults to maxPages)

Type:integer

Minimum:1

Maximum:100000

Minimum text length

minTextLength

Optional

Minimum extracted text length in characters; shorter pages are filtered out.

Type:integer

Minimum:0

Default:100

Example:200

Language filter

language

Optional

ISO 639-1 code (e.g. en). Leave empty to accept all.

Type:string

Example:en

Remove duplicates

deduplicate

Optional

When true, near-duplicate pages are removed using SimHash fingerprinting.

Type:boolean

Default:true

Max dedup fingerprints

maxFingerprints

Optional

Maximum SimHash fingerprints kept in memory (FIFO eviction when exceeded)

Type:integer

Minimum:1

Default:50000

Crawl strategy

crawlStrategy

Optional

recurse: follow links from seed pages; seeds-only: crawl only seed/sitemap URLs

Type:string

Default:seeds-only

Options:

recurseseeds-only

Stay within seed domains

stayWithinDomain

Optional

When recursing, only follow links on seed hostnames

Type:boolean

Default:true

Include URL globs

includeUrlGlobs

Optional

Optional glob patterns; enqueued URLs must match at least one when set

Type:string[]

Exclude URL globs

excludeUrlGlobs

Optional

Optional glob patterns; matching URLs are never enqueued

Type:string[]

Max concurrency

maxConcurrency

Optional

Maximum number of pages fetched in parallel

Type:integer

Minimum:1

Maximum:200

Default:10

Min request delay (seconds)

minRequestDelaySecs

Optional

Minimum delay between requests for politeness (0 = no limit)

Type:number

Minimum:0

Default:0

Max crawl depth

maxDepth

Optional

Maximum link hops from seed URLs (0 = seeds only). Leave empty for unlimited depth.

Type:integer

Minimum:0

Export rejected pages

exportRejectedPages

Optional

When true, URLs filtered out during processing are written to a separate 'rejected' dataset with filterReason

Type:boolean

Default:false

Proxy configuration

proxyConfiguration

Optional

Apify proxy settings for blocked or geo-restricted sites

Type:object

Ai Training Data Curator

omarchydev/ai-training-data-curator

Crawl websites and curate high-quality training data for LLM fine-tuning. Automatic deduplication, quality scoring, and language detection. Export to JSONL, Parquet, or CSV formats ready for OpenAI, Claude, or Llama training.

Omarchy Dev

AI Training Data Curator

ryanclinton/ai-training-data-curator

Crawl any website and extract clean, structured text data ready for LLM fine-tuning, RAG pipelines, and AI model training.

Ryan Clinton

📰 Substack Posts Monitor

skootle/substack-posts

Pull every post from any Substack publication via the public archive API: title, author, paywall flag, full body, engagement counts, reading time. For newsletter aggregators, content marketers, and AI summarizers. Export, run via API, schedule, or integrate with other tools.

Skootle

AI Training Data Quality MCP Server

ryanclinton/ai-training-data-quality-mcp

AI training data quality assessment, bias detection, and governance scoring for AI agents via the Model Context Protocol.

Ryan Clinton

Anthropic News & Research Scraper

automation-lab/anthropic-scraper

Scrapes news articles and research papers from Anthropic's website. Returns title, date, categories, description, image URL, and optionally full article text.

Stas Persiianenko

AI Training Dataset Builder: Articles, Blogs & Web Pages

turboextract/ai-training-dataset-builder

Turn any list of URLs into clean, structured training data for AI models, RAG systems, and LLM fine-tuning. Built for ML engineers and AI teams.

Moses Ndambuki

medRxiv Scraper

parseforge/medrxiv-scraper

Extract comprehensive preprint data from medRxiv, including titles, authors, abstracts, full text, DOIs, citations, and metadata. Automate access to health-science preprints with structured outputs, ideal for researchers and analysts who need reliable, large-scale article data without manual work.

ParseForge

Tatoeba Sentence Corpus Scraper

parseforge/tatoeba-sentence-corpus-scraper

Extract Tatoeba sentence corpus with millions of bilingual example sentences. Capture sentence ID, language, text, owner, audio URL, translations, tags, and license. Export to JSON, CSV, or Excel for language learning, NLP training data, translation memory, and linguistic research.

ParseForge

Genius Lyrics Scraper | Songs Artists and Annotations

parseforge/genius-lyrics-scraper

Scrape song lyrics, artist pages, album info, release dates, producers, writers, and Genius annotations. Build music datasets, lyric search apps, NLP corpora, sentiment analysis pipelines, and song recommendation tools from one of the largest lyric archives online.

ParseForge

TED Talks Transcript Scraper

jungle_synthesizer/ted-talks-transcript-scraper

Extracts full transcripts from TED.com talks in any available language. Returns timed segments (JSON), plain text, SRT, and WebVTT formats alongside speaker metadata, tags, and multi-language availability.