Corpus Coverage Gate

Score web pages for RAG corpus quality: text length, heading structure, boilerplate hints, token estimate, metadata, and warnings.

Docs → RAG Corpus Builder (LLM-ready Markdown + llms.txt)

yasaslive/docs-rag-builder

Crawl any documentation site into an embeddings-ready corpus: clean markdown chunks with heading-path metadata, generated llms.txt, corpus.jsonl, and optional OpenAI embeddings.

Knowledge Intelligence Engine — Website to Markdown for RAG

ryanclinton/website-content-to-markdown

Turn any website, documentation site or help centre into a retrieval-ready knowledge corpus for RAG and AI search. Clean Markdown plus chunks, change detection, deduplication, retrieval scoring, version awareness and a full corpus audit, in one run.

Torrent / BitTorrent DHT corpus search & crawl statistics

apricot_inference/fast-tracker

Search a live BitTorrent DHT metadata corpus and see how fast it is growing. Two endpoints: /search with category filters and relevance ranking, and /stats with a per-minute capture rate you can verify. No endpoint or key needed — the corpus is crawled and hosted by this Actor's operator.

Common Crawl Corpus Builder

zentrafoundry/common-crawl-targeted-corpus-builder

Build targeted text corpora from Common Crawl with provenance.

Crawl a whole website or docs site and get one clean, LLM-ready Markdown + JSON record per page (title, headings, content, links, token count). Built for RAG ingestion and AI knowledge bases.

YouTube & Transcript Corpus Audit for RAG

taroyamada/youtube-channel-transcript-rag-intelligence

Audit authorized transcript text for RAG readiness, produce retrieval chunks and source-linked reports, and optionally try public YouTube captions without login or cookies.

Website Crawl Preflight Auditor

sebastian-actors/website-crawl-preflight-auditor

Check robots.txt, sitemaps, static extraction quality, crawl risk, and estimated Apify costs before running a crawler or RAG ingestion job.

Package rights-declared HTML, Markdown, text, and born-digital PDFs into deterministic RAG pages, traceable chunks, and verified manifests.

Website to AI / RAG Data Extractor

scrapyx/website-rag-extractor

Turns any website into a clean, embedding-ready corpus. Strips navigation, footers and cookie banners, converts the real content to markdown, and splits it into overlapping chunks that each carry their own URL, title and metadata. Crawl by URL list, sitemap or link graph.