RAG Web Corpus Builder
Pricing
from $0.90 / 1,000 results
RAG Web Corpus Builder
Turn public web pages into clean, chunked, metadata-rich Markdown records ready for RAG ingestion and vector stores.
Pricing
from $0.90 / 1,000 results
Rating
0.0
(0)
Developer
Cynix Dev
Maintained by CommunityActor stats
0
Bookmarked
2
Total users
1
Monthly active users
a day ago
Last modified
Categories
Share
Turn any set of public web pages into clean, chunked, metadata-rich Markdown records that drop straight into a RAG pipeline, vector store, or custom-GPT knowledge base — no post-processing required.
Most crawlers hand you raw HTML or one giant Markdown blob per page. You still have to strip navigation and ads, split the text into token-bounded chunks, and attach source metadata so your LLM can cite it. This Actor does that last mile for you.
What it does
- Crawls the start URLs you provide (with configurable depth and same-domain restriction).
- Extracts the main content using a readability-style algorithm — nav, footers, and ads removed.
- Converts clean HTML to Markdown.
- Splits each page into token-aware chunks with configurable size and overlap.
- Emits one dataset record per chunk with rich metadata for citations and dedup.
Output record
| Field | Description |
|---|---|
url | Source page URL |
title | Page title |
chunkIndex / totalChunks | Position of the chunk within the page |
text | Clean Markdown chunk text |
charCount / tokenEstimate | Size metrics (~4 chars/token) |
contentHash | sha256 of the chunk, for deduplication |
crawledAt | ISO timestamp |
Input
| Field | Default | Description |
|---|---|---|
startUrls | — (required) | Public pages to crawl |
maxCrawlDepth | 1 | Link levels to follow (0 = only start URLs) |
maxPages | 50 | Hard cap on pages per run |
sameDomainOnly | true | Stay on the start URL's domain |
includeGlobs / excludeGlobs | [] | URL filters |
chunkSize | 1000 | Target chars per chunk |
chunkOverlap | 150 | Overlap chars between chunks |
minChunkChars | 200 | Minimum chunk size |
Use with AI agents (MCP)
Once published, this Actor is discoverable through Apify's Actors MCP Server, so agents in LangChain, LlamaIndex, CrewAI, or Claude Desktop can call it directly to ground themselves in fresh, structured web content.
Compliance
This Actor targets public text content only. It does not log in, does not bypass paywalls, and does not collect personal data. Respect each site's terms of use and robots directives, and only crawl content you are permitted to access.
