Website to RAG Knowledge Base (AI Chatbot Chunks)
Pricing
from $1.00 / 1,000 chunks
Website to RAG Knowledge Base (AI Chatbot Chunks)
Turn any website into clean, heading-aware text chunks for AI chatbots and vector databases (Pinecone, Qdrant, pgvector, n8n). Respects robots.txt, uses the sitemap, skips menus and cookie banners, removes duplicate boilerplate, and can return only pages that changed since the last run.
Pricing
from $1.00 / 1,000 chunks
Rating
0.0
(0)
Developer
Serkan Bal
Maintained by CommunityActor stats
0
Bookmarked
2
Total users
1
Monthly active users
11 hours ago
Last modified
Categories
Features
Share
Website to RAG Knowledge Base
Turn any website into clean, heading-aware text chunks ready for an AI chatbot or a vector database. Paste a URL, get rows like this:
{"url": "https://example.com/faq","title": "FAQ","headingPath": "FAQ > Do you rent e-bikes?","text": "## Do you rent e-bikes?\nYes, e-bikes rent for $45 per day or $220 per week, helmet and lock included.\n- Pick up from 9am\n- Return by 6pm","approxTokens": 34,"contentHash": "1f0c9a7e2b3d4c5a","crawledAt": "2026-11-15T10:00:00.000Z"}
Embed text, store url + headingPath as metadata, and your bot can answer from the website and cite the exact section.
Why this one
| This Actor | Generic page scraper | |
|---|---|---|
| Only the readable content (no menus, footers, cookie banners, scripts) | Yes | Usually raw HTML or full page text |
| Chunks split at headings, with the heading path on every chunk | Yes | No, you split it yourself |
| Tables kept as rows (prices, opening hours, plans) | Yes | Often flattened into noise |
| Duplicate boilerplate removed across pages | Yes | No |
| Only pages that changed since last run (cheap scheduled refresh) | Yes | No |
| Respects robots.txt, uses the sitemap, polite by default | Yes | Varies |
Who it is for
- Teams building a support or sales chatbot: load your help center, pricing and policy pages into Pinecone, Qdrant, Supabase pgvector, or n8n's vector store.
- Agencies: build a client's knowledge base in minutes, then schedule a weekly refresh with Only pages that changed.
- AI agents (MCP): give an agent a site's content as tidy chunks instead of raw HTML.
Input
| Option | What it does |
|---|---|
| Start URLs | Home page or a section like /help |
| Max pages / Max link depth | How far to go |
| Output | Chunks (one row per chunk) or whole pages as markdown |
| Chunk size / overlap | Defaults 1,500 / 200 characters (about 375 / 50 tokens) |
| Only follow / Skip URLs containing | Focus on /docs/ or skip /blog/ |
| Only pages that changed since last run | Return just new or edited pages on scheduled runs |
Use with n8n
Run this Actor from n8n's Apify node (or an HTTP Request node), then feed the rows into a vector store insert node with text as the document and url, title, headingPath as metadata. A ready-made RAG support chatbot workflow that uses the same idea is in our templates repo: https://github.com/steadyflow-labs/n8n-ai-templates
What it does not do
- Public pages only: no logins, no paywalls, no CAPTCHA solving. Pages disallowed by robots.txt are skipped.
- No JavaScript rendering: content must be in the page HTML (true for most marketing sites, docs and help centers). PDF files are not read in this version.
- It does not create embeddings; you choose the model.
Run report
Each run saves RUN_REPORT in the key-value store: pages fetched, pages with content, chunks returned, duplicates skipped, unchanged pages skipped, pages blocked by robots.txt, and errors.
Need this wired into your own chatbot or help desk? Reach out through the profile links.