Website to RAG Knowledge Base (AI Chatbot Chunks) avatar

Website to RAG Knowledge Base (AI Chatbot Chunks)

Pricing

from $1.00 / 1,000 chunks

Go to Apify Store
Website to RAG Knowledge Base (AI Chatbot Chunks)

Website to RAG Knowledge Base (AI Chatbot Chunks)

Turn any website into clean, heading-aware text chunks for AI chatbots and vector databases (Pinecone, Qdrant, pgvector, n8n). Respects robots.txt, uses the sitemap, skips menus and cookie banners, removes duplicate boilerplate, and can return only pages that changed since the last run.

Pricing

from $1.00 / 1,000 chunks

Rating

0.0

(0)

Developer

Serkan Bal

Serkan Bal

Maintained by Community

Actor stats

0

Bookmarked

2

Total users

1

Monthly active users

11 hours ago

Last modified

Categories

Features

Limited permissions

Share

Website to RAG Knowledge Base

Turn any website into clean, heading-aware text chunks ready for an AI chatbot or a vector database. Paste a URL, get rows like this:

{
"url": "https://example.com/faq",
"title": "FAQ",
"headingPath": "FAQ > Do you rent e-bikes?",
"text": "## Do you rent e-bikes?\nYes, e-bikes rent for $45 per day or $220 per week, helmet and lock included.\n- Pick up from 9am\n- Return by 6pm",
"approxTokens": 34,
"contentHash": "1f0c9a7e2b3d4c5a",
"crawledAt": "2026-11-15T10:00:00.000Z"
}

Embed text, store url + headingPath as metadata, and your bot can answer from the website and cite the exact section.

Why this one

This ActorGeneric page scraper
Only the readable content (no menus, footers, cookie banners, scripts)YesUsually raw HTML or full page text
Chunks split at headings, with the heading path on every chunkYesNo, you split it yourself
Tables kept as rows (prices, opening hours, plans)YesOften flattened into noise
Duplicate boilerplate removed across pagesYesNo
Only pages that changed since last run (cheap scheduled refresh)YesNo
Respects robots.txt, uses the sitemap, polite by defaultYesVaries

Who it is for

  • Teams building a support or sales chatbot: load your help center, pricing and policy pages into Pinecone, Qdrant, Supabase pgvector, or n8n's vector store.
  • Agencies: build a client's knowledge base in minutes, then schedule a weekly refresh with Only pages that changed.
  • AI agents (MCP): give an agent a site's content as tidy chunks instead of raw HTML.

Input

OptionWhat it does
Start URLsHome page or a section like /help
Max pages / Max link depthHow far to go
OutputChunks (one row per chunk) or whole pages as markdown
Chunk size / overlapDefaults 1,500 / 200 characters (about 375 / 50 tokens)
Only follow / Skip URLs containingFocus on /docs/ or skip /blog/
Only pages that changed since last runReturn just new or edited pages on scheduled runs

Use with n8n

Run this Actor from n8n's Apify node (or an HTTP Request node), then feed the rows into a vector store insert node with text as the document and url, title, headingPath as metadata. A ready-made RAG support chatbot workflow that uses the same idea is in our templates repo: https://github.com/steadyflow-labs/n8n-ai-templates

What it does not do

  • Public pages only: no logins, no paywalls, no CAPTCHA solving. Pages disallowed by robots.txt are skipped.
  • No JavaScript rendering: content must be in the page HTML (true for most marketing sites, docs and help centers). PDF files are not read in this version.
  • It does not create embeddings; you choose the model.

Run report

Each run saves RUN_REPORT in the key-value store: pages fetched, pages with content, chunks returned, duplicates skipped, unchanged pages skipped, pages blocked by robots.txt, and errors.

Need this wired into your own chatbot or help desk? Reach out through the profile links.