Website to Knowledge Base (RAG / Chatbot Ready)
Under maintenancePricing
from $0.01 / 1,000 results
Go to Apify Store
Website to Knowledge Base (RAG / Chatbot Ready)
Under maintenanceCrawl a website and produce clean Markdown, semantic chunks, FAQs and an llms.txt for chatbot and RAG development.
Pricing
from $0.01 / 1,000 results
Rating
0.0
(0)
Developer
Jared Keanche
Maintained by CommunityActor stats
0
Bookmarked
2
Total users
1
Monthly active users
a day ago
Last modified
Categories
Share
Website to Knowledge Base — RAG Chunks, FAQs & llms.txt
Paste a URL and get a chatbot-ready knowledge base in one run: clean Markdown, embedding-ready chunks, FAQs, llms.txt, products and a site profile.
Why use it
- RAG-ready chunks: heading-aware, with context headers, stable IDs and token counts. Never split inside links, tables or code.
- Less noise: removes menus, cookie banners, chat widgets, repeated blocks and near-duplicate pages.
- Auto rendering: fast HTTP first, a browser only for JavaScript or blocked pages.
- Incremental mode: scheduled runs output only new and changed pages and list removed ones. Unchanged pages are free.
- FAQs & products extracted and de-duplicated. Optional PDFs.
- Optional AI (your Anthropic key): summaries, key facts and a chatbot system prompt.
Outputs
- Dataset: one row per page, or per chunk for vector-DB integrations (Pinecone, Qdrant…)
KNOWLEDGE_BASE.md: upload to a Custom GPT or Claude Projectchunks.jsonl,faqs.jsonl,products.jsonl,llms.txt,llms-full.txtRUN_REPORT: what was skipped and why, with hints
Pricing
| Event | Per 1,000 |
|---|---|
| Page saved | $0.80 |
| + browser rendering | +$1.20 |
| $2.00 | |
| AI enrichment | $1.00 |
Thin, duplicate, blocked and unchanged pages are free. Runs stop cleanly at your cost limit.
Quick start
- Add a homepage, docs or help-center URL.
- Click Start (defaults suit most sites).
- Download
KNOWLEDGE_BASE.mdor use the chunks dataset.
Only crawl sites you may crawl. robots.txt is respected by default.
