Pricing

from $2.00 / 1,000 results

Go to Apify Store

Website to RAG Knowledge Dataset

Try for free

Convert a website sitemap into clean text and markdown rows for RAG, AI search, chatbots, and knowledge base workflows.

Pricing

from $2.00 / 1,000 results

Rating

0.0

(0)

Developer

GhostGrid

Actor stats

Bookmarked

Total users

Monthly active users

2 days ago

Last modified

What It Does

Finds URLs from a sitemap or homepage
Supports sitemap indexes, gzipped sitemaps, and robots.txt sitemap discovery
Crawls each page and removes common navigation, footer, sidebar, comment, ad, and cookie banner content
Extracts clean plain text and markdown
Captures metadata such as title, description, keywords, canonical URL, and Open Graph image
Detects page language with lightweight page signals
Counts words for filtering and chunking decisions
Optionally adds same-domain pages linked from sitemap pages

Common Use Cases

Prepare public documentation for a RAG pipeline
Build a website knowledge base for a support chatbot
Export clean website content for AI search
Audit page titles, metadata, and content coverage
Create a repeatable dataset for scheduled website content refreshes

Input

Parameter	Type	Default	Description
`sitemap_url`	string	Required	URL to a sitemap.xml file or a website homepage
`max_pages`	integer	100	Maximum number of pages to crawl
`include_linked_pages`	boolean	false	Also add same-domain pages linked from sitemap pages, up to `max_pages`

Output

Each successful page creates one dataset item.

Field	Type	Description
`url`	string	Page URL
`title`	string	Page title
`text`	string	Clean main content as plain text
`markdown`	string	Clean main content as markdown
`metadata`	object	Description, keywords, Open Graph image, and canonical URL
`crawled_at`	string	ISO timestamp
`word_count`	integer	Number of extracted words
`language`	string	Detected language code

Example Input

{
  "sitemap_url": "https://docs.apify.com/sitemap.xml",
  "max_pages": 25,
  "include_linked_pages": false
}

Recommended Pricing

Use pay per event pricing.

Suggested launch price:

Actor start: free or near free
Dataset item: $0.0015 to $0.0025 per extracted page

Start at $0.002 per extracted page if Apify allows that exact value. This keeps small tests cheap and makes scheduled knowledge base refreshes a natural recurring use case.

Notes

This actor extracts content from public pages. It does not bypass logins, paywalls, or access controls.

Sitemap to RAG Dataset

ghostgrid/sitemap-to-rag-dataset

Convert any website sitemap into an LLM-ready dataset for RAG pipelines.

GhostGrid

Website Content Crawler

crawlerbros/website-content-crawler

Crawls websites and extracts clean text, markdown, or HTML content. Ideal for LLM training data, RAG pipelines, and knowledge base building.

Crawler Bros

Website To LLM Knowledge Pack

attainable_iota/website-to-llm-knowledge-pack

Crawl any website and turn it into an LLM-ready knowledge pack. This Actor extracts clean main text + metadata, follows links with depth/URL filters, and outputs per-page dataset items plus knowledge.jsonl, knowledge.md, and manifest.json for RAG/embeddings pipelines.

M Junaid Shaukat

Site to LLM Knowledge Base

adambounhar/site-to-knowledge-base

Turn any website or docs into clean, LLM-ready Markdown for RAG and AI agents — one record per page, each with a token count. Sitemap- and robots.txt-aware, with predictable per-page pricing (no token credits). Simple knowledge-base ingestion.

Mohamed Adam BOUNHAR

Web-to-Markdown Generator for AI & RAG Pipelines

profitstack/web-to-markdown-generator-for-ai-rag-pipelines

Convert any website into clean, heading-based chunking, LLM-ready Markdown for RAG and AI agents.

Manas Mantri

Website Content Scraper: Clean Markdown for AI and RAG

scrapemint/website-content-scraper

Crawl any website and get clean markdown, text, or HTML per page, ready for RAG pipelines, chatbots, and LLM fine tuning. Plain HTTP, no browser, no API key. Pay per page.

Ken M

Website to Text & Markdown — AI / RAG Content Crawler

inexhaustible_glass/rag-website-crawler

Scrape any website into clean text & Markdown with RAG-ready chunks and token counts for LLMs, vector databases (Pinecone, Qdrant) and AI chatbots. Also extracts linked PDF/Word/Excel. Anti-block, robots.txt-aware. Website-to-text for beginners, full RAG pipeline for pros. CPU only, no API key.

Hitman studio

Knowledge Intelligence Engine — Website to Markdown for RAG

ryanclinton/website-content-to-markdown

Turn any website, documentation site or help centre into a retrieval-ready knowledge corpus for RAG and AI search. Clean Markdown plus chunks, change detection, deduplication, retrieval scoring, version awareness and a full corpus audit, in one run.