Sitemap to RAG Dataset
Pricing
from $0.80 / 1,000 page extracteds
Sitemap to RAG Dataset
Convert any website sitemap into an LLM-ready dataset for RAG pipelines.
Pricing
from $0.80 / 1,000 page extracteds
Rating
0.0
(0)
Developer
GhostGrid
Maintained by CommunityActor stats
0
Bookmarked
2
Total users
1
Monthly active users
a day ago
Last modified
Categories
Share
Feed it a sitemap URL (or just a homepage), get back clean text content from every page, formatted for RAG pipelines and LLM training.
What it does
- Parses sitemap.xml files (including sitemap indexes, gzipped sitemaps, and auto-discovery from robots.txt)
- Crawls each page and strips out navigation, footers, sidebars, ads, and other boilerplate
- Outputs clean text and markdown versions of the main content
- Extracts metadata (title, description, keywords, Open Graph image)
- Detects page language and counts words
- Splits content into RAG-ready chunks with stable chunk IDs (UUID5 based on URL + index)
- Estimates token counts per chunk (len/4 heuristic)
- Computes SHA-256 content hashes for pages and chunks
- Supports diff mode to detect new, changed, and removed pages between crawls
Input
| Parameter | Type | Default | Description |
|---|---|---|---|
sitemap_url | string | required | URL to a sitemap.xml or a website homepage (auto-detects sitemap) |
max_pages | integer | 100 | Max pages to crawl (1 - 10,000) |
include_linked_pages | boolean | false | Also crawl pages linked from sitemap pages |
chunk_strategy | string | fixed_size | How to split text: fixed_size, sentence, or paragraph |
chunk_token_limit | integer | 512 | Max tokens per chunk (100 - 4,096) |
diff_mode | boolean | false | Compare with previous crawl, report new/changed/removed pages |
include_chunks | boolean | true | Output RAG chunks as separate dataset items |
Output
The dataset contains two types of records:
Page records (type: "page")
One per crawled page:
| Field | Type | Description |
|---|---|---|
type | string | Always "page" |
url | string | Page URL |
title | string | Page title |
text | string | Clean main content as plain text |
markdown | string | Content formatted as markdown |
content_hash | string | SHA-256 hex digest of the page text |
metadata | object | Description, keywords, OG image, canonical URL |
crawled_at | string | ISO 8601 timestamp |
word_count | integer | Word count of the extracted text |
language | string | Detected language code (e.g. en) |
Chunk records (type: "chunk")
One per RAG chunk (when include_chunks is true):
| Field | Type | Description |
|---|---|---|
type | string | Always "chunk" |
url | string | Source page URL |
title | string | Source page title |
chunk_id | string | Stable UUID5 (based on url:chunk_index) |
chunk_index | integer | 0-based index within the page |
total_chunks | integer | Total chunks for this page |
text | string | The chunk text |
start_char | integer | Character offset in full page text |
end_char | integer | End character offset |
token_estimate | integer | Estimated tokens (len/4) |
content_hash | string | SHA-256 of the chunk text |
page_content_hash | string | SHA-256 of the full page text |
language | string | Page language code |
Diff records (type: "diff")
One per run when diff_mode is true:
| Field | Type | Description |
|---|---|---|
type | string | Always "diff" |
new | list | URLs found in this crawl but not the previous one |
changed | list | URLs present in both but with different content hash |
removed | list | URLs in the previous crawl but not this one |
crawl_time | string | ISO 8601 timestamp of this crawl |
previous_crawl_time | string | ISO 8601 timestamp of the previous crawl |
total_new | integer | Count of new URLs |
total_changed | integer | Count of changed URLs |
total_removed | integer | Count of removed URLs |
How to use
Run locally:
With chunking and diff mode:
Deploy to Apify:
Pricing
This actor uses pay-per-event (PPE) pricing. You are charged 1 event per page successfully crawled and extracted. Chunk records and diff records do not incur additional charges.
