# Sitemap to Changed-Only RAG JSONL (`orbiscribe/sitemap-to-rag-delta-dataset`) Actor Crawl sitemap.xml files and emit only added, changed, or deleted Markdown/JSONL chunks for cheaper RAG reindexing. - **URL**: https://apify.com/orbiscribe/sitemap-to-rag-delta-dataset.md - **Developed by:** [Orbiscribe Labs](https://apify.com/orbiscribe) (community) - **Categories:** AI, Developer tools, Automation - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, NaN bookmarks - **User rating**: No ratings yet ## Pricing $1.00 / 1,000 sitemap page processeds This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## Sitemap to Changed-Only RAG JSONL Use this Actor when you already have a public sitemap and need to know what changed before reindexing a knowledge base, search index, or chatbot corpus. It fetches sitemap URLs, follows sitemap indexes, extracts public HTML pages, hashes readable content, and compares pages against the previous snapshot stored in Apify key-value storage. ### Why use this instead of a generic crawler? Generic crawlers are useful for the first crawl. This Actor is for the second, third, and hundredth crawl, when you do not want to re-embed unchanged pages. - paste a `sitemap.xml` - keep only useful docs paths with `includeUrlPatterns` - schedule repeat runs - emit `added`, `changed`, and `deleted` records - succeed with a no-change summary when scheduled runs find nothing new - leave `includeUnchanged` off for changed-only reindexing - export `RAG_DELTA_CHUNKS_JSONL` directly to your vector pipeline - pay only for processed sitemap pages, with a small live default run ### What you get - Dataset rows for sitemap pages and chunks. - `changeType` values: `added`, `changed`, `unchanged`, `deleted`, and `no_changes`. - Clean Markdown, main text, headings, canonical URL, content hash, and source sitemap metadata. - Key-value outputs: `RAG_CHUNKS_JSONL`, `RAG_DELTA_CHUNKS_JSONL`, `DOCUMENTS_JSONL`, `URL_INVENTORY`, `CHANGE_SUMMARY`, `MARKDOWN_BUNDLE`, `BUYER_BRIEF`, and `RUN_SUMMARY`. ### Common workflows - Reindex only changed pages from a documentation or marketing site. - Schedule weekly sitemap diffs and send changed records to a webhook. - Keep an auditable URL inventory with hashes and source sitemap provenance. - Export JSONL for a vector database without reshaping Apify dataset rows. ### Input Start with one or more `sitemapUrls`. Turn on `compareToPreviousRun` for scheduled delta runs. Keep `includeUnchanged` off when you only want records that require action. The default input runs a tiny live Apify docs sitemap sample and filters to the Actor marketing playbook path, so the first run shows real content instead of a site-wide URL inventory: ```json { "sitemapUrls": ["https://docs.apify.com/sitemap_base.xml"], "includeUrlPatterns": ["/academy/actor-marketing-playbook/"], "excludeUrlPatterns": [], "compareToPreviousRun": true, "includeUnchanged": false, "maxPages": 5, "dryRun": false } ```` Use `dryRun: true` when you want bundled demo records without crawling live pages or calling custom pay-per-event charges. ### Pricing Recommended monetization: Pay per Event at **$0.001** per `sitemap-rag-page`. That is **$1 per 1,000** processed sitemap pages, plus normal Apify platform usage. When pay-per-event pricing is enabled, dry runs are uncharged and free-plan callers get the first 25 processed sources without this Actor's custom event charge. Users should still set Apify spending limits before large sitemap crawls. ### Limits and compliance Public pages only. This Actor does not bypass logins, paywalls, robots policies, or access controls. Extraction quality depends on the page structure, so run a small crawl before scheduling a large site. # Actor input Schema ## `sitemapUrls` (type: `array`): Sitemap.xml URLs or sitemap index URLs to crawl. Start with one small docs sitemap, inspect changes, then scale. ## `includeUrlPatterns` (type: `array`): Only crawl sitemap URLs containing at least one of these strings. Leave empty to allow all sitemap URLs. ## `excludeUrlPatterns` (type: `array`): Skip sitemap URLs containing any of these strings before maxPages is applied. ## `startUrls` (type: `array`): Optional direct page URLs used if a sitemap is missing or for controlled tests. ## `compareToPreviousRun` (type: `boolean`): Store snapshots and emit changed/new/deleted records on scheduled recrawls. ## `includeUnchanged` (type: `boolean`): Include unchanged page and chunk rows in the dataset. Leave off for reindexing workflows. ## `snapshotStoreName` (type: `string`): Key-value store used for previous page hashes and deleted-page detection. ## `maxPages` (type: `integer`): Maximum number of HTML pages to fetch in one run. ## `chunkSizeChars` (type: `integer`): Target character length for generated text chunks. ## `chunkOverlapChars` (type: `integer`): Character overlap between adjacent chunks. ## `requestTimeoutMs` (type: `integer`): Per-request timeout in milliseconds. ## `dryRun` (type: `boolean`): Emit bundled demo records without fetching or charging events. ## Actor input object example ```json { "sitemapUrls": [ "https://docs.apify.com/sitemap_base.xml" ], "includeUrlPatterns": [ "/academy/actor-marketing-playbook/" ], "excludeUrlPatterns": [], "startUrls": [ { "url": "https://example.com/" } ], "compareToPreviousRun": true, "includeUnchanged": false, "snapshotStoreName": "sitemap-to-rag-delta-dataset-snapshots", "maxPages": 5, "chunkSizeChars": 2500, "chunkOverlapChars": 250, "requestTimeoutMs": 15000, "dryRun": false } ``` # Actor output Schema ## `results` (type: `string`): No description ## `ragChunksJsonl` (type: `string`): No description ## `deltaChunksJsonl` (type: `string`): No description ## `documentsJsonl` (type: `string`): No description ## `changeSummary` (type: `string`): No description ## `urlInventory` (type: `string`): No description ## `markdownBundle` (type: `string`): No description ## `buyerBrief` (type: `string`): No description ## `runSummary` (type: `string`): No description # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = {}; // Run the Actor and wait for it to finish const run = await client.actor("orbiscribe/sitemap-to-rag-delta-dataset").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = {} # Run the Actor and wait for it to finish run = client.actor("orbiscribe/sitemap-to-rag-delta-dataset").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{}' | apify call orbiscribe/sitemap-to-rag-delta-dataset --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=orbiscribe/sitemap-to-rag-delta-dataset", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "Sitemap to Changed-Only RAG JSONL", "description": "Crawl sitemap.xml files and emit only added, changed, or deleted Markdown/JSONL chunks for cheaper RAG reindexing.", "version": "0.1", "x-build-id": "MbaDPrOtkNiSdq85u" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/orbiscribe~sitemap-to-rag-delta-dataset/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-orbiscribe-sitemap-to-rag-delta-dataset", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/orbiscribe~sitemap-to-rag-delta-dataset/runs": { "post": { "operationId": "runs-sync-orbiscribe-sitemap-to-rag-delta-dataset", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/orbiscribe~sitemap-to-rag-delta-dataset/run-sync": { "post": { "operationId": "run-sync-orbiscribe-sitemap-to-rag-delta-dataset", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "properties": { "sitemapUrls": { "title": "Sitemap URLs", "type": "array", "description": "Sitemap.xml URLs or sitemap index URLs to crawl. Start with one small docs sitemap, inspect changes, then scale.", "default": [ "https://docs.apify.com/sitemap_base.xml" ], "items": { "type": "string" } }, "includeUrlPatterns": { "title": "Include URL patterns", "type": "array", "description": "Only crawl sitemap URLs containing at least one of these strings. Leave empty to allow all sitemap URLs.", "default": [ "/academy/actor-marketing-playbook/" ], "items": { "type": "string" } }, "excludeUrlPatterns": { "title": "Exclude URL patterns", "type": "array", "description": "Skip sitemap URLs containing any of these strings before maxPages is applied.", "default": [], "items": { "type": "string" } }, "startUrls": { "title": "Fallback page URLs", "type": "array", "description": "Optional direct page URLs used if a sitemap is missing or for controlled tests.", "default": [ { "url": "https://example.com/" } ], "items": { "type": "object", "required": [ "url" ], "properties": { "url": { "type": "string", "title": "URL of a web page", "format": "uri" } } } }, "compareToPreviousRun": { "title": "Compare to previous run", "type": "boolean", "description": "Store snapshots and emit changed/new/deleted records on scheduled recrawls.", "default": true }, "includeUnchanged": { "title": "Include unchanged pages", "type": "boolean", "description": "Include unchanged page and chunk rows in the dataset. Leave off for reindexing workflows.", "default": false }, "snapshotStoreName": { "title": "Snapshot store name", "type": "string", "description": "Key-value store used for previous page hashes and deleted-page detection.", "default": "sitemap-to-rag-delta-dataset-snapshots" }, "maxPages": { "title": "Max pages", "minimum": 1, "maximum": 5000, "type": "integer", "description": "Maximum number of HTML pages to fetch in one run.", "default": 5 }, "chunkSizeChars": { "title": "Chunk size characters", "minimum": 500, "maximum": 12000, "type": "integer", "description": "Target character length for generated text chunks.", "default": 2500 }, "chunkOverlapChars": { "title": "Chunk overlap characters", "minimum": 0, "maximum": 2000, "type": "integer", "description": "Character overlap between adjacent chunks.", "default": 250 }, "requestTimeoutMs": { "title": "Request timeout ms", "minimum": 1000, "maximum": 60000, "type": "integer", "description": "Per-request timeout in milliseconds.", "default": 15000 }, "dryRun": { "title": "Dry run", "type": "boolean", "description": "Emit bundled demo records without fetching or charging events.", "default": false } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```