# Sitemap Intelligence API (`s3nafps/sitemap-intelligence-api`) Actor Classify public sitemap URLs into pricing, docs, blog, changelog, product, legal, support, and other page types. - **URL**: https://apify.com/s3nafps/sitemap-intelligence-api.md - **Developed by:** [mohamed senator](https://apify.com/s3nafps) (community) - **Categories:** SEO tools, Developer tools - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing Pay per usage This Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage, which gets cheaper the higher subscription plan you have. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-usage ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## Sitemap Intelligence API Turn public domains and sitemaps into a structured, classified website URL inventory. ### What this Actor does Sitemap Intelligence API reads public sitemap and robots.txt data, deduplicates URLs, classifies pages by business use, and saves one structured record per discovered URL to the Apify Dataset. It helps you quickly find pricing pages, docs, blogs, changelogs, product pages, legal pages, support pages, and other useful public website sections. ### Use cases - Build a website URL inventory for SEO or content audits. - Find competitor pricing, docs, blog, changelog, and legal pages. - Prepare clean crawl targets for monitoring or extraction workflows. - Group large sitemaps by page type before deeper analysis. ### Who this is for SEO teams, data teams, SaaS operators, agencies, competitor intelligence teams, and builders who need a fast map of public website URLs. ### Input - `startUrls` - Public domains or sitemap URLs. Domains without a protocol are treated as HTTPS. Required, 1-20 items. - `includeRobotsTxt` - When true, the Actor checks robots.txt for sitemap references. Default: `true`. - `maxUrls` - Maximum sitemap URLs to return across all start URLs. Default: `5000`, maximum: `10000`. - `classifyUrls` - When true, URLs are classified by path and optional title signals. Default: `true`. - `fetchPageTitles` - When true, the Actor fetches discovered pages to improve title-based classification. This is slower and uses more requests. Default: `false`. ### Example input ```json { "startUrls": ["https://www.apify.com"], "includeRobotsTxt": true, "maxUrls": 50, "classifyUrls": true, "fetchPageTitles": false } ```` ### Output The Actor saves one Dataset item per discovered URL. - `domain` - Hostname from the start URL. - `sitemapUrl` - Sitemap where the URL was found, or `null` when no sitemap result was found. - `url` - Discovered public URL. - `path` - URL path. - `pageType` - Heuristic page type such as `pricing`, `docs`, `blog`, `changelog`, `product`, `legal`, `support`, `landing_page`, or `unknown`. - `confidenceScore` - Classification confidence from 0 to 1. - `detectedSignals` - Regex/path signals used for classification. - `title` - Page title when `fetchPageTitles` is enabled, otherwise `null`. - `lastModified`, `priority`, `changeFrequency` - Sitemap metadata when available. - `discoveredFrom` - Sitemap or discovery source. - `scrapedAt` - ISO timestamp for the run. - `extractionStatus` - Present as `failed` when no sitemap URLs are found. - `errorMessage` - Failure reason when extraction fails. ### Example output ```json { "domain": "www.apify.com", "sitemapUrl": "https://www.apify.com/sitemap.xml", "url": "https://www.apify.com/pricing", "path": "/pricing", "pageType": "pricing", "confidenceScore": 0.7, "detectedSignals": ["\\b(pricing|plans|price)\\b"], "title": null, "lastModified": "2026-01-01", "priority": "0.8", "changeFrequency": "weekly", "discoveredFrom": "https://www.apify.com/sitemap.xml", "scrapedAt": "2026-07-04T00:00:00.000Z" } ``` ### How it works - Reads the Actor input. - Normalizes public domains and sitemap URLs. - Checks `/sitemap.xml` and, when enabled, robots.txt sitemap references. - Parses sitemap indexes and URL sets. - Deduplicates URLs and optionally fetches page titles. - Classifies URLs by page path and title signals. - Saves results to the Apify Dataset. ### Limitations - Works with public sitemaps and robots.txt only. - Does not access private pages, logins, paywalls, CAPTCHA, or restricted content. - Classification is heuristic and may be wrong on unusual URL structures. - `fetchPageTitles` improves classification but makes the run slower. - Completeness depends on what the target website exposes in public sitemaps. ### Error handling Invalid or empty `startUrls` input fails the run validation. Failed sitemap requests are captured and, when no URLs are found for a start URL, the Actor returns a failed Dataset item with `extractionStatus: "failed"` and an `errorMessage`. Page title fetch failures are skipped so sitemap extraction can continue. ### Troubleshooting - If no URLs are returned, check whether the website exposes a public sitemap or lists sitemap URLs in `robots.txt`. - If important pages are missing, try enabling `includeRobotsTxt` and increasing `maxUrls`. - If page types look too generic, enable `fetchPageTitles` for stronger classification signals. - If the run is slow, reduce `maxUrls` or keep `fetchPageTitles` disabled. ### API usage ```bash curl -X POST "https://api.apify.com/v2/acts/USERNAME~sitemap-intelligence-api/runs?token=YOUR_APIFY_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "startUrls": ["https://www.apify.com"], "includeRobotsTxt": true, "maxUrls": 50, "classifyUrls": true, "fetchPageTitles": false }' ``` ### Integration ideas - Send classified pricing and changelog URLs into a change monitor. - Feed docs and blog URLs into an AI-ready content extractor. - Export the Dataset to CSV for SEO audits. - Use scheduled runs to refresh public URL inventories. ### SEO keywords sitemap API, sitemap parser, website map, URL classifier, SEO audit, competitor URLs, pricing page finder, docs finder, changelog finder, Apify sitemap ### Ethical use This Actor is designed for public data and user-provided URLs only. Do not use it to access private, login-protected, or restricted content. # Actor input Schema ## `startUrls` (type: `array`): Public domains or sitemap URLs to inspect. ## `includeRobotsTxt` (type: `boolean`): Also read robots.txt for sitemap references. ## `maxUrls` (type: `integer`): Maximum sitemap URLs to return. ## `classifyUrls` (type: `boolean`): Classify URLs by page type using path and optional title signals. ## `fetchPageTitles` (type: `boolean`): Fetch each page and include its title. Slower and uses more requests. ## Actor input object example ```json { "startUrls": [ "https://www.apify.com" ], "includeRobotsTxt": true, "maxUrls": 5000, "classifyUrls": true, "fetchPageTitles": false } ``` # Actor output Schema ## `results` (type: `string`): No description # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "startUrls": [ "https://www.apify.com" ] }; // Run the Actor and wait for it to finish const run = await client.actor("s3nafps/sitemap-intelligence-api").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "startUrls": ["https://www.apify.com"] } # Run the Actor and wait for it to finish run = client.actor("s3nafps/sitemap-intelligence-api").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "startUrls": [ "https://www.apify.com" ] }' | apify call s3nafps/sitemap-intelligence-api --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=s3nafps/sitemap-intelligence-api", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "Sitemap Intelligence API", "description": "Classify public sitemap URLs into pricing, docs, blog, changelog, product, legal, support, and other page types.", "version": "0.1", "x-build-id": "n7umD4jBmnTJ6ezua" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/s3nafps~sitemap-intelligence-api/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-s3nafps-sitemap-intelligence-api", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/s3nafps~sitemap-intelligence-api/runs": { "post": { "operationId": "runs-sync-s3nafps-sitemap-intelligence-api", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/s3nafps~sitemap-intelligence-api/run-sync": { "post": { "operationId": "run-sync-s3nafps-sitemap-intelligence-api", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "required": [ "startUrls" ], "properties": { "startUrls": { "title": "Domains or sitemap URLs", "minItems": 1, "maxItems": 20, "type": "array", "description": "Public domains or sitemap URLs to inspect.", "items": { "type": "string" } }, "includeRobotsTxt": { "title": "Check robots.txt", "type": "boolean", "description": "Also read robots.txt for sitemap references.", "default": true }, "maxUrls": { "title": "Maximum URLs", "minimum": 1, "maximum": 10000, "type": "integer", "description": "Maximum sitemap URLs to return.", "default": 5000 }, "classifyUrls": { "title": "Classify URLs", "type": "boolean", "description": "Classify URLs by page type using path and optional title signals.", "default": true }, "fetchPageTitles": { "title": "Fetch page titles", "type": "boolean", "description": "Fetch each page and include its title. Slower and uses more requests.", "default": false } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```