# Sitemap URl Extractor (`scrapebridge/sitemap-url-extractor`) Actor 🔎 Sitemap URL Extractor pulls URLs from any sitemap quickly and accurately. Export results for SEO audits, crawling, link building & content planning. 🚀 Automate discovery, enhance indexing, boost rankings. - **URL**: https://apify.com/scrapebridge/sitemap-url-extractor.md - **Developed by:** [Scrape Bridge](https://apify.com/scrapebridge) (community) - **Categories:** SEO tools, Automation, Developer tools - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $2.99 / 1,000 results This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ### Sitemap URL Extractor ⚡ — Extract Every URL from Any XML Sitemap If you’ve ever had to manually copy page URLs out of an XML sitemap, you know how slow and error-prone it gets—especially when you need the full list for SEO audits or crawling. **Sitemap URL Extractor** automatically extracts all URLs from a sitemap (including sitemap indexes) and saves them to a dataset in Apify. This sitemap url extractor is ideal if you’re looking for an extract urls from sitemap workflow, whether you’re doing SEO sitemap URL extraction or building a URL list from sitemap data. It’s a practical tool for SEOs, marketers, and data analysts who need a clean URL inventory fast. In one run, you can turn a root sitemap or sitemap index into a structured dataset without the copy-paste grind. --- ### See the Data: Sample Output Here's a real record from a single run: ```json { "url": "https://onescales.com/blog/how-to-scale-customer-onboarding/", "lastmod": "2025-05-26", "changefreq": "weekly", "status": "success" } ```` The extracted dataset uses these fields: | Field | Type | What It Tells You | |---|---|---| | `url` | string (or `null`) | The concrete page URL captured from the sitemap so you can build lists, feeds, or crawl targets. | | `lastmod` | string (or `null`) | The sitemap’s “Last Modified” value—useful for freshness checks and prioritizing updates. | | `changefreq` | string | The sitemap’s declared change frequency (defaults to `"weekly"` when missing). | | `status` | string | Indicates whether the record was collected successfully (useful when reviewing dataset health). | | `error_message` | string (or `null`) | Any error details for troubleshooting run issues (null when everything worked for that record). | | `source` | string (or `null`) | Where the record came from in your workflow (leave null unless you enrich data downstream). | | `notes` | string (or `null`) | Optional space for your own annotations if you add transforms later. | | `rank` | number (or `null`) | Optional ordering helper if you sort URLs in post-processing. | | `category` | string (or `null`) | Optional grouping label (for example, “blog” vs “product”) if you derive it after export. | | `tags` | array (or `null`) | Optional tags you may add after export for segmentation. | | `run_id` | string (or `null`) | Optional run identifier if you track runs in your pipeline. | | `timestamp` | string (or `null`) | Optional capture time metadata if you add it in downstream steps. | Export your full dataset as JSON, CSV, or Excel from the Apify dashboard. *** ### Setting It Up Drop this into your `input.json` and you're ready to go: ```json { "root_sitemap_url": "https://onescales.com/sitemap.xml" } ``` | Parameter | Required | What It Does | |---|---|---| | `root_sitemap_url` | ✅ | The URL of the sitemap or sitemap index you want the sitemap url extractor to start from. | *** ### What It Does Sitemap URL Extractor fetches your root sitemap, detects whether it’s a direct sitemap or a sitemap index, and then processes it to output a complete list of URLs. #### Extracts URLs from both sitemap indexes and direct sitemaps If the root is a sitemap index, it recursively processes the sub-sitemaps it references, so you get the full set of URLs from the entire tree. If the root is a direct urlset, it parses the contained URLs right away. #### Automatically extracts URLs from XML sitemaps This extract sitemap links workflow focuses on XML sitemap structures and produces clean URL records in your Apify dataset. The result is well-suited for “generate URL list from sitemap” tasks and bulk extract URLs from sitemap use cases. #### Saves results live into your Apify dataset URLs are pushed to the dataset as they are collected, so you can start reviewing progress without waiting for the entire crawl to finish. This makes the actor useful when you’re building a sitemap url parser pipeline into your broader workflow. #### Includes resilience for real-world fetching The actor logs progress and handles unsupported sitemap types with warnings. If fetching a sub-sitemap fails, it won’t crash the whole run—processing continues for other available items. #### Uses residential proxy support for reliable scraping The actor is designed to work with residential proxy support to improve reliability when fetching sitemap files across the web. This helps keep bulk operations stable when you’re running sitemap URL crawler tool jobs repeatedly. Overall, Sitemap URL Extractor turns a sitemap root into a dataset of usable URLs—fast, structured, and ready for downstream SEO or analytics workflows. *** ### Why Sitemap URL Extractor? There are plenty of ways to pull data from sitemaps—here’s why Sitemap URL Extractor stands out. #### One-run URL inventory (including sitemap indexes) Many tools only handle a single urlset. Sitemap URL Extractor is built to support sitemap indexes too, which makes it a stronger fit for SEO sitemap URL extraction when you need completeness. #### Clean, structured output for analysis and crawling The actor outputs consistent records with `url`, `lastmod`, and `changefreq` (defaulting to `"weekly"` when missing). That structure makes it easy to feed into crawl planning, reporting, and data pipelines. #### Designed for automation at scale Because it saves results directly to an Apify dataset, it fits smoothly into bulk processes like exporting URL lists, validating coverage, and supporting automated SEO workflows—without manual copy-paste. *** ### Real-World Use Cases Here's how different teams put Sitemap URL Extractor to work: **SEO Specialists**\ You’re auditing a site and need the complete set of discovered URLs from the sitemap, including all sub-sitemaps referenced by a sitemap index. You run Sitemap URL Extractor once, export the dataset, and immediately compare the URL inventory against indexed pages or internal crawl schedules. It cuts the time spent on “extract URLs from sitemap” work down to minutes. **Content and Editorial Teams**\ You want to prioritize updates based on recency, so “last modified” becomes a decision input. After the actor extracts URLs from the sitemap, you sort or filter by `lastmod` and build an editorial backlog tied to actual sitemap metadata. **Marketing Analysts**\ You’re building a channel-level URL dataset for reporting, landing page tracking, and campaign attribution. Sitemap URL Extractor gives you a bulk list of URLs you can merge with campaign logs—perfect for workflows like generate URL list from sitemap and bulk extract URLs from sitemap. **Automation & Data Engineering**\ You need a reliable step in a pipeline that periodically regenerates a URL list from publicly available sitemap data. You schedule the actor to run, pull the dataset, and pass the results into your data warehouse or downstream crawlers—keeping the process consistent over time. **Web Researchers**\ You’re compiling web resources for a study and want a deterministic source of discovered URLs. The sitemap link extractor output helps you standardize input URLs before you enrich them with additional signals. *** ### How to Run It No code required. Here's how to get your first results in under 5 minutes: 1. **Open the actor page on Apify**\ Go to [console.apify.com](https://console.apify.com) and open **Sitemap URL Extractor**. 2. **Enter your inputs**\ In the input field, set `root_sitemap_url` to your sitemap file or sitemap index URL. 3. **Configure proxy settings (recommended for reliability)**\ If you use Apify’s proxy settings, enable residential proxy support for more stable fetching. 4. **Start the run and watch the live log**\ Launch the run and monitor progress in the log output as sitemaps are fetched and parsed. 5. **Open the Dataset tab to see live results**\ Extracted URLs appear as records are pushed to the dataset, including `url`, `lastmod`, and `changefreq`. 6. **Export in your preferred format**\ Download your results from the dataset as JSON, CSV, or Excel for analysis or crawling. The whole setup takes under 5 minutes — results start appearing within seconds of launch. *** ### Export & Integration Options Once your data is collected, Sitemap URL Extractor fits directly into your existing workflow. You can export the extracted URLs from the Apify dataset tab as **JSON**, **CSV**, or **Excel**. From there, you can analyze freshness with `lastmod` or filter by `changefreq` for crawl planning. For integrations, you can use **API access** to pull results programmatically, set up **webhooks** to trigger downstream actions when runs complete, and connect tools via **Zapier** or **Make**. You can also set **scheduled runs** to automatically refresh your sitemap-derived URL list on a recurring basis. *** ### Pricing Sitemap URL Extractor runs on Apify, which includes a **free tier** — no credit card needed to start. On Apify, you’ll typically begin with platform credits for several real test runs. For larger or more frequent runs, you’ll scale using Apify’s compute-based billing (Apify pricing applies). Start free at [apify.com](https://apify.com) — scale up when you need to. *** ### Reliability & Limitations | What We Handle | How | |---|---| | Loading sitemap files | Uses standard HTTP fetching with a timeout and redirects handling enabled. | | Sitemap indexes | Recursively processes sub-sitemaps referenced by a sitemap index. | | Direct urlset sitemaps | Parses urlset XML and pushes records per URL into the dataset. | | Missing XML fields | `changefreq` defaults to `"weekly"` when it’s not present. | | Unsupported XML structure | Logs a warning when sitemap type isn’t recognized. | | Partial availability | If a sub-sitemap can’t be fetched, the run continues for other sub-sitemaps. | Limitations: Sitemap URL Extractor processes sitemaps that are publicly accessible and structured as XML sitemaps. It won’t access login-gated or private content, and it depends on the correctness of the sitemap XML provided by the site owner. For enterprise-scale needs or custom configurations, reach out and we'll help. *** ### Frequently Asked Questions #### Is there a free plan? Yes, Apify offers a free tier for starting out. You can use it to run Sitemap URL Extractor and validate the output for your sitemap url extraction use case. #### Do I need to log in or create an account on the target website? No. This actor works with publicly available sitemap files—no login to the target website is required. #### How accurate is the extracted data? The output is driven by what’s present in the sitemap XML. `url` and `lastmod` come from the sitemap, and `changefreq` defaults to `"weekly"` when the sitemap doesn’t provide a value. #### How many results can I get per run? You can extract as many URLs as your sitemap (and any sitemap index tree) contains. For very large sites, results volume is ultimately determined by the sitemap structure available at `root_sitemap_url`. #### How fresh is the data? The freshness matches the sitemap content at the time the actor fetches it. If the sitemap updates frequently, your extracted URLs will reflect those updates on the next run. #### Is this legal? Does it comply with GDPR / CCPA? This actor extracts URLs from publicly available data in sitemap XML files. You’re responsible for using the extracted data in compliance with GDPR, CCPA, and applicable laws and any platform terms. #### Can I export to Google Sheets or Excel? Yes. You can export from the Apify dataset tab as JSON, CSV, or Excel, and then import into Google Sheets or other tools. #### Can I schedule this to run automatically? Yes. You can run the actor on a schedule using Apify scheduling features so your website sitemap-derived URL list stays current. #### Can I access results via the API? Yes. You can pull dataset results programmatically using the Apify API as part of your automation or data pipeline. #### What happens when the actor encounters an error? If it can’t parse or fetch the sitemap content, the actor logs the issue and may skip unsupported items (for example, failing to fetch a sub-sitemap in an index). When parsing direct urlset data, it pushes URL records as they’re processed so you can still keep the useful partial output. *** ### Get Help & Use Responsibly Got a question about Sitemap URL Extractor or a feature you’d like added? Reach out at — we’re happy to help with setup and also consider improvements based on feedback like better dataset fields for bulk extract URLs from sitemap workflows. **Sitemap URL Extractor provides publicly available data.** It does not access private accounts, login-gated pages, or password-protected content. You’re responsible for complying with GDPR, CCPA, and any applicable platform terms. For data-removal requests, contact . Use responsibly, ethically, and only for lawful purposes. # Actor input Schema ## `root_sitemap_url` (type: `string`): The URL of the sitemap or sitemap index to start with. ## Actor input object example ```json { "root_sitemap_url": "https://onescales.com/sitemap.xml" } ``` # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "root_sitemap_url": "https://onescales.com/sitemap.xml" }; // Run the Actor and wait for it to finish const run = await client.actor("scrapebridge/sitemap-url-extractor").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "root_sitemap_url": "https://onescales.com/sitemap.xml" } # Run the Actor and wait for it to finish run = client.actor("scrapebridge/sitemap-url-extractor").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "root_sitemap_url": "https://onescales.com/sitemap.xml" }' | apify call scrapebridge/sitemap-url-extractor --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=scrapebridge/sitemap-url-extractor", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "Sitemap URl Extractor", "description": "🔎 Sitemap URL Extractor pulls URLs from any sitemap quickly and accurately. Export results for SEO audits, crawling, link building & content planning. 🚀 Automate discovery, enhance indexing, boost rankings.", "version": "0.1", "x-build-id": "CqDM6eMjyEBuR9gsh" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/scrapebridge~sitemap-url-extractor/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-scrapebridge-sitemap-url-extractor", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/scrapebridge~sitemap-url-extractor/runs": { "post": { "operationId": "runs-sync-scrapebridge-sitemap-url-extractor", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/scrapebridge~sitemap-url-extractor/run-sync": { "post": { "operationId": "run-sync-scrapebridge-sitemap-url-extractor", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "required": [ "root_sitemap_url" ], "properties": { "root_sitemap_url": { "title": "Root Sitemap URL", "type": "string", "description": "The URL of the sitemap or sitemap index to start with." } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```