Pricing

from $0.25 / 1,000 url founds

Try for free

Go to Apify Store

Sitemap Extractor: Every Page URL of Any Website

Try for free

Get every page URL of a website from its sitemaps, found automatically (robots.txt, sitemap indexes, compressed, text and RSS sitemaps), with a link-following fallback when there's no sitemap. Track new and removed pages, and check pages for broken links, redirects and noindex.

Pricing

from $0.25 / 1,000 url founds

Rating

0.0

(0)

Developer

Elliott Tyson

Actor stats

Bookmarked

Total users

Monthly active users

10 days ago

Last modified

Categories

Developer tools

SEO tools

You can access the Sitemap Extractor: Every Page URL of Any Website programmatically from your own applications by using the Apify API. You can also choose the language preference from below. To use the Apify API, you’ll need an Apify account and your API token, found in API & Integrations in Apify Console.

Python

JavaScript

CLI

OpenAPI

HTTP

MCP

{
  "openapi": "3.0.1",
  "info": {
    "version": "1.0",
    "x-build-id": "X2sSYLDhGYMBIedru"
  },
  "servers": [
    {
      "url": "https://api.apify.com/v2"
    }
  ],
  "paths": {
    "/acts/planetlinux98~sitemap-extractor/run-sync-get-dataset-items": {
      "post": {
        "operationId": "run-sync-get-dataset-items-planetlinux98-sitemap-extractor",
        "x-openai-isConsequential": false,
        "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.",
        "tags": [
          "Run Actor"
        ],
        "requestBody": {
          "required": true,
          "content": {
            "application/json": {
              "schema": {
                "$ref": "#/components/schemas/inputSchema"
              }
            }
          }
        },
        "parameters": [
          {
            "name": "token",
            "in": "query",
            "required": true,
            "schema": {
              "type": "string"
            },
            "description": "Enter your Apify token here"
          }
        ],
        "responses": {
          "200": {
            "description": "OK"
          }
        }
      }
    },
    "/acts/planetlinux98~sitemap-extractor/runs": {
      "post": {
        "operationId": "runs-sync-planetlinux98-sitemap-extractor",
        "x-openai-isConsequential": false,
        "summary": "Executes an Actor and returns information about the initiated run in response.",
        "tags": [
          "Run Actor"
        ],
        "requestBody": {
          "required": true,
          "content": {
            "application/json": {
              "schema": {
                "$ref": "#/components/schemas/inputSchema"
              }
            }
          }
        },
        "parameters": [
          {
            "name": "token",
            "in": "query",
            "required": true,
            "schema": {
              "type": "string"
            },
            "description": "Enter your Apify token here"
          }
        ],
        "responses": {
          "200": {
            "description": "OK",
            "content": {
              "application/json": {
                "schema": {
                  "$ref": "#/components/schemas/runsResponseSchema"
                }
              }
            }
          }
        }
      }
    },
    "/acts/planetlinux98~sitemap-extractor/run-sync": {
      "post": {
        "operationId": "run-sync-planetlinux98-sitemap-extractor",
        "x-openai-isConsequential": false,
        "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.",
        "tags": [
          "Run Actor"
        ],
        "requestBody": {
          "required": true,
          "content": {
            "application/json": {
              "schema": {
                "$ref": "#/components/schemas/inputSchema"
              }
            }
          }
        },
        "parameters": [
          {
            "name": "token",
            "in": "query",
            "required": true,
            "schema": {
              "type": "string"
            },
            "description": "Enter your Apify token here"
          }
        ],
        "responses": {
          "200": {
            "description": "OK"
          }
        }
      }
    }
  },
  "components": {
    "schemas": {
      "inputSchema": {
        "type": "object",
        "required": [
          "websites"
        ],
        "properties": {
          "websites": {
            "title": "Websites or sitemap URLs",
            "minItems": 1,
            "maxItems": 10000,
            "type": "array",
            "description": "One per line. Type a website (<code>example.com</code>) and we find its sitemaps for you: in robots.txt, at the usual addresses, or as compressed, text or RSS sitemaps. If a site has no sitemap, we follow its links instead. You can also paste a sitemap address (<code>https://example.com/sitemap.xml</code>) to read just that one, or a section of a site (<code>https://example.com/blog/</code>) to keep only URLs under it.",
            "items": {
              "type": "string"
            }
          },
          "discoveryMode": {
            "title": "How to find URLs",
            "enum": [
              "auto",
              "sitemaps",
              "crawl",
              "both"
            ],
            "type": "string",
            "description": "<b>Sitemaps, or follow links if there's no sitemap</b> works for almost any site. <b>Sitemaps only</b> never crawls. <b>Follow links only</b> ignores sitemaps. <b>Both</b> reads the sitemaps and also follows links, and marks pages that are linked on the site but missing from its sitemap (<code>inSitemap: false</code>).",
            "default": "auto"
          },
          "maxCrawlPages": {
            "title": "Max pages to visit when following links",
            "minimum": 1,
            "maximum": 100000,
            "type": "integer",
            "description": "Only used when we follow a site's links. Each page visited is one request to the site. Reading sitemaps doesn't count against this.",
            "default": 500
          },
          "includeUrlPatterns": {
            "title": "Only keep URLs containing",
            "type": "array",
            "description": "Keep only URLs that contain at least one of these. Use <code>*</code> for any text: <code>/blog/</code>, <code>/products/*-red</code> or <code>*.pdf</code>. Matching ignores case. Leave empty to keep every URL.",
            "items": {
              "type": "string"
            }
          },
          "excludeUrlPatterns": {
            "title": "Leave out URLs containing",
            "type": "array",
            "description": "Drop URLs that contain any of these, for example <code>/tag/</code>, <code>?page=</code> or <code>/fr/</code>. Same rules as above.",
            "items": {
              "type": "string"
            }
          },
          "modifiedWithinDays": {
            "title": "Changed within the last (days)",
            "minimum": 0,
            "maximum": 36500,
            "type": "integer",
            "description": "Keep only URLs whose last-modified date (from the sitemap) is within this many days. URLs without a date are left out when this is set, because we can't tell. Leave empty or 0 for no date filter."
          },
          "includeImages": {
            "title": "Include image URLs from image sitemaps",
            "type": "boolean",
            "description": "Adds an <code>images</code> list to each URL when the sitemap lists images for that page.",
            "default": false
          },
          "checkPages": {
            "title": "Check each page (status, redirects, noindex, canonical)",
            "type": "boolean",
            "description": "Visits every URL that's saved and reports what search engines would see: the HTTP status after redirects, the redirect chain, response time, whether the page says \"noindex\", where its canonical link points, whether robots.txt blocks Google, and the page title. Problems are listed in plain words in the <code>issues</code> field. Charged as an extra event per page checked; pages we couldn't check (for example behind bot protection) aren't charged.",
            "default": false
          },
          "onlyChanges": {
            "title": "Only new, updated and removed URLs since the last run (monitoring)",
            "type": "boolean",
            "description": "Remembers the URLs it has already given you and returns only what changed: new pages, pages whose last-modified date moved forward, and pages that disappeared (<code>changeType</code>). The first run returns every URL. Ideal for scheduled runs that watch a site for new pages, products or articles. Keep the same filters between runs.",
            "default": false
          },
          "includeRemoved": {
            "title": "Also report URLs that were removed",
            "type": "boolean",
            "description": "In monitoring mode, also add a row for each URL that was there last time but is gone now (<code>changeType: removed</code>). To avoid false alarms, removals are only reported when every sitemap was read successfully.",
            "default": true
          },
          "monitorName": {
            "title": "Monitor name",
            "maxLength": 60,
            "type": "string",
            "description": "Optional. Give each monitor a name (for example <code>competitor-products</code>) so several schedules can watch the same site separately. Monitor memory is kept in a key-value store called <code>sitemap-extractor-monitors</code> in your account."
          },
          "maxUrlsPerWebsite": {
            "title": "Max URLs per website",
            "minimum": 0,
            "type": "integer",
            "description": "Stop saving URLs for a website after this many. 0 means no limit. Big sites can list millions of URLs, so a limit is a good idea for a first try.",
            "default": 0
          },
          "maxUrls": {
            "title": "Max URLs in total",
            "minimum": 0,
            "type": "integer",
            "description": "Stop once this many URLs have been saved across all websites. 0 means no limit. You can also cap spending with the run's maximum cost setting.",
            "default": 0
          },
          "maxRequestsPerSecond": {
            "title": "Max requests per second, per website",
            "minimum": 1,
            "maximum": 20,
            "type": "integer",
            "description": "How fast we may load pages from one website when following links or checking pages. The default is polite for any site. Only raise it for sites you own or manage. A site's robots.txt Crawl-delay always wins.",
            "default": 4
          }
        }
      },
      "runsResponseSchema": {
        "type": "object",
        "properties": {
          "data": {
            "type": "object",
            "properties": {
              "id": {
                "type": "string"
              },
              "actId": {
                "type": "string"
              },
              "userId": {
                "type": "string"
              },
              "startedAt": {
                "type": "string",
                "format": "date-time",
                "example": "2025-01-08T00:00:00.000Z"
              },
              "finishedAt": {
                "type": "string",
                "format": "date-time",
                "example": "2025-01-08T00:00:00.000Z"
              },
              "status": {
                "type": "string",
                "example": "READY"
              },
              "meta": {
                "type": "object",
                "properties": {
                  "origin": {
                    "type": "string",
                    "example": "API"
                  },
                  "userAgent": {
                    "type": "string"
                  }
                }
              },
              "stats": {
                "type": "object",
                "properties": {
                  "inputBodyLen": {
                    "type": "integer",
                    "example": 2000
                  },
                  "rebootCount": {
                    "type": "integer",
                    "example": 0
                  },
                  "restartCount": {
                    "type": "integer",
                    "example": 0
                  },
                  "resurrectCount": {
                    "type": "integer",
                    "example": 0
                  },
                  "computeUnits": {
                    "type": "integer",
                    "example": 0
                  }
                }
              },
              "options": {
                "type": "object",
                "properties": {
                  "build": {
                    "type": "string",
                    "example": "latest"
                  },
                  "timeoutSecs": {
                    "type": "integer",
                    "example": 300
                  },
                  "memoryMbytes": {
                    "type": "integer",
                    "example": 1024
                  },
                  "diskMbytes": {
                    "type": "integer",
                    "example": 2048
                  }
                }
              },
              "buildId": {
                "type": "string"
              },
              "defaultKeyValueStoreId": {
                "type": "string"
              },
              "defaultDatasetId": {
                "type": "string"
              },
              "defaultRequestQueueId": {
                "type": "string"
              },
              "buildNumber": {
                "type": "string",
                "example": "1.0.0"
              },
              "containerUrl": {
                "type": "string"
              },
              "usage": {
                "type": "object",
                "properties": {
                  "ACTOR_COMPUTE_UNITS": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATASET_READS": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATASET_WRITES": {
                    "type": "integer",
                    "example": 0
                  },
                  "KEY_VALUE_STORE_READS": {
                    "type": "integer",
                    "example": 0
                  },
                  "KEY_VALUE_STORE_WRITES": {
                    "type": "integer",
                    "example": 1
                  },
                  "KEY_VALUE_STORE_LISTS": {
                    "type": "integer",
                    "example": 0
                  },
                  "REQUEST_QUEUE_READS": {
                    "type": "integer",
                    "example": 0
                  },
                  "REQUEST_QUEUE_WRITES": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATA_TRANSFER_INTERNAL_GBYTES": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATA_TRANSFER_EXTERNAL_GBYTES": {
                    "type": "integer",
                    "example": 0
                  },
                  "PROXY_RESIDENTIAL_TRANSFER_GBYTES": {
                    "type": "integer",
                    "example": 0
                  },
                  "PROXY_SERPS": {
                    "type": "integer",
                    "example": 0
                  },
                  "PROXY_UNBLOCKER_UNITS": {
                    "type": "integer",
                    "example": 0
                  }
                }
              },
              "usageTotalUsd": {
                "type": "number",
                "example": 0.00005
              },
              "usageUsd": {
                "type": "object",
                "properties": {
                  "ACTOR_COMPUTE_UNITS": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATASET_READS": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATASET_WRITES": {
                    "type": "integer",
                    "example": 0
                  },
                  "KEY_VALUE_STORE_READS": {
                    "type": "integer",
                    "example": 0
                  },
                  "KEY_VALUE_STORE_WRITES": {
                    "type": "number",
                    "example": 0.00005
                  },
                  "KEY_VALUE_STORE_LISTS": {
                    "type": "integer",
                    "example": 0
                  },
                  "REQUEST_QUEUE_READS": {
                    "type": "integer",
                    "example": 0
                  },
                  "REQUEST_QUEUE_WRITES": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATA_TRANSFER_INTERNAL_GBYTES": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATA_TRANSFER_EXTERNAL_GBYTES": {
                    "type": "integer",
                    "example": 0
                  },
                  "PROXY_RESIDENTIAL_TRANSFER_GBYTES": {
                    "type": "integer",
                    "example": 0
                  },
                  "PROXY_SERPS": {
                    "type": "integer",
                    "example": 0
                  },
                  "PROXY_UNBLOCKER_UNITS": {
                    "type": "integer",
                    "example": 0
                  }
                }
              }
            }
          }
        }
      }
    }
  }
}

Sitemap Extractor: Get Every URL of a Website OpenAPI definition

OpenAPI is a standard for designing and describing RESTful APIs, allowing developers to define API structure, endpoints, and data formats in a machine-readable way. It simplifies API development, integration, and documentation.

OpenAPI is effective when used with AI agents and GPTs by standardizing how these systems interact with various APIs, for reliable integrations and efficient communication.

By defining machine-readable API specifications, OpenAPI allows AI models like GPTs to understand and use varied data sources, improving accuracy. This accelerates development, reduces errors, and provides context-aware responses, making OpenAPI a core component for AI applications.

You can download the OpenAPI definitions for Sitemap Extractor: Every Page URL of Any Website from the options below:

OpenAPI.json

If you’d like to learn more about how OpenAPI powers GPTs, read our blog post.

You can also check out our other API clients:

Sitemap Extractor: Every Page URL of Any Website API in Python

Sitemap Extractor: Every Page URL of Any Website API in JavaScript

Sitemap Extractor: Every Page URL of Any Website API through CLI

Sitemap Extractor: Every Page URL of Any Website API

Sitemap Extractor

apify/sitemap-extractor

This Apify Actor extracts all URLs from a website's sitemaps and checks their status codes via lightweight HTTP requests. It provides a clean list of valid links, acting as an ideal pre-processor to ensure your larger crawling projects target only active URLs.

Apify

403

3.2

(8)

Website Sitemap Extractor

glassventures/website-sitemap-extractor

Extract all URLs from any website's sitemap. Auto-discovers sitemaps from robots.txt, supports sitemap index files and .gz compression. Filter by URL pattern, date range.

Glass Ventures

XML Sitemap Finder & Extractor API

andok/find-sitemap-from-url

Find and extract all XML sitemaps for any domain. Automatically parses robots.txt, scans HTML tags, and recursively follows indexes. Perfect for SEO & web scraping.

Andok

Sitemap Extractor: Website → All URLs (sitemap.xml parser)

boxbox10/sitemap-extractor

Give it a website. Get every URL from its sitemap — loc, lastmod, changefreq, priority — as one clean record per URL. Auto-discovers sitemap.xml, robots.txt Sitemap: directives, and nested sitemap indexes. Perfect for SEO audits, crawl seeding, and URL discovery.

Marvin Eguilos

Sitemap Change Detector

tri_angle/sitemap-change-detector

Identify and monitor sitemaps for specified websites. Retrieve only the new, updated, or removed URLs since the last crawl.

Tri⟁angle

Sitemap URL Extractor — recursive, filters, .gz

x402opklaar/sitemap-url-extractor

Extract every URL from any sitemap: recursive sitemap-index expansion, gzipped sitemaps, robots.txt auto-discovery, include/exclude filters, lastmod output. Feed the URL list to any crawler or audit. Pay per sitemap file processed.

Opklaar

Sitemap & URL Extractor — Get Every URL of a Website

dataquarry/sitemap-url-extractor

Get every URL of a website: parses sitemap.xml and sitemap-indexes (discovered via robots.txt or the default location), with a same-site crawl fallback when there's no sitemap. Returns each URL + lastmod. No API key.

Daniel Brenner

Sitemap Scraper

pvillalva/sitemap-scraper

The Sitemap Scraper extracts and outputs all URLs from a given sitemap.

Percival Villalva

278

Sitemap Extractor

smorgi_apps/sitemap-extractor

Smorgi Apps

Sitemap URL Extractor & 404 Checker — XML Sitemap Scraper

factpipe/sitemap-url-extractor

Extract every URL from any website's XML sitemaps: robots.txt discovery, nested sitemap indexes, lastmod and priority. Optional HTTP status check finds 404s and redirects. $0.30 per 1,000 URLs.