Pricing

from $0.50 / 1,000 record scrapeds

Try for free

Go to Apify Store

Sitemap Walker Pro — Recursive URL Discovery

Try for free

Walk sitemaps and sitemap-index trees recursively. robots.txt fallback. Include/exclude globs/regex, lastmodSince filter, priorityMin filter, optional chunked output. Per-URL rows with image / video / news / hreflang metadata. Pure HTTP, no browser.

Pricing

from $0.50 / 1,000 record scrapeds

Rating

0.0

(0)

Developer

BowTiedRaccoon

Actor stats

Bookmarked

Total users

Monthly active users

18 days ago

Last modified

Categories

SEO tools

Developer tools

You can access the Sitemap Walker Pro — Recursive URL Discovery programmatically from your own applications by using the Apify API. You can also choose the language preference from below. To use the Apify API, you’ll need an Apify account and your API token, found in API & Integrations in Apify Console.

Python

JavaScript

CLI

OpenAPI

HTTP

MCP

{
  "openapi": "3.0.1",
  "info": {
    "version": "0.1",
    "x-build-id": "7ENqwrL2PiKbqytcF"
  },
  "servers": [
    {
      "url": "https://api.apify.com/v2"
    }
  ],
  "paths": {
    "/acts/jungle_synthesizer~sitemap-url-extractor-pro/run-sync-get-dataset-items": {
      "post": {
        "operationId": "run-sync-get-dataset-items-jungle_synthesizer-sitemap-url-extractor-pro",
        "x-openai-isConsequential": false,
        "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.",
        "tags": [
          "Run Actor"
        ],
        "requestBody": {
          "required": true,
          "content": {
            "application/json": {
              "schema": {
                "$ref": "#/components/schemas/inputSchema"
              }
            }
          }
        },
        "parameters": [
          {
            "name": "token",
            "in": "query",
            "required": true,
            "schema": {
              "type": "string"
            },
            "description": "Enter your Apify token here"
          }
        ],
        "responses": {
          "200": {
            "description": "OK"
          }
        }
      }
    },
    "/acts/jungle_synthesizer~sitemap-url-extractor-pro/runs": {
      "post": {
        "operationId": "runs-sync-jungle_synthesizer-sitemap-url-extractor-pro",
        "x-openai-isConsequential": false,
        "summary": "Executes an Actor and returns information about the initiated run in response.",
        "tags": [
          "Run Actor"
        ],
        "requestBody": {
          "required": true,
          "content": {
            "application/json": {
              "schema": {
                "$ref": "#/components/schemas/inputSchema"
              }
            }
          }
        },
        "parameters": [
          {
            "name": "token",
            "in": "query",
            "required": true,
            "schema": {
              "type": "string"
            },
            "description": "Enter your Apify token here"
          }
        ],
        "responses": {
          "200": {
            "description": "OK",
            "content": {
              "application/json": {
                "schema": {
                  "$ref": "#/components/schemas/runsResponseSchema"
                }
              }
            }
          }
        }
      }
    },
    "/acts/jungle_synthesizer~sitemap-url-extractor-pro/run-sync": {
      "post": {
        "operationId": "run-sync-jungle_synthesizer-sitemap-url-extractor-pro",
        "x-openai-isConsequential": false,
        "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.",
        "tags": [
          "Run Actor"
        ],
        "requestBody": {
          "required": true,
          "content": {
            "application/json": {
              "schema": {
                "$ref": "#/components/schemas/inputSchema"
              }
            }
          }
        },
        "parameters": [
          {
            "name": "token",
            "in": "query",
            "required": true,
            "schema": {
              "type": "string"
            },
            "description": "Enter your Apify token here"
          }
        ],
        "responses": {
          "200": {
            "description": "OK"
          }
        }
      }
    }
  },
  "components": {
    "schemas": {
      "inputSchema": {
        "type": "object",
        "required": [
          "sp_intended_usage",
          "sp_improvement_suggestions",
          "seeds"
        ],
        "properties": {
          "sp_intended_usage": {
            "title": "What is the intended usage of this data?",
            "minLength": 1,
            "type": "string",
            "description": "What will this data feed? E.g. lead lists, KYB checks, price tracking."
          },
          "sp_improvement_suggestions": {
            "title": "How can we improve this crawler for you?",
            "minLength": 1,
            "type": "string",
            "description": "Provide any feedback or suggestions for improvements."
          },
          "sp_contact": {
            "title": "Contact email (optional)",
            "minLength": 1,
            "type": "string",
            "description": "We'll personally help with your use case. No spam."
          },
          "seeds": {
            "title": "Seeds",
            "type": "array",
            "description": "List of sitemap URLs OR site root URLs. Site roots fall back to robots.txt sitemap discovery when fallbackToRobotsTxt is on.",
            "items": {
              "type": "string"
            }
          },
          "fallbackToRobotsTxt": {
            "title": "Fall Back To robots.txt",
            "type": "boolean",
            "description": "When a seed lacks an explicit sitemap, fetch /robots.txt and parse its Sitemap: directives.",
            "default": true
          },
          "recurseSitemapIndex": {
            "title": "Recurse Sitemap Index",
            "type": "boolean",
            "description": "Walk into nested <sitemapindex> entries (most large sites use these).",
            "default": true
          },
          "includePatterns": {
            "title": "Include Patterns",
            "type": "array",
            "description": "Glob or /regex/ patterns to include. Empty = include all.",
            "items": {
              "type": "string"
            }
          },
          "excludePatterns": {
            "title": "Exclude Patterns",
            "type": "array",
            "description": "Glob or /regex/ patterns to exclude after includes apply.",
            "items": {
              "type": "string"
            }
          },
          "lastmodSince": {
            "title": "Last-Modified Since (ISO date)",
            "type": "string",
            "description": "Only emit URLs with lastmod >= this date. Useful for incremental crawls."
          },
          "priorityMin": {
            "title": "Minimum Priority",
            "type": "number",
            "description": "Only emit URLs with priority >= this (0.0-1.0)."
          },
          "chunkSize": {
            "title": "Chunk Size",
            "type": "integer",
            "description": "If set, group output URLs into chunks of this size and tag each row with a chunk index."
          },
          "maxUrls": {
            "title": "Max URLs",
            "type": "integer",
            "description": "Hard cap on total URLs emitted (0 = unlimited). Governs the run scope; the Apify tester's maxItems guard is separate.",
            "default": 0
          },
          "maxItems": {
            "title": "Max Items (Apify Tester Cap)",
            "maximum": 1000000,
            "type": "integer",
            "description": "Apify tester safety cap — limits emitted records regardless of maxUrls. Defaults to 15 (the tester's 5-min timeout headroom). Override only if you know what you're doing.",
            "default": 15
          }
        }
      },
      "runsResponseSchema": {
        "type": "object",
        "properties": {
          "data": {
            "type": "object",
            "properties": {
              "id": {
                "type": "string"
              },
              "actId": {
                "type": "string"
              },
              "userId": {
                "type": "string"
              },
              "startedAt": {
                "type": "string",
                "format": "date-time",
                "example": "2025-01-08T00:00:00.000Z"
              },
              "finishedAt": {
                "type": "string",
                "format": "date-time",
                "example": "2025-01-08T00:00:00.000Z"
              },
              "status": {
                "type": "string",
                "example": "READY"
              },
              "meta": {
                "type": "object",
                "properties": {
                  "origin": {
                    "type": "string",
                    "example": "API"
                  },
                  "userAgent": {
                    "type": "string"
                  }
                }
              },
              "stats": {
                "type": "object",
                "properties": {
                  "inputBodyLen": {
                    "type": "integer",
                    "example": 2000
                  },
                  "rebootCount": {
                    "type": "integer",
                    "example": 0
                  },
                  "restartCount": {
                    "type": "integer",
                    "example": 0
                  },
                  "resurrectCount": {
                    "type": "integer",
                    "example": 0
                  },
                  "computeUnits": {
                    "type": "integer",
                    "example": 0
                  }
                }
              },
              "options": {
                "type": "object",
                "properties": {
                  "build": {
                    "type": "string",
                    "example": "latest"
                  },
                  "timeoutSecs": {
                    "type": "integer",
                    "example": 300
                  },
                  "memoryMbytes": {
                    "type": "integer",
                    "example": 1024
                  },
                  "diskMbytes": {
                    "type": "integer",
                    "example": 2048
                  }
                }
              },
              "buildId": {
                "type": "string"
              },
              "defaultKeyValueStoreId": {
                "type": "string"
              },
              "defaultDatasetId": {
                "type": "string"
              },
              "defaultRequestQueueId": {
                "type": "string"
              },
              "buildNumber": {
                "type": "string",
                "example": "1.0.0"
              },
              "containerUrl": {
                "type": "string"
              },
              "usage": {
                "type": "object",
                "properties": {
                  "ACTOR_COMPUTE_UNITS": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATASET_READS": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATASET_WRITES": {
                    "type": "integer",
                    "example": 0
                  },
                  "KEY_VALUE_STORE_READS": {
                    "type": "integer",
                    "example": 0
                  },
                  "KEY_VALUE_STORE_WRITES": {
                    "type": "integer",
                    "example": 1
                  },
                  "KEY_VALUE_STORE_LISTS": {
                    "type": "integer",
                    "example": 0
                  },
                  "REQUEST_QUEUE_READS": {
                    "type": "integer",
                    "example": 0
                  },
                  "REQUEST_QUEUE_WRITES": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATA_TRANSFER_INTERNAL_GBYTES": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATA_TRANSFER_EXTERNAL_GBYTES": {
                    "type": "integer",
                    "example": 0
                  },
                  "PROXY_RESIDENTIAL_TRANSFER_GBYTES": {
                    "type": "integer",
                    "example": 0
                  },
                  "PROXY_SERPS": {
                    "type": "integer",
                    "example": 0
                  },
                  "PROXY_UNBLOCKER_UNITS": {
                    "type": "integer",
                    "example": 0
                  }
                }
              },
              "usageTotalUsd": {
                "type": "number",
                "example": 0.00005
              },
              "usageUsd": {
                "type": "object",
                "properties": {
                  "ACTOR_COMPUTE_UNITS": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATASET_READS": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATASET_WRITES": {
                    "type": "integer",
                    "example": 0
                  },
                  "KEY_VALUE_STORE_READS": {
                    "type": "integer",
                    "example": 0
                  },
                  "KEY_VALUE_STORE_WRITES": {
                    "type": "number",
                    "example": 0.00005
                  },
                  "KEY_VALUE_STORE_LISTS": {
                    "type": "integer",
                    "example": 0
                  },
                  "REQUEST_QUEUE_READS": {
                    "type": "integer",
                    "example": 0
                  },
                  "REQUEST_QUEUE_WRITES": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATA_TRANSFER_INTERNAL_GBYTES": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATA_TRANSFER_EXTERNAL_GBYTES": {
                    "type": "integer",
                    "example": 0
                  },
                  "PROXY_RESIDENTIAL_TRANSFER_GBYTES": {
                    "type": "integer",
                    "example": 0
                  },
                  "PROXY_SERPS": {
                    "type": "integer",
                    "example": 0
                  },
                  "PROXY_UNBLOCKER_UNITS": {
                    "type": "integer",
                    "example": 0
                  }
                }
              }
            }
          }
        }
      }
    }
  }
}

Sitemap Walker Pro — Recursive URL Discovery OpenAPI definition

OpenAPI is a standard for designing and describing RESTful APIs, allowing developers to define API structure, endpoints, and data formats in a machine-readable way. It simplifies API development, integration, and documentation.

OpenAPI is effective when used with AI agents and GPTs by standardizing how these systems interact with various APIs, for reliable integrations and efficient communication.

By defining machine-readable API specifications, OpenAPI allows AI models like GPTs to understand and use varied data sources, improving accuracy. This accelerates development, reduces errors, and provides context-aware responses, making OpenAPI a core component for AI applications.

You can download the OpenAPI definitions for Sitemap Walker Pro — Recursive URL Discovery from the options below:

OpenAPI.json

If you’d like to learn more about how OpenAPI powers GPTs, read our blog post.

You can also check out our other API clients:

Sitemap Walker Pro — Recursive URL Discovery API in Python

Sitemap Walker Pro — Recursive URL Discovery API in JavaScript

Sitemap Walker Pro — Recursive URL Discovery API through CLI

Sitemap Walker Pro — Recursive URL Discovery API

Website URL Crawler & Link Extractor

maged120/get-urls-pro

Crawl any website and extract every URL with its depth, parent page and anchor text. Build a sitemap or audit internal links. Works on static and JavaScript sites.

Maged

221

Sitemap Generator - Crawl Website & Create XML Sitemap

scrappy_garden/sitemap-generator

Generate an XML sitemap for any website. Crawls internal pages from start URLs (with depth + page limits), deduplicates URLs, and stores a ready-to-submit sitemap.xml plus a structured dataset and summary for SEO audits.

Bikram Adhikari

Website Sitemap Extractor

glassventures/website-sitemap-extractor

Extract all URLs from any website's sitemap. Auto-discovers sitemaps from robots.txt, supports sitemap index files and .gz compression. Filter by URL pattern, date range.

Glass Ventures

Sitemap Generator

datawinder/sitemap-generator

Automatically crawl a website and generate an SEO-ready sitemap in XML, HTML, or TXT format. Supports crawl depth limits, URL include/exclude patterns, and optional merging with an existing sitemap.xml. Ideal for SEO audits, site migrations, and automation.

DatawinderLabs

Sitemap Extractor

cerebral_aluminum/sitemap-extractor

Extract all URLs from website sitemaps. Pages, images, PDFs. Handles sitemap indexes and WordPress.

Benny

XML Sitemap Scraper & URL Extractor API - SEO Crawler

pink_comic/sitemap-url-extractor

Parse XML website sitemaps and extract URLs for SEO crawls, audits, migrations, and RAG indexing. Auto-discovers sitemap files from robots.txt, follows nested indexes, and exports URL, lastmod, priority, changefreq, and image metadata.

Ava Torres

Sitemap URL Extractor

mikolabs/sitemap-url-extractor

Extract every URL and its metadata from any sitemap.xml in seconds. Paste one or more sitemap URLs, run the Actor, and get a clean, structured dataset with url, lastmod, changefreq, priority, and more — ready to export as CSV, JSON, or Excel.

Mikolabs

Sitemap URL Extractor - XML Sitemap Scraper

benthepythondev/sitemap-url-extractor

Extract URLs from XML sitemaps and sitemap indexes. Get URL, lastmod, changefreq, priority and source sitemap.

Ben

Sitemap URL Extractor

seemuapps/sitemap-extractor

Extract every URL from a website's sitemap.xml. Recursively walks nested sitemap indexes and returns loc, lastmod, changefreq, and priority for each page.

Seemu Scraping

Sitemap URL Extractor - List All URLs in a Sitemap

dltik/sitemap-url-extractor

Extract every URL from any XML sitemap, with lastmod, changefreq and priority. Resolves sitemap indexes recursively. Pass a sitemap.xml or just a site root to auto-discover its sitemaps. Pure HTTP, no browser — fast and cheap.