Pricing

from $0.01 / 1,000 url records

Try for free

Go to Apify Store

URL & Sitemap Link Extractor: Site Map, Health Checks & Monitor

Try for free

Extract every URL from a site's sitemap or a page, no sitemap link needed. Auto-discovers via robots.txt, follows sitemap-index nesting, keeps lastmod, changefreq, and priority. Adds a site-structure summary, link-health checks, and change monitoring. Export as JSON, CSV, or Excel. $0.00001 per URL.

Pricing

from $0.01 / 1,000 url records

Rating

0.0

(0)

Developer

GetAScraper

Actor stats

Bookmarked

Total users

Monthly active users

5 days ago

Last modified

Categories

SEO tools

Developer tools

Automation

You can access the URL & Sitemap Link Extractor: Site Map, Health Checks & Monitor programmatically from your own applications by using the Apify API. You can also choose the language preference from below. To use the Apify API, you’ll need an Apify account and your API token, found in API & Integrations in Apify Console.

Python

JavaScript

CLI

OpenAPI

HTTP

MCP

{
  "openapi": "3.0.1",
  "info": {
    "version": "0.1",
    "x-build-id": "0POxGXtqbje0Fm3Kv"
  },
  "servers": [
    {
      "url": "https://api.apify.com/v2"
    }
  ],
  "paths": {
    "/acts/getascraper~url-link-extractor/run-sync-get-dataset-items": {
      "post": {
        "operationId": "run-sync-get-dataset-items-getascraper-url-link-extractor",
        "x-openai-isConsequential": false,
        "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.",
        "tags": [
          "Run Actor"
        ],
        "requestBody": {
          "required": true,
          "content": {
            "application/json": {
              "schema": {
                "$ref": "#/components/schemas/inputSchema"
              }
            }
          }
        },
        "parameters": [
          {
            "name": "token",
            "in": "query",
            "required": true,
            "schema": {
              "type": "string"
            },
            "description": "Enter your Apify token here"
          }
        ],
        "responses": {
          "200": {
            "description": "OK"
          }
        }
      }
    },
    "/acts/getascraper~url-link-extractor/runs": {
      "post": {
        "operationId": "runs-sync-getascraper-url-link-extractor",
        "x-openai-isConsequential": false,
        "summary": "Executes an Actor and returns information about the initiated run in response.",
        "tags": [
          "Run Actor"
        ],
        "requestBody": {
          "required": true,
          "content": {
            "application/json": {
              "schema": {
                "$ref": "#/components/schemas/inputSchema"
              }
            }
          }
        },
        "parameters": [
          {
            "name": "token",
            "in": "query",
            "required": true,
            "schema": {
              "type": "string"
            },
            "description": "Enter your Apify token here"
          }
        ],
        "responses": {
          "200": {
            "description": "OK",
            "content": {
              "application/json": {
                "schema": {
                  "$ref": "#/components/schemas/runsResponseSchema"
                }
              }
            }
          }
        }
      }
    },
    "/acts/getascraper~url-link-extractor/run-sync": {
      "post": {
        "operationId": "run-sync-getascraper-url-link-extractor",
        "x-openai-isConsequential": false,
        "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.",
        "tags": [
          "Run Actor"
        ],
        "requestBody": {
          "required": true,
          "content": {
            "application/json": {
              "schema": {
                "$ref": "#/components/schemas/inputSchema"
              }
            }
          }
        },
        "parameters": [
          {
            "name": "token",
            "in": "query",
            "required": true,
            "schema": {
              "type": "string"
            },
            "description": "Enter your Apify token here"
          }
        ],
        "responses": {
          "200": {
            "description": "OK"
          }
        }
      }
    }
  },
  "components": {
    "schemas": {
      "inputSchema": {
        "type": "object",
        "properties": {
          "domain": {
            "title": "Domain",
            "type": "string",
            "description": "A bare domain (e.g. example.com). The Actor checks robots.txt for its declared sitemap(s) and follows any sitemap index automatically, no need to know the exact sitemap URL. Ignored whenever Specific Sitemap or Page URLs below is filled in, so clear this field or add your own URLs there, not both.",
            "default": "apify.com"
          },
          "links": {
            "title": "Specific Sitemap or Page URLs",
            "type": "array",
            "description": "One or more exact sitemap URLs or webpage URLs to extract from, when you already know them. A sitemap index is still followed automatically. A plain webpage URL returns the links found on that one page (no further crawling).",
            "default": [],
            "items": {
              "type": "string"
            }
          },
          "urlPattern": {
            "title": "Include Path Pattern",
            "type": "string",
            "description": "Only keep URLs whose path contains one of these comma-separated patterns (e.g. \"/blog, /products\"). Leave blank to keep every URL.",
            "default": ""
          },
          "customExcludePattern": {
            "title": "Exclude Path Pattern",
            "type": "string",
            "description": "Drop URLs whose path contains any of these comma-separated patterns (e.g. \"/tag, /author\"). Leave blank to skip this filter.",
            "default": ""
          },
          "excludeExtensions": {
            "title": "Exclude File Extensions",
            "type": "string",
            "description": "Drop URLs ending in these comma-separated file extensions (e.g. \"pdf, jpg, zip\"). Leave blank to skip this filter.",
            "default": ""
          },
          "checkLinkHealth": {
            "title": "Check Link Health",
            "type": "boolean",
            "description": "For a bounded sample of the returned URLs, send a request to each and record its HTTP status, so you can spot broken links. Costs one extra request per checked URL, so it defaults off.",
            "default": false
          },
          "maxUrls": {
            "title": "Max URLs",
            "minimum": 1,
            "maximum": 5000,
            "type": "integer",
            "description": "Maximum number of URLs to return in this run, after filters are applied.",
            "default": 25
          },
          "maxSitemapsToFollow": {
            "title": "Max Sitemaps to Follow",
            "minimum": 1,
            "maximum": 200,
            "type": "integer",
            "description": "Maximum number of sitemap or page URLs to actually fetch. A sitemap index can list hundreds or thousands of nested sitemaps (a real large site can have 2,000+); this caps how many of those are followed in one run.",
            "default": 5
          },
          "onlyNewOrChanged": {
            "title": "Incremental Monitor Mode",
            "type": "boolean",
            "description": "Only return URLs that are new or whose lastmod/changefreq/priority changed since your last run with the same monitor state name. Pair with an Apify schedule to catch new or updated pages without re-downloading the whole sitemap every time.",
            "default": false
          },
          "stateName": {
            "title": "Monitor State Name",
            "type": "string",
            "description": "Name for this run's incremental state, used only when Incremental Monitor Mode is on. Use different names to track separate domains or filters independently.",
            "default": "default"
          },
          "proxyConfiguration": {
            "title": "Proxy Configuration",
            "type": "object",
            "description": "This Actor makes plain HTTP requests to fetch sitemaps and pages. Datacenter proxy (the default) is fast and works for every target checked during development; switch to Residential only if a specific site you're targeting starts blocking datacenter IPs.",
            "default": {
              "useApifyProxy": true
            }
          }
        }
      },
      "runsResponseSchema": {
        "type": "object",
        "properties": {
          "data": {
            "type": "object",
            "properties": {
              "id": {
                "type": "string"
              },
              "actId": {
                "type": "string"
              },
              "userId": {
                "type": "string"
              },
              "startedAt": {
                "type": "string",
                "format": "date-time",
                "example": "2025-01-08T00:00:00.000Z"
              },
              "finishedAt": {
                "type": "string",
                "format": "date-time",
                "example": "2025-01-08T00:00:00.000Z"
              },
              "status": {
                "type": "string",
                "example": "READY"
              },
              "meta": {
                "type": "object",
                "properties": {
                  "origin": {
                    "type": "string",
                    "example": "API"
                  },
                  "userAgent": {
                    "type": "string"
                  }
                }
              },
              "stats": {
                "type": "object",
                "properties": {
                  "inputBodyLen": {
                    "type": "integer",
                    "example": 2000
                  },
                  "rebootCount": {
                    "type": "integer",
                    "example": 0
                  },
                  "restartCount": {
                    "type": "integer",
                    "example": 0
                  },
                  "resurrectCount": {
                    "type": "integer",
                    "example": 0
                  },
                  "computeUnits": {
                    "type": "integer",
                    "example": 0
                  }
                }
              },
              "options": {
                "type": "object",
                "properties": {
                  "build": {
                    "type": "string",
                    "example": "latest"
                  },
                  "timeoutSecs": {
                    "type": "integer",
                    "example": 300
                  },
                  "memoryMbytes": {
                    "type": "integer",
                    "example": 1024
                  },
                  "diskMbytes": {
                    "type": "integer",
                    "example": 2048
                  }
                }
              },
              "buildId": {
                "type": "string"
              },
              "defaultKeyValueStoreId": {
                "type": "string"
              },
              "defaultDatasetId": {
                "type": "string"
              },
              "defaultRequestQueueId": {
                "type": "string"
              },
              "buildNumber": {
                "type": "string",
                "example": "1.0.0"
              },
              "containerUrl": {
                "type": "string"
              },
              "usage": {
                "type": "object",
                "properties": {
                  "ACTOR_COMPUTE_UNITS": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATASET_READS": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATASET_WRITES": {
                    "type": "integer",
                    "example": 0
                  },
                  "KEY_VALUE_STORE_READS": {
                    "type": "integer",
                    "example": 0
                  },
                  "KEY_VALUE_STORE_WRITES": {
                    "type": "integer",
                    "example": 1
                  },
                  "KEY_VALUE_STORE_LISTS": {
                    "type": "integer",
                    "example": 0
                  },
                  "REQUEST_QUEUE_READS": {
                    "type": "integer",
                    "example": 0
                  },
                  "REQUEST_QUEUE_WRITES": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATA_TRANSFER_INTERNAL_GBYTES": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATA_TRANSFER_EXTERNAL_GBYTES": {
                    "type": "integer",
                    "example": 0
                  },
                  "PROXY_RESIDENTIAL_TRANSFER_GBYTES": {
                    "type": "integer",
                    "example": 0
                  },
                  "PROXY_SERPS": {
                    "type": "integer",
                    "example": 0
                  }
                }
              },
              "usageTotalUsd": {
                "type": "number",
                "example": 0.00005
              },
              "usageUsd": {
                "type": "object",
                "properties": {
                  "ACTOR_COMPUTE_UNITS": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATASET_READS": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATASET_WRITES": {
                    "type": "integer",
                    "example": 0
                  },
                  "KEY_VALUE_STORE_READS": {
                    "type": "integer",
                    "example": 0
                  },
                  "KEY_VALUE_STORE_WRITES": {
                    "type": "number",
                    "example": 0.00005
                  },
                  "KEY_VALUE_STORE_LISTS": {
                    "type": "integer",
                    "example": 0
                  },
                  "REQUEST_QUEUE_READS": {
                    "type": "integer",
                    "example": 0
                  },
                  "REQUEST_QUEUE_WRITES": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATA_TRANSFER_INTERNAL_GBYTES": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATA_TRANSFER_EXTERNAL_GBYTES": {
                    "type": "integer",
                    "example": 0
                  },
                  "PROXY_RESIDENTIAL_TRANSFER_GBYTES": {
                    "type": "integer",
                    "example": 0
                  },
                  "PROXY_SERPS": {
                    "type": "integer",
                    "example": 0
                  }
                }
              }
            }
          }
        }
      }
    }
  }
}

Sitemap URL Extractor: Site Audit and Monitor Tool OpenAPI definition

OpenAPI is a standard for designing and describing RESTful APIs, allowing developers to define API structure, endpoints, and data formats in a machine-readable way. It simplifies API development, integration, and documentation.

OpenAPI is effective when used with AI agents and GPTs by standardizing how these systems interact with various APIs, for reliable integrations and efficient communication.

By defining machine-readable API specifications, OpenAPI allows AI models like GPTs to understand and use varied data sources, improving accuracy. This accelerates development, reduces errors, and provides context-aware responses, making OpenAPI a core component for AI applications.

You can download the OpenAPI definitions for URL & Sitemap Link Extractor: Site Map, Health Checks & Monitor from the options below:

OpenAPI.json

If you’d like to learn more about how OpenAPI powers GPTs, read our blog post.

You can also check out our other API clients:

URL & Sitemap Link Extractor: Site Map, Health Checks & Monitor API in Python

URL & Sitemap Link Extractor: Site Map, Health Checks & Monitor API in JavaScript

URL & Sitemap Link Extractor: Site Map, Health Checks & Monitor API through CLI

URL & Sitemap Link Extractor: Site Map, Health Checks & Monitor API

Sitemap URL Extractor — robots.txt + sitemap.xml Crawl

v0iddo/sitemap-url-extractor

Discover every URL a site exposes via its public sitemap chain. Reads robots.txt, follows Sitemap declarations, recursively descends sitemap-index files, extracts URLs with lastmod, changefreq, priority.

vøiddo

Sitemap URL Extractor

lost.fany/sitemap-url-extractor

Extract every URL from a website's sitemap.xml, sitemap index and robots.txt in seconds — with lastmod, changefreq, priority and hreflang. No crawling, no browser: fast, cheap, reliable.

Natalia Ivanova

Sitemap Extractor: Website → All URLs (sitemap.xml parser)

boxbox10/sitemap-extractor

Give it a website. Get every URL from its sitemap — loc, lastmod, changefreq, priority — as one clean record per URL. Auto-discovers sitemap.xml, robots.txt Sitemap: directives, and nested sitemap indexes. Perfect for SEO audits, crawl seeding, and URL discovery.

Marvin Eguilos

Sitemap URL Extractor

ninhothedev/sitemap-url-extractor

$0.5/1K 🔥 Fast sitemap URL extractor! Every URL from any site's sitemap.xml & robots.txt — with lastmod & priority. JSON, CSV, Excel or API in seconds. Paste sites & pull thousands of URLs for SEO & crawling ⚡

ninhothedev

Sitemap URL Extractor — Every URL in a sitemap.xml as JSON

eliai/sitemap-url-extractor

Get every URL out of a sitemap.xml, a sitemap index (followed recursively), or a bare domain via robots.txt. Returns loc, lastmod, changefreq and priority per URL. Priced per FILE, not per URL: $0.005 per sitemap plus $0.015 per 1,000 URLs. Failures never charged.

Broke to Built

Sitemap URL Extractor - List All URLs in a Sitemap

dltik/sitemap-url-extractor

Extract every URL from any XML sitemap, with lastmod, changefreq and priority. Resolves sitemap indexes recursively. Pass a sitemap.xml or just a site root to auto-discover its sitemaps. Pure HTTP, no browser — fast and cheap.

Walid

Sitemap URL Extractor

mikolabs/sitemap-url-extractor

Extract every URL and its metadata from any sitemap.xml in seconds. Paste one or more sitemap URLs, run the Actor, and get a clean, structured dataset with url, lastmod, changefreq, priority, and more — ready to export as CSV, JSON, or Excel.

Mikolabs

Sitemap URL Extractor - XML Sitemap Scraper

benthepythondev/sitemap-url-extractor

Extract URLs from XML sitemaps and sitemap indexes. Get URL, lastmod, changefreq, priority and source sitemap.

Ben

XML Sitemap URL Extractor

devilscrapes/xml-sitemap-url-extractor

Turn any site's sitemap.xml, sitemap index, or robots.txt Sitemap directive into a clean URL dataset with loc, lastmod, changefreq and priority per row. Recursively expands sitemap indexes (bounded depth), follows redirects, and decompresses .xml.gz automatically.

DevilScrapes

Sitemap & URL Extractor — Get Every URL of a Website

dataquarry/sitemap-url-extractor

Get every URL of a website: parses sitemap.xml and sitemap-indexes (discovered via robots.txt or the default location), with a same-site crawl fallback when there's no sitemap. Returns each URL + lastmod. No API key.