Sitemap URL Finder & New URL Monitor

Extract sitemap URLs into a source-linked inventory for SEO audits, migrations and content ingestion. The Actor reads robots.txt, XML sitemap indexes, namespace-prefixed XML, CDATA, gzip and text sitemaps without fetching every destination page.

Example input

{"websites":[{"url":"https://docs.apify.com"}],"maxResults":5,"maxRequestsPerCrawl":5}

The existing inventory mode is the default. websites discovers sitemap locations; existing API clients can instead pass startUrls with direct sitemap or robots URLs. Do not send the default websites value when using direct startUrls: website discovery retains its existing precedence. All original aliases, filters and row fields remain available.

Monitor newly observed URLs

{"startUrls":[{"url":"https://docs.apify.com/sitemap.xml"}],"mode":"changes","historyName":"my-docs-watchlist","maxResults":5,"maxRequestsPerCrawl":5}

Use the same inputs and history for serial repeat runs. The first delivered observations are baseline; later new URLs are new, and changed sitemap metadata is metadata_changed. An unchanged check can correctly return no rows. previousLastmod supplies the previous published hint when available. A sitemap hint does not prove the destination page changed, and missing URLs never mean removal.

Each watchlist retains at most 10,000 compact URL observations. Older evicted URLs can appear new again. Separate source/filter combinations use separate history records; use distinct history names for independent clients. Only confirmed delivered positives enter history. Source, output and history writes are not one transaction: interrupted jobs can produce repeats, and uncertain writes are not retried blindly.

Coverage and crawler handoff

OUTPUT reports per-sitemap outcomes, delivered count, limits, failures and whether history was updated. complete:false and stoppedReasons explain request, result or charge limits and source errors. A configured result cap is not a promise that the complete website was scanned. Invalid XML, unsupported declarations and invalid gzip are reported rather than treated as useful empty results. All failed sources fail the run.

START_URLS contains {"startUrls":[{"url":"..."}],"deliveredCount":5,"truncated":false} for downstream crawlers. It contains only confirmed delivered URLs, with a 10,000-entry handoff cap and explicit truncation. Destination pages are not fetched. No new schedule, proxy subscription or AI service is created.

Existing result and startup prices remain unchanged; the current Store tier is authoritative. Empty or unchanged checks still consume platform resources and may incur the existing startup event. Charge-limited rows are not added to history. Start with the bounded preview above, then raise limits only for a useful workload.

Example Output

{
  "url": "https://docs.apify.com/platform/actors",
  "domain": "docs.apify.com",
  "path": "/platform/actors",
  "sourceUrl": "https://docs.apify.com/sitemap_base.xml",
  "sourceDomain": "docs.apify.com",
  "lastmod": "2026-06-10",
  "changefreq": "weekly",
  "priority": "0.8",
  "filterType": "contains",
  "filterValue": "/platform/",
  "matchedAt": "2026-06-11T13:55:51.865Z"
}

Python API Example

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")

run = client.actor("thescrapelab/sitemap-target-url-extractor").call(run_input={
    "websites": [{"url": "https://docs.apify.com"}],
    "includeUrlText": "/platform/",
    "maxResults": 5,
})

if run is None:
    raise RuntimeError("Actor run failed")

items = client.dataset((run["defaultDatasetId"] if isinstance(run, dict) else run.default_dataset_id)).list_items().items

for item in items:
    print(item["url"], item["sourceUrl"])

Limits And Caveats

• The Actor reads sitemap files; it does not crawl every HTML page to discover links.
• Some websites do not publish complete or valid sitemaps.
• Password-protected, blocked, or private sitemaps may return no results.
• Very large sites can contain many sitemap indexes. Use maxResults and maxRequestsPerCrawl to keep runs predictable.
• Sitemap metadata is included only when the website provides it. Image, video, and alternate-language sitemap extensions are not currently included.

Support

Use the Actor issue/support link in Apify Console. Include the run ID and summary stop reason, without private account details or credentials.