Sitemap URL Extractor & Broken Link Checker

Sitemap links scraper returns page URLs and lastmod values from public sitemaps and robots.txt discovery. Sitemap links API results can also include HTTP status, redirect, and request error details when status checking is enabled.

Input example

{
  "startUrls": [
    "https://www.sitemaps.org/"
  ],
  "checkStatus": false,
  "maxItems": 20
}

The input keys and allowed values follow .actor/input_schema.json.

Output fields

Field	Type	Illustrative example
rowType	string	"url"
url	string	"https://www.sitemaps.org/protocol.html"
lastmod	string	"2022-12-15"
sitemapUrl	string	"https://www.sitemaps.org/sitemap.xml"
inputUrl	string	"https://www.sitemaps.org/"
checkStatus	string	"not_checked"
statusCode	integer or null	null
isBroken	boolean or null	null
finalUrl	string or null	null
redirectChain	array	[]
requestMethod	string or null	null
headStatusCode	integer or null	null
responseTimeMs	number or null	null
checkedAt	string or null	null
errors	array	[]

Output example

No sample_output.json is present in this actor folder. The following illustrative row is preserved from the previous README and is not a recorded actor run.

{
  "rowType": "url",
  "url": "https://www.sitemaps.org/protocol.html",
  "lastmod": "2022-12-15",
  "sitemapUrl": "https://www.sitemaps.org/sitemap.xml",
  "inputUrl": "https://www.sitemaps.org/",
  "checkStatus": "not_checked",
  "statusCode": null
}

Limits

Setting	Default	Accepted range
maxItems (all output rows)	1000	1–100000
concurrency	10	1–50
requestTimeoutSecs (each network operation)	20	1–120
maxRetries (extra attempts per hop)	2	0–5
maxSitemaps (global sitemap fetches)	1000	1–10000
maxDepth (initial source = zero)	10	0–50
maxRedirects (followed hops)	10	0–30

A sitemap document is limited to 10 MiB before and after gzip decompression. Larger files, DTD/entity declarations, malformed XML, private IP targets and non-HTTP URLs are rejected. robots.txt requests are additional to maxSitemaps. maxItems includes free diagnostics; errors can therefore reduce the number of extracted URLs. At most 1000 starting URLs are accepted.

Retrievable sitemap entries determine coverage. Pages omitted from sitemaps are not discovered. The actor does not create sitemaps, crawl HTML links, render JavaScript, authenticate, solve CAPTCHAs or verify content-level soft 404s. robots.txt Sitemap directives are used for discovery; Disallow rules are not evaluated for page checks. Use it on sites you are authorized to check.

FAQ

What are the output limits? maxItems caps extracted URLs and diagnostics. The actor rejects malformed, oversized, non-public or non-HTTP sitemap targets.

Do I need a proxy or login? Public sitemaps need no login or proxy. Private network targets are rejected.

How is this priced? Proposed pay per event: $0.00008 per unique extracted url row, including one with a failed optional status check. Discovery diagnostics and duplicates are uncharged. These are proposals in PRICING.md; the owner must configure Store billing.

How are rate limits handled? concurrency limits HTTP requests; selected transient failures are retried with backoff and Retry-After.

How fresh is the data? Sitemaps contain only URLs the publisher exposes; this actor does not crawl page links or render JavaScript. lastmod is the publisher’s value, not an independently verified update date. Check only sites you are authorized to query.

Use with AI agents / MCP

After the actor is published, connect your agent to the Apify MCP server with an Apify API token. Use call-actor with the published actor identifier and the JSON input shown above. Read the returned dataset with get-dataset-items; the run response itself contains status and storage IDs, not the result rows. Alternatively, use the Apify API to start a run and read its default dataset. Keep the actor input bounded with maxItems, and inspect source links and any error fields before using results in an answer.