Seo Sitemap Url Extractor

Sitemap URL Extractor

One small SEO job: read a public XML sitemap and return one dataset item per distinct page URL, including optional lastmod and the source sitemap. Sitemap indexes are followed one level only, with at most 20 child files. Supports .xml, .xml.gz, and gzip-compressed HTTP responses.

Who this is for

SEO specialists, content inventory owners, and small-site operators who need a quick, CSV-ready set of sitemap URLs without a site crawler or search engine scraping.

Input (copy into the Actor input form)

{
  "sitemapUrl": "https://www.djangoproject.com/sitemap.xml",
  "maxUrls": 5
}

The prefilled URL is a public Django project sitemap (checked as publicly reachable on 2026-11-01). maxUrls is optional (default 5, min 1, max 500). Only one sitemap URL is accepted per run.

Output (example schema)

Every successfully extracted URL has the following stable fields:

{
  "url": "https://www.djangoproject.com/",
  "lastmod": null,
  "sourceSitemap": "https://www.djangoproject.com/sitemap.xml",
  "status": "ok",
  "error": null
}

The example illustrates the field layout; individual URLs and dates depend on the live sitemap. lastmod is null when the tag is missing. Duplicate URLs are returned only once, in sitemap order. With no page URLs, the run returns an empty dataset.

If a sitemap cannot be read or parsed, one error item is produced instead of a traceback, for example:

{
  "url": null,
  "lastmod": null,
  "sourceSitemap": "https://site.example/sitemap.xml",
  "status": "error",
  "error": "The sitemap returned HTTP 404."
}

A bad child within an index also produces an error item while other children may still succeed. status: error items are not page URLs; filter on status: ok for URL exports.

Operation and limits

• Uses only the Python standard library (urllib, xml.etree, gzip, etc.). No Apify SDK, browser, search engine scraping, login, or installed packages.
• Makes requests only to the input host; redirects and child sitemap references to other hosts are rejected. Page URLs merely listed in the XML are not visited.
• Reads /robots.txt first and skips blocked sitemap paths. A missing robots.txt (404 or 410) permits fetching; 401/403 and ambiguous errors fail closed. Uses a descriptive Actor user agent.
• Up to 20 child sitemaps from an index, up to 500 unique page URLs, 4 MB transfer per sitemap, 12 MB decompressed, 500 KB robots.txt, 10 s/request, at most four same-host redirects. Only one index layer is followed; a nested index produces an error item. Later children are not fetched after the URL cap is reached.
• HTTPS downgraded to HTTP, private network targets, credentials in URLs, nonstandard ports, oversized XML, and XML DTD/entity declarations are refused. Test-only localhost access is gated behind SEO_TEST_ALLOW_LOOPBACK=1; never set this variable in production.
• Extracts <loc> and <lastmod> from XML sitemap <url> elements. It does not validate destination pages, read image/video tags, or guarantee that URLs are indexable.
• For public operation, HTTP requests require working DNS and internet access.

Use locally / test

No dependencies to install:

$ACTOR_INPUT_JSON='{"sitemapUrl":"https://www.djangoproject.com/sitemap.xml","maxUrls":5}' python src/main.py

JSON item files are written under storage/datasets/default/. Alternatively create storage/key_value_stores/default/INPUT.json and run python src/main.py. Cloud runs read the run input from the default key-value store and append results through the Apify REST API, requiring the platform's APIFY_TOKEN, APIFY_DEFAULT_KEY_VALUE_STORE_ID, and APIFY_DEFAULT_DATASET_ID environment variables (no SDK).

For offline, actual script-mode checks with local HTTP fixtures:

python tests/run_smoke.py
python -m unittest discover -s tests -p 'test_*.py' -v

tests/evidence/ contains the actual smoke input, terminal output, and dataset item(s). Smoke uses a local fixture server plus a dedicated test-only localhost override because public DNS was unavailable in the packaging environment. This is not a live Apify or live public-network test.

Store configuration suggestion (NOT applied)

• Categories (max 3): SEO tools, Marketing, Developer tools.
• Pricing: $0.001 per successfully extracted URL (status: ok); don't charge for error rows. This is an indicative proposal, not an Apify billing setup.
• Full suggestion is in store_metadata.json (documentation metadata; it is not a billing command).

No publishing, billing configuration, or platform run was performed.