Sitemap New Urls avatar

Sitemap New Urls

Pricing

$2.00 / 1,000 results

Go to Apify Store
Sitemap New Urls

Sitemap New Urls

Pricing

$2.00 / 1,000 results

Rating

0.0

(0)

Developer

Eichan Jolly

Eichan Jolly

Maintained by Community

Actor stats

0

Bookmarked

2

Total users

1

Monthly active users

13 days ago

Last modified

Categories

Features

Limited permissions

Share

Sitemap New URLs Snapshot

Read one public XML sitemap URL and export URL records. Optionally compare every URL to a caller-supplied knownUrls baseline, exposing isNew. A sitemap index is supported for the first five direct child urlset files.

Who it is for

SEO analysts, website owners, publishing operations, and automation builders tracking which public URLs appear in a sitemap.

Input example

{
"sitemapUrl": "https://docs.apify.com/sitemap.xml",
"maxUrls": 100
}

The form prefills one sitemap URL. For a meaningful new URL comparison, provide previously known URL strings:

{
"sitemapUrl": "https://docs.apify.com/sitemap.xml",
"knownUrls": ["https://docs.apify.com/"],
"maxUrls": 100,
"onlyNew": true
}
FieldRequiredMeaning
sitemapUrlYesOne public XML urlset or sitemapindex URL.
knownUrlsNoPreviously known URL strings (up to 2,000). If omitted, isNew is null. If explicitly [], every URL is new. Matching is exact and case-sensitive.
maxUrlsNoMaximum unique entries in document order, 1–200, default 100.
onlyNewNoWrite only records with isNew: true; requires the knownUrls field, even if empty. Default false.

Output example

One Dataset item per qualifying URL (illustrative values):

{
"url": "https://docs.apify.com/actors/",
"lastModified": "2026-10-31",
"isNew": true,
"sourceSitemap": "https://docs.apify.com/sitemap.xml",
"checkedAt": "2026-11-01T09:00:00Z"
}
  • url: absolute public HTTP(S) page URL from the XML <loc> element.
  • lastModified: unmodified <lastmod> string if present (maximum 100 characters), otherwise null. Sitemap timestamps are publisher-supplied, not independently verified.
  • isNew: true / false when a knownUrls baseline is explicitly present; otherwise null (unknown). This is not automatic cross-run new URL detection.
  • sourceSitemap: XML file that contained this <url> record (useful when reading a sitemap index).
  • checkedAt: UTC ISO 8601 timestamp assigned to all records from the run.

Scheduled use case

Schedule daily to export a snapshot of your sitemap. Preserve previous run URLs in your workflow or external storage, then supply them as knownUrls to the next run and set onlyNew: true if you only need additions. The Actor does not read previous runs or persist a baseline by itself, and knownUrls is capped at 2,000, so it is intended for small sitemaps or bounded comparisons.

Limits and behavior

  • XML urlset sitemaps and one-level sitemapindex files only. A sitemap index reads up to five child sitemap files in index order. Nested indexes are rejected with a clear error.
  • Source download limit 2 MB per sitemap, max 200 output candidates, 20-second HTTP timeout per request. Processing is sequential; server rate limits and connectivity may affect results.
  • HTML landing pages, HTML error pages, RSS feeds, unrelated XML, malformed XML, and DTD/entity declarations are explicitly rejected with plain-language messages. In particular an HTML page is not silently treated as an empty sitemap.
  • The Actor does not fetch listed pages, crawl websites, render JavaScript, follow canonical tags, or prove that listed URLs are live. Output URL strings are checked for HTTP(S) syntax and obviously private literal IPs but not DNS-resolved.
  • Only public source HTTP(S) web URLs on standard ports; redirects to private networks are blocked. DNS can change between validation and connection; do not treat this as a hardened network security boundary.
  • Deduplication is by exact URL string within one run; content is emitted in XML order. For an empty sitemap there are zero rows. maxUrls may truncate the snapshot; an index with more than five child sitemaps is explicitly noted in terminal output.
  • BadStatusLine and other malformed HTTP replies are reported as plain download errors, never tracebacks.
  • No login, external API keys, persistent state, alerting, third-party scraping services, or enabled billing events.

Run locally and verify

Create storage/key_value_stores/default/INPUT.json with the input JSON, or set ACTOR_INPUT_JSON. From the Actor root run:

python src/main.py
python -m unittest discover -s tests -p 'test_*.py' -v
python tests/smoke_run.py

Rows are saved in storage/datasets/default/000000001.json, etc. tests/smoke_run.py starts a local HTTP fixture and executes python src/main.py in a subprocess, then prints actual terminal output and Dataset items. It sets EICHAN_TEST_LOOPBACK=1 only for isolated non-cloud testing, not production; cloud and ordinary local runs reject loopback URLs.

Deploy to Apify

Upload this complete folder to a new Actor, build the included Dockerfile, and run it. The Apify runtime supplies the input key-value-store and default Dataset identifiers plus a run token; the code uses only urllib to access its own run storage. The runtime token is not a user-managed external API key.

Store categories and suggested pricing

  • Category: Automation (primary)
  • Secondary category: Developer tools
  • Suggested pricing model: Pay per event — per-run actor-start event, provisional $0.005/run, for one bounded sitemap query. Review usage and performance before setting a real price.
  • Actual billing: Not configured or charged; no billing events emitted. Pricing and category selection would be set manually only if this Actor is later published.