Sitemap New Urls
Pricing
$2.00 / 1,000 results
Sitemap New Urls
Pricing
$2.00 / 1,000 results
Rating
0.0
(0)
Developer
Eichan Jolly
Maintained by CommunityActor stats
0
Bookmarked
2
Total users
1
Monthly active users
13 days ago
Last modified
Categories
Features
Share
Sitemap New URLs Snapshot
Read one public XML sitemap URL and export URL records. Optionally compare every URL to a caller-supplied knownUrls baseline, exposing isNew. A sitemap index is supported for the first five direct child urlset files.
Who it is for
SEO analysts, website owners, publishing operations, and automation builders tracking which public URLs appear in a sitemap.
Input example
{"sitemapUrl": "https://docs.apify.com/sitemap.xml","maxUrls": 100}
The form prefills one sitemap URL. For a meaningful new URL comparison, provide previously known URL strings:
{"sitemapUrl": "https://docs.apify.com/sitemap.xml","knownUrls": ["https://docs.apify.com/"],"maxUrls": 100,"onlyNew": true}
| Field | Required | Meaning |
|---|---|---|
sitemapUrl | Yes | One public XML urlset or sitemapindex URL. |
knownUrls | No | Previously known URL strings (up to 2,000). If omitted, isNew is null. If explicitly [], every URL is new. Matching is exact and case-sensitive. |
maxUrls | No | Maximum unique entries in document order, 1–200, default 100. |
onlyNew | No | Write only records with isNew: true; requires the knownUrls field, even if empty. Default false. |
Output example
One Dataset item per qualifying URL (illustrative values):
{"url": "https://docs.apify.com/actors/","lastModified": "2026-10-31","isNew": true,"sourceSitemap": "https://docs.apify.com/sitemap.xml","checkedAt": "2026-11-01T09:00:00Z"}
url: absolute public HTTP(S) page URL from the XML<loc>element.lastModified: unmodified<lastmod>string if present (maximum 100 characters), otherwisenull. Sitemap timestamps are publisher-supplied, not independently verified.isNew:true/falsewhen aknownUrlsbaseline is explicitly present; otherwisenull(unknown). This is not automatic cross-run new URL detection.sourceSitemap: XML file that contained this<url>record (useful when reading a sitemap index).checkedAt: UTC ISO 8601 timestamp assigned to all records from the run.
Scheduled use case
Schedule daily to export a snapshot of your sitemap. Preserve previous run URLs in your workflow or external storage, then supply them as knownUrls to the next run and set onlyNew: true if you only need additions. The Actor does not read previous runs or persist a baseline by itself, and knownUrls is capped at 2,000, so it is intended for small sitemaps or bounded comparisons.
Limits and behavior
- XML
urlsetsitemaps and one-levelsitemapindexfiles only. A sitemap index reads up to five child sitemap files in index order. Nested indexes are rejected with a clear error. - Source download limit 2 MB per sitemap, max 200 output candidates, 20-second HTTP timeout per request. Processing is sequential; server rate limits and connectivity may affect results.
- HTML landing pages, HTML error pages, RSS feeds, unrelated XML, malformed XML, and DTD/entity declarations are explicitly rejected with plain-language messages. In particular an HTML page is not silently treated as an empty sitemap.
- The Actor does not fetch listed pages, crawl websites, render JavaScript, follow canonical tags, or prove that listed URLs are live. Output URL strings are checked for HTTP(S) syntax and obviously private literal IPs but not DNS-resolved.
- Only public source HTTP(S) web URLs on standard ports; redirects to private networks are blocked. DNS can change between validation and connection; do not treat this as a hardened network security boundary.
- Deduplication is by exact URL string within one run; content is emitted in XML order. For an empty sitemap there are zero rows.
maxUrlsmay truncate the snapshot; an index with more than five child sitemaps is explicitly noted in terminal output. BadStatusLineand other malformed HTTP replies are reported as plain download errors, never tracebacks.- No login, external API keys, persistent state, alerting, third-party scraping services, or enabled billing events.
Run locally and verify
Create storage/key_value_stores/default/INPUT.json with the input JSON, or set ACTOR_INPUT_JSON. From the Actor root run:
python src/main.pypython -m unittest discover -s tests -p 'test_*.py' -vpython tests/smoke_run.py
Rows are saved in storage/datasets/default/000000001.json, etc. tests/smoke_run.py starts a local HTTP fixture and executes python src/main.py in a subprocess, then prints actual terminal output and Dataset items. It sets EICHAN_TEST_LOOPBACK=1 only for isolated non-cloud testing, not production; cloud and ordinary local runs reject loopback URLs.
Deploy to Apify
Upload this complete folder to a new Actor, build the included Dockerfile, and run it. The Apify runtime supplies the input key-value-store and default Dataset identifiers plus a run token; the code uses only urllib to access its own run storage. The runtime token is not a user-managed external API key.
Store categories and suggested pricing
- Category: Automation (primary)
- Secondary category: Developer tools
- Suggested pricing model: Pay per event — per-run
actor-startevent, provisional $0.005/run, for one bounded sitemap query. Review usage and performance before setting a real price. - Actual billing: Not configured or charged; no billing events emitted. Pricing and category selection would be set manually only if this Actor is later published.