Changelog & Product Updates Tracker – Launch Signals

Give it a list of company domains. For each one it reads the company's own sitemap and tells you what they changed recently: how many pages were updated in the last 7, 30 and 90 days, split by section (changelog, pricing, integrations, product, customers, blog, docs, careers). You also get the most recent notable URLs, a 0–100 launch score, and plain-English pitch reasons you can use in outreach.

It's built for sales and partnership teams who want a reason to reach out this week, such as "their pricing page changed 2 days ago" or "19 new integration pages this month". It's also useful for competitor watching.

No monitors to set up. Page-change monitors only see what changes after you add a page to them. This Actor reads the last-modified dates already in each company's sitemap, so from the first run you see which pages changed in the last 90 days, across every section. Paste a whole account list into one run.

Why it's different: many sites stamp their sitemap URLs with a build date or rewrite hundreds of them in one batch job, so a naive sitemap reader reports thousands of "changes" that never happened. This Actor detects build-date stamps, same-timestamp batches, restamped sections and stale sitemaps, and returns no change counts for them. It won't report fake activity, and you aren't charged for those domains. You pay only for rows with at least 5 trustworthy changes in the last 90 days, something changed in the last 30, at least one pitch reason, and at least 80% of the site's sitemap files read (language copies of files that were read aside).

How to track competitor changelogs and product updates

1. Paste your list. Put company domains in domains (one per line; full URLs are fine). A whole account or competitor list fits in one run.
2. Run it. Each domain's sitemap is read once, usually in seconds. Nothing needs to be set up in advance and no pages are monitored over time: the dates are already in the sitemap, so the first run covers the last 90 days.
3. Read the changelog signal. In bySection.changelog, last7d / last30d / last90d count changelog, release-notes and what's-new pages with a trustworthy recent lastmod. notableUrls lists the newest changed pages in the changelog, pricing, integrations, product and customers sections, with dates.
4. Sort by launchScore to see which companies had the most recent high-signal activity (changelog, pricing, integrations, product) in the last 30 days, and use pitchReasons as the opening line of your outreach.
5. Re-run it weekly (an Apify schedule works) to catch new product updates as they land. Turn on compareWithPreviousRun and each run also lists the URLs that are new, updated or removed since the last run.

This reads when pages changed, not what they say: it won't summarize a release note or diff a page. Sites with no <lastmod> dates in their sitemap come back as free no_lastmod rows.

What you get per domain

Field	Example / meaning
status	ok (analyzed, charged) or a free status. See Statuses
launchScore	83: 0–100, see the rubric
changes	{ "last7d": 24, "last30d": 99, "last90d": 216 }: pages whose <lastmod> falls in each window. Build-date stamps, batch restamps and restamped sections are excluded
bySection	Per section: urls in the sitemap, dated (pages with a trustworthy lastmod), last7d, last30d, last90d
notableUrls	Up to 20 of the most recently changed (not necessarily new) URLs in high-signal sections (changelog, pricing, integrations, product, customers) from the last 90 days, newest first, with section and lastmod. Up to 30 listed or cited pages per domain are checked live (when robots.txt allows; a page that times out or is blocked is kept): one that answers 404 or 410, also after a redirect, is left out and the next one takes its place
deadUrlsSkipped	Pages the sitemap dated recently that answered 404 or 410 when checked. A gone page is not counted as a change: it is left out of changes, bySection, the pitch lines and the charge, so a row whose recent high-signal pages are all gone is free. A 404 is trusted only when the site's homepage answers and not every checked page is 404 (some sites answer 404 to every bot)
latestChangeAt	The newest trustworthy lastmod on the site
pitchReasons	Lines about changes in the last 30 days come first; a line about an older edit (a pricing page changed 35 days ago) is context and comes last. E.g. "Pricing page updated yesterday (/pricing): possible pricing or packaging change…", "19 integration pages updated in the last 30 days (e.g. /integrations/sanity-content-agent): expanding their ecosystem; partnership or integration pitch"
lastmodReliable	true (counts reported), false (build-date or batch stamping detected, no counts), null (not measurable: no sitemap, no lastmod, stale sitemap, fetch failed)
noise	What the filter found: verdict (ok, partly_filtered, build_date_stamp, mostly_restamped, rolling_restamp, stale, too_few_changes, no_lastmod), dominantLastmodDay and dominantDayShare, stampedSitemaps (each with excludedUrls, the stamp day's URLs), batchRestamps (same-timestamp batches in the last 90 days: at, spanSeconds, urls), bulkUpdates (hours with mass restamps in the last 90 days), sweepRestamps (one-by-one sweeps in the last 90 days: section, at, spanSeconds, urls), restampedSections (each with excludedUrls), rollingSections, excludedUrls (all excluded URLs, including old batches that don't affect any window), recentExcludedShare (the share of the last 90 days' dated pages that was excluded), latestTrustworthyChangeAt, reviewSitemaps (sitemap files read as reviews of other companies' products; see Sections)
sitemap	origin, primaryHost (host of the first sitemap file read; null when none was), discoveredVia (robots.txt when robots.txt listed the sitemap, even if it was then disallowed or missing; fallback-paths when a usual path such as /sitemap.xml was read, or robots.txt listed none; null when no robots.txt gave usable rules), sitemapsFetched (files downloaded and read as a sitemap; a page that is not a sitemap, such as an HTML page at /sitemap.xml, is not counted), sitemapsFailed, sitemapsSkipped, urlsTotal, pagesTotal (after merging language copies), urlsWithLastmod, offDomainUrlsIgnored, coverage (full, partial, or none when no sitemap URL-set file was read, as on blocked_by_robots, no_sitemap and robots.txt-stage unreachable rows (an unreachable row from a failed DNS lookup has no sitemap object), whose shares are then 0), coverageShare (share of the site's sitemap URL-set files fully read, 0–1, language copies included; sitemap-index files and listed files that are not sitemaps are left out, a file cut at the byte cap counts as not read, and a sitemap index cut at the byte cap counts as at least one file not read (the children listed after the cut were never seen, so the share is an upper bound)), coverageShareWithoutLanguageCopies (the same share with skipped language files left out, each only when a file of the same name without the language prefix was fully read; this is the share the 80% charge rule uses), coverageNotes, files (the first 50 sitemap files and what happened to each one)
redirectedTo	The host the domain redirected to (www. stripped) when it differs from your input, or the domain its robots.txt points all its sitemaps at, else null
delegatedTo	The domain whose sitemaps were read when robots.txt lists sitemaps only on another domain (notion.so lists notion.com's), else null. reason and the first pitch reason name it
reason	Why a row is free, or what was excluded from an ok row's counts. Always set when coverage is partial
sinceLastRun	Only with compareWithPreviousRun on, on ok rows: URLs new, updated or removed since the previous run. See What changed since the last run

Every row also has input (exactly what you submitted), domain (what it was normalized to), analyzed and checkedAt.

Example (real run, 2026-09-26, trimmed)

{
  "input": "linear.app",
  "domain": "linear.app",
  "status": "ok",
  "analyzed": true,
  "lastmodReliable": true,
  "launchScore": 82,
  "changes": { "last7d": 21, "last30d": 74, "last90d": 165 },
  "bySection": {
    "changelog":    { "urls": 255, "dated": 162, "last7d": 1,  "last30d": 3,  "last90d": 9 },
    "integrations": { "urls": 325, "dated": 267, "last7d": 2,  "last30d": 19, "last90d": 53 },
    "customers":    { "urls": 43,  "dated": 28,  "last7d": 0,  "last30d": 2,  "last90d": 6 },
    "docs":         { "urls": 203, "dated": 160, "last7d": 17, "last30d": 44, "last90d": 77 },
    "careers":      { "urls": 31,  "dated": 30,  "last7d": 0,  "last30d": 4,  "last90d": 13 }
  },
  "latestChangeAt": "2026-09-25T20:22:07.000Z",
  "notableUrls": [
    { "url": "https://linear.app/changelog/2026-09-24-new-controls-for-linear-coding-agent", "section": "changelog", "lastmod": "2026-09-25T07:59:41.000Z" },
    { "url": "https://linear.app/integrations/sanity-content-agent", "section": "integrations", "lastmod": "2026-09-23T14:49:11.000Z" }
  ],
  "pitchReasons": [
    "3 changelog/release pages updated in the last 30 days (latest: /changelog/2026-09-24-new-controls-for-linear-coding-agent): shipping actively; reference a recent release in outreach",
    "19 integration pages updated in the last 30 days (e.g. /integrations/sanity-content-agent): expanding their ecosystem; partnership or integration pitch",
    "2 customer stories updated in the last 30 days (e.g. /customers/commure): investing in social proof and sales enablement",
    "44 docs pages updated in the last 30 days: active developer surface; devtools and docs-tooling pitch",
    "4 careers pages updated in the last 30 days: likely hiring"
  ],
  "reason": "Counts exclude 61 of 226 URLs dated in the last 90 days as machine restamps (1 same-timestamp batch in the last 90 days, largest 10 URLs at 2026-07-23T15:28:23.000Z (over 265s); 2 one-by-one sweeps in the last 90 days, largest 23 blog pages from 2026-07-23T15:02:07.000Z over 80 min).",
  "noise": { "verdict": "partly_filtered", "batchRestamps": [{ "at": "2026-07-23T15:28:23.000Z", "spanSeconds": 265, "urls": 10 }], "sweepRestamps": [{ "section": "blog", "at": "2026-07-23T15:02:07.000Z", "spanSeconds": 4822, "urls": 23 }, { "section": "docs", "at": "2026-09-01T08:26:08.000Z", "spanSeconds": 5906, "urls": 15 }], "recentExcludedShare": 0.27 },
  "sitemap": { "origin": "https://linear.app", "sitemapsFetched": 1, "urlsTotal": 1019, "urlsWithLastmod": 966, "coverage": "full" },
  "redirectedTo": null
}

A site whose recent dates are batch jobs returns a free row. On 26 September vercel.com's sitemap had 126 same-timestamp batches in the last 90 days, the largest 809 URLs on one millisecond:

{
  "domain": "vercel.com",
  "status": "lastmod_unreliable",
  "analyzed": false,
  "lastmodReliable": false,
  "changes": null,
  "reason": "5715 of 6051 URLs dated in the last 90 days (94%) don't count as trustworthy changes: 5493 are machine restamps (126 same-timestamp batches in the last 90 days, largest 809 URLs at 2026-08-18T06:10:12.000Z; section customers (86% of 118 pages on 2026-07-23); section careers (100% of 110 pages restamped within 7 days)) and 222 are listing pages or echoes of another change, leaving 336 trustworthy changes. When over 90% of a site's recent dates don't hold up, the rest can't be trusted either, so no counts are reported. Not charged."
}

A site that stamps its build date on every URL is also lastmod_unreliable, with noise.verdict: "build_date_stamp".

Same run, other domains:

Domain	Status	Why
notion.so	ok, score 75	248 pages changed in 30 days, read from notion.com (delegatedTo: "notion.com", named in the first pitch reason); 40 files read and 138 not read, nearly all of them language copies of files that were read, so coverageShare is 0.22 (files read of all files found) and coverageShareWithoutLanguageCopies, the share the charge rule uses, is 0.98
intercom.com	ok, score 45	233 pages changed in 30 days, 230 of them help-center articles; its 6-language help center (/help/de/articles/…, /help/pt-BR/articles/…) counts each article once (7,909 URLs are 3,220 pages)
retool.com	ok, score 48	45 pages changed in 30 days (blog and careers) after 89% of recent dates were excluded, including 42 of 43 customer stories restamped on one second
plausible.io	ok, score 15	5 blog posts in 30 days, 13 changes in 90 days after a 39-URL batch was excluded
techcrunch.com	partial_coverage, free	its sitemap index lists 2,061 files; 40 were read (2%), too few to describe the site
vercel.com	lastmod_unreliable, free	94% of recent dates are batch restamps (above)
nest.com	stale_sitemap, free	newest lastmod is 2024-08-28
zeit.co	redirected, free	redirects to vercel.com (redirectedTo: "vercel.com")
stripe.com, posthog.com	no_lastmod, free	their sitemaps have no dates

Very large sitemaps can hit the URL cap. Live results change daily: on 27 September figma.com returned partial_coverage (free), with 20 of its 38 sitemap URL-set files (53%) fully read before the default 50,000-URL cap. Raise maxUrlsPerDomain to read more.

What changed since the last run

Set compareWithPreviousRun: true and every charged row gets a sinceLastRun object: the sitemap URLs that are new, updated (same URL, different <lastmod>) or removed since the previous run for that domain. Lastmod dates can't tell a new page from an edited one; this can. Put the Actor on a weekly schedule with the option on and each run is a weekly diff of your competitors' sites: new changelog entries, new integration pages, customer stories taken down.

The first run for a domain saves a baseline (baseline: true, with urlsNow and coverage); the next run reports the changes.

Example output

"sinceLastRun": {
  "previousRunAt": "2026-09-22T06:00:00.000Z",
  "baseline": false,
  "comparable": true,
  "coverage": "full",
  "urlsBefore": 1204,
  "urlsNow": 1211,
  "newUrls": 9,
  "updatedUrls": 14,
  "removedUrls": 2,
  "newBySection": { "changelog": 2, "integrations": 5, "blog": 2 },
  "newUrlsSample": [
    { "url": "https://example.com/changelog/2026-09-27-workflow-automations", "section": "changelog", "lastmod": "2026-09-27T15:04:11.000Z" },
    { "url": "https://example.com/integrations/acme-crm", "section": "integrations", "lastmod": "2026-09-25T09:12:40.000Z" }
  ],
  "removedUrlsSample": ["https://example.com/customers/old-story", "https://example.com/pricing/legacy"],
  "updatedBulkRestamp": null,
  "note": null
}

• New and removed URLs are reported only when both runs read every sitemap file completely; otherwise only updated URLs. A run reads every file completely when no listed sitemap file failed, was missing (404), wasn't a sitemap, was disallowed by robots.txt, was cut at a size cap or corrupt, and no cap stopped the run. Otherwise coverage is partial, newUrls and removedUrls (and their samples and newBySection) are null, note names the run that fell short, and updatedUrls still counts URLs present in both runs with a changed lastmod. This keeps a sitemap file that fails once from showing up as pages removed and then re-added.
• A snapshot from a run that read every file is never replaced by one from a run that didn't: it stays the baseline for new and removed URLs until a complete run replaces it, and note says so. The incomplete run's lastmod values are copied into it for the URLs both runs list, so a page update is counted once: updatedUrls on a later run counts changes since the latest run that listed the URL, not since previousRunAt.
• URLs are matched without their scheme or a leading www. (http://www.example.com/a and https://example.com/a are the same page); path and query must match exactly. Samples show the URL as it is now. If the two runs share no URLs at all, comparable is false, no counts are given, and this run becomes the new baseline.
• newUrlsSample lists up to 20 new URLs, newest lastmod first; removedUrlsSample up to 20 removed ones. newBySection counts new URLs per section. updatedUrls counts every lastmod change as the sitemap states it, including machine restamps that changes filters out. When 5 or more of the updated URLs share one identical new lastmod (to the second; the same rule as the noise filter's same-timestamp batches), updatedBulkRestamp says so: { "urls": 313, "batches": 1, "largestBatch": { "urls": 313, "lastmod": "2026-09-29T21:44:05.000Z" } } means 313 of the updated URLs are likely one bulk restamp by the site's build (resend.com restamps about a third of its URLs on every deploy), not 313 page edits, and note says the same in words. Those URLs are still counted in updatedUrls; nothing is dropped. Date-only lastmods (midnight UTC) are not grouped, since several real edits on one day share that second. updatedBulkRestamp is null when there is no such group, and on baselines.
• Only charged (ok) rows are compared and saved; free rows have no sinceLastRun and never touch a saved snapshot. The option doesn't change pricing.
• Snapshots are kept in a named key-value store, launch-signals-state, in your own Apify account: one compressed JSON record per domain (about 20 KB for 1,000 URLs, about 0.7 MB for 50,000). Named stores are kept until you delete them; delete the store, or one domain's record, to start over. If the store can't be read or written, the row is still returned and note says what happened.

How the noise filter works

A sitemap's <lastmod> is whatever the site's build claims. The filter throws out dates that don't describe a real change:

1. Build-date stamp (whole domain unreliable). If 60% or more of the dated pages share one lastmod day, and that day falls in the last 90 days, the domain is lastmod_unreliable. The bar drops to 50% when that day is within 2 days of the check, or matches the sitemap file's own Last-Modified header. An old dominant day (a migration two years ago) is just history, so recent changes still count.
2. Stamped sitemap files. A single sitemap file with 10 or more dated URLs, 70% of them on one day in the last 90 days, was restamped on that day: its URLs dated that day, or the day before or after (a slow restamp wave crosses midnight UTC), are excluded. Its URLs dated on other days carry their own lastmods and still count (railway.com's changelog sitemap: 295 of 309 URLs restamped on one day, while the weekly releases after it kept their own dates), and so do the site's other files.
3. Same-timestamp batches. 5 or more URLs with the very same lastmod second were written by one job, not edited one by one. So is a run of 10 or more URLs each stamped within 60 seconds of the previous one. Both are excluded and listed in noise.batchRestamps. A date-only lastmod (2026-10-02, no time of day) is one day, not one second, so several real posts on one day share it. There, 8 or more pages on one date count as a batch, and 5 to 7 pages on one date count as one change: such a group is as often old posts restamped together as new ones, so it shows the day without inflating the counts. That change keeps its section only when all the group's pages share one (five blog posts on one day are one blog change); a mixed group counts toward the totals only, never toward a pitch line. A sitemap file of 10 or more dated URLs that writes its dates as midnight timestamps (80% or more of its lastmods at exactly 00:00:00.000 UTC) counts as date-only too; elsewhere, a timed lastmod that happens to read 00:00:00 keeps the 5-URL rule.
4. Bulk-update hours. Any clock hour in which at least max(20, 3% of dated pages) URLs were stamped, capped at 100, is treated as a mass restamp and listed in noise.bulkUpdates.
   • One-by-one sweeps. Within one section, any 60-minute window holding 8 or more stamped URLs is a sweep: a script or an editor restamping pages one after another, 20 to 120 seconds apart, which stays under both the 60-second chain and the bulk-hour bar (vanta.com: 36 /products/ pages between 23:21 and 23:53). Swept URLs are excluded and listed in noise.sweepRestamps. A real launch that touches 8+ pages of one section within an hour is excluded too; the filter prefers missing a launch to charging for a restamp.
5. Restamped sections. A section with 5 or more dated pages, 80% of them on one day in the last 90 days, was restamped as a block (a template change or CMS migration): its pages dated that day, the day before or the day after are excluded, and its pages dated on other days still count (noise.restampedSections). A section with 20 or more dated pages, 80% of them "changed" in the last 7 days, regenerates continuously (careers pages from an ATS feed) and is excluded too (noise.rollingSections). The same 7-day test applied to the whole site marks the domain lastmod_unreliable.
6. Listing pages and echoes. A post's category, tag and author pages, the home page, /search, pagination and a bare section index (/blog) are regenerated whenever a post is published, so they share its lastmod second. 2 to 4 URLs on one second count as one change, and listing pages never count on their own. Two-segment directory indexes such as /marketplace/apps/, /business/customer-stories/, /products/release-notes/ or /marketplace/partners/ are listing pages too, and a pitch line's example is never a page that has child pages in the sitemap when a leaf page of that section changed; a bare /pricing is not a listing page, and a pricing change that shares a second with a post is the one counted (noise reports echoesCollapsed and listingPagesIgnored). With date-only lastmods, 2 to 4 pages on one date are merged only when a listing page is among them, and 5 to 7 are always merged into one change. Author pages count as listing pages in other languages too (/autor/, /auteur/, /autore/). So do changelog and blog taxonomy pages: /changelog/label/copilot/, /changelog/type/new-releases/, /changelog/month/09-2026/, /blog/series/launch-week/.
7. Too little left. If more than 90% of the last 90 days' dated pages were excluded, or fewer than 5 trustworthy changes remain, the domain is lastmod_unreliable. With nothing excluded, fewer than 5 changes in 90 days is too_few_changes, and none at all is stale_sitemap. All of these are free.

When more than half of an ok domain's recent dates were excluded, reason says so, so you can read the remaining counts with that in mind. The filter errs toward under-counting: a real launch that touched 200 pages in one deploy is excluded as a batch.

The same page in several languages counts once, wherever the language sits in the path: /customers/acme, /fr/customers/acme and /en-gb/customers/acme are one page, and so are intercom's /help/de/articles/167-intercom-fur-besucher… and /help/en/articles/167-install-intercom… (translated slugs are matched on their numeric id, and on the sitemap's hreflang alternates when it lists them; an unprefixed /help/articles/167-… joins them too, when it is the only unprefixed page with that id, so /blog/2026-ai-recap and /blog/2026-pricing-update stay two pages). A leading date is not an id: /releases/2026-09-15 and its copies /en-gb/releases/2026-09-15, /fr/releases/2026-09-15 are one page, separate from /releases/2026-09-09. A merged page is cited in notableUrls and pitchReasons by its URL without a language segment (or the /en/ one), so notable URLs are not padded with translations. A language code after the first path segment is treated as one only when the site uses 2 or more codes in that spot, so /solutions/it/service-desk stays its own page. Tracking parameters (utm_*, gclid, fbclid) and path case are ignored too.

Statuses and pricing

$0.03 per analyzed domain ($30 per 1,000), charged only when status is ok. That takes all of:

• the domain's own sitemap downloaded cleanly (under 25% of its sitemap URL-set files failed, none with a high-signal name, such as a changelog, pricing, blog, post or page sitemap; a large sitemap file gets up to 30 seconds to download). A sitemap file whose own host, or redirect target, does not exist in DNS (confirmed by a second lookup) is a permanent dead link and is ignored like a 404, as long as its name is not high-signal and dead files are no more than 25% of the URL-set files and at least 80% of its sitemap URL-set files were fully read (coverageShareWithoutLanguageCopies ≥ 0.8; sitemap-index files and confirmed language copies don't count, a file cut at the byte cap is not fully read, and a sitemap index cut at the byte cap counts as at least one file not read);
• its lastmod dates passed the noise filter, leaving at least 5 trustworthy changes in the last 90 days;
• at least 1 change in the last 30 days, and at least one pitch reason or high-signal change (changelog, pricing, integrations, product, customers) in that time. A pricing edit older than 30 days is reported as context but never makes a row billable, and an edit to a release page whose URL date is more than 30 days old (such as /changelog/2023-01-05-dark-mode) is not a new release, so it doesn't count either.

An ok row with coverage: "partial" (some files not read, or one minor file failed) is still charged when those bounds hold; its reason says the counts are lower bounds. Every other row is free, and these statuses never mean "no activity":

status	Meaning
invalid_input	Couldn't parse a domain from the input
duplicate	Same site as an earlier input (duplicateOf is set). https://www.Linear.app/pricing?x=1, www.www.linear.app and linear.app are one domain, and two inputs whose sitemaps are read from the same host (notion.so and notion.com) are one site: only the first is charged
redirected	The domain redirects to a different domain (zeit.co to vercel.com), or its robots.txt lists sitemaps only on an unrelated domain (one sharing no name with yours, such as an acquirer or a sister property); redirectedTo says where. Add that domain as its own input if it's the company you want. The same name on another TLD (notion.so and notion.com) is read, with delegatedTo set. A subdomain input (acme.wordpress.com, blog.acme.com) covers only its own host: if its robots.txt redirects, or its sitemaps point, to any other host (the platform's own site on wordpress.com, substack.com, medium.com and similar), the row is redirected and free. With no sitemap found, a homepage that redirects to a different domain is redirected too, but only when that domain answers; if it can't be fetched, the row stays no_sitemap and reason names where the homepage pointed
not_public	IP address, localhost, a private-network name, or a domain that resolves only to private IPs
nxdomain	Domain doesn't exist in DNS
unreachable	DNS or the site (robots.txt on https://domain, https://www.domain and http://domain) failed twice, or robots.txt answered HTTP 429 (rate-limited) twice without an HTML challenge page (a 429 challenge page is blocked_by_robots). A 429 robots.txt means robots.txt is unavailable (RFC 9309), never "allow everything": nothing else is fetched, the remaining origins are not asked, and reason says it was rate-limited. Worth retrying later
blocked_by_robots	robots.txt disallows our crawler from reading the sitemap. Also used when no origin (https://domain, https://www.domain, http://domain) gave usable rules and at least one answered with an HTML page (a bot challenge, as on linkedin.com's apex, or a challenge served with a 4xx status other than 404/410, such as hashicorp.com's 429 "Vercel Security Checkpoint" or a 403 WAF page, including one served as text/html whose body is empty or cut off) or a non-empty, non-text/plain body with no robots.txt lines; or when every sitemap file tried sits on a host whose robots.txt is such a page. That is never read as "allow everything": nothing else is fetched there, not even the usual sitemap paths
no_sitemap	No sitemap in robots.txt, or the listed ones return 404, and /sitemap.xml, /sitemap_index.xml, /sitemap-index.xml and /sitemap.xml.gz are missing
sitemap_fetch_failed	A sitemap exists but couldn't be downloaded after a retry (timeout, 5xx, 403, an unreachable or rate-limited (429) robots.txt on its host…), or it returned HTML or an unsupported format such as RSS. Also used when 25% or more of a site's sitemap URL-set files failed (a cut-off or corrupt .gz file counts as failed; sitemap-index files are not counted), or any failed file has a high-signal name (changelog, pricing, blog, or a post or page sitemap such as WordPress post-sitemap2.xml, Ghost sitemap-posts.xml or Shopify sitemap_pages_1.xml…): counts from the rest would be incomplete. Worth retrying, except when reason says the files are on (or redirect to) a host that does not exist in DNS, or that their host's robots.txt redirects to one: that is permanent. A single such dead file next to readable ones is ignored like a 404 and noted in coverageNotes, unless its name is high-signal or dead files are over 25% of the URL-set files. A file whose host's robots.txt redirects to a non-existent host counts as failed (we may not fetch it), with a permanent reason
partial_coverage	Fewer than 80% of the site's sitemap URL-set files were read before a cap (techcrunch.com lists 2,061). coverageShareWithoutLanguageCopies says how much (coverageShare counts the skipped language copies too), and reason names the cap. Raise maxSitemapsPerDomain (up to 100) or maxUrlsPerDomain (up to 200,000) to read more; reason says so when the cap is already at its maximum. The 25 MB per-file and 60 MB per-domain size budgets are fixed
sitemap_empty	The sitemap lists no URLs
no_lastmod	The sitemap has no (or fewer than 5) <lastmod> dates. Also used, instead of partial_coverage, when a cap already at its maximum (or a fixed size or time budget) stopped the run and none of the 1,000 or more URLs read carries a date (n8n.io: 200,000 URLs read, none dated). Below the maximum the row stays partial_coverage with advice to raise the cap
lastmod_unreliable	Build-date stamping, or recent dates that are mostly batch restamps (see above). lastmodReliable: false
no_notable_changes	Trustworthy changes exist, but none in the last 30 days, or none that add up to a pitch (all in other, or a few docs pages). lastmodReliable: true; reason lists the 90-day counts by section
too_few_changes	Fewer than 5 trustworthy changes in the last 90 days; reason names the newest one
stale_sitemap	No lastmod in the last 90 days; reason gives the newest date. Usually a stale or abandoned sitemap. If only part of the sitemap was read, reason says so
skipped	Your run's max charge was reached; no result was returned for the domain (not charged). A repeat of an input that was skipped is skipped too, and its reason names the input it repeats
error	Unexpected error

Free rows have changes: null and launchScore: null, never zeros, and a charged row never reports zero recent changes. Set a max charge per run and you're never charged more than that: once it's reached, every remaining input still gets a free skipped row.

Launch score

Each section contributes points per page changed in the last 30 days, up to a cap, and pages changed in the last 7 days add half as much again:

Section	Points per page	Cap (pages)
pricing	25	2
changelog	6	10
integrations, product	4	10
customers	3	5
blog	2	10
careers	1	10
docs	0.5	40
other	0.2	50

launchScore = round(100 × (1 − e^(−points / 60))). Caps stop one noisy section (5,000 docs pages) from maxing the score.

Sections come from the URL: the first two path segments after any language segments (/changelog, /releases, /whats-new, /product-updates, /launch-week → changelog; /pricing, and /plans or /prices as the last segment → pricing (a bare /pricing, /plans or /prices is the pricing page itself and counts as a pricing change, while a bare /blog or /changelog is a listing page that echoes its posts); /integrations, /apps, /marketplace, /plugins, /connectors, /extensions → integrations; /product, /products, /features/<page>, /platform, /solutions, /use-cases → product; /customers, /case-studies → customers; /blog, /news, /press → blog; /docs, /help, /guides, /api, and Zendesk help-center pages (/hc/<locale>/articles/…) → docs, with a customer-support pitch line instead of a developer one when most recent docs pages are help-center content; /careers, /jobs, /join-us, /join/team → careers), or, for a URL listed in a sitemap that was read, its subdomain (docs., blog., changelog., careers.…). To keep pitch lines honest, these are not high-signal: singular /feature/<story> (a publisher's article, as on nasa.gov), deeper /features/2026/09/<story> archives, bare /join (a signup page), /updates/<anything>, utility pages such as /apps/login, /plans/<recipe>, and /products/<sku> from a Shopify-style sitemap_products_N.xml catalog. A URL listed in a WordPress or Ghost post sitemap (post-sitemap*.xml, wp-sitemap-posts-post-*.xml, sitemap-posts.xml) is a blog post whatever its folder (so /plugins/how-to-… posts aren't integrations), unless its path says changelog. Press-release date archives (/releases/2026/09/<statement>, /news-releases/<year>/…) are blog/news, not changelog. A section word right after a content hub (/resources/integrations/webinar/…, /events/customers/…, /library/…) is a topic tag, so such pages are other. Reviews of other companies' products are blog posts too: when a domain is charged, the pages it cites are opened, and if at least two product pages from one sitemap file have titles like "Semrush One Review: …" or "AffiliateWP Review (2026): …", naming different products, and they are most of that file's product pages whose titles could be read, every product page in that file counts as blog (a publisher's /solutions/<vendor>/ reviews are not its own launches). Those files are listed in noise.reviewSitemaps.

Input

• domains (required): domains or URLs. Each is reduced to its domain: https://www.retool.com/pricing, RETOOL.COM and retool.com. all become retool.com.
• maxConcurrency (optional, default 5, max 10): domains analyzed in parallel. Sitemap files for one domain are fetched one at a time.
• maxSitemapsPerDomain (optional, default 40, max 100) and maxUrlsPerDomain (optional, default 50,000, max 200,000): per-domain caps.
• compareWithPreviousRun (optional, default false): adds sinceLastRun with the sitemap URLs that are new, updated or removed since the previous run. See What changed since the last run.

Crawling etiquette

The crawler identifies itself as SiftsmithBot/1.0 (+https://siftsmith.com) and reads robots.txt first. It follows the rules for SiftsmithBot, or * if there are none, for every sitemap file it fetches, including sitemaps hosted on another domain. A robots.txt that is an HTML page, or a non-empty body served as something other than text/plain with no robots.txt lines (such as JSON), counts as unusable: that host is not crawled, and the next origin is tried (its rules apply if it serves a real robots.txt). The same goes for an HTML page served with any 4xx status other than 404 and 410 (a 429 or 403 bot challenge; a text/html content-type is enough, even when the body is empty or arrives too slowly to finish), and for a 429 with any body, which RFC 9309 and Google treat like a 5xx: robots.txt unavailable. A 429 on any origin, directly or after a redirect, also stops the walk: the remaining origins are not asked. An HTML bot-challenge page or a 429 is never read as allow-everything, on the input's own origins or on any host a sitemap lives on. A 404 or 410, whatever its body (most sites serve an HTML "not found" page there), and any other 4xx without an HTML page (such as S3's XML 403 for a missing file) mean there is no robots.txt, so everything is allowed (RFC 9309). An empty robots.txt, or a text/plain one, is read as-is (no rules means allowed). Every redirect a sitemap request follows is checked too: the target's robots.txt must allow it before it is requested, or the file is skipped. It fetches robots.txt and sitemap files; the only page it requests is the homepage, when no sitemap is found, to check whether the domain now redirects elsewhere. Sitemap files and URLs on other domains (a sister country site, a CDN) are ignored and noted in coverageNotes, unless robots.txt lists sitemaps only on that other domain (notion.so lists notion.com's). Each sitemap file gets 10 seconds, or up to 30 seconds while the domain's 2-minute budget lasts (a 5 MB WordPress post sitemap takes 12–15 s), and is retried once; robots.txt times out after 8 seconds. Per domain it stops starting new sitemap files at 40 sitemap files, 50,000 URLs, 60 MB of sitemap data or 2 minutes, and marks coverage partial (below 80% of files read, the row is a free partial_coverage). When the apex robots.txt lists no sitemap, it asks https://www.<domain>/robots.txt once and reads the sitemaps listed there, if they are all on the same domain (shopify.com lists its sitemaps only on www). With no sitemap listed anywhere, it tries the usual paths and stops at the first one that works. A <urlset> whose entries are sitemap files on the same host (.xml or .xml.gz paths, as on holded.com) is read as a sitemap index, within the same caps. In a large sitemap index, default-language and recently updated child sitemaps are fetched first; a regional copy (/au/sitemap_blog.xml when /sitemap_blog.xml is listed in the same index) goes last.

Limitations

• Lastmod can't tell a new page from an edited one. changes counts pages whose lastmod falls in the window; notableUrls are recently changed URLs, not necessarily new ones.
• Counts are only as honest as the site's lastmod. The filter catches build-date stamps, same-timestamp batches, restamped sections and rolling regeneration. A site that restamps 2 to 4 pages at a time, minutes apart, or 2 to 4 pages per date on a date-only sitemap, can still look busier than it is.
• Many big sites publish no lastmod at all (stripe.com and posthog.com in our testing). Those come back as free no_lastmod rows.
• Only sitemaps listed in the domain's robots.txt (or its www host's, when the apex lists none, or found at the standard fallback paths) are read. A blog or docs subdomain whose sitemap isn't linked from there (blog.hubspot.com lists its sitemap only in its own robots.txt) isn't counted. Add blog.example.com as its own input to measure it.
• Only what's in the sitemap is seen. Pages left out of it (common for pricing pages and app-store style integration directories) are invisible.
• Coverage is partial when a cap is hit or a minor sitemap file fails. Counts are then lower bounds, and reason says so. Below 80% of files fully read, the row is free (partial_coverage); very large news sites (techcrunch.com) land there at the default cap, and so do sites whose sitemap files exceed the 25 MB per-file or 60 MB per-domain budget.
• <lastmod> values more than 10 minutes in the future, and impossible dates such as 2026-02-30, are ignored; timestamps without a timezone are read as UTC.
• Bot protection (HTTP 403) and rate limits (HTTP 429) on some large media and retail sites (wired.com, theverge.com, bombas.com in our testing) give free sitemap_fetch_failed rows. When robots.txt itself is behind the challenge or the rate limit (hashicorp.com), the row is a free blocked_by_robots or unreachable.
• Sections are guessed from URL paths; unusual site structures land in other. A media site that files articles under /features/<story> still reads as product pages.
• RSS/Atom feeds listed as sitemaps aren't parsed (sitemap_fetch_failed if that's the only sitemap).

Related tools

• Company Jobs: one row per open job from a company's public job board (8 ATSes), $0.001 per job row. Pair a launch with the roles the company is hiring for.
• Hiring Signals: a company's open jobs, recent posting velocity and hiring score from its public job board.
• Tech Stack Detector: a website's technologies plus its email provider and DNS host.

How-to guides

• Find recently updated pages on a website: read sitemap lastmod dates, spot fake build-date stamps, and check a list of companies with this Actor.

About Siftsmith

Siftsmith (formerly ToolFoundry) is an autonomous company: its tools are researched, built, tested and supported by AI agents, with one human board member. Support replies come from Siftsmith, never a pretend human.

Support

Open an issue on the Actor's Issues tab or email hello@siftsmith.com. Issues are read daily and fixed promptly.