Versions follow MAJOR.MINOR.PATCH (src/version.py); Apify shows MAJOR.MINOR from .actor/actor.json.
Every run logs its version and records it in the RUN_STATS key-value record.
- Input field descriptions rewritten for AI agents (Apify's MCP server shows agents only the description, not the
form): each now states its default, its allowed range, the value of each option and how it combines with other
fields. No change to field names, types, defaults or behaviour.
- Links mode: links to different sites are checked in turns, at most 2 at a time per site, and a worker never waits
on a busy site while another site's link is ready, so one slow or rate-limiting site no longer holds up every
check. A site that answers HTTP 429 (after the retry) is left alone for the rest of the run: its other links
aren't requested, aren't charged, and are listed in
NOT_CHECKED with outcome rate-limited; the site is named
in linkHostsThrottled in RUN_STATS and in the status message. (The same scheduling SEO Audit Crawler uses,
now shared in common/mms_linkcheck.) The URLs and sitemap modes are unchanged.
- The URL checking and link parsing code moved to the shared
common/mms_linkcheck (SEO Audit Crawler uses it
too). No behaviour change.
First release.
- Checks URLs and reports the status code, every redirect hop (address, status, target, time), final URL,
canonical URL (HTML
<link rel=canonical> or the HTTP Link header), response time, content type and size,
with a clear outcome: ok, broken, refused, robots-blocked, not-public, invalid-url, dns-error or error.
- Three modes: a list of URLs; every URL in a site's sitemaps (the same discovery as Sitemap URL Extractor:
robots.txt
Sitemap: lines, /sitemap.xml, indexes, .xml.gz); or every link on given pages, following same-site
pages up to a crawl depth and page cap, with the pages each link was found on.
- HEAD first, GET when the HEAD answer is an error or the page's canonical URL is wanted; HEAD-only and GET-only
options.
- "Only report URLs that changed since the last run", remembered per check in a key-value store in the user's
account; only URLs actually written to the results update the memory.
- robots.txt respected on every hop (a disallowed URL is reported, not requested); public addresses on ports 80
and 443 only; at most 2 requests at a time per site; Crawl-delay honoured.
- URLs that are never requested (disallowed by robots.txt, a private address or non-standard port, an invalid
address, a site asking for more than 30 s between requests) aren't results and aren't charged: they're listed
in the
NOT_CHECKED key-value record, counted in RUN_STATS and logged. They don't use up the max charge
either. A URL that was requested and then redirected into one of those is a result, with its redirect chain.
- A HEAD request that fails outright (dropped connection, timeout) is retried with GET before the URL is reported
as an error. HEAD answers carrying
Content-Encoding: gzip with no body (as CDNs send them) are read correctly.
- Charged per URL in the results, through Apify's standard
apify-default-dataset-item event. "Max results per
run" and the maximum cost per run are honoured before anything is requested.
- Failure isolation: a failing URL, page or sitemap only affects itself.
