# Changelog of Bulk URL & Broken Link Checker (`humble-echidna/url-checker`) Actor

- **URL**: https://apify.com/humble-echidna/url-checker/changelog.md
- **Full Actor documentation**: https://apify.com/humble-echidna/url-checker.md

## Changelog

Versions follow MAJOR.MINOR.PATCH (`src/version.py`); Apify shows MAJOR.MINOR from `.actor/actor.json`.
Every run logs its version and records it in the `RUN_STATS` key-value record.

### 1.0.3 (2026-09-25)

- Input field descriptions rewritten for AI agents (Apify's MCP server shows agents only the description, not the
  form): each now states its default, its allowed range, the value of each option and how it combines with other
  fields. No change to field names, types, defaults or behaviour.

### 1.0.2 (2026-09-24)

- Links mode: links to different sites are checked in turns, at most 2 at a time per site, and a worker never waits
  on a busy site while another site's link is ready, so one slow or rate-limiting site no longer holds up every
  check. A site that answers HTTP 429 (after the retry) is left alone for the rest of the run: its other links
  aren't requested, aren't charged, and are listed in `NOT_CHECKED` with outcome `rate-limited`; the site is named
  in `linkHostsThrottled` in `RUN_STATS` and in the status message. (The same scheduling SEO Audit Crawler uses,
  now shared in `common/mms_linkcheck`.) The URLs and sitemap modes are unchanged.

### 1.0.1 (2026-09-24)

- The URL checking and link parsing code moved to the shared `common/mms_linkcheck` (SEO Audit Crawler uses it
  too). No behaviour change.

### 1.0.0 (2026-09-24)

First release.

- Checks URLs and reports the status code, every redirect hop (address, status, target, time), final URL,
  canonical URL (HTML `<link rel=canonical>` or the HTTP `Link` header), response time, content type and size,
  with a clear outcome: ok, broken, refused, robots-blocked, not-public, invalid-url, dns-error or error.
- Three modes: a list of URLs; every URL in a site's sitemaps (the same discovery as Sitemap URL Extractor:
  robots.txt `Sitemap:` lines, /sitemap.xml, indexes, .xml.gz); or every link on given pages, following same-site
  pages up to a crawl depth and page cap, with the pages each link was found on.
- HEAD first, GET when the HEAD answer is an error or the page's canonical URL is wanted; HEAD-only and GET-only
  options.
- "Only report URLs that changed since the last run", remembered per check in a key-value store in the user's
  account; only URLs actually written to the results update the memory.
- robots.txt respected on every hop (a disallowed URL is reported, not requested); public addresses on ports 80
  and 443 only; at most 2 requests at a time per site; Crawl-delay honoured.
- URLs that are never requested (disallowed by robots.txt, a private address or non-standard port, an invalid
  address, a site asking for more than 30 s between requests) aren't results and aren't charged: they're listed
  in the `NOT_CHECKED` key-value record, counted in `RUN_STATS` and logged. They don't use up the max charge
  either. A URL that was requested and then redirected into one of those is a result, with its redirect chain.
- A HEAD request that fails outright (dropped connection, timeout) is retried with GET before the URL is reported
  as an error. HEAD answers carrying `Content-Encoding: gzip` with no body (as CDNs send them) are read correctly.
- Charged per URL in the results, through Apify's standard `apify-default-dataset-item` event. "Max results per
  run" and the maximum cost per run are honoured before anything is requested.
- Failure isolation: a failing URL, page or sitemap only affects itself.