# Changelog of Website SEO Audit: On-Page SEO Checker & Site Crawler (`fecklesspleb/website-seo-audit`) Actor

- **URL**: https://apify.com/fecklesspleb/website-seo-audit/changelog.md
- **Full Actor documentation**: https://apify.com/fecklesspleb/website-seo-audit.md

## Changelog

Version stays "1.0" in .actor/actor.json: Apify accepts only MAJOR.MINOR there and assigns the patch (build) number at the next build, which is the patch bump for this release.

### Unreleased

- The sample site is now `https://www.python.org/` (was `https://example.com`): `SAMPLE_URL` in `my_actor/main.py`, the `startUrls` input schema `default` and `prefill`, and a new actor.json `exampleRunInput` all hold this one value, so `{}`, the prefill and the example input are all sample runs (15 s deadline, free warning row if the site cannot be read). Only the exact list `["https://www.python.org/"]` is a sample run; other spellings (for example without the trailing slash) are normal runs.
- A sample run now audits at most 2 pages (`SAMPLE_MAX_PAGES`), even when `maxPagesPerSite` is higher (its default stays 10, and 10 is also the code fallback when it is missing or 0), so `{}`, the prefill and the example cost at most 2 x $0.003 + start fee. `maxPagesPerSite` prefill and `exampleRunInput` are 2. This also caps a run whose start URL list is exactly the sample URL with a larger `maxPagesPerSite`. README Default column and `maxPagesPerSite` row updated. New test `tests/test_sample_1102v.py`.

### 2026-11-02 (actor.json version unchanged: 1.0): private-address (SSRF) guard

- On the Apify platform, every request (start pages, crawled pages, each redirect hop, and robots.txt) is refused when the host
  resolves to a private, loopback, link-local, carrier-grade NAT, reserved, multicast or unspecified address
  (including IPv4-mapped IPv6), or when any of the host's addresses is one. Same rules as Bulk HTTP Request.
- Such a page is output free like other pages with no response: `statusCode: null`, issue `http-error`,
  `error: "Blocked: private, loopback and link-local addresses are not allowed"`. Nothing is crawled
  from it.
- A host the guard cannot resolve fails as a DNS error, as before (`error` now reads
  "DnsLookupFailed: DNS lookup failed for <host>" instead of "ConnectError: ..."); the row stays free.
- A robots.txt that redirects to a private address counts as unavailable (no robots.txt rules), like a robots.txt
  that cannot be fetched.
- Local runs (APIFY_IS_AT_HOME not set) are unchanged.
- Files: `my_actor/common.py` (`ip_blocked`, `host_problem`, `check_public`, `guard_request` httpx request hook,
  `guard_site`, `Blocked`, `DnsLookupFailed`; `Http._robots_for` checks the site before fetching robots.txt),
  `my_actor/main.py` (hook on the httpx client, `Blocked` handled), `tests/test_ssrf_guard_1102t.py` (new),
  README Limits (one line).

### Unreleased (next 1.0.x build): error pages are free (round 1030b)

- **Charges (lower):** pages that return HTTP 400 or higher, or give no response (DNS or connection errors, timeouts), are still saved with the `http-error` issue but are no longer charged. Before, they were charged like any other page. Pages blocked by robots.txt are still not fetched, not saved and not charged.
- Unchanged: the rows output, their order and content, **Only pages with issues**, and **Max pages in dataset** (free pages still count toward it). Tested by comparing the rows of the previous code with the new code on the same inputs.
- Under a **Maximum cost per run**, the crawl stops at the same point as before (error pages still count when the Actor decides to stop crawling), so the rows are the same and only the charge is lower.

### Unreleased (next 1.0.x build): index pages counted once, clean title length, crawl stops at the limit (round 1030)

- **BREAKING (fewer rows and charges):** a directory and its index file (`/` and `/index.html`, also `index.htm`, `index.shtml`, `index.php`, `index.asp(x)`, and `default.htm(l)`, `default.shtml`, `default.php`, `default.asp(x)`) are one page: audited and charged once, and no longer reported as duplicate titles of each other. The canonical check treats them as the same URL too.
- **BREAKING (output values):** titles are whitespace-normalized (runs of spaces, tabs and line breaks become one space, ends trimmed) before `title` and `titleLength` are set, so `title-too-long` / `title-too-short` follow the visible title.
- **BREAKING (fewer requests):** the crawl stops when **Max results** or your maximum cost per run is reached, instead of fetching pages that can no longer be saved. RUN_SUMMARY `truncated` then says "stopped by the result or cost limit". This also fixes a crash (KeyError 'score') when output stopped part way through a site.
- **BREAKING (output values):** `checkedAt` is now the time each page was fetched, not the run start time.
- **BREAKING (input editor):** `startUrls` uses the plain URL list editor (`stringList`); the default and prefill are now `["https://example.com"]`. Plain strings and `{"url": ...}` objects both work.
- **BREAKING (fewer requests):** redirects are followed one hop at a time (at most 10) and every target is checked against its own robots.txt; a page that redirects to a disallowed page is not fetched and not charged.
- Empty input and the prefilled sample: 15 second limit; if the sample site is down, errors or is slow, the run succeeds and the row is free with a `warning` field (or one free warning row).