# Changelog of SiteVerdict — AI Search Readiness Checker (`siteverdict/siteverdict-ai-search-readiness`) Actor

- **URL**: https://apify.com/siteverdict/siteverdict-ai-search-readiness/changelog.md
- **Full Actor documentation**: https://apify.com/siteverdict/siteverdict-ai-search-readiness.md

## Changelog

### 2.0.6 (2026-10-16) — Private-network (SSRF) protection

- URL privacy: the dataset, reports, `OUTPUT`, monitoring snapshots, alerts and the run log show URLs without query strings, fragments or credentials (`?[redacted]`); each row has a `urlHash` (first 16 hex characters of the SHA-256 of the checked URL) to match rows to your input. Same-page links (`#anchor`, empty or same-query links) are left out of the llms.txt draft and link lists, so the query string of the URL you entered is not repeated there.

- Every start URL is checked before anything is loaded (IP literals in every encoding, internal names, DNS answer); a private, loopback, link-local, cloud-metadata or internal target gets an `error` row "refused: private-network protection; not charged" and is never contacted or charged.

- All plain HTTP fetches (pages, robots.txt, sitemaps, llms.txt, access-log download, IP-range lists) use a DNS-pinned transport that re-checks every redirect hop; the same applies to Chromium (rendering, screenshots, PDF logos), which now runs through a local checking proxy (shared SiteVerdict guard, `src/ssrf-guard.ts` + `src/netguard.ts`).

- `webhookUrl`, `slackWebhookUrl`, `reportLogoUrl` and `accessLogUrl` must resolve to public hosts; otherwise they are ignored with a warning. Webhooks never follow redirects.

- OUTPUT gains `privateNetworkProtection` (refused targets, browser requests blocked). No change to scoring or pricing.

### Unreleased (prepared 2026-10-16, NOT pushed) — URL privacy and reporting fixes

- Dataset `url` / `inputUrl`, per-page `url`, `OUTPUT.results[].url`, reports, monitoring snapshots, alerts and log lines no longer contain the query string, fragment or credentials of the URL you entered: they become `?[redacted]` / `#[redacted]`, paths are capped at 120 characters. New `urlHash` field (first 16 hex of SHA-256 of the full checked URL) as a join key. Monitoring baselines are still keyed by the full URL, so existing baselines keep matching.
- Final URLs, request evidence and error texts are shown without query strings; the suggested `og:url` and canonical snippets use the page address without its query string.
- `OUTPUT.privateNetworkProtection.browserRequestsBlocked` is now filled in (it was read after the browser had closed and was always empty).
- No change to scoring, inputs or pricing.

### Memory default (15 Oct 2026)

- 15 Oct 2026 (build after this entry): default run memory is now 2048 MB via `defaultMemoryMbytes` (was 4096 MB (no key set); peak 260 MB). Measured on the example input: same output, see the public-15 remediation log. You can still choose more memory per run. No change to inputs, outputs or pricing.

### 2.0.0 — 2026-10-11

Flagship upgrade. Benchmarked on 25 public sites; see BENCHMARK.md.

**Removed (compliance)**

- The opt-in CDN probe that sent other vendors' AI user-agent strings is gone. We never impersonate crawlers.

**AI crawlers**

- 35 agents, each re-verified against vendor documentation on 11 Oct 2026. New: Google-Agent, Google-GeminiNotebook, Meta-WebIndexer, MistralAI-Index/User/Training, Amzn-User, YouBot, OAI-AdsBot, GoogleOther, Google-CloudVertexBot and facebookexternalhit.
- Every agent now carries a role (search / user / training / other), a status (official / unofficial / legacy) and per-row confidence.

**CDN / WAF AI blocking (new honest model)**

- Verdict levels: `confirmed` / `likely` / `possible` / `unlikely`, built from evidence:
  - challenges, 401/403/429 and 402 served to our honest client
  - Cloudflare AI Crawl Control defaults
  - Cloudflare managed robots.txt
- Access-log verification (new): combined-log or JSON-lines files (e.g. Cloudflare Logpush). Each request's source IP is checked against the vendors' official IP-range lists and the result shows the status codes real AI bots received.
- Bot-challenge detection now also catches Akamai bot-failover pages served with HTTP 200.

**AI usage preferences (new)**

- Reads Cloudflare Content-Signal (per group), IETF aipref Content-Usage lines and W3C TDMRep (header, meta tag and /.well-known/tdmrep.json).

**Raw vs JavaScript-rendered (rewritten)**

- Reports the share of rendered text present in the raw HTML, plus structured data and headings that exist only after JavaScript.
- Flags title, meta, canonical, noindex and nosnippet changes made by JavaScript.
- Optional screenshot evidence.

**Entity and structured data**

- Checks the @id graph, WebSite→publisher links, dangling references, knowledge-graph sameAs and NAP phone consistency.
- More required-property checks: Event, Recipe, JobPosting, Offer, Person.
- A bare `{"@id"}` reference node is no longer reported as missing @type.

**Citability (new sub-score)**

- Heuristic checks: answer-first opening paragraph, fact density, front-loading, lists and tables, outbound citations, sentence length.

**Multi-page audit (new)**

- Up to 50 pages per site, sampled across site sections and skipping boilerplate pages.
- Robots rules are evaluated per path, with a site-level rollup and the most common issues.

**llms.txt**

- New spec-valid generator built from audited pages.
- Link checker that tells broken links apart from links refused to our client.

**Monitoring and alerts (new)**

- Snapshots in a named store, severity-ranked diff and only-changed output.
- Slack and webhook alerts (https only, secret inputs).

**Reports**

- White-label HTML and PDF (prepared-by, logo, accent colour).
- Evidence: a request log (HAR-lite) with timestamps, and the screenshot.
- Agent-ready OUTPUT summary.

**AI fix (new, optional, user's own key)**

- Anthropic (Claude, with server-side refusal fallback) or OpenAI.
- Drafts llms.txt, a meta description, an answer-first paragraph and FAQ + FAQPage JSON-LD.
- FAQ answers must quote a sentence from the page or they are dropped.

**Scoring**

- Five categories: Access 30, Extractability 20, Structure 20, Trust 15, Citability 15.
- Eligibility caps: unreachable ≤20, noindex ≤35, major AI search blocked or CDN block confirmed ≤45.
- Blockers are now limited to major AI agents.
- All meta tags are read case-insensitively; for example `<meta name="Description">` was missed before.

**New pay-per-event events**

- `page-audited`, `render-comparison`, `pdf-report`, `ai-fix-generated`, `site-rechecked-unchanged`, `change-alert-sent`, `access-log-analysed`

### 1.0.0 — 2026-10-11

First release: robots.txt matrix for 20 AI agents, passive CDN detection, opt-in UA probe, raw/rendered word ratio, structure and trust checks, llms.txt validation, 0–100 score, top fixes, HTML report.