# Changelog of SiteVerdict — Web Page to Clean Markdown for AI (LLM-ready) (`siteverdict/siteverdict-web-to-markdown`) Actor

- **URL**: https://apify.com/siteverdict/siteverdict-web-to-markdown/changelog.md
- **Full Actor documentation**: https://apify.com/siteverdict/siteverdict-web-to-markdown.md

## Changelog

### 2.0.5 (2026-10-16) — Private-network (SSRF) protection, completed

- The existing host check (name filter + DNS lookup) is replaced by the shared SiteVerdict guard: internal names (localhost, \*.local, \*.internal, single-label names, cloud-metadata names), and private / loopback / link-local / CGNAT / multicast / reserved / metadata addresses in every IPv4/IPv6 encoding (decimal/octal/hex, IPv4-mapped, NAT64, 6to4, Teredo, unicode dots) are refused for start URLs, every redirect hop, robots.txt and the webhook.
- HTTP fetching now uses a DNS-pinned transport: the address that was checked is the address that is connected to (no rebinding / TTL-0 window). Refused pages keep the existing wording ("Host resolves to a private or reserved address; refused.") and are not charged.
- Browser rendering: Chromium sends every request (navigations, redirects, meta refresh, JS navigations, frames, scripts, XHR) through a local checking proxy; blocked requests are counted in `SUMMARY.privateNetworkBlocked`.
- No change to inputs, outputs (one additive SUMMARY field) or pricing.

### Memory default (15 Oct 2026)

- 15 Oct 2026 (build after this entry): default run memory is now 1024 MB via `defaultMemoryMbytes` (was 4096 MB (no key set); peak 138-190 MB). Measured on the example input: same output, see the public-15 remediation log. You can still choose more memory per run. No change to inputs, outputs or pricing.

### 2.0.0 — 2026-10-11

Flagship upgrade. Measured on 30 hand-checked pages (`BENCHMARK.md`): main-content F1 90.4% -> **91.9%**, key facts 96.6% -> **98.9%**, output tokens -8%.

- **Extraction v2**: Readability-guided recovery (gap filling, adjacent-prose extension, stop at related / popular / newsletter rails, junk trimming), widget pre-clean (share bars, newsletter and back-in-stock sign-ups, ad slots, cookie notices), MediaWiki citation-noise removal (`removeReferenceLists`, default on), schema.org **product facts** line + `product` field, tracking parameters stripped from links, decorative / duplicate images dropped from Markdown, HTTP-200 waiting-room / bot-failover interstitials detected as `bot-protection`.
- **PDF and DOCX** conversion (unpdf / pdf.js, mammoth): document metadata, page markers, per-page chunks with `page` and `#page=N` citation URLs; scanned PDFs reported as `no-text-layer` (not charged). `followDocumentLinks` for crawl / sitemap modes. New event `document-converted`.
- **Embedding presets and exact token counts**: js-tiktoken cl100k / o200k; presets for OpenAI, Cohere v3, Claude/Voyage and 512-token open models; `tokens` + `tokenizer` on pages and chunks.
- **Monitoring / incremental re-crawl**: named state store per `monitorId`, conditional GET (ETag / Last-Modified, 304 = unchanged), sitemap `<lastmod>` delta, content-hash change detection with a section-level `changeSummary`, `removed` pages, `CHANGES` record, HTTPS webhook alerts (Slack-compatible). Unchanged / removed rows are never charged.
- **AI enrichment (bring your own key)**: summary, key points and grounded Q\&A pairs via Anthropic (official SDK, structured output, server-side refusal fallback) or OpenAI; secret input, off by default; new event `ai-enrichment` charged only on valid output.
- **Citations**: every chunk now carries a `citation` object (anchor or page URL); the page citation gains `modifiedAt` and a ready-made `text`.
- New dataset views: Changes, AI summary & Q\&A, Products; the chunk view is CSV / vector-DB ready.
- Benchmark harness in `benchmark/` (not shipped in the image): snapshot fetcher, references, scorer, compliance check.

### 1.0.0 — 2026-10-11

First release.

- Modes: list of URLs, sitemap (robots.txt `Sitemap:` / `/sitemap.xml`, sitemap indexes, gzip, text sitemaps), same-site crawl with depth, include/exclude globs and max pages.
- Fast HTTP mode with Mozilla Readability + semantic `<main>/<article>` fallback; optional headless Chromium (`auto` / `browser`).
- Markdown (GFM tables, fenced code with language, images with alt), plain text, title/author/dates/language/canonical/site name, heading outline with anchors, internal/external links, images, JSON-LD, word count, token estimate, SHA-256 content hash, citation object.
- Heading-aware RAG chunking (token size, sentence overlap within sections, tables split by rows with header repeated, code fences kept balanced, heading path + anchor per chunk).
- De-duplication by canonical URL and content hash.
- llms.txt + llms-full.txt generation.
- Compliance: robots.txt (RFC 9309) on every redirect hop and browser navigation, Crawl-delay, honest `SiteVerdictBot` UA, paywall / login / bot-protection / `noai` / TDM-reservation detection with no content returned, SSRF guard, per-host politeness, host skipped after 3 bot-protection blocks.
- Pay-per-event: `page-converted`, `page-converted-browser`; max total charge respected; never charged for blocked, paywalled, duplicate, too-short or failed pages.
- Resumable crawl state across migrations.