# Changelog of Website to Markdown Crawler for LLMs and RAG (`fecklesspleb/website-to-markdown-crawler`) Actor

- **URL**: https://apify.com/fecklesspleb/website-to-markdown-crawler/changelog.md
- **Full Actor documentation**: https://apify.com/fecklesspleb/website-to-markdown-crawler.md

## Changelog

### 2026-11-02 (actor.json version unchanged: 1.0): private-address (SSRF) guard

- On the Apify platform, every request (start URLs, discovered links, each redirect hop, and robots.txt) is refused
  when the host resolves to a private, loopback, link-local, carrier-grade NAT, reserved, multicast or unspecified
  address (including IPv4-mapped IPv6), or when any of the host's addresses is one. Same rules as Bulk HTTP Request.
- Such a page is not charged and counts in `failed`. With `includeFailedPages` it gets a free row with
  `error: "not fetched: private, loopback and link-local addresses are not allowed"`. If no page is saved, the usual
  free status row (`recordType: "status"`, `message: "Not charged: no page was saved. ..."`) lists that reason.
- A site whose own host is private is reported as blocked (not as a robots.txt problem). A robots.txt that redirects
  to a private address makes the site disallowed, like an unreachable robots.txt
  ("the site could not be reached (Blocked on robots.txt)").
- A host that cannot be resolved still fails as before ("DNS or connection error"); if that happens on robots.txt
  the reason now names `DnsLookupFailed` instead of `ConnectError`.
- Local runs (APIFY_IS_AT_HOME not set) are unchanged.
- Files: `my_actor/common.py` (guard helpers; `Http._robots_for` checks the site first), `my_actor/main.py` (hook
  on the httpx client, `Blocked` handled), `tests/test_ssrf_guard_1102t.py` (new), README Limits (one line).