Crawl a whole website, docs section or blog and save every page as clean Markdown or text for LLMs, RAG, vector databases and AI chatbots. Depth, page and path limits, include and exclude globs. Stays on the site, respects robots.txt, no browser. $0.80 per 1,000 pages.
2026-11-02 (actor.json version unchanged: 1.0): private-address (SSRF) guard
On the Apify platform, every request (start URLs, discovered links, each redirect hop, and robots.txt) is refused
when the host resolves to a private, loopback, link-local, carrier-grade NAT, reserved, multicast or unspecified
address (including IPv4-mapped IPv6), or when any of the host's addresses is one. Same rules as Bulk HTTP Request.
Such a page is not charged and counts in failed. With includeFailedPages it gets a free row with
error: "not fetched: private, loopback and link-local addresses are not allowed". If no page is saved, the usual
free status row (recordType: "status", message: "Not charged: no page was saved. ...") lists that reason.
A site whose own host is private is reported as blocked (not as a robots.txt problem). A robots.txt that redirects
to a private address makes the site disallowed, like an unreachable robots.txt
("the site could not be reached (Blocked on robots.txt)").
A host that cannot be resolved still fails as before ("DNS or connection error"); if that happens on robots.txt
the reason now names DnsLookupFailed instead of ConnectError.
Local runs (APIFY_IS_AT_HOME not set) are unchanged.
Files: my_actor/common.py (guard helpers; Http._robots_for checks the site first), my_actor/main.py (hook
on the httpx client, Blocked handled), tests/test_ssrf_guard_1102t.py (new), README Limits (one line).
