# Changelog of Web Page to Markdown & Text - URL Reader for LLMs & RAG (`entranced_gelato/ai-web-page-reader`) Actor

- **URL**: https://apify.com/entranced_gelato/ai-web-page-reader/changelog.md
- **Full Actor documentation**: https://apify.com/entranced_gelato/ai-web-page-reader.md

## Changelog

### 2026-11-02: Pricing notes

- README: a notice that the new prices start on 16 November 2026 at 14:41 UTC, and what applies until then.

### 2026-11-01

#### Added

- **Several pages per run.** New input `urls` ("More page URLs", up to 100). Each page becomes its own row and is charged as one "Page read"; pages that can't be read are free, and the run stops before a page the maximum total charge can't cover. One page behaves exactly as before.

### 2026-10-17: Compliance update

#### Added

- **robots.txt, on by default.** New input `respectRobotsTxt` (default `true`). Before reading, the Actor fetches the site's robots.txt once and applies the rules for `AIDevsBot`, otherwise those for `*` (RFC 9309: the longest matching rule wins). Redirect targets are checked before they are requested. A disallowed page is never fetched or charged and is reported like a block page: `ROBOTS_DISALLOWED`, or `ROBOTS_UNREACHABLE` when robots.txt can't be loaded (429, 5xx, network error, timeout). A 401/403 answer for robots.txt closes the whole site; a 404 allows everything.
- **TDM reservation.** New output fields `tdmReserved` and `tdmPolicy`, read from the TDMRep `tdm-reservation` / `tdm-policy` headers and meta tags (the meta tag wins). Reserved pages are still converted to text and Markdown but never sent to AI (no AI call, no AI charge; `aiError` says why).
- **AI labels.** AI summaries also carry `aiGeneratedAt` (ISO 8601, UTC) and `aiDigitalSourceType` (`trainedAlgorithmicMedia`); items without AI have both set to `null`.

#### Changed

- **Honest identification.** Requests send `Mozilla/5.0 (compatible; AIDevsBot/1.0; +https://apify.com/entranced_gelato/ai-web-page-reader)` instead of a fixed Chrome 126 browser string. Compressed (gzip) responses are still requested.
- **AI privacy and model choice.** Email addresses and phone numbers are masked in every AI prompt. Gateway requests include `"provider": { "data_collection": "deny" }`. On Apify's gateway only `openai/...` and `anthropic/...` models are used, never `:free` or stealth variants; any other `aiModel` falls back to `openai/gpt-4o-mini`, with a status message.
- `aiNotice` now reads "AI-generated; may contain errors. Check facts before relying on them."
- **Licence.** The code is closed source: `package.json` declares `"license": "UNLICENSED"` and `"private": true` (it used to say ISC), and a `LICENSE` file reserves all rights.
- **README and input texts.** New "AI and your data" section; rewritten legal FAQ with the content licence note (for example Wikipedia's CC BY-SA), personal data such as bylines and your role as data controller; robots.txt, opt-out and TDM explanations; a data-protection line; wrong charges go to the Issues tab. No links to outside services.

### 2026-10-11

#### Changed

- **Apify's standard proxies only.** Residential proxy groups and the `GOOGLE_SERP` group are dropped from the proxy settings and custom proxy URLs are ignored. If nothing usable is left, the run uses Apify Proxy in automatic mode (datacenter proxies). When settings are dropped the run logs a warning and sets a status message; it never fails because of it. "No proxy" still means no proxy.

### 2026-10-10

#### Added

- **Built-in AI summary.** With "Generate AI summary" on and no key, the TL;DR and key points are written through Apify's AI gateway (`openai/gpt-4o-mini` by default, override with the new `aiModel` input) for $0.02 per summary. The gateway is used only when the "AI summary" event is priced, and the event is charged only after a summary is produced.
- AI output is labelled with `aiGenerated`, `aiModel` and `aiNotice`; `aiError` explains a summary that was requested but not produced.
- `.actor/store-listing.json`, dataset views (overview, Markdown, status), output schema links, CI workflow and offline tests (unit + local end-to-end).

#### Changed

- With your own `openaiApiKey` the summary no longer triggers the "AI summary" charge (OpenAI bills you directly).
- "Page read" ($0.004, as low as $0.0028) is charged only for pages read successfully, after the record is saved. Failed, blocked, empty (`EMPTY_CONTENT`) and non-HTML responses (`UNSUPPORTED_CONTENT_TYPE`, e.g. PDFs) are reported for free. The run respects the max total charge.
- Pages showing a CAPTCHA, bot challenge or login wall are reported (`BLOCKED` / `LOGIN_REQUIRED`) and never bypassed; HTTP 401/403 are no longer retried with a new proxy session. Downloads are capped at 30 MB.
- `image` is `null` when a page has no `og:image` (it used to repeat the page URL).
- README rewritten: pricing with worked examples and when to use Website to Markdown Crawler instead.