# Changelog of Competitor Website Analyzer (`entranced_gelato/ai-competitive-brief-generator`) Actor

- **URL**: https://apify.com/entranced_gelato/ai-competitive-brief-generator/changelog.md
- **Full Actor documentation**: https://apify.com/entranced_gelato/ai-competitive-brief-generator.md

## Changelog

### 2026-11-02: Pricing notes

- README: a notice that the new prices start on 16 November 2026 at 14:41 UTC, and what applies until then; the automatic Actor start fee ($0.00005 per GB of run memory) is listed in the cost table.

### 2026-10-17: Compliance update

#### Added

- **robots.txt is respected by default.** New input `respectRobotsTxt` ("Respect robots.txt", default `true`). Before the homepage, each subpage and every redirect, the Actor reads the site's robots.txt (RFC 9309) and follows the `AIDevsBot` group if there is one, otherwise the `*` group; the longest matching rule wins. A disallowed homepage is not fetched: the run writes a free error row (`errorCode` `ROBOTS_DISALLOWED`, `blocked: true`), sets a clear status message, charges nothing and still succeeds. Disallowed subpages are skipped and listed in the new output field `skippedSubpages`. robots.txt answering 401/403 counts as disallowing the whole site; 429, 5xx, network errors and timeouts make it unreachable, so the site is skipped for the run (`ROBOTS_UNREACHABLE`); other 4xx (e.g. 404) allow everything.
- **Text-and-data-mining reservations (TDMRep).** Every entry in `pages` has `tdmReserved` and `tdmPolicy`, read from the `tdm-reservation` / `tdm-policy` HTTP headers and `<meta>` tags (the tag wins). Reserved pages still get plain extraction but are never sent to the AI; if no page is left, the AI step is skipped with `aiError` "Skipped: the site reserves text-and-data-mining rights (TDMRep)." and no AI charge.
- AI labels `aiGeneratedAt` (ISO 8601 UTC) and `aiDigitalSourceType` (IPTC "trainedAlgorithmicMedia"); AI briefs end with a visible line "*AI-generated by <model> on <date>. May contain errors; check facts before relying on them.*". The overview view shows `aiNotice`.
- `LICENSE` with an all-rights-reserved notice.

#### Changed

- **Honest identification:** requests send `Mozilla/5.0 (compatible; AIDevsBot/1.0; +https://apify.com/entranced_gelato/ai-competitive-brief-generator)` instead of a fixed Chrome 126 User-Agent.
- `aiNotice` now reads "AI-generated; may contain errors. Check facts before relying on them."
- **AI gateway model allowlist:** only `openai/...` and `anthropic/...` models, never `:free` or stealth variants; any other `aiModel` falls back to `openai/gpt-4o-mini` with a status message. Gateway requests include `"provider": { "data_collection": "deny" }`. The own-key path is unchanged.
- Emails and phone numbers are masked in every part of the AI prompt (title, description, headings, prices, text, URL) unless personal data is on, not only in the page text. Page text keeps neighbouring blocks apart, so a number followed directly by other text is masked too.
- Licence: `package.json` is `"license": "UNLICENSED"` and `"private": true` (was ISC).
- README: "AI and your data", robots.txt and TDM section, a legal FAQ that names competitor sites' terms, wrong-charge and data-protection lines, a not-affiliated line and no outside links. Prices are unchanged: an AI brief is $0.05 ("Brief generated"), also with your own OpenAI key; extractive briefs are free.

### 2026-10-11

#### Changed

- **Apify's standard proxies only.** Residential proxy groups and the `GOOGLE_SERP` group are dropped from the proxy settings and custom proxy URLs are ignored. If nothing usable is left, the run uses Apify Proxy in automatic mode (datacenter proxies). When settings are dropped the run logs a warning and sets a status message; it never fails because of it. "No proxy" still means no proxy.

### 2026-10-10

#### Changed

- **Extractive briefs are now free.** The "Brief generated" event ($0.05 per brief, as low as $0.035 on higher plans) is charged only for AI-written briefs, once, after the brief is saved. Failed, blocked or AI-less runs cost nothing.
- **AI is off by default.** New input `useAi` ("Generate AI-written brief"). Without a key the brief is written through Apify's AI gateway (`openai/gpt-4o-mini` by default, override with `aiModel`); the gateway is used only when the "Brief generated" event is priced. Supplying `openaiApiKey` still switches AI on, as before.
- AI output is labelled with `aiGenerated`, `aiModel` and `aiNotice`; `aiError` explains when a requested AI brief could not be written.
- Sites that show a CAPTCHA, bot challenge or login wall are reported (`errorCode: BLOCKED` / `LOGIN_REQUIRED`) and skipped. HTTP 401/403 responses are no longer retried with a new proxy session.
- Personal data is off by default: links to individual people's profiles are dropped, emails and phone numbers in page text are masked, and the AI names roles, not people. New input `includePersonalData` (GDPR note in its description).
- Social links are matched by exact host (a link to netflix.com is no longer reported as x.com) and subpages must be on the same origin.
- README, Store listing (`.actor/store-listing.json`), dataset views and output schema rewritten to match the billing.

#### Added

- Unit tests and local end-to-end tests (no live traffic), schema checks and a CI workflow (`.github/workflows/ci.yml`).