# Changelog of Google Trends Scraper & API (`suslabs/google-trends-scraper`) Actor

- **URL**: https://apify.com/suslabs/google-trends-scraper/changelog.md
- **Full Actor documentation**: https://apify.com/suslabs/google-trends-scraper.md

## Changelog and output contract

### Monitor noise tolerance — 2026-10-21

- Added monitorSensitivity (low/normal/high, default normal). Numeric moves must exceed max(12 index points, 15% of the larger value) at normal sensitivity (low 16/25%, high 3/5%). New complete periods always trigger updates; removed/partial buckets and status-only differences do not.
- Rising top-five and Breakout arrivals require two consecutive qualifying panels. Numeric growth must clear the prior fifth rising score by tolerance; Breakouts also clear a 5000% boundary (high uses 5000% alone). This filters ties and promotions from vanished leaders. Unknown old growth uses categorical confirmation. Near-tied region swaps use a margin, and direction labels need numeric evidence plus hysteresis. Trending volume buckets/lifecycle changes still register.
- Monitoring output adds changeReasons and newCompletePoints; newRisingQueries/newBreakoutQueries contain confirmed arrivals. UNCHANGED holds the prior reported data baseline so gradual drift accumulates. Missing panels retain their baseline; newly available panels seed free. Old snapshot layouts load without mass NEW.
- Empty or wholly unmeasurable numeric panels preserve their previous baseline even when an unrelated panel triggers UPDATED. A regression covers confirmed query changes alongside empty timeline/region panels.
- Legacy city snapshots lacking coordinates migrate through unique city names while retaining old values; numeric changes remain detectable. Wholly unmeasurable old numeric panels seed free.
- Query confirmation additionally requires elapsed time from a continuously qualifying candidate: normal one hour, low six hours, high no minimum. New complete periods and numeric moves remain immediate. Candidate times use the existing staged-snapshot transaction and stay out of previous. This adds explicit query-alert latency and avoids treating cached samples minutes apart as temporal evidence.
- Stage/push/commit/revert and \_base semantics, complete-only billing, daily billing, request/retry/proxy behavior and RUN_STATE persistence are unchanged. New offline noise, real-change, legacy and member/trending tests accompany the existing restart and billing tests. Live validation and spend are recorded in GAP_PROGRESS.md.
- Final private build 0.2.61: 29 NEW, then 28/29 UNCHANGED (96.55%) on both repeats. Two above-tolerance region revisions remain unverified as real demand; noise classification is not perfect. All ten final tasks and the baseline complete; prefill 23.507 s. 79 parser/orchestration tests and 10 daily tests pass. GAP_PROGRESS retains all failed prototypes, every run/build cost and the hourly completion evidence.
- Live hourly follow-up on the same monitor/input, 68.21 minutes after its first run finished: all 29 UPDATED with new complete period reasons, 53 new complete hourly points, zero NEW resets across the older snapshot format.
- Recorded full task spend: $0.267453522/$0.30, including all 62 runs and five builds. All 29 real v1 snapshots also replay as UNCHANGED against their matching original reports.

### Phase 3 review fixes — 2026-10-21

**Restart and billing safety**

- Flat and named-dataset rows are pushed **before** the charged envelope. Previously a restart between the two left a charged job with no data.
- A report the max-charge budget can't cover is detected before anything is pushed (`calculate_max_event_charge_count_within_limit`). It is not marked done, so it appears in FAILED_KEYWORDS.
- After the durable push, later steps (multi-unit charge, monitor commit, expiry rows) are logged on failure instead of turning a delivered, charged report into INTERNAL_ERROR.
- Optional named-dataset copies can't fail the main delivery.
- Chain normalization only uses groups that were actually delivered.
- The second-pass stop branch, the final pending loop and the chain loop no longer crash the run on one bad item.
- `maxRunSeconds` counts time from `ACTOR_STARTED_AT`, so a migration doesn't reset the budget.
- `widgetRetries` survive the pending/second pass.
- A restart no longer holds ids of free flat rows in memory (~94 MiB per million rows). Trending dedup no longer copies the run-wide id set per country.
- Direct-first: a direct panel refusal that a residential panel retry rescues still counts as a direct failure, so the run-wide switch works for the dominant failure (relatedsearches 429).

**Daily estimates**

- The last window is always a full 240 days. With `all`, short tails were unresolvable on 121 of 365 days; now 0. Custom 10-year ranges: 23/74 → 0/74.
- A window whose coarse reference is all zero is resolved as zeros (a new term: 7/38 → 38/38 windows). `hasData:false` points with a numeric value count as measured zeros.
- Failed windows are not checkpointed, so a restart retries them. Window requests still count.
- **Billing:** windows are separate units, charged as 1 + delivered windows even when the report is partial. One failed window no longer makes a 39-unit report free; under the previous rule, `all` would have been paid only ~8% of the time.
- The job id no longer includes the window list, and plans use the run start date. A restart after UTC midnight produced a new id and a second N+1 charge.
- README and form now state the `all` multiplier (up to 39 charges per keyword) and how Max results counts daily reports.

**Restart edge cases (fixed the same day):**

- Monitor snapshots are staged before the push, tagged with the report id and run id, and finalized after it. On a restart the run checks whether the report reached its dataset: if so, the staged snapshot stands; if not, it compares against the previous snapshot. A stage left by an older run counts as delivered, so the worst case is a free UNCHANGED row, never a second charge. A failed push reverts the stage. Trending snapshots are written once per country.
- Restart recovery rebuilds `partialFree`, `KEYWORD_INSIGHTS` and `ERRORS` from the dataset. `widgetRetries`, `secondPass`, `rowsOmitted`, `unchanged` and `stoppedBy` are saved on Apify's PERSIST_STATE event (also emitted right before a migration) as `RUN_STATE`.
- **Live proof:** run NoWRuCPbgdbQrfDc9 (30 monitored keywords, rebooted after 13 reports): 29 distinct keywords delivered once each ("coffee" was duplicated in the input), no duplicates, 13 skipped on resume, SUMMARY delivered 29 / COMPLETE, 29 insights. The repeat run D0jea8kI8vjufbiok had no NEW rows.
- **Still open:** multi-unit charges (sampling, daily) come after the push, so a restart in between can under-charge (never double-charge).

**pytrends-suslabs client:**

- pytrends HTTP arguments are ignored with a warning (they used to turn the Actor's retries off).
- `interest_by_region` follows pytrends' resolution rule, so a US query returns states.
- One run fetches all three panels and is cached per payload.
- Local `pn` → country map (turkey, hong_kong, czech_republic work).
- Keywords are stripped; a missing keyword raises.
- Cities are keyed by coordinates; no-data regions are 0.
- int64 / datetime64\[ns] dtypes like pytrends; `geoCode` is the last column.
- Bad timeframes are rejected locally; the Actor's error detail is included in errors.
- Python ≥3.11. 11 tests.

### Complete-only billing and partial-report fix — 2026-10-21

- **Pay only for complete results.** A report (or comparison group) missing any requested panel is delivered free: `status: partial`, `missingDataTypes` listed, `charged: false`. SUMMARY counts these as `partialFree`. Daily-resolution windows are still charged per delivered window.
- **Root cause of partial reports, fixed.** In the 2,000-keyword load test, 398 reports were partial: 90% a 429 on `relatedsearches`, 10% on `comparedgeo`, all after exhausting their retries.
  - One attempt sends the cookie, explore and every panel over one kept-alive connection, so a single IP bursts 5+ requests and the last panel hits Google's per-IP limit.
  - A panel failure restarted the whole attempt, costing 2 extra requests per retry.
  - Panel timeouts and dropped connections were swallowed with no retry at all.
- **Now:** a refused, timed-out or dropped panel is retried alone, up to 3 times, on fresh connections (new residential IPs), reusing the cookie and token. It falls back to a full new attempt if that fails or Google rejects the token. SUMMARY adds `widgetRetries`.
- **Stress A/B** (100 load-test keywords, concurrency 20, residential only, maxRetries 2, interleaved 2 runs per build): partial 7/200 → 1/200 (that one a timeout, also fixed); full retries 38 → 1; proxy+compute $0.212–0.245 → $0.203–0.206 per 1k; runtime 68–87 s → 54–66 s.

### Phase 3 (Hard) — 2026-10-21

- Fixed trending restart caps and completion when duplicate items were skipped; free country checkpoints do not consume result slots. Added minimal durable chain timelines so normalization survives compact-output restarts, using deterministic planned group order.
- Separate output datasets now use run-scoped unnamed aliases, following unnamed storage retention. Pending raw sections live in the default KVS; both passes use bounded workers honoring maxConcurrency.
- Added opt-in dailyResolution: overlapping 240-day windows fitted to Google's coarse curve, diagnostics, restart checkpoints and free flat daily points. Reserve baseline plus all window events before work; only delivered baseline/usable windows count as normal keyword-result events. No new event or price.
- Added separate unpublished TrendReq client in D:\Apify\pytrends-suslabs, with pandas conversion, mocked API tests and a migration example. All six methods were checked through live Actor responses.
- Sampling metadata now preserves per-replica complete averages and original Google averages. docs/ACCURACY.md reports the ten-keyword, three-range study, including two refused replicas.
- Retained direct-first, rotating residential, HTTP-only production and all three form panels. Browser research runs use a separate tagged image; no browser dependency enters this Actor. Gap 1.15/1.16/1.17 product notes are in D:\Apify\ideas. Final measurements and research verdicts are in GAP_PROGRESS.md.
- Persisted the run-wide direct switch across reboot. A late in-flight direct success/failure cannot re-enable it after the third refusal; successful RSS/autocomplete/explore/full-feed requests now reset the consecutive counter consistently.

### Phase 2 review fixes — 2026-10-21

- **Direct-first is now the default** (`directFirst: true`). Each report starts on the Actor's own connection and switches to rotating residential on any failure (refusal, block page, consent redirect, 5xx, timeout). The switch is free: no retry is used and there is no wait. After 3 consecutive direct failures across reports, the run skips direct. A direct success resets the counter. SUMMARY adds `directFirst` and `directTurnedOff`. Direct-first is turned off automatically for sampling, for a proxy exit country (`viewedFrom`) and for custom proxies. The full Trending Now feed follows the same policy and now retries timeouts and 5xx.
  - Evidence: Codex's 5-round interleaved benchmark (median $0.171 vs $0.430 per 1k keywords; 100/100 vs 99/100 delivered) plus 3 review runs (60/60 delivered, $0.025–0.13 per 1k, 3.6–66 s).
- **Removed** the impit transport (0/100 delivered in the benchmark), the legacy 1 s backoff (median $0.511 per 1k) and dead code (`_get`, `scrape_keyword`, an unreachable 3xx branch, the unused `report_ids` and `in_flight`).
- **Monitoring:** keyword and comparison reports are never expired, so a weekly schedule no longer re-charges unchanged reports as NEW (only trending entries expire). Corrupt snapshot entries are dropped instead of failing the job. EXPIRED rows get their own id. `monitorName` and `cacheName` have no shared defaults anymore.
- **Billing safety:** if the SDK drops a charged row because the max-charge budget is exhausted, the report is counted as skipped (`stoppedBy: budget`), not delivered, and monitor state is not committed. The end-of-run path for rate-limited reports now records `seen`, requests and retries.
- **Inputs:** `trendingHours` values other than 4/24/48/168 are rounded up with a free note (the form can't restrict an integer to a list). Bad import files and non-string `urlsFromFile` now give INVALID_INPUT instead of crashing. Debug key names are sanitized.
- **Form demo:** restored all three panels (direct-first makes it 3.5–4.6 s). Flat rows are built only when flat output or named datasets need them.

### Phase 2 (Medium) — 2026-10-21

- Added opt-in incumbent input/output compatibility, including comma-separated isMultiple groups, empty legacy time range, numeric string categories and raw array-valued output keys. Category ID remains available in the form as categoryId; API category accepts either type.
- Added full Trending Now alongside the unchanged RSS default: growth, lifecycle, breakdown, filters, four windows, language, sorting, per-country limits, separate concurrency and observed US-CA support. Full feed uses direct cloud requests before residential on refusal.
- Added named monitoring with semantic snapshots, NEW/UPDATED/UNCHANGED/EXPIRED, previous values, expiry/reset and only-new/only-changed filtering. Unchanged and monitoring-only rows are free.
- Added anchor/bridge comparisons with a free normalized envelope, low-precision warnings and normal billing per underlying group. Added optional multi-level region panels, per-query settings, descriptive seasonality and age-limited free cached results on refusal.
- Added opt-in repeated sampling with mean/consensus, standard error and agreement; successful replicas use normal events, with the full multiplier reserved before requests. Benchmarked direct-first, impit TLS and legacy backoff (see the Phase 2 review entry for the outcome).
- Added local n8n/Make JSON and Sheets recipe. Extended offline parsers, math, state, cache, budget and billing regressions; real full-feed fixtures came from Apify only.
- The form demo still compares ChatGPT/Gemini/Claude in the US and now preselects timelines only; the API default remains all three panels. Full-panel prefill runs took 49–74 seconds during refusal-heavy tests; the timeline demo took 2.8–9.5 seconds in the first three tests.
- Fetch the selected primary panel before secondary panels, preserving its data if a secondary request is refused. Compact output also removes the long compatible-layout arrays and optional extra region panels.
- Existing defaults and four event names remain. New fields/arrays are opt-in; all details, normalized envelopes, cached/unchanged/status/summary rows are free. No Store, account, publication or pricing settings changed.

### 0.2 review fixes — 2026-10-21

- **Proxy:** residential IPs rotate per request again (no sticky `session_id`). A/B on the same 20 keywords: sticky had 20 retries, 156 requests, 88 s and $0.41 per 1k; rotating had 3–7 retries, 109–121 requests, 31–42 s and $0.24–0.28 per 1k.
- **Retry backoff:** now 0.5 s doubling, capped at 30 s (was 1 s doubling). Every retry already uses a new IP. The default input (3-keyword compare) now runs in 8–21 s.
- **Input form:** human titles; the main fields (What to scrape, Keywords, Country, Time range, Data to collect) come first; everything else is in collapsible sections; date pickers for start and end date. `isMultiple`, `comparisonMode` and `relatedData` are API-only aliases now (still accepted). The prefill is `mode: compare` with ChatGPT/Gemini/Claude in the US; the default `mode` stays `keywords`.
- **Docs:** README rewritten for the Store page; the previous technical README moved to `docs/TECHNICAL.md`. `.actor/input_schema.json` and `docs/INPUT_FIELDS.md` are now hand-maintained (the `tools/build_schema.py` and `tools/prepare_docs.py` generators were removed because they overwrite edits). Added `examples/tasks/` (10 published-task inputs, all verified on Apify).
- No output or billing contract changes.

### 0.2 / schemaVersion 1.1 — 2026-10-20

Added comparison batching, shared location/period comparison inputs, URL parsing, country aliases, explicit region levels and low-volume flags, current RSS, autocomplete and raw explore modes. Added offline validation, descriptive statistics, stable IDs, optional flat/compact output, dataset views and free KVS/run diagnostics.

Original nested fields retain their names, meanings and defaults. Default HTTP work remains one cookie bootstrap, one explore and three section widgets per keyword. Every run now adds a free end-of-run summary; consumers must filter that row. All successful comparison groups have one report envelope, with member rows under perKeyword. Flat mode adds free details after a compact envelope.

Event contract: keyword-result, comparison-result, trending-item, autocomplete-result. The owner controls Console prices; no deployment changes them. A primary-success partial report may be billed; missing-primary and no-data reports are free. A comparison needs every member's primary section to qualify.

Nullable/optional fields: Google's missing averages/coordinates/formatting are null; absent requested sections are named in missingDataTypes with a status/errorCode. Zero-denominator percentages and insufficient samples are null rather than infinity/NaN. Disabled sections remain omitted. excludeEmptyFields intentionally removes null/empty values. Unknown upstream fields are not copied unless explicitly requested through explore/debug modes.

Stable IDs use the validated query group, category/property/language/timezone, sections, geography, resolution and low-volume request setting. Report IDs are stable across repeated runs with the same task settings. Flat IDs add the member index and a natural identity: timestamp, region code/name/coordinates, or related query plus TOP/RISING list. RSS IDs use country/query settings, term and publication date. Ranking changes do not change detail IDs.

### 0.1

Independent keyword exploration with timeline, regions, related queries and retries. Supplied historical benchmark: 120/120 keywords across two tests; roughly $0.30 operating cost per 1,000 keywords. No claim is made that this success rate persists across Google changes.