Google Trends scraper and API in bulk: interest over time, compare up to 5 keywords, daily trending searches, states, metros and cities, rising and Breakout queries. A pytrends alternative that handles 429 errors for you. You pay only for complete results, plus a tiny start fee.
Added monitorSensitivity (low/normal/high, default normal). Numeric moves must exceed max(12 index points, 15% of the larger value) at normal sensitivity (low 16/25%, high 3/5%). New complete periods always trigger updates; removed/partial buckets and status-only differences do not.
Rising top-five and Breakout arrivals require two consecutive qualifying panels. Numeric growth must clear the prior fifth rising score by tolerance; Breakouts also clear a 5000% boundary (high uses 5000% alone). This filters ties and promotions from vanished leaders. Unknown old growth uses categorical confirmation. Near-tied region swaps use a margin, and direction labels need numeric evidence plus hysteresis. Trending volume buckets/lifecycle changes still register.
Monitoring output adds changeReasons and newCompletePoints; newRisingQueries/newBreakoutQueries contain confirmed arrivals. UNCHANGED holds the prior reported data baseline so gradual drift accumulates. Missing panels retain their baseline; newly available panels seed free. Old snapshot layouts load without mass NEW.
Empty or wholly unmeasurable numeric panels preserve their previous baseline even when an unrelated panel triggers UPDATED. A regression covers confirmed query changes alongside empty timeline/region panels.
Legacy city snapshots lacking coordinates migrate through unique city names while retaining old values; numeric changes remain detectable. Wholly unmeasurable old numeric panels seed free.
Query confirmation additionally requires elapsed time from a continuously qualifying candidate: normal one hour, low six hours, high no minimum. New complete periods and numeric moves remain immediate. Candidate times use the existing staged-snapshot transaction and stay out of previous. This adds explicit query-alert latency and avoids treating cached samples minutes apart as temporal evidence.
Stage/push/commit/revert and _base semantics, complete-only billing, daily billing, request/retry/proxy behavior and RUN_STATE persistence are unchanged. New offline noise, real-change, legacy and member/trending tests accompany the existing restart and billing tests. Live validation and spend are recorded in GAP_PROGRESS.md.
Final private build 0.2.61: 29 NEW, then 28/29 UNCHANGED (96.55%) on both repeats. Two above-tolerance region revisions remain unverified as real demand; noise classification is not perfect. All ten final tasks and the baseline complete; prefill 23.507 s. 79 parser/orchestration tests and 10 daily tests pass. GAP_PROGRESS retains all failed prototypes, every run/build cost and the hourly completion evidence.
Live hourly follow-up on the same monitor/input, 68.21 minutes after its first run finished: all 29 UPDATED with new complete period reasons, 53 new complete hourly points, zero NEW resets across the older snapshot format.
Recorded full task spend: $0.267453522/$0.30, including all 62 runs and five builds. All 29 real v1 snapshots also replay as UNCHANGED against their matching original reports.
Phase 3 review fixes — 2026-10-21
Restart and billing safety
Flat and named-dataset rows are pushed before the charged envelope. Previously a restart between the two left a charged job with no data.
A report the max-charge budget can't cover is detected before anything is pushed (calculate_max_event_charge_count_within_limit). It is not marked done, so it appears in FAILED_KEYWORDS.
After the durable push, later steps (multi-unit charge, monitor commit, expiry rows) are logged on failure instead of turning a delivered, charged report into INTERNAL_ERROR.
Optional named-dataset copies can't fail the main delivery.
Chain normalization only uses groups that were actually delivered.
The second-pass stop branch, the final pending loop and the chain loop no longer crash the run on one bad item.
maxRunSeconds counts time from ACTOR_STARTED_AT, so a migration doesn't reset the budget.
widgetRetries survive the pending/second pass.
A restart no longer holds ids of free flat rows in memory (~94 MiB per million rows). Trending dedup no longer copies the run-wide id set per country.
Direct-first: a direct panel refusal that a residential panel retry rescues still counts as a direct failure, so the run-wide switch works for the dominant failure (relatedsearches 429).
Daily estimates
The last window is always a full 240 days. With all, short tails were unresolvable on 121 of 365 days; now 0. Custom 10-year ranges: 23/74 → 0/74.
A window whose coarse reference is all zero is resolved as zeros (a new term: 7/38 → 38/38 windows). hasData:false points with a numeric value count as measured zeros.
Failed windows are not checkpointed, so a restart retries them. Window requests still count.
Billing: windows are separate units, charged as 1 + delivered windows even when the report is partial. One failed window no longer makes a 39-unit report free; under the previous rule, all would have been paid only ~8% of the time.
The job id no longer includes the window list, and plans use the run start date. A restart after UTC midnight produced a new id and a second N+1 charge.
README and form now state the all multiplier (up to 39 charges per keyword) and how Max results counts daily reports.
Restart edge cases (fixed the same day):
Monitor snapshots are staged before the push, tagged with the report id and run id, and finalized after it. On a restart the run checks whether the report reached its dataset: if so, the staged snapshot stands; if not, it compares against the previous snapshot. A stage left by an older run counts as delivered, so the worst case is a free UNCHANGED row, never a second charge. A failed push reverts the stage. Trending snapshots are written once per country.
Restart recovery rebuilds partialFree, KEYWORD_INSIGHTS and ERRORS from the dataset. widgetRetries, secondPass, rowsOmitted, unchanged and stoppedBy are saved on Apify's PERSIST_STATE event (also emitted right before a migration) as RUN_STATE.
Live proof: run NoWRuCPbgdbQrfDc9 (30 monitored keywords, rebooted after 13 reports): 29 distinct keywords delivered once each ("coffee" was duplicated in the input), no duplicates, 13 skipped on resume, SUMMARY delivered 29 / COMPLETE, 29 insights. The repeat run D0jea8kI8vjufbiok had no NEW rows.
Still open: multi-unit charges (sampling, daily) come after the push, so a restart in between can under-charge (never double-charge).
pytrends-suslabs client:
pytrends HTTP arguments are ignored with a warning (they used to turn the Actor's retries off).
interest_by_region follows pytrends' resolution rule, so a US query returns states.
One run fetches all three panels and is cached per payload.
Local pn → country map (turkey, hong_kong, czech_republic work).
Keywords are stripped; a missing keyword raises.
Cities are keyed by coordinates; no-data regions are 0.
int64 / datetime64[ns] dtypes like pytrends; geoCode is the last column.
Bad timeframes are rejected locally; the Actor's error detail is included in errors.
Python ≥3.11. 11 tests.
Complete-only billing and partial-report fix — 2026-10-21
Pay only for complete results. A report (or comparison group) missing any requested panel is delivered free: status: partial, missingDataTypes listed, charged: false. SUMMARY counts these as partialFree. Daily-resolution windows are still charged per delivered window.
Root cause of partial reports, fixed. In the 2,000-keyword load test, 398 reports were partial: 90% a 429 on relatedsearches, 10% on comparedgeo, all after exhausting their retries.
One attempt sends the cookie, explore and every panel over one kept-alive connection, so a single IP bursts 5+ requests and the last panel hits Google's per-IP limit.
A panel failure restarted the whole attempt, costing 2 extra requests per retry.
Panel timeouts and dropped connections were swallowed with no retry at all.
Now: a refused, timed-out or dropped panel is retried alone, up to 3 times, on fresh connections (new residential IPs), reusing the cookie and token. It falls back to a full new attempt if that fails or Google rejects the token. SUMMARY adds widgetRetries.
Stress A/B (100 load-test keywords, concurrency 20, residential only, maxRetries 2, interleaved 2 runs per build): partial 7/200 → 1/200 (that one a timeout, also fixed); full retries 38 → 1; proxy+compute $0.212–0.245 → $0.203–0.206 per 1k; runtime 68–87 s → 54–66 s.
Phase 3 (Hard) — 2026-10-21
Fixed trending restart caps and completion when duplicate items were skipped; free country checkpoints do not consume result slots. Added minimal durable chain timelines so normalization survives compact-output restarts, using deterministic planned group order.
Separate output datasets now use run-scoped unnamed aliases, following unnamed storage retention. Pending raw sections live in the default KVS; both passes use bounded workers honoring maxConcurrency.
Added opt-in dailyResolution: overlapping 240-day windows fitted to Google's coarse curve, diagnostics, restart checkpoints and free flat daily points. Reserve baseline plus all window events before work; only delivered baseline/usable windows count as normal keyword-result events. No new event or price.
Added separate unpublished TrendReq client in D:\Apify\pytrends-suslabs, with pandas conversion, mocked API tests and a migration example. All six methods were checked through live Actor responses.
Sampling metadata now preserves per-replica complete averages and original Google averages. docs/ACCURACY.md reports the ten-keyword, three-range study, including two refused replicas.
Retained direct-first, rotating residential, HTTP-only production and all three form panels. Browser research runs use a separate tagged image; no browser dependency enters this Actor. Gap 1.15/1.16/1.17 product notes are in D:\Apify\ideas. Final measurements and research verdicts are in GAP_PROGRESS.md.
Persisted the run-wide direct switch across reboot. A late in-flight direct success/failure cannot re-enable it after the third refusal; successful RSS/autocomplete/explore/full-feed requests now reset the consecutive counter consistently.
Phase 2 review fixes — 2026-10-21
Direct-first is now the default (directFirst: true). Each report starts on the Actor's own connection and switches to rotating residential on any failure (refusal, block page, consent redirect, 5xx, timeout). The switch is free: no retry is used and there is no wait. After 3 consecutive direct failures across reports, the run skips direct. A direct success resets the counter. SUMMARY adds directFirst and directTurnedOff. Direct-first is turned off automatically for sampling, for a proxy exit country (viewedFrom) and for custom proxies. The full Trending Now feed follows the same policy and now retries timeouts and 5xx.
Evidence: Codex's 5-round interleaved benchmark (median $0.171 vs $0.430 per 1k keywords; 100/100 vs 99/100 delivered) plus 3 review runs (60/60 delivered, $0.025–0.13 per 1k, 3.6–66 s).
Removed the impit transport (0/100 delivered in the benchmark), the legacy 1 s backoff (median $0.511 per 1k) and dead code (_get, scrape_keyword, an unreachable 3xx branch, the unused report_ids and in_flight).
Monitoring: keyword and comparison reports are never expired, so a weekly schedule no longer re-charges unchanged reports as NEW (only trending entries expire). Corrupt snapshot entries are dropped instead of failing the job. EXPIRED rows get their own id. monitorName and cacheName have no shared defaults anymore.
Billing safety: if the SDK drops a charged row because the max-charge budget is exhausted, the report is counted as skipped (stoppedBy: budget), not delivered, and monitor state is not committed. The end-of-run path for rate-limited reports now records seen, requests and retries.
Inputs:trendingHours values other than 4/24/48/168 are rounded up with a free note (the form can't restrict an integer to a list). Bad import files and non-string urlsFromFile now give INVALID_INPUT instead of crashing. Debug key names are sanitized.
Form demo: restored all three panels (direct-first makes it 3.5–4.6 s). Flat rows are built only when flat output or named datasets need them.
Phase 2 (Medium) — 2026-10-21
Added opt-in incumbent input/output compatibility, including comma-separated isMultiple groups, empty legacy time range, numeric string categories and raw array-valued output keys. Category ID remains available in the form as categoryId; API category accepts either type.
Added full Trending Now alongside the unchanged RSS default: growth, lifecycle, breakdown, filters, four windows, language, sorting, per-country limits, separate concurrency and observed US-CA support. Full feed uses direct cloud requests before residential on refusal.
Added named monitoring with semantic snapshots, NEW/UPDATED/UNCHANGED/EXPIRED, previous values, expiry/reset and only-new/only-changed filtering. Unchanged and monitoring-only rows are free.
Added anchor/bridge comparisons with a free normalized envelope, low-precision warnings and normal billing per underlying group. Added optional multi-level region panels, per-query settings, descriptive seasonality and age-limited free cached results on refusal.
Added opt-in repeated sampling with mean/consensus, standard error and agreement; successful replicas use normal events, with the full multiplier reserved before requests. Benchmarked direct-first, impit TLS and legacy backoff (see the Phase 2 review entry for the outcome).
Added local n8n/Make JSON and Sheets recipe. Extended offline parsers, math, state, cache, budget and billing regressions; real full-feed fixtures came from Apify only.
The form demo still compares ChatGPT/Gemini/Claude in the US and now preselects timelines only; the API default remains all three panels. Full-panel prefill runs took 49–74 seconds during refusal-heavy tests; the timeline demo took 2.8–9.5 seconds in the first three tests.
Fetch the selected primary panel before secondary panels, preserving its data if a secondary request is refused. Compact output also removes the long compatible-layout arrays and optional extra region panels.
Existing defaults and four event names remain. New fields/arrays are opt-in; all details, normalized envelopes, cached/unchanged/status/summary rows are free. No Store, account, publication or pricing settings changed.
0.2 review fixes — 2026-10-21
Proxy: residential IPs rotate per request again (no sticky session_id). A/B on the same 20 keywords: sticky had 20 retries, 156 requests, 88 s and $0.41 per 1k; rotating had 3–7 retries, 109–121 requests, 31–42 s and $0.24–0.28 per 1k.
Retry backoff: now 0.5 s doubling, capped at 30 s (was 1 s doubling). Every retry already uses a new IP. The default input (3-keyword compare) now runs in 8–21 s.
Input form: human titles; the main fields (What to scrape, Keywords, Country, Time range, Data to collect) come first; everything else is in collapsible sections; date pickers for start and end date. isMultiple, comparisonMode and relatedData are API-only aliases now (still accepted). The prefill is mode: compare with ChatGPT/Gemini/Claude in the US; the default mode stays keywords.
Docs: README rewritten for the Store page; the previous technical README moved to docs/TECHNICAL.md. .actor/input_schema.json and docs/INPUT_FIELDS.md are now hand-maintained (the tools/build_schema.py and tools/prepare_docs.py generators were removed because they overwrite edits). Added examples/tasks/ (10 published-task inputs, all verified on Apify).
No output or billing contract changes.
0.2 / schemaVersion 1.1 — 2026-10-20
Added comparison batching, shared location/period comparison inputs, URL parsing, country aliases, explicit region levels and low-volume flags, current RSS, autocomplete and raw explore modes. Added offline validation, descriptive statistics, stable IDs, optional flat/compact output, dataset views and free KVS/run diagnostics.
Original nested fields retain their names, meanings and defaults. Default HTTP work remains one cookie bootstrap, one explore and three section widgets per keyword. Every run now adds a free end-of-run summary; consumers must filter that row. All successful comparison groups have one report envelope, with member rows under perKeyword. Flat mode adds free details after a compact envelope.
Event contract: keyword-result, comparison-result, trending-item, autocomplete-result. The owner controls Console prices; no deployment changes them. A primary-success partial report may be billed; missing-primary and no-data reports are free. A comparison needs every member's primary section to qualify.
Nullable/optional fields: Google's missing averages/coordinates/formatting are null; absent requested sections are named in missingDataTypes with a status/errorCode. Zero-denominator percentages and insufficient samples are null rather than infinity/NaN. Disabled sections remain omitted. excludeEmptyFields intentionally removes null/empty values. Unknown upstream fields are not copied unless explicitly requested through explore/debug modes.
Stable IDs use the validated query group, category/property/language/timezone, sections, geography, resolution and low-volume request setting. Report IDs are stable across repeated runs with the same task settings. Flat IDs add the member index and a natural identity: timestamp, region code/name/coordinates, or related query plus TOP/RISING list. RSS IDs use country/query settings, term and publication date. Ranking changes do not change detail IDs.
0.1
Independent keyword exploration with timeline, regions, related queries and retries. Supplied historical benchmark: 120/120 keywords across two tests; roughly $0.30 operating cost per 1,000 keywords. No claim is made that this success rate persists across Google changes.
