Google Trends Scraper

Interest over time, interest by region and related queries from Google Trends.

The point of this one is that the run ends. Google answers an un-warmed request with 429, and it rate limits per address. A scraper that does not handle both does not fail quickly, it hangs, and the run gets killed at the platform timeout after you have already paid for the compute.

What comes back

dataType	Fields	Rows per term
interestOverTime	date, timestamp, value (0-100), isPartial	53 for 12 months, 93 for 90 days
interestByRegion	region, regionCode, value (0-100)	51 for the US, up to ~250 worldwide
relatedQueries	query, value, formattedValue, isBreakout, link	up to 50

Every row also carries searchTerm, geo, timeframe and category, so a multi-term run is one flat table you can filter rather than a nested blob.

value is Google's own 0-100 index, not a search volume. 100 is the peak of the range you asked for, and the numbers are only comparable inside a single term and timeframe.

Related topics is not offered

Google currently returns an empty list for the related-topics widget, whether the term is sent as text or resolved to a topic entity first. Rather than list it as a feature and hand you nothing, it is left out. If Google starts answering it again, it will appear as an option.

Reliability

This is the part that is different, so here is exactly what it does.

A cookie is fetched first. trends.google.com sets an NID cookie on the HTML page, and the API refuses requests that do not carry it. Warm-up happens automatically and again after any rejection.

Retries change address, not just wait. Backing off on a rate-limited address only spends time. From the second attempt each retry asks the proxy for a new session and re-warms the cookie for it.

Use the Apify proxy. This is not a formality: during development a single address that had made a few hundred requests was refused with 429 continuously for thirty minutes, checked every 45 seconds. No backoff schedule survives that. Without a proxy the retries have nothing to rotate to, and every term will eventually come back exhausted.

Every term has a deadline. secondsPerTerm is a hard budget. When it passes, the term is recorded as skipped and the run moves to the next one. This is what stops one difficult term from consuming the whole run.

One bad term does not kill the run. Terms are fetched independently. Nineteen good terms and one Google will not answer gives you nineteen terms of data and a line in the summary, not a failed run.

Partial results are labelled as partial. If the time series came back but the regional breakdown did not, that term is reported under partial with the reason. A short row count is never left looking like a small trend.

You are not charged for nothing. Billing is per row that reaches the dataset. A term that returns no data costs nothing.

The run summary

Every run writes RUN_SUMMARY to the key-value store:

{
  "requested": 3,
  "collected": 2,
  "rows": 372,
  "partial": [],
  "skipped": [{ "term": "zzzz-nonexistent-term-xyz", "reason": "empty-from-google" }]
}

The reason separates "Google has nothing to say about this" from "we could not get an answer", which is the distinction that matters when you are deciding whether to re-run:

reason	Meaning	Worth retrying
empty-from-google	Google answered, with nothing in it. Usually a term with no measurable interest in that region or timeframe.	No
not-offered-by-google	Google did not offer this widget for this term at all.	No
bad-request	Google rejected the term, geo or timeframe as invalid.	No, fix the input
deadline	The secondsPerTerm budget ran out.	Yes, with a longer budget
exhausted	Rate limited past the retry count.	Yes, with a proxy enabled

If a scheduled run needs to alert on trouble, watch for deadline and exhausted in skipped and partial. The other three are Google telling you something true about the term, not a failure of the run.

A run where nothing at all came back fails loudly rather than writing an empty dataset quietly.

Input

{
  "searchTerms": ["bitcoin", "ethereum"],
  "geo": "US",
  "timeframe": "today 12-m",
  "outputs": ["interestOverTime", "interestByRegion", "relatedQueries"],
  "category": 0,
  "language": "en-US",
  "secondsPerTerm": 120,
  "proxyConfiguration": { "useApifyProxy": true }
}

geo takes a country code such as US, GB, KR, or a sub-region such as US-CA. Leave it empty for worldwide.

timeframe decides the bucket size: today 12-m gives weekly points, today 3-m gives daily, now 7-d gives hourly, all goes back to 2004 monthly.

category is a Google Trends category id. 0 is everything, 7 is Finance, 71 is Food & Drink, 5 is Computers & Electronics.

Notes

Google Trends is an index, not a dataset with an SLA. Values for the most recent bucket move as Google finishes counting, which is what isPartial marks. If you are storing history, re-fetch the tail rather than trusting the last point.