YouTube Niche Gap Finder

Given a niche or topic, surfaces YouTube sub-topics with rising search interest but low upload volume or stale top-result content — real content gaps, ranked by opportunity score.

How it works

1. Expand — pulls real completion terms from YouTube's public autosuggest endpoint for your niche (and, optionally, for each seed/competitor channel).
2. Signal — checks each candidate sub-topic against Google Trends relative interest, via a real headless browser that intercepts Trends' own internal widgetdata/multiline network call (not DOM scraping).
3. Supply — runs a real YouTube search for each candidate and parses YouTube's own ytInitialData JSON to get the total result count and the upload-recency of the top ~10 ranked videos.
4. Score — combines interest vs. supply vs. freshness into one opportunityScore per sub-topic (formula below).

Inputs

Field	Default	Description
niche	"AI explainers for Gen Z"	The niche/topic to explore. Required.
seedChannels	[]	Optional competitor channel handles/URLs to benchmark against.
region	"GB"	Two-letter region code for autosuggest + search localization.
maxCandidates	10	Caps how many candidates are fully scored per run (each does a Trends fetch + a YouTube search fetch — keeps runtime/cost predictable).
opportunityThreshold	0.5	opportunityScore at/above this is flagged as a real gap (high-opportunity-scored); below it is low-opportunity-scored.

Output fields (per candidate sub-topic)

Field	Description
subTopic	The candidate term.
searchVolumeSignal	0–1 relative interest score.
searchVolumeSource	Where the signal came from — trends-api (real Google Trends data), trends-fallback-autosuggest-rank (Trends unreachable — see below), or unavailable.
existingVideoCount	Total YouTube search results for the term (supply).
avgUploadRecencyDays	Average days-since-upload across the top ~10 ranked results (freshness of existing competition).
opportunityScore	Composite score — see formula below.
opportunityTier	high or low, relative to opportunityThreshold.
topResultTitles	Titles of the top ranked results found for the term.

The opportunityScore formula, in plain language

opportunityScore = searchVolumeSignal / (log(existingVideoCount + 2) * freshnessPenalty)

• searchVolumeSignal (0–1): how much interest the topic has right now.
• log(existingVideoCount + 2): how saturated the topic already is. A log scale is used deliberately — the difference between 500 and 5,000 existing videos matters far less than the difference between 5 and 50. More existing videos pushes the score down.
• freshnessPenalty: rewards topics where existing coverage is old, not just topics with few videos.
  • Top results < 30 days old → penalty 1.5 (fresh competition, harder gap, score pulled down)
  • 30–180 days → 1.0 (neutral)
  • 180–365 days → 0.7 (getting stale, score pulled up)
  • > 365 days → 0.5 (stale, score pulled up more)
  • No recency data → 1.0 (no adjustment)

In short: high interest + few existing videos + old top results = highest score. Low interest + thousands of existing videos + freshly-uploaded top results = lowest score. The exact same formula is implemented (and commented) in src/scoring/opportunityScorer.ts.

What we found investigating Google Trends live

Google Trends has no official API. The /trends/explore page loads its interest-over-time chart from two internal, undocumented endpoints the page itself calls after load (/trends/api/explore for tokens, then /trends/api/widgetdata/multiline for the actual timeseries — both prefixed with a )]}', XSSI-protection line before the JSON body). This actor intercepts that second call directly via Playwright's page.on('response'), rather than scraping the rendered DOM.

Confirmed live during development: Google Trends returns a hard HTTP 429 immediately — on the very first request, with no prior request volume — to requests from this environment's outbound network (and by extension, typical cloud/datacenter IP ranges, which is what most Actor runs use by default). This was verified three independent ways: a raw curl against the rendered page, a raw curl against the internal /trends/api/explore endpoint, and a full Playwright browser run with response interception. All three hit the same 429 before any real data loaded. This matches Trends' well-known aggressive IP-reputation gating of non-residential traffic.

Because of this, the actor tries the real interception approach first on every run (so it self-heals automatically if Trends ever stops blocking this IP range, or if you configure a residential proxy), and falls back to a documented, clearly-labeled signal when blocked: autosuggest rank position. YouTube's own autosuggest ranks completions by observed popularity, so a term's position in the list it came from is a real, if less precise, interest proxy. Every output record's searchVolumeSource field tells you exactly which signal actually produced that row's score — nothing is silently substituted.

Pricing (Pay-Per-Event)

Event	Price	Charged when
apify-actor-start	$0.05	Run starts (built-in, one-time per run)
apify-default-dataset-item	$0.002	A real candidate sub-topic is generated from autosuggest expansion (built-in, automatic per dataset item)
low-opportunity-scored	$0.005	A candidate is fully scored and its opportunityScore falls below the threshold
high-opportunity-scored	$0.015	A candidate is fully scored and flagged as a real content gap (primary event — this is the actor's core value)

Notes

• maxRequestRetries is capped at 1 on every browser fetch in this actor (both the Trends fetcher and the YouTube search fetcher), with a 30s requestHandlerTimeoutSecs. A single blocked request fails fast and falls through to its documented fallback rather than compounding into minutes of retries.
• maxCandidates bounds total browser fetches per run — a broad niche won't spiral into dozens of slow fetches.