# YouTube Niche Gap Finder (`conceivable_extension/youtube-niche-gap-finder`) Actor

Given a niche or topic, surfaces YouTube sub-topics with rising search interest but low upload volume or stale top-result content — real content gaps, ranked by opportunity score.

- **URL**: https://apify.com/conceivable\_extension/youtube-niche-gap-finder.md
- **Developed by:** [joseph fadero](https://apify.com/conceivable_extension) (community)
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $15.00 / 1,000 sub-topic scored high opportunities

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## YouTube Niche Gap Finder

Given a niche or topic, surfaces YouTube sub-topics with rising search interest but low upload volume or stale top-result content — real content gaps, ranked by opportunity score.

### How it works

1. **Expand** — pulls real completion terms from YouTube's public autosuggest endpoint for your niche (and, optionally, for each seed/competitor channel).
2. **Signal** — checks each candidate sub-topic against Google Trends relative interest, via a real headless browser that intercepts Trends' own internal `widgetdata/multiline` network call (not DOM scraping).
3. **Supply** — runs a real YouTube search for each candidate and parses YouTube's own `ytInitialData` JSON to get the total result count and the upload-recency of the top ~10 ranked videos.
4. **Score** — combines interest vs. supply vs. freshness into one `opportunityScore` per sub-topic (formula below).

### Inputs

| Field | Default | Description |
|---|---|---|
| `niche` | `"AI explainers for Gen Z"` | The niche/topic to explore. Required. |
| `seedChannels` | `[]` | Optional competitor channel handles/URLs to benchmark against. |
| `region` | `"GB"` | Two-letter region code for autosuggest + search localization. |
| `maxCandidates` | `10` | Caps how many candidates are fully scored per run (each does a Trends fetch + a YouTube search fetch — keeps runtime/cost predictable). |
| `opportunityThreshold` | `0.5` | `opportunityScore` at/above this is flagged as a real gap (`high-opportunity-scored`); below it is `low-opportunity-scored`. |

### Output fields (per candidate sub-topic)

| Field | Description |
|---|---|
| `subTopic` | The candidate term. |
| `searchVolumeSignal` | 0–1 relative interest score. |
| `searchVolumeSource` | Where the signal came from — `trends-api` (real Google Trends data), `trends-fallback-autosuggest-rank` (Trends unreachable — see below), or `unavailable`. |
| `existingVideoCount` | Total YouTube search results for the term (supply). |
| `avgUploadRecencyDays` | Average days-since-upload across the top ~10 ranked results (freshness of existing competition). |
| `opportunityScore` | Composite score — see formula below. |
| `opportunityTier` | `high` or `low`, relative to `opportunityThreshold`. |
| `topResultTitles` | Titles of the top ranked results found for the term. |

### The `opportunityScore` formula, in plain language

```
opportunityScore = searchVolumeSignal / (log(existingVideoCount + 2) * freshnessPenalty)
```

- **`searchVolumeSignal`** (0–1): how much interest the topic has right now.
- **`log(existingVideoCount + 2)`**: how saturated the topic already is. A *log* scale is used deliberately — the difference between 500 and 5,000 existing videos matters far less than the difference between 5 and 50. More existing videos pushes the score down.
- **`freshnessPenalty`**: rewards topics where existing coverage is old, not just topics with few videos.
  - Top results < 30 days old → penalty **1.5** (fresh competition, harder gap, score pulled down)
  - 30–180 days → **1.0** (neutral)
  - 180–365 days → **0.7** (getting stale, score pulled up)
  - \> 365 days → **0.5** (stale, score pulled up more)
  - No recency data → **1.0** (no adjustment)

In short: **high interest + few existing videos + old top results = highest score.** Low interest + thousands of existing videos + freshly-uploaded top results = lowest score. The exact same formula is implemented (and commented) in `src/scoring/opportunityScorer.ts`.

### What we found investigating Google Trends live

Google Trends has no official API. The `/trends/explore` page loads its interest-over-time chart from two internal, undocumented endpoints the page itself calls after load (`/trends/api/explore` for tokens, then `/trends/api/widgetdata/multiline` for the actual timeseries — both prefixed with a `)]}',` XSSI-protection line before the JSON body). This actor intercepts that second call directly via Playwright's `page.on('response')`, rather than scraping the rendered DOM.

**Confirmed live during development:** Google Trends returns a hard HTTP 429 immediately — on the very first request, with no prior request volume — to requests from this environment's outbound network (and by extension, typical cloud/datacenter IP ranges, which is what most Actor runs use by default). This was verified three independent ways: a raw `curl` against the rendered page, a raw `curl` against the internal `/trends/api/explore` endpoint, and a full Playwright browser run with response interception. All three hit the same 429 before any real data loaded. This matches Trends' well-known aggressive IP-reputation gating of non-residential traffic.

Because of this, the actor tries the real interception approach first on every run (so it self-heals automatically if Trends ever stops blocking this IP range, or if you configure a residential proxy), and falls back to a documented, clearly-labeled signal when blocked: **autosuggest rank position**. YouTube's own autosuggest ranks completions by observed popularity, so a term's position in the list it came from is a real, if less precise, interest proxy. Every output record's `searchVolumeSource` field tells you exactly which signal actually produced that row's score — nothing is silently substituted.

### Pricing (Pay-Per-Event)

| Event | Price | Charged when |
|---|---|---|
| `apify-actor-start` | $0.05 | Run starts (built-in, one-time per run) |
| `apify-default-dataset-item` | $0.002 | A real candidate sub-topic is generated from autosuggest expansion (built-in, automatic per dataset item) |
| `low-opportunity-scored` | $0.005 | A candidate is fully scored and its `opportunityScore` falls below the threshold |
| `high-opportunity-scored` | $0.015 | A candidate is fully scored and flagged as a real content gap (primary event — this is the actor's core value) |

### Notes

- `maxRequestRetries` is capped at 1 on every browser fetch in this actor (both the Trends fetcher and the YouTube search fetcher), with a 30s `requestHandlerTimeoutSecs`. A single blocked request fails fast and falls through to its documented fallback rather than compounding into minutes of retries.
- `maxCandidates` bounds total browser fetches per run — a broad niche won't spiral into dozens of slow fetches.

# Actor input Schema

## `niche` (type: `string`):

The niche or topic to explore for content gaps, e.g. 'AI explainers for Gen Z'. Used to seed YouTube autosuggest expansion.

## `seedChannels` (type: `array`):

Optional competitor channel handles or URLs to benchmark against (e.g. '@channelname' or a full channel URL). Not required — the actor works from the niche alone.

## `region` (type: `string`):

Two-letter region/country code used for YouTube autosuggest and search localization, e.g. 'GB', 'US'.

## `maxCandidates` (type: `integer`):

Caps how many candidate sub-topics are fully scored per run (each one does a Trends fetch + a YouTube search fetch). Keeps runtime and cost predictable — a broad niche can otherwise expand into dozens of slow browser fetches.

## `opportunityThreshold` (type: `number`):

opportunityScore at or above this value is flagged as a real content gap (high-opportunity-scored). Below it, a candidate is still scored and returned but billed at the lower low-opportunity-scored rate.

## Actor input object example

```json
{
  "niche": "AI explainers for Gen Z",
  "seedChannels": [],
  "region": "GB",
  "maxCandidates": 10,
  "opportunityThreshold": 0.5
}
```

# Actor output Schema

## `resultsDatasetUrl` (type: `string`):

Dataset of scored sub-topic candidate records produced by this run.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "niche": "AI explainers for Gen Z",
    "seedChannels": [],
    "region": "GB"
};

// Run the Actor and wait for it to finish
const run = await client.actor("conceivable_extension/youtube-niche-gap-finder").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "niche": "AI explainers for Gen Z",
    "seedChannels": [],
    "region": "GB",
}

# Run the Actor and wait for it to finish
run = client.actor("conceivable_extension/youtube-niche-gap-finder").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "niche": "AI explainers for Gen Z",
  "seedChannels": [],
  "region": "GB"
}' |
apify call conceivable_extension/youtube-niche-gap-finder --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,conceivable_extension/youtube-niche-gap-finder"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/MmeqQLdNiv77xB9EZ/builds/kqzCTcb2miwgNr9fp/openapi.json