# YouTube Keyword Rank Tracker (`sarangofficial64/youtube-keyword-rank-tracker`) Actor Searches YouTube for a list of keywords and returns the ranked organic video results in the same order a real user would see them, for a given country. - **URL**: https://apify.com/sarangofficial64/youtube-keyword-rank-tracker.md - **Developed by:** [Sarang V](https://apify.com/sarangofficial64) (community) - **Categories:** Social media - **Stats:** 7 total users, 5 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing Pay per usage This Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage, which gets cheaper the higher subscription plan you have. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-usage ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js/docs.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python/docs.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## YouTube Keyword Rank Tracker (Phase 1 — Apify Actor) Data collection engine for a YouTube keyword rank tracking platform (Share of Search, à la Ahrefs/Semrush but for YouTube Search). This Actor's only job is to search YouTube for a list of keywords and return the ranked video results in the same top-to-bottom order a real, logged-out user would see them for a given country. Storage, analysis, and Share-of-Search math belong in the application layer, not here. Built with [Crawlee](https://crawlee.dev)'s `PlaywrightCrawler` on top of the Apify SDK. Uses real DOM scraping of the rendered search page (not the YouTube Data API), because the API's result set and ordering do not reliably match what users actually see in the UI — and matching the UI is the whole point. ### Table of contents 1. [How it works](#how-it-works) 2. [Input — full reference](#input--full-reference) 3. [Output — full reference](#output--full-reference) 4. [Computing Share of Search downstream](#computing-share-of-search-downstream) 5. [Why RESIDENTIAL proxy](#why-residential-proxy-not-datacenter-not-google-serp) 6. [Fresh-session / no cross-request bias guarantees](#fresh-session--no-cross-request-bias-guarantees) 7. [Exact counts, and what a short count means](#exact-counts-and-what-a-short-count-means) 8. [Fidelity: what a real user sees](#fidelity-what-a-real-user-sees) 9. [Known limitations (Phase 1 / MVP scope)](#known-limitations-phase-1--mvp-scope) 10. [Validating against real YouTube Incognito (India)](#validating-against-real-youtube-incognito-india) 11. [Local development](#local-development) 12. [Deploying](#deploying) --- ### How it works 1. For each keyword, build the **bare** search URL a real user would land on: `youtube.com/results?search_query=` — no `gl`/`hl`, no filters, no login. See [Fidelity](#fidelity-what-a-real-user-sees). 2. Load the page through a fresh Apify Proxy session tied to the target `country`, in a brand-new incognito browser context, with `locale`/`timezoneId`/fingerprint (and, for India, `geolocation`) set to match. 3. Accept the cookie-consent interstitial if YouTube shows one. 4. **Verify against YouTube's own `ytcfg`** that it resolved the target country and a logged-out session. If not, retire the session and retry — don't return foreign results. 5. Scroll the results feed until `maxResultsPerKeyword` **distinct** videos have loaded, waiting patiently for each continuation rather than guessing a fixed delay. 6. If the attempt came up short, retire the proxy session and retry the keyword on a fresh one. Real keywords do not run out (one was measured at **148+** videos), so a short count is treated as OUR failure and only YouTube's own continuation stream may ever declare a real shortage — see [Exact counts](#exact-counts-and-what-a-short-count-means). 7. Read the video results straight out of the DOM, in visual order — only top-level `ytd-video-renderer` items are captured (actual video results; playlists, channels, mixes, and nested Shorts shelves are skipped so that "position" has one unambiguous meaning: rank among distinct video results). 8. Push one dataset row per video, and one entry per keyword into `KEYWORD_SUMMARIES` recording the status, every attempt's count, and YouTube's resolved locale. Source layout: | File | Responsibility | |---|---| | [`src/main.js`](src/main.js) | Actor entry point: reads input, builds the crawler, orchestrates the run | | [`src/searchPage.js`](src/searchPage.js) | Page-level mechanics: URL building, consent dismissal, scrolling, DOM extraction | | [`src/proxyCheck.js`](src/proxyCheck.js) | Proxy geolocation self-check and input sanity warnings | | [`.actor/input_schema.json`](.actor/input_schema.json) | Defines the Actor's input form on Apify Console | | [`.actor/actor.json`](.actor/actor.json) | Actor manifest (name, version, entrypoints) | | [`AUDIT_PROMPT.md`](AUDIT_PROMPT.md) | Self-contained brief for an **independent** auditor to verify the output is genuinely what an India incognito user sees. Includes the traps that produce a false PASS — read it before trusting any result, including your own. | --- ### Input — full reference Input is a single JSON object (this is what fills the form on Apify Console, or the body of an API "Run Actor" call). Full example: ```json { "keywords": ["best women trimmer", "python for beginners"], "country": "IN", "language": "en", "maxResultsPerKeyword": 20, "proxyConfiguration": { "useApifyProxy": true, "apifyProxyGroups": ["RESIDENTIAL"], "apifyProxyCountry": "IN" }, "maxRequestRetries": 5, "navigationTimeoutSecs": 90, "resultsTimeoutSecs": 45, "loadBudgetSecs": 240 } ```` #### Field-by-field | Field | Type | Required | Default | Notes | |---|---|---|---|---| | `keywords` | `string[]` | **Yes** | — | One YouTube search is run per entry. Duplicates and blank/whitespace-only entries are silently removed before running. At least one non-empty keyword is required or the run fails immediately with a clear error. No hard cap in code, but very large lists (hundreds+) will take proportionally longer and use proportionally more proxy traffic — see [Local development](#local-development) for timing estimates. | | `country` | `string` | No | `"IN"` | ISO 3166-1 alpha-2 code (e.g. `IN`, `US`, `GB`). Feeds three things: the browser `locale`/`Accept-Language` (`-`) and fingerprint, the proxy's `apifyProxyCountry` default if you don't set `proxyConfiguration` explicitly, and the country the Actor **verifies against YouTube's own resolved `GL`** on every page (a mismatch retires the session and retries rather than returning foreign results). It is deliberately **not** sent as a `gl` URL parameter — see [Fidelity](#fidelity-what-a-real-user-sees). Only `IN` currently gets a tuned `timezoneId`/`geolocation` profile (see `COUNTRY_PROFILES` in `src/main.js`). | | `language` | `string` | No | `"en"` | BCP-47-ish language code. Combined with `country` into the browser locale (`en-IN`) used for the fingerprint and `Accept-Language`. Not sent as an `hl` URL parameter; YouTube resolves `HL` itself (for India it resolves `en-IN`, not a bare `en`). | | `maxResultsPerKeyword` | `integer` | No | `20` | How many **distinct** video results to collect per keyword, min `1`, max `200` (enforced by the input schema on Apify Console; not hard-enforced via raw API). This is a guarantee, not a target: YouTube does not run out of results for real keywords, so a short count is reported as a loud `FAILED_INCOMPLETE` failure rather than as a fact about YouTube — see [Exact counts](#exact-counts-and-what-a-short-count-means). Counting is by distinct video ID because YouTube genuinely repeats videos across continuation batches. | | `proxyConfiguration` | `object` | **Yes** | `{"useApifyProxy": true, "apifyProxyGroups": ["RESIDENTIAL"], "apifyProxyCountry": ""}` if omitted | Standard [Apify Proxy configuration object](https://docs.apify.com/platform/proxy). On Apify Console this renders as the built-in proxy picker UI. See [Why RESIDENTIAL proxy](#why-residential-proxy-not-datacenter-not-google-serp) for why the default is what it is. Passing `{"useApifyProxy": false}` disables the proxy entirely (requests exit from wherever the Actor is physically running) — useful only for local mechanics testing, never for real India validation. | | `maxRequestRetries` | `integer` | No | `5` | How many **extra attempts, each with a fresh proxy session**, a keyword gets. Spent on *measuring* YouTube, and on nothing else: sessions that never rendered, resolved the wrong country, or got cut off mid-feed are **refunded** (up to 6) so bad luck can't eat the budget. Raising this is the main lever when a keyword comes back `FAILED_INCOMPLETE`. Retries are only spent when needed — a keyword that hits its target on attempt 1 costs one attempt. | | `navigationTimeoutSecs` | `integer` | No | `90` | Per-attempt timeout (seconds) for the search page to finish its initial load. Residential proxies are slow; don't set this low. | | `resultsTimeoutSecs` | `integer` | No | `45` | How long to wait for the first video result to *render* after the page loads. YouTube ships ~1MB of HTML and then builds the results client-side, so a slow session can deliver the page but never hydrate it. On expiry the attempt is abandoned, a `DIAG-*` snapshot (screenshot + element counts + resolved locale) is written to the key-value store, and the keyword is retried on a fresh session. | | `loadBudgetSecs` | `integer` | No | `240` | Max time one attempt may spend scrolling. YouTube renders results a few at a time (the first paint holds only ~4), so 50 results takes ~25s on a fast line and longer through a residential proxy. Raise this for large `maxResultsPerKeyword`. | #### Notes on `proxyConfiguration` This is the one field worth understanding in more depth beyond the table above — see [Why RESIDENTIAL proxy](#why-residential-proxy-not-datacenter-not-google-serp). In short: always use `groups: ["RESIDENTIAL"]` with `country` matching the top-level `country` input field. *** ### Output — full reference Output is split across two places, deliberately: - The **dataset** (Console: run → *Output* tab) holds **only video results** — every row has the exact same shape. This is what you export as JSON/CSV/Excel, and what a rank tracker actually needs. - The **key-value store** (Console: run → *Storage* → *Key-value store* tab) holds per-run **diagnostics**: proxy verification and per-keyword success/failure summaries. These aren't results you'd scroll through row by row, so they don't live in the dataset — mixing row shapes there just produces a wall of blank/"undefined" columns in Apify Console's table view. #### The dataset: one row per video Rows are **denormalized**: `keyword`, `country`, `language`, and `searchedAt` are repeated on every row so each row is independently meaningful — you don't need to join against anything else to know which keyword/run a row belongs to. `position` is 1-based rank, top to bottom, in the exact order the video appeared on the page. ```json { "keyword": "best women trimmer", "country": "IN", "language": "en", "searchedAt": "2026-07-15T07:58:05.192Z", "position": 1, "youtubeResolvedGl": "IN", "youtubeResolvedHl": "en-IN", "youtubeLoggedIn": false, "title": "Best Trimmer For Women in India⚡Facial, Underarm & Private Hair Removal Made Easy", "videoId": "AbCdEfGhIjK", "videoUrl": "https://www.youtube.com/watch?v=AbCdEfGhIjK&pp=...", "channelName": "Product Bazar", "channelUrl": "https://www.youtube.com/@ProductBazar", "channelId": null, "channelHandle": "@ProductBazar", "viewCountText": "72K views", "publishedTimeText": "11 months ago", "lengthText": "6:19", "thumbnailUrl": "https://i.ytimg.com/vi/AbCdEfGhIjK/hqdefault.jpg", "isAd": false } ``` | Field | Type | Always present? | Description | |---|---|---|---| | `keyword` | `string` | Yes | The search query this result came from. | | `country` | `string` | Yes | Echo of the input `country` for this run. | | `language` | `string` | Yes | Echo of the input `language` for this run. | | `searchedAt` | `string` (ISO 8601) | Yes | Timestamp the page was scraped — shared across all rows for the same keyword. | | `position` | `integer` | Yes | 1-based rank among **distinct** video results, top to bottom. This is the number a rank tracker cares about. YouTube repeats some videos further down the feed; repeats are collapsed to their first (highest) appearance so a repeat can't push everything below it down a slot. | | `youtubeResolvedGl` | `string \| null` | Yes | The country **YouTube itself** resolved for this session, read from its `ytcfg` — not an echo of your input. Must be `"IN"` for a valid India run. | | `youtubeResolvedHl` | `string \| null` | Yes | The locale YouTube itself resolved, e.g. `"en-IN"`. Note this is `en-IN`, not a bare `en` — that is what a real Indian user gets. | | `youtubeLoggedIn` | `boolean \| null` | Yes | Must be `false`. Proof the results carry no account personalization. | | `title` | `string \| null` | Almost always | Video title. `null` only if the title element itself failed to parse (rare/layout-change indicator). | | `videoId` | `string` | Yes | YouTube's 11-character video ID, extracted from the result's link. Results without a parsable ID are dropped (they can be neither ranked nor de-duplicated) and counted in `unidentifiableItemsSkipped`. | | `videoUrl` | `string \| null` | Almost always | Full `youtube.com/watch?v=...` URL (includes YouTube's internal tracking query params, harmless to keep or strip). | | `channelName` | `string \| null` | Usually | Display name of the channel/byline, as text. For multi-channel co-attributed videos (e.g. two channels credited on one video), this can be a combined string like `"Channel A and Channel B"` with no matching `channelId`/`channelHandle` — see [Known limitations](#known-limitations-phase-1--mvp-scope). | | `channelUrl` | `string \| null` | Usually | Full URL to the channel, whichever form YouTube linked (handle or ID based). `null` for the multi-attribution case above. | | `channelId` | `string \| null` | **Rarely** | The stable `UC...` channel ID, **only populated when YouTube happens to link the channel via `/channel/UC…`**. Most current search results link via handle instead — see `channelHandle`. | | `channelHandle` | `string \| null` | Usually | The `@handle`-style channel identifier (e.g. `"@ProductBazar"`), extracted whenever `channelId` isn't available. **Use this as the primary channel-matching key for MVP**, not `channelId`. | | `viewCountText` | `string \| null` | Usually | View count exactly as displayed, e.g. `"72K views"`, `"2.1 crore views"` (India-locale numbering), or `"11K watching"` for live streams. Deliberately left as display text, not parsed to a number, since formats vary by locale/live-status — parse downstream if you need a numeric value. | | `publishedTimeText` | `string \| null` | Usually | Relative publish time as displayed, e.g. `"11 months ago"`, `"3 years ago"`. `null` for some live/premiere states where YouTube doesn't show this. | | `lengthText` | `string \| null` | Usually | Video duration as displayed, e.g. `"6:19"`, `"1:02:21"`. `null` for livestreams/premieres, which don't show a fixed duration badge. | | `thumbnailUrl` | `string \| null` | Almost always | Deterministically constructed as `https://i.ytimg.com/vi//hqdefault.jpg` (not scraped from the page — see [Known limitations](#known-limitations-phase-1--mvp-scope) for why). `null` only if `videoId` itself is `null`. | | `isAd` | `boolean` | Yes | `true` if the result carried an "Ad" badge (in-feed promoted video). These are still included in the ranked `position` sequence since that's what a real user visually scrolls past — filter them out downstream if your Share of Search should be organic-only. | #### Key-value store: `KEYWORD_SUMMARIES` One entry, written once at the end of the run: an array with one object per keyword, telling you what actually happened for it. This is the **true denominator** for Share of Search — don't assume it always equals `maxResultsPerKeyword`, and always check `status` before trusting a keyword's numbers. ```json [ { "status": "exact", "keyword": "hair styler machine use", "country": "IN", "language": "en", "searchedAt": "2026-07-15T13:41:02.417Z", "totalVideoResults": 50, "requestedMaxResults": 50, "attempts": 1, "attemptCounts": [50], "sessionsUsed": ["session_Md2pIhxPJE"], "youtubeResolvedGl": "IN", "youtubeResolvedHl": "en-IN", "youtubeLoggedIn": false, "duplicatesSkipped": 0, "unidentifiableItemsSkipped": 0, "nonVideoBlocksSeen": { "shelves": 0, "reelShelves": 0 }, "sourceUrl": "https://www.youtube.com/results?search_query=hair+styler+machine+use" }, { "status": "FAILED_INCOMPLETE", "keyword": "zxqwjk vlurbn qqpzt fictional nonsense phrase 9182", "totalVideoResults": 21, "requestedMaxResults": 50, "attempts": 4, "attemptCounts": [21, 18, 15, 16], "note": "Could not collect 50 results after 4 measured attempt(s) [21,18,15,16]. YouTube never signalled end-of-results, so it almost certainly HAS more - this is a scraper/proxy failure, NOT a fact about YouTube. ..." } ] ``` | Field | Present when | Description | |---|---|---| | `status` | Always | See the four values below. **Check this before trusting a count.** | | `keyword`, `country`, `language`, `searchedAt` | Always | Same meaning as in dataset rows. | | `totalVideoResults` | Always | Number of dataset rows pushed for this keyword — the correct Share-of-Search denominator. `0` when `status` is `error`. | | `requestedMaxResults` | Always | Echo of the input `maxResultsPerKeyword`. | | `attempts` / `attemptCounts` / `sessionsUsed` | Always | How many independent proxy sessions were used and what each one returned. **The spread here is the honest confidence interval.** `[50]` is a clean first-attempt hit; `[21, 18, 15, 16]` means the sessions disagreed and the number should not be treated as a fact about YouTube. | | `youtubeResolvedGl` / `youtubeResolvedHl` / `youtubeLoggedIn` | Always | What **YouTube itself** resolved for the session that produced these rows, read from its own `ytcfg`. For a valid India run these must read `IN` / `en-IN` / `false`. This is the fidelity receipt, not our assumption. | | `duplicatesSkipped` | Always | Repeats collapsed so `position` stays a clean rank. A number comparable to `totalVideoResults` is a throttling signal. | | `errorMessage` | `error` / salvaged rows | Human-readable failure reason. | | `sourceUrl` | Always | The bare YouTube search URL — paste it into a browser in-country to re-check by hand. | ##### `status` values | Value | Meaning | Trust the count? | |---|---|---| | `exact` | Got exactly `maxResultsPerKeyword` distinct videos. | **Yes.** | | `short_content_shortage` | Fewer than requested, and **YouTube's own continuation stream ended with no further page** on ≥2 independent sessions. | **Only for nonsense/ultra-niche queries.** On a normal keyword this should never appear — YouTube does not run out (one ordinary keyword measured **148+** and was still going). If you see it on a real keyword, verify by hand. | | `FAILED_INCOMPLETE` | Fewer than requested, and **YouTube never signalled end-of-results** — so it almost certainly has more. A throttled/slow proxy beat us. | **No — this is not a measurement.** The count is an artifact of our failure, not a property of YouTube. Re-run the keyword, or raise `maxRequestRetries` / `loadBudgetSecs`. | | `error` | The keyword never produced a usable page across every attempt. No dataset rows. | No rows to trust; see `errorMessage` and any `DIAG-*` records. | **Downstream handling tip**: use only `status: "exact"` (and, for genuinely niche queries you've verified by hand, `short_content_shortage`) when computing Share of Search. **Never** feed a `FAILED_INCOMPLETE` count into a denominator: it is a scraper failure wearing the costume of a small result set, and using it will silently overstate that keyword's share. ##### Why the bar for "content shortage" is set so high Earlier versions let **session agreement** prove a shortage. That was actively dangerous. Measured on Apify: three throttled sessions stalled at 13, 19 and 21 results and "agreed", so the run reported a **confirmed content shortage of 21** for a keyword that returns **35+** on a healthy session. Three wrong answers that match are still three wrong answers — agreement between throttled sessions measures the throttling, not the catalogue. So agreement can no longer prove anything by itself. The only admissible proof that results ran out is **YouTube saying so**, via its continuation stream (see [Exact counts](#exact-counts-and-what-a-short-count-means)). Everything else is reported as our failure, loudly, because on a real keyword that is exactly what it is. #### Other key-value store entries | Key | Written | Purpose | |---|---|---| | `PROXY_GEO_CHECK` | Once per run, before any keyword is scraped | Pre-flight proof of where the run's proxy actually exits from — see [Why RESIDENTIAL proxy](#why-residential-proxy-not-datacenter-not-google-serp). Shape: `{ expectedCountry, checked, matched, geo: { country, city, ip, org }, endpoint }`. A real India run looks like `{ "matched": true, "geo": { "country": "IN", "city": "Delhi", "org": "AS55836 Reliance Jio Infocomm Limited" } }`. This is a plain HTTP check, not a browser launch, and it is **advisory only** — if every geolocation endpoint is rate-limited it reports `checked: false` and the run continues, because the authoritative gate is YouTube's own resolved `GL` on each page (`youtubeResolvedGl`). | | `DIAG--a` and `DIAG--a-SHOT` | Only when an attempt's page never rendered results | Why an attempt failed, instead of leaving you to guess: resolved locale, element counts, page title, body-text snippet, and a JPEG screenshot. Typical throttled-session signature: ~1MB of `htmlLength` and a valid `GL: "IN"` (so YouTube served the real page) but every element count `0` — i.e. the page arrived and never hydrated. That is a slow/bad session, and the fresh-session retry is the fix. | *** ### Computing Share of Search downstream For a target channel, per keyword: ``` share_of_search(keyword) = count(dataset rows for that keyword where channelId/channelHandle == target) / totalVideoResults (from KEYWORD_SUMMARIES) ``` Overall SOS = average of `share_of_search(keyword)` across all keywords whose `status` in `KEYWORD_SUMMARIES` is `"exact"` or `"short_content_shortage"` — this averaging (not a single weighted ratio) matches the platform spec's example: `(10% + 25% + 0%) / 3 = 11.67%`. Exclude `"error"` **and `"FAILED_INCOMPLETE"`**. A `FAILED_INCOMPLETE` keyword is a measurement failure, not a small denominator: dividing by its count would silently overstate that keyword's share. YouTube does not run out of results for real keywords, so a short count there means the scraper lost, not that the topic is thin. *** ### Why RESIDENTIAL proxy (not datacenter, not Google SERP) Apify offers three proxy types. For this Actor, `RESIDENTIAL` is the right one, not a default picked out of convenience: - **`RESIDENTIAL`** (used here) — real IP addresses from home/mobile ISPs in the target country. This is what actually makes YouTube's location detection believe the request is a genuine visitor from India, the same way a real user's ISP-assigned IP would. Slower and more expensive per GB than datacenter, but it's the only option that reliably reproduces India-specific ranking, ad mix, and number formatting (e.g. views shown in lakh/crore rather than raw counts). - **`DATACENTER`** — cheap, fast, but the IP ranges are well-known hosting provider blocks. YouTube is more likely to rate-limit or serve a generic/non-localized result set to these IPs regardless of the `gl` country parameter you pass, since IP geolocation is a stronger signal than URL parameters. - **`GOOGLE_SERP`** — purpose-built for `google.com/search`; per Apify's own docs, its country-routing only activates for hostnames starting with `google`. It does **not** apply to `youtube.com`, so it's not an option here at all. #### Proving the proxy is actually India, every run Trusting the `country: "IN"` input field alone isn't enough — proxy pools can have partial coverage, and it's easy to accidentally leave `useApifyProxy: false` or the wrong group selected. So before scraping any keywords, the Actor: 1. Makes one plain HTTP request through the configured proxy to a geolocation-IP service (`ipapi.co`, falling back to `ipinfo.io`) and asks where it's really exiting from. A real India run reports e.g. `exit IP 157.49.13.16 is in IN (Delhi, AS55836 Reliance Jio Infocomm Limited)`. 2. Logs `PROXY CHECK OK` if it matches, or a loud `PROXY CHECK MISMATCH` / `PROXY CHECK SKIPPED` / `PROXY CHECK INCONCLUSIVE` warning otherwise. 3. Saves the result to the run's key-value store under `PROXY_GEO_CHECK`. 4. Separately, warns immediately (before even trying the network check) if the input's `proxyConfiguration` isn't using `RESIDENTIAL` or its country doesn't match the `country` input field. This pre-flight is **advisory only**, for two reasons learned the hard way: - It used to launch its own throwaway Chromium, which silently failed on *every* production run (`Executable doesn't exist at /pw-browsers/…`) because the installed Playwright didn't match the browsers baked into the base image. The check existed but only ever reported "couldn't check", and nobody noticed. It's now a browser-free HTTP call with no version coupling. - Even when it worked, it verified a *different* browser session than the ones used for scraping, and geolocation endpoints themselves get rate-limited (observed: a 200 response with no country field, reported as a terrifying "resolves to undefined, NOT IN"). The **authoritative** gate is therefore per-page, not pre-flight: every scraped page must have YouTube's own `ytcfg` resolve to the target `GL` and a logged-out session, or that session is retired and the keyword retried. See [Fidelity](#fidelity-what-a-real-user-sees). None of these checks abort the run — they're diagnostics, since a proxy hiccup shouldn't silently waste the rest of a multi-keyword run. Always check the run log / `PROXY_GEO_CHECK` before trusting a given run's data for India validation. *** ### Fresh-session / no cross-request bias guarantees Every keyword search is designed to look like a brand-new, logged-out Incognito window — never influenced by any earlier search in the same run. This was verified empirically (not just assumed) while building the Actor: | Mechanism | What it prevents | How it's enforced | |---|---|---| | `launchContext.useIncognitoPages: true` | Cookies/localStorage/sessionStorage/IndexedDB carrying over between requests | Crawlee actually **defaults this to `false`**, which was caught via a local test: a cookie set on one request was still readable on the next by default. Explicitly set `true` in `src/main.js`; re-tested and confirmed clean (cookies, localStorage, sessionStorage, and IndexedDB all came back empty on a follow-up request after being written on a prior one). | | `persistCookiesPerSession: false` | Crawlee's own session-pool layer re-injecting a request's cookies into a *later* request that reuses the same proxy session | Crawlee defaults this to `true` when session pooling is on (the default). Explicitly disabled. | | Apify `SessionPool` default (`maxPoolSize: 1000`) | The same proxy IP being reused across many keyword searches in one run | Not modified — Crawlee's own default already hands out a brand-new session (and thus a new residential IP) for every request until 1000 sessions exist, which no realistic keyword list will reach. | | No login, ever | Any personalized/watch-history-influenced ranking | The Actor never authenticates to YouTube; every search is anonymous by construction. | Net effect: no cookies, no cached identifiers, no shared browser storage, and (in practice) no shared IP carry across keyword searches within a run. *** ### Exact counts, and what a short count means `maxResultsPerKeyword` is a **guarantee, not a target**. If you ask for 50 you get 50 distinct videos, unless YouTube genuinely does not have 50 for that query — and the Actor is not allowed to simply *assume* which of those two it is hit. That distinction is hard, because **YouTube gives no usable end-of-results signal**. This was measured, not guessed: - The first paint holds only ~**4** results; the rest stream in **1–4 at a time** as you scroll. Reaching 50 takes ~25s on a fast line, longer through a residential proxy. - At genuine exhaustion the loading spinner (`ytd-continuation-item-renderer`) **stays present and visible forever**, and no "no more results" message appears. A stalled feed and an exhausted feed look **identical** in the DOM. Verified by holding an exhausted query at a hard plateau for 3+ minutes. - The same exhausted query returned **21 results one session and 22 the next**, so even "genuine" counts are not perfectly stable. **Crucially, YouTube does not run out of results for real keywords.** One ordinary commercial keyword was measured at **148 distinct videos and was still producing** when the test stopped. So on a normal keyword, a short count is this Actor failing — it is *not* a fact about YouTube, and must never be reported as one. The only thing that may ever prove results ran out is **YouTube's own continuation stream**. Each extra page arrives as a `/youtubei/v1/search` response that either offers a further page or doesn't (verified on an exhausted query: its responses ran `[true × 7, null]` — the last carried no continuation). So: - YouTube's last word was **"more available"** and results stopped anyway → `cut_off`. We were throttled. The session is retired and the keyword retried, and this can **never** count toward a shortage. - YouTube offered **no further page** → genuine end. Still requires two independent sessions to agree before it is reported as `short_content_shortage`. - Anything else short → **`FAILED_INCOMPLETE`**, reported loudly as our failure. Session agreement alone proves nothing — see [Why the bar is set so high](#why-the-bar-for-content-shortage-is-set-so-high). Two sessions are additionally disqualified from ever corroborating a shortage: - **Throttled sessions.** When YouTube throttles a session it stops serving fresh continuations and starts re-serving videos it already sent — the feed plateaus early while padding itself with repeats. Observed directly: one session plateaued at 16 distinct videos with 8 repeats, while the very next fresh session reached the full 50. A throttled plateau says nothing about YouTube's supply. - **Sessions beaten by another session.** If any attempt reached a higher count, a lower plateau cannot corroborate a shortage — something clearly had more to give. This is the whole point: a proxy problem must never be laundered into a "fact" about YouTube. Reporting 11 results as a content shortage when YouTube has 50 is the single worst failure this Actor can make, so when it cannot prove a shortage, it says so instead of guessing. ### Fidelity: what a real user sees A real user physically in India opens youtube.com and types a keyword. Their URL has **no `gl`/`hl` parameters**, so the Actor doesn't send any either — it requests the bare `https://www.youtube.com/results?search_query=…`. That's safe because YouTube resolves the locale itself from the exit IP, which was verified on a real Indian residential line: a bare URL resolved to `GL=IN`, `HL=en-IN`, `LOGGED_IN=false`. Forcing `hl=en&gl=IN` (as this Actor used to) was both a deviation from that real URL *and* actively wrong — it pinned `HL` to a bare `en` when a real Indian user resolves to `en-IN`. Rather than assume any of this held, every scraped page is checked against **YouTube's own resolved state** (`ytcfg`), and the answer is recorded on every row as `youtubeResolvedGl` / `youtubeResolvedHl` / `youtubeLoggedIn`. If YouTube resolves a country other than the target, or reports a logged-in session, the session is retired and the keyword retried — unfaithful results are never returned. Independent corroboration that this works: view counts come back in Indian numbering (`"3.4 lakh views"`, `"2.1 crore views"`). One consequence worth knowing: `country`/`language` inputs shape the browser locale, fingerprint and proxy, but they are **requests, not commands**. The exit IP is what actually decides, and `youtubeResolvedGl` is what actually happened. *** ### Known limitations (Phase 1 / MVP scope) - **Channel matching**: search results overwhelmingly link channels via `/@handle` now, not the stable `/channel/UC…` ID. Match on `channelHandle` for MVP; resolving a handle to its canonical `channelId` (via one extra page load per unique channel) is a reasonable Phase 2 addition if handles ever change. A small fraction of results (multi-channel co-attributed videos) have neither populated — see the `video` row field table above. - **Shorts shelves and ads**: only top-level video results are ranked. Shorts shelves and in-feed ad units that appear interleaved in the results are currently skipped, not because they're unimportant, but because including them raises open questions (does a Shorts shelf count as one "slot" or as N videos?) that should be a deliberate product decision, not an artifact of the scraper. In-feed ads that render as a normal `ytd-video-renderer` (i.e. inline promoted videos) **are** captured, with `isAd: true`. - **Result variance**: YouTube's organic ranking is not perfectly deterministic between two back-to-back runs of the same keyword — expect minor position drift even with identical proxy/locale settings. Plan for the platform to track trends over multiple runs, not treat any single run as ground truth. - **Thumbnail images and video length are best-effort**: `lengthText` is `null` for livestreams/premieres (no fixed duration to show), and `thumbnailUrl` is derived from `videoId` rather than scraped (image loading is intentionally blocked for speed — see `src/main.js`). *** ### Validating against real YouTube Incognito (India) This is the actual Phase 1 acceptance test — do this before building anything on top of the Actor's output: 1. Push this project to Apify (`apify login`, then `apify push` from this directory) and run it on the platform with `proxyConfiguration` set to Apify Proxy, `RESIDENTIAL` group, `IN` country, and 3-5 representative keywords. 2. Check the run log for `PROXY CHECK OK` (or the `PROXY_GEO_CHECK` key-value store entry) to confirm the run's proxy really exited from India before trusting anything else about it. 3. On a separate machine/VPN with a real Indian IP, open an Incognito window and run the same searches manually on youtube.com. 4. Compare the Actor's `video` rows (in `position` order) for each keyword against what you see on-screen in Incognito, checking: same videos, same top-to-bottom order, same channels. 5. Expect close alignment on the first ~10-15 results; beyond that, organic variance (per [Known limitations](#known-limitations-phase-1--mvp-scope)) is normal. If entire keywords come back empty or the top results look US/generic rather than India-specific, the proxy/locale isn't taking effect — recheck step 2 first. *** ### Local development ```bash npm install npx playwright install --with-deps chromium # first time only apify run # or: node src/main.js, using storage/key_value_stores/default/INPUT.json ``` Local runs without `apify login` won't have Apify Proxy access, so `proxyConfiguration` can be set to `{ "useApifyProxy": false }` for a quick mechanics smoke test — but that hits YouTube from whatever IP the machine running it has, so it's not a substitute for the India-proxy validation above unless you already know that machine's IP is genuinely Indian. Rough timing observed in local testing: ~10-15 seconds per keyword (dominated by the initial page load + scroll-to-load-more), largely independent of `maxResultsPerKeyword` for values under ~20 since most results load in the first page without needing many scroll rounds. ### Deploying ```bash apify login apify push ``` # Actor input Schema ## `keywords` (type: `array`): List of search keywords to look up on YouTube. One search will be run per keyword. ## `country` (type: `string`): Country to simulate the search from. Sets the Apify Proxy exit country and the browser locale/fingerprint, and is the country the Actor VERIFIES against YouTube's own resolved GL on every page (a mismatch retires the session and retries). It is deliberately NOT sent as a 'gl' URL parameter - a real user typing a keyword has no such parameter, and YouTube resolves the country from the exit IP by itself. Use the ISO 3166-1 alpha-2 code, e.g. IN for India. ## `language` (type: `string`): Language hint, combined with Country into the browser locale (e.g. en + IN = en-IN) used for the fingerprint and Accept-Language header. Not sent as an 'hl' URL parameter: YouTube resolves HL itself, and for a real Indian user it resolves 'en-IN', not a bare 'en'. ## `maxResultsPerKeyword` (type: `integer`): How many DISTINCT video results to collect per keyword, counted in the same top-to-bottom order a user would scroll through. Shorts are not counted (they are ignored by design), and videos YouTube repeats further down the feed are collapsed to their first appearance. This is a guarantee, not a target: if fewer are returned, the Actor will not call it a content shortage unless two independent proxy sessions confirm it - check 'status' in KEYWORD\_SUMMARIES. ## `proxyConfiguration` (type: `object`): Required. Use Apify Proxy with the 'RESIDENTIAL' group and the matching country selected below for the most accurate, geo-consistent results. ## `maxRequestRetries` (type: `integer`): How many extra attempts (each with a FRESH proxy session) a keyword gets. Used both to recover from a failed page load and to corroborate a short result count: a shortfall is only reported as a real content shortage once two independent sessions independently run out of results at the same count. Lower values make short counts less trustworthy. ## `navigationTimeoutSecs` (type: `integer`): Max time to wait for the YouTube search page to load per attempt. Residential proxies can be slow; do not set this too low. ## `resultsTimeoutSecs` (type: `integer`): Max time to wait for the first video result to render after the page loads. If it expires, the attempt is abandoned, a diagnostic snapshot (screenshot + page state) is stored in the key-value store under DIAG-\*, and the keyword is retried with a fresh proxy session. ## `loadBudgetSecs` (type: `integer`): Max time to spend scrolling to load results for one attempt. YouTube India result pages are dominated by Shorts (measured: 160 Shorts vs 35 videos on one keyword), and the video count can sit frozen for several scrolls while a block of Shorts streams in, so reaching a high target legitimately takes many scrolls. Raise this for large maxResultsPerKeyword values; a run that stops on this budget is reported as a shortfall, never as a content shortage. ## Actor input object example ```json { "keywords": [ "how to tie a tie", "python for beginners" ], "country": "IN", "language": "en", "maxResultsPerKeyword": 20, "proxyConfiguration": { "useApifyProxy": true, "apifyProxyGroups": [ "RESIDENTIAL" ], "apifyProxyCountry": "IN" }, "maxRequestRetries": 5, "navigationTimeoutSecs": 90, "resultsTimeoutSecs": 45, "loadBudgetSecs": 240 } ``` # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "keywords": [ "how to tie a tie", "python for beginners" ], "country": "IN", "language": "en", "maxResultsPerKeyword": 20, "proxyConfiguration": { "useApifyProxy": true, "apifyProxyGroups": [ "RESIDENTIAL" ], "apifyProxyCountry": "IN" } }; // Run the Actor and wait for it to finish const run = await client.actor("sarangofficial64/youtube-keyword-rank-tracker").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "keywords": [ "how to tie a tie", "python for beginners", ], "country": "IN", "language": "en", "maxResultsPerKeyword": 20, "proxyConfiguration": { "useApifyProxy": True, "apifyProxyGroups": ["RESIDENTIAL"], "apifyProxyCountry": "IN", }, } # Run the Actor and wait for it to finish run = client.actor("sarangofficial64/youtube-keyword-rank-tracker").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "keywords": [ "how to tie a tie", "python for beginners" ], "country": "IN", "language": "en", "maxResultsPerKeyword": 20, "proxyConfiguration": { "useApifyProxy": true, "apifyProxyGroups": [ "RESIDENTIAL" ], "apifyProxyCountry": "IN" } }' | apify call sarangofficial64/youtube-keyword-rank-tracker --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=sarangofficial64/youtube-keyword-rank-tracker", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "YouTube Keyword Rank Tracker", "description": "Searches YouTube for a list of keywords and returns the ranked organic video results in the same order a real user would see them, for a given country.", "version": "0.1", "x-build-id": "TjuK8do19sbx0iaya" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/sarangofficial64~youtube-keyword-rank-tracker/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-sarangofficial64-youtube-keyword-rank-tracker", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/sarangofficial64~youtube-keyword-rank-tracker/runs": { "post": { "operationId": "runs-sync-sarangofficial64-youtube-keyword-rank-tracker", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/sarangofficial64~youtube-keyword-rank-tracker/run-sync": { "post": { "operationId": "run-sync-sarangofficial64-youtube-keyword-rank-tracker", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "required": [ "keywords", "proxyConfiguration" ], "properties": { "keywords": { "title": "Keywords", "type": "array", "description": "List of search keywords to look up on YouTube. One search will be run per keyword.", "items": { "type": "string" } }, "country": { "title": "Country", "type": "string", "description": "Country to simulate the search from. Sets the Apify Proxy exit country and the browser locale/fingerprint, and is the country the Actor VERIFIES against YouTube's own resolved GL on every page (a mismatch retires the session and retries). It is deliberately NOT sent as a 'gl' URL parameter - a real user typing a keyword has no such parameter, and YouTube resolves the country from the exit IP by itself. Use the ISO 3166-1 alpha-2 code, e.g. IN for India.", "default": "IN" }, "language": { "title": "Language", "type": "string", "description": "Language hint, combined with Country into the browser locale (e.g. en + IN = en-IN) used for the fingerprint and Accept-Language header. Not sent as an 'hl' URL parameter: YouTube resolves HL itself, and for a real Indian user it resolves 'en-IN', not a bare 'en'.", "default": "en" }, "maxResultsPerKeyword": { "title": "Max results per keyword", "minimum": 1, "maximum": 200, "type": "integer", "description": "How many DISTINCT video results to collect per keyword, counted in the same top-to-bottom order a user would scroll through. Shorts are not counted (they are ignored by design), and videos YouTube repeats further down the feed are collapsed to their first appearance. This is a guarantee, not a target: if fewer are returned, the Actor will not call it a content shortage unless two independent proxy sessions confirm it - check 'status' in KEYWORD_SUMMARIES.", "default": 20 }, "proxyConfiguration": { "title": "Proxy configuration", "type": "object", "description": "Required. Use Apify Proxy with the 'RESIDENTIAL' group and the matching country selected below for the most accurate, geo-consistent results.", "default": { "useApifyProxy": true, "apifyProxyGroups": [ "RESIDENTIAL" ], "apifyProxyCountry": "IN" } }, "maxRequestRetries": { "title": "Max retries per keyword", "minimum": 0, "type": "integer", "description": "How many extra attempts (each with a FRESH proxy session) a keyword gets. Used both to recover from a failed page load and to corroborate a short result count: a shortfall is only reported as a real content shortage once two independent sessions independently run out of results at the same count. Lower values make short counts less trustworthy.", "default": 5 }, "navigationTimeoutSecs": { "title": "Navigation timeout (seconds)", "minimum": 10, "type": "integer", "description": "Max time to wait for the YouTube search page to load per attempt. Residential proxies can be slow; do not set this too low.", "default": 90 }, "resultsTimeoutSecs": { "title": "First results timeout (seconds)", "minimum": 5, "type": "integer", "description": "Max time to wait for the first video result to render after the page loads. If it expires, the attempt is abandoned, a diagnostic snapshot (screenshot + page state) is stored in the key-value store under DIAG-*, and the keyword is retried with a fresh proxy session.", "default": 45 }, "loadBudgetSecs": { "title": "Result loading budget (seconds)", "minimum": 30, "type": "integer", "description": "Max time to spend scrolling to load results for one attempt. YouTube India result pages are dominated by Shorts (measured: 160 Shorts vs 35 videos on one keyword), and the video count can sit frozen for several scrolls while a block of Shorts streams in, so reaching a high target legitimately takes many scrolls. Raise this for large maxResultsPerKeyword values; a run that stops on this budget is reported as a shortfall, never as a content shortage.", "default": 240 } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```