---

status: verified last_verified: 2026-08-25 owner: operator depends_on: [05-monetization/legal-gate.md, 08-instincts/project-instincts.md]

Play Charts Scraper — S9 first scraper

First revenue-path code in this project. Gated on ../../05-monetization/legal-gate.md, which records a PASS with accepted ToS risk for this target on 2026-08-25. Read that entry before changing scope — in particular, the field list is a legal boundary, not a design preference.

Layout

src/charts.py            core extraction — shared, no I/O, no CLI
src/main.py              Apify Actor entrypoint (async, httpx, proxy-aware)
scrape_play_charts.py    standalone CLI for local/nightly runs (requests)
.actor/                  actor.json, input_schema.json, dataset_schema.json
Dockerfile               apify/actor-python:3.11

Both entrypoints import the same src/charts.py, so the Actor and the CLI cannot drift apart — a parser fix lands in both at once.

What it does

Fetches Google Play top charts for one or more countries and emits one record per app: rank, package_id, title, rating, price_usd, is_free, country, chart, collected_at.

Local:

$python3 scrape_play_charts.py --countries PK,IN,AE --out charts.jsonl

Flag	Default	Meaning
--countries	US	comma-separated ISO codes
--chart	topselling_free	label written into each record
--contact	operator email	contact address embedded in the User-Agent
--out	-	output path, - for stdout
--delay	3.0	seconds between countries
--min-free-pct	15.0	abort before writing if disk is fuller than this

As an Apify Actor: input is countries, chart, contactEmail, delaySeconds, proxyConfiguration. Defaults to PK, IN, AE on DATACENTER proxies — see the economics note below, that default is a margin decision, not a convenience.

Output record:

{"rank": 1, "package_id": "com.underthing.focus.friend",
 "title": "Focus Friend by Hank Green", "rating": 4.5, "price_usd": null,
 "is_free": true, "country": "US", "chart": "topselling_free",
 "collected_at": "2026-08-25T13:58:02.114+00:00"}

Two design decisions worth not undoing

1. Parsing is anchored on accessibility attributes, not CSS classes. Google Play's class names (Epkrse, LrNMN, ULeU3b) are build-generated and rotate without notice; aria-label="Rated 4.4 stars out of five stars", title=, and href= are driven by accessibility requirements and are far more stable. A parser written against the class names looks cleaner and breaks on Google's next deploy.

[VERIFIED 2026-08-25] There is no JSON-LD on this page — an early plan assumed there was. The machine-readable payload is an AF_initDataCallback ds:4 block: ~856KB of deeply nested positional arrays, where a field is identified by its index path (e.g. (0,1,0,21,0,2)). It parses, but any change to Google's array shape silently shifts every index. The HTML-attribute route was chosen deliberately over it.

2. It aborts rather than emitting a partial or empty result set. Per silent-failure-over-loud-failure: a run that errors can be excluded from marketplace statistics; a run that "succeeds" with empty rows lands in a customer's download and generates a one-star review that cannot be undone. Checks enforced before any output is written:

• >= 20 apps per country, else ParserBreak
• no empty titles
• >= 50% of records carry a rating (catches a broken rating anchor)
• all ratings within 0.0–5.0
• if any country fails, nothing is written at all — a nightly run that quietly emits 2 of 4 countries would corrupt a time series nobody rechecks

Verified working: an intentionally broken rating anchor was rejected with only 0/66 records carry a rating — the rating anchor probably broke.

Why regional, not generic

[VERIFIED 2026-08-25] Live measurement of chart divergence against the US chart:

Country	Apps	Not in US chart	Divergence
PK	58	35	60%
IN	55	42	76%
AE	48	38	79%

Apify already hosts several generic Play scrapers. This divergence is the evidence that a South Asia / MENA focus is an actual gap rather than a repositioning of the same product.

Deploying to Apify

[VERIFIED 2026-08-25] Pushed and running live at ghostlabpk/google-play-regional-charts (https://console.apify.com/actors/vrlac7utdieQhU4Tf). Confirmed end to end: build succeeds, a real run against countries: ["PK"] returned 58 apps and pushed them to the dataset, all on Apify's own infrastructure (not ghostlab).

Deploy tooling lives on ghostlab — no system Node/npm (no passwordless sudo), so apify-cli runs via nvm:

export NVM_DIR="$HOME/.nvm"; . "$NVM_DIR/nvm.sh"   # nvm-managed Node, not system
apify login -t <token>       # from Apify Console -> Settings -> Integrations
cd 09-code/play-charts-scraper
apify push --force
apify call --input-file <input.json> --json   # test a real run before listing

Then in the Apify Console: set pricing to pay-per-result at $0.50/1K (the market anchor — see ../../06-finance/apify-unit-economics.md; do not undercut it) and publish.

Two upstream dependency bugs, pinned around [VERIFIED 2026-08-25]

crawlee 0.6.12 (pulled in transitively by apify) declares compatibility ranges in its own package metadata that are wrong — both failures looked identical (AttributeError/TypeError deep in an unrelated import chain, nothing to do with this Actor's own code) and both were root-caused the same way: bisect in an isolated venv, pin below the real break.

Package	Declared range	Actually breaks at	Pinned to
pydantic	>=2.8.0 (no ceiling)	2.12.0+ (RootModel HttpHeaders: "cannot specify both default and default_factory")	<2.12
browserforge	>=1.2.3 (no ceiling)	1.2.4 (_browserforge_workaround.py: module 'browserforge.download' has no attribute 'DATA_FILES')	==1.2.3

Do not raise either ceiling without re-testing a real Actor run (not just an import) — the first pydantic<2.13 attempt still resolved to a broken 2.12.5 and looked fine until actually run.

[RISK] DATACENTER proxy group not yet accessible on this account

A real run with proxyConfiguration: {"apifyProxyGroups": ["DATACENTER"]} was rejected at input validation: "You currently do not have access to proxy groups: DATACENTER". The proxy-free run above works — this is an account-level restriction, not a code or config problem. Apify's pricing page lists 5 free datacenter IPs on the Free tier, so this is most likely a new-account hold (email verification / account-maturity gate against signup abuse), not a paywall. Check Console → Proxy → Datacenter before assuming a paid plan is required. The whole margin model in ../../06-finance/apify-unit-economics.md assumes datacenter proxies — running proxy-free or on residential changes the unit economics materially, so don't list publicly until this is confirmed working.

[VERIFIED 2026-08-25] Apify's flat-monthly rental model is retired — it stopped accepting new listings 2026-04-01 and shuts down 2026-10-01. Pay-per-result/pay-per-event is the only route for a new listing; do not design around rental.

Not yet done

• [RISK] Not yet listed publicly. Pushed, built, and verified running — see above — but not published to the Store, and shouldn't be until DATACENTER proxy access is confirmed (the pricing model depends on it).
• [PLANNED] Nightly scheduling + the historical time series that is the actual Method 2 asset. The disk guard now exists (--min-free-pct, default 15%) so the unattended-run precondition is met; scheduling itself is not set up.
• [RISK] The local CLI runs from the home residential IP. The Apify Actor does not — it runs on Apify's infrastructure through their proxies, which is one of the reasons Method 1 prefers Apify over self-hosting.
• [RISK] Only topselling_free is collected today; the chart field is a label, not a selector. Paid/grossing charts would need a different URL and are not implemented.
• [RESOLVED 2026-08-25] The planning price points ($0.25 basic / $2.20–3.90 deep) were checked against live listings and are 4–8x too high. Market anchor is $0.50/1K. Full cost model, including the proxy line that decides margin, is in ../../06-finance/apify-unit-economics.md. Use datacenter proxies, not residential — at 2.33MB/request, residential bandwidth eats ~76% of net revenue.