Fixes from independent review round 1.
- Duplicate removal by title and publisher now needs the two publication times to be less than 12 hours apart, so a daily column that reuses its headline is kept once per day. Days whose articles were all duplicates of other feeds no longer count toward the empty-day stop.
- A 400 or 404 for one feed (for example an unknown topic ID) no longer counts toward the 3-feeds-in-a-row breaker.
- Topic subsection URLs (
/topics/<id>/sections/<id>) are skipped with a warning instead of failing the run.
- On the Apify platform, with no proxy set, a block, consent redirect or non-RSS page on the direct connection switches the run once to Apify Proxy (datacenter). OUTPUT has
proxyFallbackUsed.
- The cookie consent error now suggests a US datacenter proxy.
queries and topics refuse non-string items (schema items.type: string and in code).- Runs without a cost cap save each page of rows with one
pushData call and one url-resolved charge.
First version.
- Google News search feeds (with full search operators) and topic feeds (top stories, 8 sections, topic IDs and topic URLs).
- Real publisher URLs: old article IDs decode offline; current IDs go through the article page signature and Google's
batchexecute lookup, 10 per call.
- 85 editions (country + language), checked live.
- Date filters (absolute or relative) on exact UTC publication times, sent to Google as US Pacific-day
after: / before: operators.
- More than 100 results per search, read one day at a time, with days at Google's cap reported (
fullDays); at most 400 days per query and 5,000 feed pages per run.
- Duplicate removal by Google ID, publisher URL, and title + publisher.
- Related coverage from Google story clusters.
- Pay per event:
article $0.0015 and url-resolved $0.0005 (only when found). Max-cost stop happens before resolution.
- Breakers for blocked resolution (5 failures in a row; any success resets) and failing feeds (3 in a row); restart-safe without double charges; per-feed run report in
OUTPUT.
