Universal Data Harvester

Rebuilt as a multi-engine search harvester that refuses datacenter IPs. Same live Actor name. All-in-one keeps the old web / Maps / Apollo / Instagram modes.

Type queries, pick engines, pick residential countries. The browser locale, timezone, and GPS are forced to match that exit so the site does not see “Italy IP + Phoenix clock + en-US headers.”

Engines

Engine	Notes
Brave	Extra engine. Real index, fewer walls than Google. Default on.
Bing	Solid HTML SERP. Default on.
DuckDuckGo	Uses the HTML endpoint. Default on.
Yandex	Useful when Western indexes hide things.
Google	Still there. Needs residential. Datacenter dies immediately.

Why this scrapes when the others die

Search engines and cloaked sites block AWS / DigitalOcean / generic datacenter ranges. This actor:

• Forces Apify RESIDENTIAL. Datacenter groups or “no proxy” throw instead of running a doomed crawl.
• Recreates the proxy + crawler per country (country is frozen on one ProxyConfiguration). Defaults: US + NL.
• US state targeting (AZ, NY, CA…) when a site is blocking one state pool.
• Matches the browser to the IP: Accept-Language, navigator.language, Chrome --lang, CDP locale, emulateTimezone, GPS setGeolocation.
• Timezone trap: sites that read Intl.DateTimeFormat().resolvedOptions().timeZone get the proxy country’s zone, not the Mac’s.
• Geo-IP interceptor: logs calls to ipapi / maxmind / ipify / ipstack and writes them on the row.
• Uncloak: force-reveals display:none result / player / gallery nodes.
• Evidence: full-page PNG + raw HTML in the key-value store when a SERP is blocked or empty. A fake “removed” page still leaves a screenshot.
• Geo comparison: if you run more than one country, a GEO_FENCE_REPORT flags size / status / result-count splits (US empty, Italy full).

Residential traffic is billed by GB. Images/fonts/media are blocked by default. Turn that off only if you need pretty screenshots.

Input

{
  "queries": ["osint telegram channels", "qwen local models"],
  "engines": ["brave", "bing", "duckduckgo"],
  "proxyCountries": ["US", "NL"],
  "forceResidential": true,
  "proxyConfiguration": {
    "useApifyProxy": true,
    "apifyProxyGroups": ["RESIDENTIAL"]
  }
}

Italy / Vietnam / Netherlands are the extra geos from the residential playbook. Add them when you want a side-by-side, not for every cheap run.

Output

One dataset row per organic hit, plus empty/error rows:

• engine, query, rank, title, url, snippet
• geoId, proxyCountry, proxySubdivision, locale, timezone
• geoLookups, hiddenRevealed, pageBytes
• evidenceScreenshotKey, evidenceHtmlKey when evidence ran
• rowType=geo-report after a multi-country sweep

Local check

$npm test