Google Dataset Search Scraper

This Apify Actor searches the public Google Dataset Search page and returns normalized dataset results. It keeps the existing server-rendered HTML path and uses the current card DOM as a browser fallback.

Input

{
  "query": "artificial intelligence",
  "maxItems": 3,
  "hl": "en",
  "maxRequestRetries": 0,
  "requestTimeoutSecs": 20,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}

Records include the displayed title, provider, description, formats, update values, repository thumbnail, public dataset URL, Google result URL, stable identity, query context, and retrieval metadata. The browser detail panel also supplies authors, license, citation count, and dataset size when Google exposes them. Optional fields are omitted when the public page does not expose them.

The example matches the public form's bounded direct prefill. API defaults remain 20 items, three retries, and a 30-second request timeout. The form's incompatible GOOGLE_SERP default has been corrected to direct routing; both proxy group aliases reject GOOGLE_SERP. Explicit compatible proxy settings use the browser transport so the selected route is preserved.

HTTP 401/403/407/429/451, detected verification/sign-in gates, and TLS certificate failures stop without transport fallback, retries, or session rotation. Ordinary static parsing or transient network failures may use the browser fallback. The browser checks access before extraction and after detail-panel interactions; it does not save a partially processed page after a gate.

Historical local HTML contained dataset cards that required browser details to expose their external URLs. Local browser launch then failed because its executable was unavailable. Offline fixtures do not verify current source access or live extraction; a fresh cloud run and dataset review are required before recovery can be claimed.

Local checks

npm ci
npm test
npm run lint
npm run validate:dataset
npm run audit:dataset

Unit tests include real local SDK/crawler initialization with networking disabled. The separate audit:dataset command requires a nonempty real dataset directory, supplied as npm run audit:dataset -- /path/to/dataset; it fails when records are missing. validate:dataset checks schema structure. The locked Playwright version matches the Docker image, whose build checks that Chromium is present. Local dependencies, datasets, logs, and temporary test files are excluded from deployment.