Startpage Search Scraper

Collect organic results from Startpage's public Web search as normalized, API-ready records. The Actor supports one search or a fair batch of up to 20 searches, native session-preserving pagination, result filters, Safe Search, and bounded browser runtime controls.

The Actor extracts Startpage's embedded structured result payload first, accepts both verified web-google and web-bing groups, maps validated first-party JSON responses when available, and retains a semantic DOM fallback. Advertisements, challenge pages, signed URLs, raw responses, and unrelated result widgets are never stored.

Input

Provide either query or queries.

{
  "queries": ["privacy search", "secure browsers"],
  "maxItems": 50,
  "maxPages": 5,
  "safeSearch": "moderate",
  "sortBy": "relevance",
  "includeDomains": [],
  "excludeKeywords": [],
  "requireSitelinks": false,
  "language": "english",
  "maxConcurrency": 3,
  "proxyConfiguration": {
    "useApifyProxy": true,
    "apifyProxyGroups": ["RESIDENTIAL"],
    "apifyProxyCountry": "US"
  }
}

Important fields:

Field	Purpose
query / queries	One phrase or up to 20 unique phrases.
maxItems	Global output limit, 1–500.
maxPages	Native Startpage pages inspected per query, 1–10.
safeSearch	moderate, strict, or off.
sortBy	Startpage relevance, domain, or title.
includeDomains / excludeDomains	Domain allow/block filters.
includeKeywords / excludeKeywords	Case-insensitive title/snippet filters.
requireSitelinks	Keep only results with source-provided sitelinks.
language	Startpage language value such as english or deutsch.
region	Optional country provenance label; proxy country controls network geolocation.
maxConcurrency	Parallel query limit, memory-capped to protect browser startup.
proxyConfiguration	No proxy, Apify Residential, or custom proxy URLs.

GOOGLE_SERP is not supported. It is an HTTP proxy for supported Google Search/Shopping requests, while this Actor visits Startpage through an HTTPS Firefox session. Use Apify Residential or custom proxies when direct access is unreliable.

Output

Each record contains non-empty values only. The 27-field contract includes:

• stable id, global position, query/source positions, and source page
• title, canonical url, domain, displayed URL, and full snippet
• source name, direct public favicon, source engine index, and source-provided sitelinks when available
• organic result type, Startpage source group, retrieval method, and extraction method
• primary query, every matched query, public search URL, Safe Search, locale provenance, and scrape time

Optional fields are omitted when Startpage does not provide them; the Actor does not emit null, empty strings, empty arrays, fabricated favicons, placeholder values, or signed proxy links.

Pagination and reliability

Startpage pagination is a page-generated POST form containing session state. The Actor clicks and validates that native next-page form instead of reconstructing a fragile page= URL. Results from all requested pages and queries are buffered, normalized, filtered, and deduplicated before one atomic dataset write. If a later page is blocked or changes shape, the run fails without publishing a partial dataset.

Sessions, persistent cookies, consistent Crawlee Firefox fingerprints, bounded retries, and optional Residential/custom proxies improve ordinary compatibility. CAPTCHA and unusual-traffic pages are detected and rejected; the Actor does not solve or bypass CAPTCHAs.

Local development

npm ci
npx playwright install firefox
npm test
apify run --purge --input-file qa-inputs/local-pagination.json
node validate-datasets.js

The qa-inputs directory includes single, batch, pagination, filtered, sitelink, clean-empty, low-concurrency, invalid-input, incompatible-proxy, and cloud Residential cases.