Startpage Shopping Scraper

Collect the sponsored product cards displayed in Startpage's Shopping tab. Version 4 defaults to products mode and retains the previous scored web-search contract as explicit web-intent mode.

Products mode

The Actor starts with an ordinary Startpage web query, clicks the visible source-issued Shopping control, and lets that page submit its own POST form. It never reconstructs or imports the generated search URL, hidden sc value, cookies, or other session controls. It reads only the main product grid, excluding the separate sponsored carousel and hidden responsive cards. It does not click product, advertising, review, merchant, or image links.

The bounded checked-in input is:

{
  "mode": "products",
  "query": "iphone 15 price",
  "maxItems": 10,
  "maxPages": 1,
  "maxConcurrency": 1,
  "maxRequestRetries": 0,
  "language": "english",
  "region": "us",
  "proxyConfiguration": { "useApifyProxy": false }
}

Provide query, queries, or both, with up to 20 unique queries. maxItems applies across all queries; selection alternates between query buckets. Products mode currently supports one loaded main-grid page. It rejects maxPages above 1, domain filters, and shopping-intent sorting rather than silently ignoring them. Relevance, ascending price, and descending price ordering are supported. Numeric price sorting does not convert currencies.

language is a requested search preference. region is retained as a requested-region provenance label and does not force the browser's network region or establish the products' currency. Product rows require a visible title, merchant, current price, and the inspected sponsorship label.

Product output

• title, merchant, sourceName, numeric price, priceText, and priceRaw
• currencySymbol when displayed; currency only when the source price explicitly names an ISO currency or unambiguous country-prefixed token. A bare $, £, or ¥ is not promoted to an ISO currency.
• Optional originalPrice / originalPriceRaw when a separate, higher crossed-out price accompanies a deal price
• Optional rating, ratingScale, exact reviewCount, visible reviewCountText, and shippingText
• sponsored: true, sponsoredLabel, resultType: "sponsored-product", resultSection: "main-shopping-grid", and retrievalMethod: "startpage-shopping"
• A derived offer id and idMethod, dataset/source positions, query provenance, a sanitized source-page route, requested language/region, and scrape time

The current-price selector is .actual-price. The shopping-price test ID may point to a crossed-out original price and is not used as the current price. Exact review counts come from the source's accessibility label when available; a visible (1k+) alone is retained as text without inventing an exact count.

Offer IDs are derived from normalized title, displayed merchant, displayed currency token, and price. They are not source-issued product IDs and can change when an offer's price changes.

Link and image limitations

The inspected cards expose Bing advertising redirects and Startpage image-proxy references, not verified merchant or original-image destinations. Product rows therefore omit url, merchantUrl, imageUrl, and usable advertising-click URLs. They retain explicitly named origin/path provenance fields, parameter names without values, and omission reasons. A provenance origin/path is not the original complete link. destinationUrlVerified is false. Tracking or session query values are never saved in product records. imageReferenceStatus is observed only when the selected DOM image element exposes a usable HTTP(S) src at extraction, otherwise not-observed. It does not establish whether the source has or lacks an image. Images are not downloaded or verified.

Legacy web-intent mode

Use "mode": "web-intent" for the previous contract: ordinary Startpage web results scored for storefront, product-path, purchasing, and snippet-price evidence. This mode supports pagination, domain filters, minShoppingIntentScore, requirePrice, and shopping-intent sorting. Its records have resultType: "shopping-intent-web-result" and retrievalMethod: "startpage-web-search", direct web-result URLs, and shopping-intent scores. Optional prices still come from snippets and retain the legacy currency mapping.

{
  "mode": "web-intent",
  "query": "wireless headphones buy",
  "maxItems": 25,
  "maxPages": 5,
  "minShoppingIntentScore": 0.25,
  "proxyConfiguration": { "useApifyProxy": false }
}

The legacy DOM fallback prefers the observed result-title anchor and excludes hidden cards. Legacy API defaults remain available when this mode is explicit; the default result mode intentionally changed in version 4.

Browser and access boundaries

Docker and the pinned dependency use stock Playwright Firefox 1.62.1. TLS and sandbox checks remain enabled. The installed Apify Cloud SDK's Chromium-oriented no-sandbox default is reset before launch; runtime hooks reject modified browser paths, launch flags, fingerprints, imported authentication state, TLS exceptions, and unsafe sandbox environment overrides. Session/fingerprint rotation and automatic blocked-page retries are disabled. Direct access is the default. An explicitly configured supported proxy is resolved once and pinned; it never rotates or falls back to direct access. GOOGLE_SERP and TLS-intercepting proxies are rejected.

Only normal Startpage main-frame navigation is permitted. Product, merchant, ad, other-frame, and off-origin document navigation is blocked. Image, media, and font requests are not needed for text-card extraction and are blocked.

Products mode permits one passive 45-second wait for the observed automatic “Verifying your request / Calculating” state on each distinct normal search navigation, with a maximum of 90 seconds for the entire run. It does not reload the page, inspect challenge state, compute proof-of-work, transfer tokens, or click challenge controls. Explicit HTTP 401/403/407/418/429/451, manual CAPTCHA or verification routes, browser/environment security failures, or verification that remains after its allowed wait stop the run. Legacy web-intent mode does not use this passive allowance. Ordinary transient failures alone may use configured retries; the preview sets zero retries.

The observed initial “Verifying your request / Loading” stage uses that same allowance. It requires page-level verification wording, no manual controls, and no actual visible web or Shopping results, so an ordinary result snippet quoting these words does not qualify. Persistent Loading or an exhausted wait budget remains terminal. This fixture correction does not identify the cause of any earlier cloud failure and does not increase the navigation or total wait limits.

Run outcome and verification

OUTPUT reports mode, completed/no-results/blocked/failed status, acknowledged record count, and passive wait time. Successful summaries also include completed-query and observed-product-card counts. An explicit no-products page produces an empty dataset. Missing selectors, missing sponsorship, wrong displayed query, unsupported form navigation, or source access failures do not become successful empty product datasets. Results are published only after all requested query handlers succeed.

When an access boundary is reached, firstBoundaryCause retains its first reason from a fixed enum, optional HTTP status, resource type, main-frame/navigation flags, and coarse source-host classification. Exact hostnames are retained only for a small fixed public-host allowlist. No paths, query values, raw errors, dynamic host labels, headers, or session data are retained. intentionalAbortCounts reports successfully enforced local request blocks by resource type. A later event cannot replace the original cause.

Unneeded subframes and image/media/font requests remain blocked using explicit local cancellation. Their exact request identities are tracked so confirmed cancellation is not presented as a main-source denial. TLS, certificate, proxy-authentication, environment-denial, unexpected frame identity, failed local abort, and denied document responses remain terminal, including when they occur on a tracked request. Security classification is synchronous to preserve event order. The first cloud candidate stopped before its initial post-navigation access check and produced zero records; its original log does not identify the triggering event. The local cancellation scenario is reproducible in offline tests, but has not been established as that cloud run's cause.

On 2026-10-02, separate ordinary browser inspection showed the Shopping control, its POST navigation, and 50 main-grid sponsored products for iphone 15 price, plus 20 separate carousel cards. privacy search engine showed an explicit no-products state. No product/ad links were clicked, and no signed/session values were transferred. This source evidence supports the selectors and output contract, but does not certify a fresh automated Actor run.

Local regressions exercise sanitized DOM fixtures, sale/current prices, currency precision, review counts, carousel exclusion, source-form navigation, query matching, omission of sensitive URL values, bounded passive verification, access denials, legacy extraction, and the actual installed Apify Cloud/Crawlee launch path. Cloud-SDK tests block networking and intercept the final browser-launch boundary; they do not claim a live Firefox launch or source dataset.

npm ci
npm test
npm run check

Run live validation separately with the bounded input and inspect both OUTPUT and the resulting dataset. No live cloud result is included in this offline candidate.