Startpage Shopping-Intent Scraper

Discover product pages, storefronts, offers, and price-bearing pages through Startpage web search. Startpage does not provide a native Shopping vertical, so this Actor does not invent a shopping feed or fabricate product metadata. It searches Startpage's web results, keeps only records with visible shopping evidence, and labels the retrieval method in every record.

Features

• Single-query (query) and fair multi-query (queries) modes
• Pagination, overall maxItems, concurrency, retry, and timeout controls
• Shopping-intent threshold, domain allow/block lists, optional price requirement, and sorting
• Embedded structured-data extraction with a DOM fallback
• Stable IDs, canonical URLs, global deduplication, and matched-query provenance
• Optional Apify Residential or custom proxy configuration
• Consistent browser fingerprints, cookie-persistent sessions, and bounded retries
• Challenge and access-denied pages fail closed and are never stored

GOOGLE_SERP is intentionally rejected: that proxy group is for supported Google HTTP requests, while this Actor visits Startpage over HTTPS. This Actor does not solve or bypass CAPTCHAs.

Input

Provide query, queries, or both. Up to 20 unique queries are accepted. maxItems is an overall run limit; multi-query results are selected round-robin so one query cannot consume the whole dataset. The recommended minShoppingIntentScore is 0.25.

Examples:

{
  "query": "iphone 15 price",
  "maxItems": 25,
  "maxPages": 5,
  "minShoppingIntentScore": 0.25,
  "requirePrice": false,
  "sortBy": "relevance",
  "proxyConfiguration": { "useApifyProxy": false }
}

{
  "queries": ["iphone 15 price", "wireless headphones buy"],
  "maxItems": 40,
  "maxConcurrency": 2
}

Output

Required records include stable identity, source and dataset positions, title, canonical destination URL, domain, source name, shopping-intent score, retrieval and extraction methods, query provenance, Startpage search URL, locale, and scrape time. snippet, price, currency, and priceRaw are included only when Startpage actually exposes them. Missing optional values are omitted instead of emitted as null or empty strings.

Prices are parsed only from the Startpage snippet; the Actor does not visit merchant pages and does not claim live inventory, ratings, reviews, images, discounts, or shipping details.

Local development

npm ci
npm test
node validate-datasets.js
apify run --purge --input-file INPUT.json

Inspect storage/datasets/default after every run. Docker uses the Apify Playwright Chrome image with matching Playwright dependencies.

Use at least 1 GB of Actor memory; 2 GB is recommended for the most reliable Chromium startup under concurrent platform load.

Limitations

Search relevance and optional fields depend on what Startpage returns for the selected network and locale. A clean no-results search produces an empty dataset. Explicit blocking, challenges, malformed content, and selector drift fail the run rather than publishing misleading records.