Startpage News Scraper

Collect dedicated Startpage News results as normalized, API-ready records. The Actor accepts one search or a batch of searches, follows native news pagination, applies optional date/publisher/domain/image filters, and selects records fairly across batch queries.

The Actor extracts Startpage's structured news payload first. A DOM fallback is retained for compatible page changes. Ordinary web-search results, advertisements, error pages, and challenge pages are never stored as news records.

Input

Provide either query or queries.

{
  "queries": ["world news", "technology news"],
  "maxItems": 50,
  "maxPages": 5,
  "dateFilter": "week",
  "sortBy": "newest",
  "requireImage": false,
  "language": "english",
  "maxConcurrency": 3,
  "proxyConfiguration": {
    "useApifyProxy": true,
    "apifyProxyGroups": ["RESIDENTIAL"],
    "apifyProxyCountry": "US"
  }
}

Important options:

Field	Purpose
query / queries	One search phrase or up to 20 phrases.
maxItems	Global output limit, 1–500.
maxPages	Native news pages per query, 1–10.
dateFilter	anytime, day, week, or month.
publishedAfter / publishedBefore	Optional client-side YYYY-MM-DD limits.
sortBy	relevance, newest, or oldest.
includeDomains / excludeDomains	Article-domain filters.
includePublishers / excludePublishers	Case-insensitive publisher filters.
requireImage	Keep only records with a source-provided image.
language	Startpage result-language preference.
region	Locale provenance attached to output.
maxConcurrency	Parallel query limit, 1–8 and memory-capped.
proxyConfiguration	No proxy, Apify Residential, or custom proxy URLs.

GOOGLE_SERP is not supported. It is an HTTP proxy for supported Google Search/Shopping requests, while this Actor visits Startpage through an HTTPS browser session. Use Apify Residential or custom proxies when direct access is unreliable.

Output

Records contain only non-empty values. Core fields include:

• stable id, global position, source page, and sourcePosition
• title, canonical article url, domain, and description
• publisher, publisherDomain, publishedAt, and publishedAtRaw
• source-provided thumbnailUrl, imageUrl, sourceLogoUrl, authors, tags, and category when available
• query, queryPosition, matchedQueries, and sanitized searchUrl
• extraction provenance, locale, date filter, and scrapedAt

Example:

{
  "id": "news_...",
  "position": 1,
  "queryPosition": 1,
  "sourcePosition": 1,
  "page": 1,
  "title": "Example headline",
  "url": "https://publisher.example/article",
  "domain": "publisher.example",
  "description": "Example summary",
  "publisher": "Example Publisher",
  "publishedAt": "2026-08-30T00:00:00.000Z",
  "resultType": "news-search-result",
  "resultSource": "startpage-news-bing",
  "extractionMethod": "embedded-structured-data",
  "query": "world news",
  "matchedQueries": ["world news"],
  "language": "english",
  "country": "global",
  "sourceSearchDomain": "startpage.com",
  "scrapedAt": "2026-08-30T00:00:05.000Z"
}

Optional fields are omitted when the source does not provide them; the Actor does not emit null, empty strings, empty arrays, placeholders, raw responses, or signed proxy URLs.

Runtime behavior

• Sessions, stable Crawlee browser fingerprints, bounded retries, and persistent cookies are enabled.
• Images, media, and fonts are blocked only after verifying structured news extraction remains complete.
• All records are normalized, filtered, deduplicated, fairly selected, and written atomically after crawling.
• A genuine no-result search completes successfully with an empty dataset.
• CAPTCHA or unusual-traffic pages fail the run and are never stored. The Actor does not solve or bypass CAPTCHAs.

Local development

npm ci
npx playwright install firefox
npm test
apify run --purge --input-file qa-inputs/local-single.json
node validate-datasets.js

The qa-inputs directory also contains batch, pagination, filtered, empty-result, invalid-input, and cloud Residential-proxy cases.