# Changelog of Startpage Search Scraper (`searchapi/startpage-search-scraper`) Actor

- **URL**: https://apify.com/searchapi/startpage-search-scraper/changelog.md
- **Full Actor documentation**: https://apify.com/searchapi/startpage-search-scraper.md

## Changelog

All notable changes to the Startpage Search Scraper are documented here.

### \[3.0.0] - 2026-08-30

#### Changed

- Follow Startpage's verified session-bound POST pagination instead of reconstructing `page=` URLs.
- Accept current `web-google` and `web-bing` organic groups while excluding ads and unrelated widgets.
- Add single and up-to-20-query batch modes with fair selection, stable IDs, canonical-URL deduplication, and atomic writes.
- Add domain, keyword, sitelink, sorting, Safe Search, concurrency, retry, and timeout controls.
- Replace the sparse/mismatched wide output with a truthful 27-field no-empty-value contract.
- Align exact Apify, Crawlee, Playwright, Firefox image, and security override versions.

#### Security and reliability

- Respect direct, Apify Residential, and custom proxy configuration; reject incompatible `GOOGLE_SERP` input.
- Enable consistent Firefox fingerprints, sessions, persistent cookies, memory-aware concurrency, and bounded retries.
- Validate navigation status/content type and first-party JSON responses before parsing.
- Remove raw URL logging, manual browser fingerprint contradictions, raw partial page pushes, and silent proxy fallback.
- Strip tracking parameters, reject signed URLs, detect challenge pages, and fail closed without storing challenge content.

#### Validation

- Add nine unit tests and single, batch, three-page, filtered, sitelink, empty-result, low-concurrency, invalid-input, and proxy fixtures.
- Add dataset auditing for required fields, types, empty values, canonical URLs, sensitive parameters, duplicates, positions, and block-page content.

### \[2.1.0] - 2026-06-13

#### Fixed

- **Replaced DOM scraping with JSON extraction**: Startpage renders the full search-result JSON server-side inside `<script>` tags. The extractor now parses the `web-google` group from `regions.mainline` and maps every field from there.
- **Query-highlight PUA markers stripped** (`U+E000`/`U+E001`) from titles and snippets — no more `¯ÇÇ…¯Çü` artifacts in output strings.
- **Optional `favicon` field**: 9/10 results in our test corpus include a base64 `faviconData`; the Google result doesn't. The handler now only emits `favicon` when the source data carries it (no more `favicon: null` for the missing case).
- **`siteLinks` normalized to `[]`**: Startpage always returns an empty `siteLinks` array in their web-search response. The field is preserved as an array (not `null`) so consumers can distinguish "no sitelinks" from "field missing".

#### Removed

- **`date` field dropped**: Startpage does not expose a publication date for web search results. The previous output `date: null` for every record was always meaningless, so the field has been removed entirely from the web-search output.

#### Added

- **`source` field** (publisher/site name) — populated from `siteTitleData` in Startpage's response.

### \[2.0.1] - 2026-06-13

#### Changed

- Verified live extraction against Startpage.com DOM (June 2026).
- Confirmed all 35 fields populate correctly from page-1 results.
- No code changes from 2.0.0; re-pushed as a verified release.

### \[2.0.0] - 2026-06-12

#### Added

- **Schema expanded** from 14 to **35 fields**.
- **New fields**: `type`, `resultType`, `titleHtml`, `displayUrl`, `host`, `favicon`, `snippetHtml`, `description`, `publishedAt`, `publishedAtRaw`, `relativeTime`, `imageUrl`, `imageUrl`, `anonymousViewUrl`, `isAd`, `isSponsored`, `searchUrl`, `searchMetadata`, `query`, `country`.
- **New view**: `fullDetails` — all 35 fields.
- **INPUT.json** added at project root for local validation.

### \[1.0.0] - 2025-05-20

#### Added

- Initial production release of the Startpage Search Scraper.
- `PlaywrightCrawler`-based architecture using Firefox.
- Modular project structure.
- `validate-datasets.js` for local dataset output validation.
- `Dockerfile` using `apify/actor-node-playwright-firefox` image.
- `README.md` with full Apify usage guide.