# Changelog of Startpage News Scraper (`searchapi/startpage-news-scraper`) Actor

- **URL**: https://apify.com/searchapi/startpage-news-scraper/changelog.md
- **Full Actor documentation**: https://apify.com/searchapi/startpage-news-scraper.md

## Changelog

All notable changes to the Startpage News Scraper are documented here.

### \[3.0.0] - 2026-08-30

#### Changed

- Extract only dedicated Startpage News groups; ordinary web results can no longer be mislabeled as news.
- Add single-query and up-to-20-query batch modes with fair global selection and canonical-URL deduplication.
- Add native pagination, date, publisher, domain, image, sorting, concurrency, retry, and timeout controls.
- Normalize source relative dates into ISO-8601 while retaining the original display value.
- Replace nullable wide output with a validated 31-field contract that omits unavailable optional values.
- Align the Firefox container and Playwright dependency at version 1.60.0 and pin all production dependencies.

#### Security and reliability

- Respect direct, Apify Residential, and custom proxy configuration; reject incompatible `GOOGLE_SERP` input.
- Enable stable Crawlee fingerprints, sessions, persistent cookies, bounded retries, and memory-aware concurrency.
- Validate first-party response status/content type before JSON parsing and retain a structured DOM fallback.
- Remove raw HTML/body persistence and raw input/URL logging; strip tracking parameters and reject signed URLs.
- Detect block/challenge pages and fail closed without storing challenge content.

#### Validation

- Add unit coverage plus single, batch, pagination, filtered, empty-result, invalid-input, and proxy QA fixtures.
- Add a dataset auditor for required fields, types, empty values, duplicate URLs, malformed URLs, challenge content, and sensitive query parameters.

### \[2.1.0] - 2026-06-13

#### Fixed

- **Replaced DOM scraping with JSON extraction**: Startpage renders the full news-result JSON server-side inside `<script>` tags. The extractor now parses the `news-bing` group from `regions.mainline` and maps every field from there.
- **`thumbnail` now populated** for all records that have a Startpage thumbnail. The previous DOM scraper missed thumbnails when the `<img>` URL didn't contain the substring `proxy-image` (which was the only filter). Records without a thumbnail in the source (e.g. News18) still return `null` for that field — by design.
- **Query-highlight PUA markers stripped** (`U+E000`/`U+E001`) from titles and descriptions — no more `¯ÇÇ…¯Çü` artifacts in output strings.
- **`publishedAt` is now ISO date** (`"2026-06-13"`) from the source `date` field, with `publishedAtRaw` carrying the naturalized format string (e.g. `{ format: "13 hours ago", args: ["13"] }`) for human display.

#### Added

- **`description` field** — the article's description / lead text.
- **`publishedAtRaw` field** — the naturalized date format (e.g. `"13 hours ago"`) for display, alongside the ISO `publishedAt`.

### \[2.0.1] - 2026-06-13

#### Changed

- Verified live extraction against Startpage.com News results (June 2026).
- Page 1 returns ~7 articles with `title`, `link`, `source`, `publishedAt`. Page 2+ is captcha-blocked from non-residential IPs.

### \[2.0.0] - 2026-06-12

#### Added

- **Schema expanded** from 12 to **37 fields**.
- **New fields**: `type`, `resultType`, `titleHtml`, `link`, `host`, `favicon`, `sourceUrl`, `sourceLogo`, `sourceDomain`, `snippetHtml`, `description`, `date`, `publishedAtRaw`, `relativeTime`, `imageUrl`, `images`, `category`, `tags`, `region`, `authors`, `anonymousViewUrl`, `searchUrl`, `searchMetadata`, `query`, `country`.
- **New view**: `fullDetails` — all 37 fields.
- **INPUT.json** added at project root for local validation.

### \[1.0.0] - 2025-05-20

#### Added

- Initial production release of the Startpage News Scraper.
- `PlaywrightCrawler`-based architecture using Firefox.
- Modular project structure.
- `validate-datasets.js` for local dataset output validation.
- `Dockerfile` using `apify/actor-node-playwright-firefox` image.
- `README.md` with full Apify usage guide.