Brave News Scraper

Extract normalized articles from Brave News. Version 3.1 reads the structured SvelteKit news response embedded in Brave's validated HTML document and retains current DOM extraction as a fallback.

Output

Available source-backed values are preserved and unavailable optional values are omitted. Common fields include:

• stable article-URL SHA-256 ID and global/query/request positions;
• canonical article URL, title, description/snippet, source/publisher, source domain, and favicon;
• displayed publication age plus an ISO timestamp derived from the one stable scrape timestamp;
• thumbnail/article image when Brave provides one;
• live-story status and optional source/profile/image metadata;
• query, locale, Safe Search, time filter, search URL, provenance, retrieval method, and scrape metadata.

Opaque debugging, click/tracking actions, cookies, proxy credentials, and hydration request state are excluded. Ad, sponsored, and breaking flags are only emitted when Brave exposes a positive or explicit value; missing markers are not fabricated as false.

Search modes

• Single query: set query.
• Multiple queries: set queries, optionally together with query. Duplicates are removed case-insensitively.
• One article: set maxItems to 1.
• Fair batch: concurrent query results are interleaved until the total maxItems limit is reached.
• Time filter: any, day, week, month, or year maps to Brave's current News filter.
• Safe Search: strict, moderate, or off is applied through Brave's preference cookie and verified after navigation.

Brave currently returns one bounded result set per News query (observed from roughly 20 to 50 records depending on the query/filter) and exposes no usable continuation cursor. Multi-query mode is the supported way to collect a larger, fairly distributed dataset.

Example

{
  "queries": ["artificial intelligence", "climate change"],
  "maxItems": 20,
  "maxConcurrency": 2,
  "country": "us",
  "language": "en",
  "safeSearch": "moderate",
  "timeRange": "week",
  "maxRequestRetries": 1,
  "proxyConfiguration": {
    "useApifyProxy": true,
    "apifyProxyGroups": ["RESIDENTIAL"],
    "apifyProxyCountry": "US"
  }
}

Reliability and performance

The primary path parses the structured response without downloading images. Image, media, and font requests are blocked after verifying that this does not affect the embedded results. Queries share one consistent Firefox session and use bounded page concurrency. Dataset writes are all-or-nothing, so a failed query cannot leave a misleading partial batch.

Temporary network failures, HTTP 429, and HTTP 5xx responses use bounded progressive retry. Invalid input, unexpected payloads, response/query mismatches, and challenge pages fail clearly. CAPTCHA or access-control pages are never solved or stored.

Direct, Apify Residential, and authorized custom proxy access are supported. Google SERP Proxy is not applicable to Brave.

Local validation

npm ci
npm test
node validate-datasets.js
apify validate-schema
apify run --purge --input-file INPUT.json
apify run --purge --input-file INPUT_BATCH.json
apify run --purge --input-file INPUT_FILTERED.json
apify run --purge --input-file INPUT_SINGLE_ITEM.json
apify run --purge --input-file INPUT_NO_RESULTS.json

The Docker image and local dependencies use Playwright Firefox 1.62.1.