Brave Search Scraper

Search Brave Search using a keyword and save normalized organic web results from the rendered results page. The Actor validates the HTML response and extracts the visible result cards with Firefox.

What is captured

Each record includes source-backed listing data where Brave displays it:

• Identity: type, resultType, position, page.
• Article and publisher: title, titleHtml, link, url, displayUrl, displayedUrl, domain, source, publisher, sourceUrl, sourceDomain, favicon.
• Text and navigation: snippet, description, snippetHtml, descriptionHtml, emphasis, sitelinks, breadcrumbs.
• Dates and rich details: author, date, publishedAt, publishedAtRaw, relativeTime, richSnippet, richResultType, rating, ratingCount, reviewCount, price, priceNumeric, currency, cacheLink, additionalInfo, category, tags.
• Images and flags: thumbnail, thumbnailUrl, imageUrl, imageAlt, imageWidth, imageHeight, isAd, isSponsored.
• Search context: engineId, searchQuery, query, country, language, market, safeSearch, searchUrl, scrapedAt, searchMetadata, canonicalUrl.

Unavailable optional fields are omitted recursively. Valid zero values and false flags are preserved. The Actor does not fabricate ratings, prices, authors, categories, or summaries.

Input

Field	Type	Default	Description
query	string	required	Search keywords.
maxItems	integer	50	Maximum records, from 1 to 500.
maxPages	integer	derived	Maximum Brave result pages, from 1 to 50.
country	string	us	Two-letter Brave country code.
language	string	en	Brave language code.
safeSearch	string	moderate	strict, moderate, or off; applied through Brave's official preference and verified.
maxRequestRetries	integer	3	Bounded retries for temporary 5xx navigation responses; rate limits and access denials are terminal.
navigationTimeoutSecs	integer	60	Navigation timeout, from 10 to 180 seconds.
headless	boolean	true	Disable only for local browser inspection.
proxyConfiguration	object	disabled locally	Apify Proxy or custom proxy configuration.

Example:

{
    "query": "artificial intelligence",
    "maxItems": 50,
    "country": "gb",
    "language": "en",
    "safeSearch": "strict",
    "proxyConfiguration": {
        "useApifyProxy": true,
        "apifyProxyGroups": ["RESIDENTIAL"],
        "apifyProxyCountry": "GB"
    }
}

Pagination and validation

Brave renders 20 organic cards per page and exposes a page-indexed Next link. The Actor follows only an enabled, forward-moving next link, preserves the requested query/country/language/Safe Search values on every page, deduplicates canonical URLs, and stops at maxItems, maxPages, or a reliable end-of-results signal.

Before extraction it validates HTTP status, HTML content type, block/challenge markers, search parameters, and the Safe Search preference cookie. Images are retained; only fonts and media are skipped for efficiency.

Local checks

npm ci
npm test
npm run lint
node validate-datasets.js
npm run audit:dataset
apify validate-schema
apify run

Inspect storage/datasets/default after a run. The local INPUT.json is a representative search; a valid no-results query completes with an empty dataset. Local and Docker execution use the Firefox executable bundled with Playwright 1.61.1.

Terminal access restrictions

HTTP 401/403/407/429, explicit CAPTCHA, and access-denial pages stop later actor-initiated requests. They are never treated as temporary retries, and no alternate source, session, or proxy is selected. Existing admitted requests observe the shared stop state before continuing or accepting results; ordinary transport and server failures remain separately bounded.

Bounded direct preview

The public prefill and INPUT.json request artificial intelligence, at most three records from one page, and zero retries over a direct connection. Runtime input defaults remain unchanged. Search already used a direct proxy prefill; explicitly requested proxies remain supported and fail before browser launch if configuration or URL creation fails.

The runtime uses the matching Apify Firefox 22-1.61.1 image and existing pinned Playwright 1.61.1, with standard sandbox and TLS behavior. Launch is bounded to 30 seconds. The existing locale, viewport, Safe Search cookie, access-denial guard, date normalization, and DOM provenance are retained. Chrome executable overrides are no longer used. Upload and Docker exclusions omit local credentials, storage, dependencies, caches, logs, and QA artifacts.

A new direct Firefox Brave News run returned real articles on October 2. This establishes availability for that route only; Search still requires its own bounded cloud run and real-row QA before a status change. Any new challenge or denial stops the run without a new identity or connection.