Bing News Scraper

Search Bing News through its normal search interface and collect the data shown on each article card. The Actor supports Bing sorting, time range, market, numbered pagination, result limits, proxies, sessions, and consistent browser fingerprints.

Input

Search mode is the recommended workflow; users provide the values they would enter or select on Bing News.

Field	Default	Purpose
query	UI prefill: artificial intelligence	Search keywords
market	en-US	Language-country market
sortBy	date	date or relevance
timeRange	any	any, day, week, or month
maxItems	50	Maximum unique records across the run
maxPages	50	Maximum numbered result pages
maxScrolls	60	Lazy-loading scroll attempts per page
maxStallRounds	2	Consecutive scrolls with no new cards before stopping
scrollDelayMs	1000	Wait after each scroll
maxConcurrency	1	Concurrent result pages
maxRequestRetries	3	Temporary-failure retries
navigationTimeoutSecs	90	Navigation timeout
requestHandlerTimeoutSecs	600	Page-processing timeout
proxyConfiguration	proxy disabled	Apify or custom proxy settings

The backward-compatible urls mode accepts only https://www.bing.com/news/search URLs. Search mode does not require users to construct URLs.

Dataset

The documented 43-field contract contains:

• Bing IDs: article ID and topic cluster ID.
• Listing text: title, snippet/description, publisher, category when Bing shows one.
• Links: article URL, domain, and Bing's publisher-search link.
• Publication data: Bing's raw/relative text plus a derived ISO-8601 timestamp when the text is parseable.
• Media: Bing's public high-quality thumbnail URL and source dimensions.
• Listing flags: breaking, paywall, and sponsored badges when present.
• Search context: query, market, country, language, page, sorting, time filter, extraction method, and one consistent scrape timestamp.

Optional fields are omitted when the current card does not expose them. Embedded base64 publisher logos and fabricated third-party favicon URLs are not stored. The Actor never opens article pages or invents article details that are absent from Bing's listing.

Extraction and pagination

Bing currently returns search and lazy-loaded news batches as HTML, not a stable structured JSON results API, so the Actor uses a focused DOM mapper with a conservative external-link fallback. Market-specific Bing pages often expose 8-10 results per numbered page; the Actor follows first offsets until maxItems, maxPages, or an explicit no-results page is reached. Records are deduplicated by Bing article ID or canonical article URL and are ordered deterministically before one dataset push.

Reliability and security

• Validates response status and HTML content type before extraction.
• Detects block/challenge pages and marks the browser session bad.
• Distinguishes explicit no-results pages from selector failures.
• Uses Bing's current interval and sortbydate controls and independently rejects records outside a requested day/week/month range.
• Uses bounded retries, persistent per-session cookies, Apify/custom proxies, and consistent Crawlee fingerprints.
• Logs counts and sanitized targets without response bodies, cookies, credentials, or proxy URLs.
• Restricts output URLs to HTTP(S) and excludes embedded data URLs.

Local verification

npm ci
npm test
apify validate-schema
apify run --purge --input-file INPUT.json
node test/audit-dataset.js --expected=20

Fixtures under test/fixtures cover minimal, filtered multi-page, concurrent URL-mode, empty-result, and invalid inputs.