Yahoo News Scraper

Search public Yahoo News results and export normalized, API-ready article records. The Actor uses Yahoo's server-rendered result cards, decodes Yahoo tracking redirects into real article URLs, supports one query or a fair batch, and completes typical runs in seconds without launching a browser.

Features

• Single-query and deduplicated multiple-query modes
• Fair interleaving and global/per-query limits
• Real Yahoo pagination with up to 50 pages per query
• US, UK, Canada, Australia, and India regional Yahoo search hosts
• Source-backed past-day, past-week, past-month, and custom date filters
• Yahoo relevance order or newest ordering across fetched candidates
• Stable article IDs and canonical URL deduplication
• Publisher and “via Yahoo News/Finance/Sports” separation
• Thumbnails, dimensions, alternative text, favicons, snippets, highlights, and normalized publication times
• Direct, Apify Residential/Datacenter, and custom proxy operation
• Bounded retries, status/content validation, block detection, and clean no-results handling

Input

Provide query, queries, or both. Duplicate queries are removed case-insensitively.

Field	Description	Default
query	One Yahoo News query	—
queries	Up to 20 queries	—
maxItems	Maximum records across all queries	100
maxItemsPerQuery	Optional cap for each query	fair share of maxItems
maxPages	Maximum pages inspected per query, 1–50	10
country	Two-letter country code	us
language	Language or language-country locale	en
timeRange	any, day, week, or month	any
startDate	Inclusive YYYY-MM-DD publication date	—
endDate	Inclusive YYYY-MM-DD publication date	—
sortBy	relevance or newest	relevance
maxConcurrency	Concurrent Yahoo page requests, 1–10	5
maxRequestRetries	Temporary retries, 0–3	2
requestTimeoutSecs	Per-request timeout	30
proxyConfiguration	Apify or custom proxy configuration	direct

Example batch input:

{
  "queries": ["artificial intelligence", "climate technology"],
  "maxItems": 20,
  "maxItemsPerQuery": 10,
  "maxPages": 5,
  "country": "us",
  "language": "en",
  "timeRange": "week",
  "sortBy": "newest",
  "proxyConfiguration": {
    "useApifyProxy": true,
    "apifyProxyGroups": ["RESIDENTIAL"],
    "apifyProxyCountry": "US"
  }
}

Set maxItems to 1 for a single search item.

Output

The dataset schema documents 52 possible fields. A normal result can include:

• stable ID and global, query, page, and page-card positions
• decoded article URL and destination domain
• article title, snippet, description, and highlighted query terms
• publisher, original Yahoo source label, and Yahoo distribution property
• thumbnail URL, alternative text, title, dimensions, and aspect ratio
• publisher favicon
• normalized publication time, exact Yahoo label, relative time, and age in seconds
• Yahoo-hosted and explicit sponsorship flags
• exact query, region, language, filters, ordering, retrieval method, and aggregate search metadata

Unavailable optional values are omitted instead of being emitted as null, empty strings, zero counts, or false flags. The Actor does not fabricate authors, categories, paywall status, engagement counts, or full article content that Yahoo's search cards do not expose.

Filter and ordering behavior

Yahoo's former age and s=4 URL parameters no longer change the current News Search cards. The Actor therefore verifies Yahoo's displayed publication labels, normalizes them against one stable run timestamp, and enforces recency/custom dates after extraction. Records without a usable Yahoo timestamp are excluded when a date filter is active.

newest sorts the candidates fetched within maxPages; it does not claim to sort Yahoo's entire index. relevance preserves Yahoo's page and card order.

Country codes us, gb, ca, au, and in select their regional Yahoo search hosts. Other valid two-letter codes use the global host with locale headers and remain visible in dataset context.

Performance and resilience

Yahoo serves the result cards in the initial HTML, so the Actor fetches and parses that response directly. Independent pages and queries run concurrently with one stable proxy session per query. Responses are checked for status, HTML content type, final Yahoo host, maximum size, challenge content, and current result structure before parsing or storing data.

Challenge pages are never retried or stored. Temporary rate limits, network errors, and server failures use bounded exponential backoff.

Proxy notes

Use Apify Residential or Datacenter proxies when your direct network is rate-limited. GOOGLE_SERP cannot be used because it only supports selected Google Search/Shopping requests, not Yahoo.

Local run

npm ci
apify run --purge --input-file INPUT.json

Results are written to storage/datasets/default.

Limitations

Yahoo News search cards contain summaries rather than full publisher article bodies. Some records omit thumbnails, favicons, highlights, or “via” labels because Yahoo does not supply them. Search inventory and regional ranking can change over time. The Actor does not bypass authentication, paywalls, CAPTCHAs, or access controls.