DuckDuckGo News Scraper

Search public DuckDuckGo News results and collect clean, source-backed article metadata through DuckDuckGo's structured news endpoint.

Highlights

• One primary query plus up to 19 additional queries
• Fair per-query limits and interleaved global output
• DuckDuckGo region, publication-time, date-sort, and Safe Search controls
• Bounded continuation pagination, concurrency, retries, and timeouts
• Direct access by default, with authorized Residential/custom proxy support
• Stable query-specific IDs, recursive empty-value omission, and no raw payloads or opaque tokens

The Actor loads the public search page to obtain the current News token and then requests the same structured records used by the website. It validates HTTP status, content type, payload shape, and continuation URLs before parsing. CAPTCHA/challenge pages fail closed and are never stored.

Example

{
  "query": "artificial intelligence",
  "queries": ["robotics"],
  "maxItems": 20,
  "maxItemsPerQuery": 10,
  "maxPages": 2,
  "region": "us-en",
  "timeRange": "w",
  "sortBy": "date",
  "safeSearch": "moderate",
  "maxConcurrency": 2,
  "proxyConfiguration": { "useApifyProxy": false }
}

Output

Records include stable identity and ranking, article title/URL/domain, snippet, publisher, image, ISO and epoch publication times, relative time, exact search/filter context, provenance, and timestamps. Key fields include id, title, link, source, snippet, thumbnail, publishedAt, publishedAtEpoch, queryPosition, globalPosition, searchUrl, dataSource, retrievalMethod, and scrapedAt.

Unavailable optional fields are omitted instead of emitted as null, empty strings, or fabricated placeholders. DuckDuckGo's opaque image and continuation tokens and internal relevance flags are intentionally not stored.

Notes

• Ranking, availability, publisher labeling, and source coverage are controlled by DuckDuckGo.
• timeRange, sortBy, region, and Safe Search are sent to DuckDuckGo and repeated in output context.
• The public endpoint does not provide trustworthy author, category, engagement, breaking-news, sponsored, or copyright metadata, so the Actor does not invent them.