DuckDuckGo News Scraper

Search public DuckDuckGo News results and collect clean, source-backed article metadata through DuckDuckGo's structured news endpoint.

Highlights

• One primary query plus up to 19 additional queries
• Fair per-query limits and interleaved global output
• DuckDuckGo region, publication-time, date-sort, and Safe Search controls
• Bounded continuation pagination, concurrency, retries, and timeouts
• Direct access by default, with authorized Residential/custom proxy support
• Stable query-specific IDs, recursive empty-value omission, and no raw payloads or opaque tokens

The Actor loads the public search page to obtain the current News token and then requests the same structured records used by the website. It validates HTTP status, content type, payload shape, and continuation URLs before parsing. CAPTCHA/challenge pages fail closed and are never stored.

Example

{
  "query": "artificial intelligence",
  "queries": ["robotics"],
  "maxItems": 20,
  "maxItemsPerQuery": 10,
  "maxPages": 2,
  "region": "us-en",
  "timeRange": "w",
  "sortBy": "date",
  "safeSearch": "moderate",
  "maxConcurrency": 2,
  "proxyConfiguration": { "useApifyProxy": false }
}

Output

Records include stable identity and ranking, article title/URL/domain, snippet, publisher, image, ISO and epoch publication times, relative time, exact search/filter context, provenance, and timestamps. Key fields include id, title, link, source, snippet, thumbnail, publishedAt, publishedAtEpoch, queryPosition, globalPosition, searchUrl, dataSource, retrievalMethod, and scrapedAt.

Unavailable optional fields are omitted instead of emitted as null, empty strings, or fabricated placeholders. DuckDuckGo's opaque image and continuation tokens and internal relevance flags are intentionally not stored.

Notes

• Ranking, availability, publisher labeling, and source coverage are controlled by DuckDuckGo.
• timeRange, sortBy, region, and Safe Search are sent to DuckDuckGo and repeated in output context.
• The public endpoint does not provide trustworthy author, category, engagement, breaking-news, sponsored, or copyright metadata, so the Actor does not invent them.

Bounded preview and run status

The prefilled input requests five records from one page with one query worker, no retries, a 25-second request timeout, and direct access. It reproduces a hash-verified historical input; it does not certify current source availability. Omitted API values retain the documented runtime defaults.

Ordinary temporary failures may retry on the same route and one fixed authorized proxy session. HTTP 401/403/407/429/451, redirects, login or verification pages, and TLS/proxy-authentication failures stop new requests across all query workers. No CAPTCHA is solved and redirects are not followed. Already completed or successfully in-flight result pages are retained.

The OUTPUT key-value record reports SUCCEEDED, NO_RESULTS, PARTIAL, BLOCKED, or FAILED, with completed/failed/skipped query counts and safe failure codes. Partial rows are retained when a later request fails; a partial status does not claim all requested queries or pages succeeded. Empty blocked/failed runs exit unsuccessfully.

Source text fields decode HTML entities and remove search highlight markup. Article URLs, source timestamps, ranks and query provenance remain source-backed and unchanged.