Duckduckgo Scraper

DuckDuckGo Scraper — Web, News and Video Search Results

DuckDuckGo Scraper extracts search results from DuckDuckGo's web, news and video engines without an API key, returning title, url, description and hostname on every row — plus publisherName and date on news rows, duration and viewCount on video rows. Every field ships as typed JSON, with missing values left null instead of guessed. Set a keyword, pick a mode and region below, and start the run.

🦆 What is DuckDuckGo Scraper?

DuckDuckGo Scraper is an Apify Actor that queries DuckDuckGo's own web, news and video result endpoints and returns the parsed results as structured JSON rows — no HTML to parse, no CSS selectors to maintain. It requires no DuckDuckGo account, login or API key: it calls the same public endpoints the duckduckgo.com website itself uses. It's built for SEO analysts tracking rankings outside Google, marketers monitoring brand and news mentions, and AI engineers who need clean search data for agents or RAG pipelines.

🌐 What DuckDuckGo search result data is publicly available to scrape?

Every DuckDuckGo search result — web, news or video — is visible to any visitor without signing in; the only real gate on this data is DuckDuckGo's anti-automation defenses, not a login wall.

Data category	Publicly available	Blocked for automated clients
Web organic results (title, URL, snippet, host)	Yes	—
News results (headline, publisher, timestamp)	Yes	Requires a browser-matching TLS fingerprint to load
Video results (title, duration, channel, views)	Yes	Requires a browser-matching TLS fingerprint to load
Session tokens (vqd, dp) that unlock the result endpoints	Yes, embedded in the page HTML	Must be harvested and, on web mode, an inline JS challenge solved per session
Image results	Yes, in a browser	Not offered by this Actor — searchMode only accepts default, news or videos
Instant answers / knowledge panel	Yes	Not returned by this Actor
Sponsored / ad results	Yes	Not returned by this Actor

DuckDuckGo Scraper only returns publicly visible data — what any visitor sees. Nothing behind a login wall.

📦 What data can I extract with DuckDuckGo Scraper?

DuckDuckGo Scraper writes one of three row schemas per result, depending on searchMode — all three share query, searchMode, title and url, and each mode adds its own source-attribution and metadata fields.

Field	Description
query	The keyword you searched, echoed on every row.
searchMode	Which mode produced the row: "default", "news" or "videos".
title	Result title (HTML entities unescaped, tags stripped).
url	Destination URL (web/news) or watch URL (video).
description	Snippet with DuckDuckGo's highlight tags stripped — web and video rows.
rawDescription	The same snippet exactly as DuckDuckGo returned it, highlight markup intact — web rows only.
hostname	Host parsed from the destination URL; null if unparseable — web rows only.
siteName	DuckDuckGo's display name for the site; null when absent — web rows only.
canonicalUrl	DuckDuckGo's canonical/click URL for the listing, when present — web rows only.
resultSource	Upstream result-source tag, e.g. "bingv7aa" — web rows only.
faviconUrl	Derived, not scraped: DuckDuckGo's favicon-proxy URL built from hostname; null with no host — web rows only.
publisherName	Outlet name as DuckDuckGo reports it — news rows only.
date	Publication time, Unix epoch seconds; null when absent — news rows only.
relativeTime	Human phrasing, e.g. "3 days ago" — news rows only.
excerpt	Article excerpt, highlight tags stripped — news rows only.
rawExcerpt	Excerpt exactly as returned, markup intact — news rows only.
image	Thumbnail URL; null on items without one — news and video rows.
provider	Upstream provider tag, e.g. "bing" (news) or "Bing" (video).
isOld	DuckDuckGo's own "older item" flag — news rows only.
duration	Video length, e.g. "51:05" — video rows only.
published	ISO-8601 publication timestamp — video rows only.
publisher	Hosting platform, e.g. "YouTube" — video rows only.
uploader	Channel / uploader name — video rows only.
viewCount	View count; null when absent — video rows only.

🔍 Web result fields (searchMode: "default")

query, searchMode, title, url, description, rawDescription, hostname, siteName, canonicalUrl, resultSource, faviconUrl.

📰 News result fields (searchMode: "news")

query, searchMode, title, url, publisherName, date, relativeTime, excerpt, rawExcerpt, image, provider, isOld.

🎬 Video result fields (searchMode: "videos")

query, searchMode, title, url, description, image, duration, published, publisher, uploader, viewCount, provider.

🤖 Add-on: Need additional search-engine data?

Comparing rankings across engines? Pair this Actor with Google Search Results Scraper or Google Search Results (SERP) Scraper for Google's index, or Baidu Search Scraper for China's leading engine — all three live in the same Scraper Engine account. See Related scrapers below for the full list.

🛠️ Why not build this yourself?

DuckDuckGo does not publish a general search API — its Instant Answer API only returns zero-click infobox data, not organic web, news or video results. Building this yourself means reverse-engineering three separate endpoints (links.duckduckgo.com/d.js, duckduckgo.com/news.js, duckduckgo.com/v.js) that each demand a different TLS fingerprint: the web endpoint only answers on a plain OpenSSL stack and hard-blocks Chrome-impersonating clients, while news and video do the opposite and return HTTP 403 on anything that isn't impersonating a real browser. On top of that, the web endpoint gates behind an inline JavaScript arithmetic challenge that has to be solved on every session, and a genuine anomaly-detection block only clears with a fresh exit IP. DuckDuckGo Scraper already runs both transports, solves the challenge deterministically offline, and escalates through direct, datacenter and residential proxies automatically — so you aren't maintaining that pipeline, funding your own proxy budget from scratch, or re-deriving the challenge logic every time DuckDuckGo changes it.

▶️ How to use DuckDuckGo Scraper

DuckDuckGo Scraper runs entirely inside the Apify platform — there's no separate signup or API key to obtain first.

1. Open DuckDuckGo Scraper on the Apify Store (or in your own Apify account) and click Start.
2. Enter the required searchKeyword — a plain query or a search operator such as site:example.com or filetype:pdf.
3. Set searchMode, region and safeSearch — all four inputs are required, and each ships with a working default.
4. Optionally cap maxItems and endPage, and set proxyConfiguration if you want to start on a stronger proxy tier than the default.
5. Click Start, then download results as JSON, CSV, Excel or XML from the Apify Console, or pull them with the Apify API or apify-client.

How to scale to bulk search extraction

The input schema takes a single searchKeyword string — there's no built-in array or list field for multiple queries in one run. To cover several keywords, loop over the Apify API (or the apify-client SDK) and start one run per keyword, optionally in parallel, then merge the resulting datasets; or use an Apify schedule to trigger repeated runs against a fixed keyword list.

💡 What can you do with DuckDuckGo search result data?

• 📈 An SEO analyst tracking rankings uses title, url and resultSource from repeated runs to see which hosts hold positions on DuckDuckGo, independent of Google's index.
• 🏷️ A brand monitoring team uses hostname and siteName across multiple region values to spot where a brand or competitor surfaces outside its home market.
• 📰 A PR analyst tracking coverage uses publisherName, date and relativeTime on news rows to bound press mentions by outlet and recency.
• 🔎 A compliance reviewer runs site:yourdomain.com and filetype:pdf queries and checks url and canonicalUrl to audit what of a domain is publicly indexed.
• 🤖 An AI engineer builds a RAG pipeline that indexes title, description and excerpt as typed JSON — no HTML parsing required — as retrieval context for an agent.

Because every run produces a typed JSON dataset reachable over the Apify API, agent frameworks can call DuckDuckGo Scraper directly as a tool — see Integrations below.

🛡️ How does DuckDuckGo Scraper handle rate limits and blocking?

DuckDuckGo Scraper starts on a direct connection and, on a block, escalates through Apify datacenter and then residential proxies, minting a fresh exit-IP session on every rotation. When the web endpoint answers with its inline JavaScript arithmetic challenge, the Actor solves it deterministically offline and replays the request — a genuine anomaly-detection block, by contrast, triggers a full exit-IP rotation rather than a retry on the same connection. Each page fetch retries up to five times, re-harvesting session tokens after every rotation, with a short randomized delay between attempts. ⚠️ If every proxy tier is exhausted without a single parseable page, the run fails explicitly instead of reporting an empty dataset as a success. A query that legitimately returns zero rows (for example, an adult keyword under STRICT safe search) is logged as a warning, not a failure — DuckDuckGo answered, it simply had nothing to show.

⬇️ Input

Parameter	Required	Type	Description	Example value
searchKeyword	Yes	string	What DuckDuckGo should search for. Plain keywords or operators (site:, filetype:, inurl:, intitle:, "exact phrase") — verified working on web mode. Default "javascript".	"apify web scraping"
searchMode	Yes	string (enum)	default (web results), news, or videos. Default "default".	"news"
region	Yes	string (enum)	DuckDuckGo locale, sent as the l parameter. 63 codes, from wt-wt (all regions) to specific country/language pairs. Default "wt-wt".	"us-en"
safeSearch	Yes	string (enum)	OFF, MODERATE or STRICT, sent as the p filter parameter. Default "OFF".	"OFF"
maxItems	No	integer	Hard cap on dataset rows, 1–1000. Authoritative — the Actor paginates until it reaches this number, hits endPage, or DuckDuckGo runs out of results. Default 20.	50
endPage	No	integer	Optional extra ceiling on result pages fetched, 0–20 (0 = no page cap). Page sizes: web 10, news ~30, videos ~60. Default 0.	0
proxyConfiguration	No	object	Apify proxy settings. Left off, the Actor starts direct and escalates to datacenter, then residential, automatically. Default {"useApifyProxy": false}.	{"useApifyProxy": true, "apifyProxyGroups": ["RESIDENTIAL"]}

Example input

{
  "searchKeyword": "apify web scraping",
  "searchMode": "default",
  "region": "us-en",
  "safeSearch": "OFF",
  "maxItems": 50,
  "endPage": 0,
  "proxyConfiguration": { "useApifyProxy": false }
}

⬆️ Output

Every run writes one flat, typed JSON row per result to the default dataset — the field set is fixed per searchMode and stays consistent across runs. Export to JSON, CSV, Excel (XLSX) or XML from the Apify Console, or read the dataset with the Apify API or apify-client.

Example output

Web mode (searchMode: "default"):

{
  "query": "apify web scraping",
  "searchMode": "default",
  "title": "Apify: Full-stack web scraping and data extraction platform",
  "url": "https://apify.com/",
  "description": "Cloud platform for web scraping, browser automation, and data for AI.",
  "rawDescription": "Cloud platform for <b>web scraping</b>, browser automation, and data for AI.",
  "hostname": "apify.com",
  "siteName": "Apify",
  "canonicalUrl": "https://apify.com/",
  "resultSource": "bingv7aa",
  "faviconUrl": "https://external-content.duckduckgo.com/ip3/apify.com.ico"
}

News mode (searchMode: "news"):

{
  "query": "artificial intelligence",
  "searchMode": "news",
  "title": "The real work of leading in the age of AI",
  "url": "https://www.forbes.com/sites/example-article/",
  "publisherName": "Forbes",
  "date": 1786075200,
  "relativeTime": "3 days ago",
  "excerpt": "That's the true labor of leading in the age of artificial intelligence.",
  "rawExcerpt": "That&#x27;s the true labor of leading in the age of <b>artificial</b> <b>intelligence</b>.",
  "image": "https://external-content.duckduckgo.com/iu/?u=example",
  "provider": "bing",
  "isOld": false
}

Video mode (searchMode: "videos"):

{
  "query": "javascript tutorial",
  "searchMode": "videos",
  "title": "JavaScript Crash Course for Beginners",
  "url": "https://www.youtube.com/watch?v=example",
  "description": "A complete beginner-friendly introduction to JavaScript.",
  "image": "https://tse.mm.bing.net/th/id/example",
  "duration": "51:05",
  "published": "2024-02-10T00:00:00.0000000",
  "publisher": "YouTube",
  "uploader": "Traversy Media",
  "viewCount": 2100000,
  "provider": "Bing"
}

Run summary

Each run also writes a RUN_SUMMARY record to the key-value store: query, searchMode, region, safeSearch, requestedMaxItems, pageBudget, pagesAnswered, rowsSaved, duplicatesSkipped, httpRequests, jsChallengesSolved, blocksSeen, exitIpRotations, finalProxyTier, runtimeSeconds and runId — the audit trail for what the run actually did.

⚙️ How does it work?

DuckDuckGo Scraper first loads the DuckDuckGo homepage with your query to harvest the vqd session token (and, for web mode, a dp token) embedded in the page HTML. It then calls the mode's own result endpoint — links.duckduckgo.com/d.js for web, duckduckgo.com/news.js for news, duckduckgo.com/v.js for video — over whichever transport that endpoint accepts, a plain HTTP stack or a Chrome-impersonating one. If DuckDuckGo answers with its inline JavaScript challenge, the Actor solves it offline and replays the request; a genuine block instead triggers a fresh exit IP. It follows DuckDuckGo's own continuation offsets, drops URLs already seen, and pushes each new row. Only publicly visible results are ever returned, and the output schema stays fixed regardless of how DuckDuckGo's page layout changes, because the Actor parses the underlying result payload, not rendered HTML.

🔌 Integrations

DuckDuckGo Scraper works with whatever already calls the Apify API — no DuckDuckGo-specific glue required.

Calling DuckDuckGo Scraper programmatically

from apify_client import ApifyClient

client = ApifyClient("<YOUR_APIFY_API_TOKEN>")
run = client.actor("<your-username>/duckduckgo-scraper").call(run_input={
    "searchKeyword": "apify web scraping",
    "searchMode": "default",
    "region": "us-en",
    "safeSearch": "OFF",
    "maxItems": 50,
})
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item["title"], item["url"])

Works in Go, Ruby, Node.js, cURL — any language that can make an HTTP request against the Apify API.

MCP integration for AI agents

DuckDuckGo Scraper is reachable through Apify's hosted MCP server. Point an MCP-compatible client (Claude Desktop, Cursor, VS Code) at:

https://mcp.apify.com?tools=<your-username>/duckduckgo-scraper

authorize with your Apify API token, and the Actor becomes a callable tool the agent can invoke with its own input.

No-code tools (n8n, Make, LangChain)

In n8n, use the HTTP Request node pointed at the Actor's run-sync-get-dataset-items endpoint with your Apify token, or Apify's own community node. In Make, use the Apify app's "Run an Actor and get dataset items" module with duckduckgo-scraper as the Actor ID. In LangChain, wrap the Actor with the langchain-apify integration (ApifyActorsTool / ApifyWrapper) to expose it as a callable tool inside an agent chain.

⚖️ Is it legal to scrape DuckDuckGo search results?

Scraping publicly available search results is generally lawful — DuckDuckGo Scraper only retrieves data any visitor sees on duckduckgo.com without signing in, and does not target or infer information about private individuals. Search result rows (titles, URLs, snippets, publisher names) are business/factual data rather than personal data, so GDPR and CCPA's lawful-basis requirements do not attach to what this Actor returns. Your use is still governed by DuckDuckGo's own Terms of Service, which restrict automated querying, and by database-rights law where it applies — review both before deploying at scale. Consult legal counsel if your use case involves bulk storage of personal data, such as named individuals who appear incidentally in news or video results.

❓ Frequently asked questions

What DuckDuckGo search result fields does DuckDuckGo Scraper return?

The core fields are title, url, description (or excerpt on news rows) and hostname (or publisherName on news, uploader on video). See What data can I extract with DuckDuckGo Scraper? above for the full per-mode field list.

Does DuckDuckGo Scraper require a DuckDuckGo account or login?

No. DuckDuckGo Scraper calls DuckDuckGo's public result endpoints directly and needs no DuckDuckGo account, cookie or API key — only an Apify account to run the Actor.

How many search results can I extract in one run?

Up to 1,000 rows — maxItems (1–1000) is the authoritative cap, and the Actor paginates until it reaches that number, hits the optional endPage page cap, or DuckDuckGo runs out of results. Page sizes are roughly 10 for web, ~30 for news and ~60 for video, and consecutive pages overlap, so unique rows land somewhat below raw page-size math.

What happens if a query returns zero results?

The run finishes with an empty dataset, not an error. DuckDuckGo genuinely returning zero rows for a query — common under STRICT safe search or a narrow keyword — is logged as a warning and treated as a valid empty result, distinct from a block. A block, by contrast, fails the run explicitly after every proxy tier has been tried.

Can I scrape multiple DuckDuckGo searches at once?

Not in a single run — searchKeyword accepts one query string. Loop over the Apify API to start one run per keyword (see How to scale to bulk search extraction above), or trigger repeated runs on an Apify schedule.

Does DuckDuckGo Scraper work with Claude, ChatGPT, and other AI agent tools?

Yes — every run produces a typed JSON dataset reachable over the Apify API, and the Actor is registerable as a tool through Apify's MCP server for MCP-compatible clients such as Claude Desktop and Cursor. See Integrations above for the exact registration URL.

How does DuckDuckGo Scraper compare to other DuckDuckGo scrapers?

As observed on the Apify Store on 2026-08-16: DuckDuckGo SERP Scraper (jaybird) covers six content types including images and instant answers, with a 97.2% run success rate shown on its own listing. DuckDuckGo Scraper (epctex) covers four modes including images but is billed as a $20/month rental on top of usage. DuckDuckGo SERP (s-r) caps results at 30 per run. DuckDuckGo Scraper here covers web, news and video up to 1,000 rows per run on pay-per-event pricing with no rental fee, and documents its anti-bot handling (deterministic JS-challenge solving, automatic proxy escalation) and honest field nulls in the open.

Does DuckDuckGo Scraper return data in a format LLMs can use directly?

Yes. Every row is typed, normalized JSON with the same field names on every run — no HTML to parse and no CSS selectors to write. Pass rows straight into an LLM prompt, index them into a vector store, or hand them to an agent tool call.

What happens when DuckDuckGo changes its layout or anti-bot system?

The output schema is built to stay stable — DuckDuckGo Scraper parses DuckDuckGo's underlying result payloads rather than screen-scraping rendered HTML, so a visual redesign alone does not change the field names you get. No specific update turnaround time is published.

Can I use DuckDuckGo Scraper without managing proxies or browser infrastructure?

Yes. DuckDuckGo Scraper starts on a direct connection and automatically escalates to Apify datacenter and then residential proxies if DuckDuckGo blocks an exit IP — you don't have to configure or pay for proxies unless you choose to override proxyConfiguration yourself.

Which DuckDuckGo search result fields work best for AI training data and RAG indexing?

For RAG indexing, title and description (or excerpt on news rows) carry the highest-information text. For training data, hostname, resultSource / publisherName / publisher, and searchMode are the most consistently structured fields across every row — every field returns as a typed primitive (string, integer or boolean), never a mixed type.

🔗 Related scrapers

Scraper	What it extracts
Google Search Results Scraper	Google organic search results by keyword or full search URL
Google Search Results (SERP) Scraper	Clean Google SERP results with pagination and location/proxy controls
Baidu Search Scraper	Baidu organic listings, answer boxes, related videos and related searches
Google Scholar Scraper	Academic paper search results from Google Scholar sources

💬 Your feedback

Found a bug, or need a DuckDuckGo field this Actor doesn't return yet? Open an issue on the Actor's Issues tab in Apify Console, or message Scraper Engine support directly — reports like these are what keep the field list and anti-bot handling current.