# DuckDuckGo Scraper: Web, News, Images & Videos (`arman-bd/duckduckgo-search-scraper`) Actor Search DuckDuckGo across web, news, images and videos in one run, with a single flat record shape for all four: title, URL, domain, snippet and rank, plus dimensions, durations and publish dates where they exist. Sponsored rows are labelled, never sold as organic. - **URL**: https://apify.com/arman-bd/duckduckgo-search-scraper.md - **Developed by:** [Arman Hossain](https://apify.com/arman-bd) (community) - **Categories:** SEO tools, AI, MCP servers - **Stats:** 1 total users, 0 monthly users, 0.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $0.18 / 1,000 result scrapeds This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. - **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use). - **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`. - **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`. - **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`. - **Any other language** — the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## DuckDuckGo Scraper: Web, News, Images & Videos ![DuckDuckGo Search: Web, news, images and videos from one query, one flat row per result, ads labelled not ranked](https://api.apify.com/v2/key-value-stores/ZQOcNAOHrIgTacAmy/records/duckduckgo-search-scraper.jpg) **DuckDuckGo Scraper** searches **DuckDuckGo** across four surfaces — web, news, images and videos — and returns one flat row per result, with the same shape on all four: title, URL, domain, snippet and the rank it held for your query, plus pixel dimensions, durations, view counts and publish dates on the verticals that have them. Every selected vertical runs for every query, so one run gives you a single table you can filter instead of four exports to join. Advertising rows sit in the same list as ranked results upstream. Here they are excluded by default and flagged `isSponsored: true` when you ask for them — never counted as rank 3. **No API key, no quota, no proxy setup and no browser.** **Agent skill: [SKILL.md](https://api.apify.com/v2/key-value-stores/t7YoTxpZEJOWvw4Ug/records/duckduckgo-search-scraper.md)** ``` https://api.apify.com/v2/key-value-stores/t7YoTxpZEJOWvw4Ug/records/duckduckgo-search-scraper.md ``` ### What you get - **Four surfaces from one query.** Web, news, images and videos each run for every keyword you give, in a single run. - **One table, not four.** Every row carries the same 25 keys whichever vertical produced it, so the result is something you filter rather than four exports to join. - **Rank you can trust.** Advertising rows are excluded by default, and flagged `isSponsored` rather than renumbered when you ask for them, so position 3 is always the third organic result. - **The fields each surface actually has** — pixel dimensions and formats on images, durations, uploaders and view counts on videos, outlets and publish dates on news — with `null` meaning *this surface has no such value*, never a quietly dropped column. - **A `RUN_SUMMARY`** recording every query run, what each vertical returned and saved, duplicates collapsed, throttling seen, and any query the source could not answer. - **Nothing to configure**: no API key, no quota, no proxy setup and no browser. ### Input | Field | Type | Default | Notes | |---|---|---|---| | `queries` | array | required | Search terms, one per line. Operators work: `"quoted phrase"`, `site:reuters.com`, `intitle:changelog`, `-excluded`, `a OR b` | | `verticals` | array | `["web"]` | Any of `web`, `news`, `images`, `videos`. Every selected vertical runs for every query | | `maxResultsPerVertical` | integer | `50` | Cap per query **per vertical**, 1–1000. Four verticals at 50 is up to 200 rows for one query, and this is the main cost control | | `region` | string | `us-en` | Locale for news, images and videos. **Does not apply to the web vertical** — see Limits | | `timeRange` | string | `any` | `any`, `day`, `week`, `month`, `year`. **Web vertical only** — see Limits | | `includeSponsored` | boolean | `false` | Keep advertising rows, flagged `isSponsored: true` | | `proxyConfiguration` | object | off | Off by default. Turn it on for deep web paging — see Limits | Operators compose exactly as you would type them, and they were each verified against the live source rather than assumed: `site:reuters.com openai` returns reuters hosts only, `"large language model" benchmark` narrows to papers, `jaguar -car` drops the car maker, and `intitle:` and `OR` both behave. ```json { "queries": ["lithium battery recycling", "site:reuters.com openai"], "verticals": ["web", "news", "images"], "maxResultsPerVertical": 100, "region": "us-en" } ``` ### Output One record per result, the same 25 keys on every row whichever vertical produced it. Every key is always present, and `null` always means *this surface does not have that value* — never that a column was quietly dropped. | Output field | Meaning | Populated on | |---|---|---| | `query` | Which input produced the row | all | | `vertical` | `web`, `news`, `images` or `videos` | all | | `position` | Rank within its query and vertical. **`null` on an advertising row**, which holds no rank — the organic rows stay 1, 2, 3… whether ads were kept or dropped | all except sponsored rows | | `title` | The result title | all | | `url` | Web: the result. News: the article. Images: the page hosting the picture. Videos: the watch page | all | | `domain` | Host of `url`, lowercased and without `www.` — the field you group by. On an advertising row it names the advertiser instead, because the link itself points at the ad network | all | | `snippet` | Web: the result summary. News: the article excerpt. Videos: the description. **`null` on images** | web, news, videos | | `displayUrl` | The shortened URL as shown. A *display* string and truncated, which is why `domain` is derived from the real link instead | web | | `isSponsored` | `true` only on advertising rows, and only when you asked for them | all | | `source` | News: the outlet. Videos: the platform hosting the video | news, videos | | `publishedAt` | ISO 8601, UTC | news, videos | | `relativeTime` | The source's own wording, e.g. `3 hours ago`, kept unparsed beside `publishedAt` | news | | `thumbnailUrl` | Preview image. **News carries one on about a third of rows** — see Limits | news, images, videos | | `imageUrl` | The full-size picture | images | | `width` / `height` | Pixel dimensions of `imageUrl` | images | | `encodingFormat` | `jpeg`, `png`, … | images | | `discoveredAt` | ISO 8601, when the picture entered the index. Not a publish date, and not named like one | images | | `duration` | As given, e.g. `11:57` | videos | | `uploader` | Channel name | videos | | `viewCount` | Integer | videos | | `embedUrl` | Player URL | videos | | `syndicate` | The upstream index the row is credited to, as the source itself labels it | news, images, videos | | `region` | The locale actually applied. **`null` on web**, which is not region-targetable | news, images, videos | | `scrapedAt` | Run timestamp, ISO 8601 | all | A web row and an image row, so you can see what `null` looks like before paying to find out: ```json { "query": "renewable energy storage", "vertical": "web", "position": 1, "title": "Renewable Energy Storage: Complete Guide to Technologies, Benefits ...", "url": "https://solartechonline.com/blog/renewable-energy-storage-guide/", "domain": "solartechonline.com", "snippet": "Comprehensive guide to renewable energy storage technologies, costs, benefits, and applications. Compare battery, mechanical, and thermal storage systems for 2025.", "displayUrl": "solartechonline.com/blog/renewable-energy-storage-guide/", "isSponsored": false, "source": null, "publishedAt": null, "relativeTime": null, "thumbnailUrl": null, "imageUrl": null, "width": null, "height": null, "encodingFormat": null, "discoveredAt": null, "duration": null, "uploader": null, "viewCount": null, "embedUrl": null, "syndicate": null, "region": null, "scrapedAt": "2026-08-09T12:41:07.000Z" } ``` ```json { "query": "renewable energy storage", "vertical": "images", "position": 1, "title": "Press releases, events and articles - GreenPowerMonitor", "url": "https://www.greenpowermonitor.com/resources/insights/", "domain": "greenpowermonitor.com", "snippet": null, "displayUrl": null, "isSponsored": false, "source": null, "publishedAt": null, "relativeTime": null, "thumbnailUrl": "https://tse3.mm.bing.net/th/id/OIP.glZCZRNZzuRoHhRNiA9AtgHaFl?r=0&pid=Api", "imageUrl": "https://www.greenpowermonitor.com/wp-content/uploads/2025/04/AdobeStock_624255795-scaled.jpeg", "width": 2560, "height": 1931, "encodingFormat": "jpeg", "discoveredAt": "2024-02-27T00:00:00.000Z", "duration": null, "uploader": null, "viewCount": null, "embedUrl": null, "syndicate": "Bing", "region": "us-en", "scrapedAt": "2026-08-09T12:41:07.000Z" } ``` A `RUN_SUMMARY` record in the key-value store holds the queries and verticals requested, the filters actually applied and which verticals each one applied to, requests made per surface, results saved per query per vertical with the reason each one stopped, duplicates collapsed, advertising rows dropped or labelled, which query-and-vertical pairs came back empty, every point at which the source stopped answering **with the offset it stopped at**, inputs that were rejected and why, and any field that failed its shape check. Failures are named there rather than left to look like a thin result set. ### Use cases - **A search tool for an agent.** Keyword in, ranked URLs and one-line summaries out, in one API call, with nothing to provision. - **Entity enrichment.** Resolve thousands of company or product names to a canonical homepage and a one-line description; group by `domain` and take rank 1. - **Image and video corpus building.** One image request returns roughly 95 rows and one video request roughly 58, so a keyword list becomes a labelled media set quickly. `width`, `height` and `encodingFormat` let you filter before downloading anything. - **Rank tracking off Google.** Watch which domains hold the top ten for your terms on an index that is not Google's, and diff runs on `url`. - **Coverage checks across countries.** Run the same news query at several locales and compare which outlets answer. ### Limits and behaviour - **This is DuckDuckGo's index, and it is not Google.** The results are largely Bing-derived — the payloads say so themselves in the `syndicate` field, which reads `bing` on news rows and `Bing` on image and video rows. If you need Google's ranking specifically, this is the wrong Actor. - **`region` does nothing on the web vertical.** Web results track where the run itself is, not the locale you ask for, so the parameter is not sent and web rows carry `region: null` rather than claiming a targeting they did not get. On news it decides which country's outlets answer; on images and videos it changes the result set without translating it. The list holds only locales that were verified to answer for their own country — twenty of the sixty-three offered upstream returned nothing at all, and two returned US English results under another country's name, so none of those are offered here. - **`timeRange` applies to the web vertical only, and cannot be shown per row.** It measurably changes which results come back, but the web surface publishes no per-row date, so `publishedAt` stays `null` on web and no row can prove which window it came from. The Actor passes the preference through and says so rather than inventing a date. - **There is no safe-search control.** Toggling it upstream returned the same number of rows with no observable difference, so shipping it as a field would be shipping a switch that does nothing. Several competing Actors offer one. - **Deep web paging needs the proxy option.** The index goes deep — the same query keeps returning fresh results hundreds of positions in — but an unrotated run gets roughly six web pages before the source stops answering it, and that does not clear inside the run. Without the proxy the Actor stops web paging there, keeps everything it already collected, records the point it stopped at in `RUN_SUMMARY`, and carries on with the other verticals, which are unaffected. **That allowance belongs to the run, not to each query**, so a web run over a long query list spends it on the first few queries and the rest come back with no web rows — they are named in `RUN_SUMMARY.emptyResults` and the run logs a warning at the start. Turn the proxy on for the web vertical whenever you have more than about six queries, or want more than about six pages. An images-only or videos-only run never needs a proxy at any depth. - **Depth is not identical run to run.** The same query can return a different count an hour later. Treat result counts as a range, not a promise. - **News rows mostly have no picture.** `thumbnailUrl` is populated on roughly a third of them. That is the source, not a parse failure, and nothing is substituted to make the column look fuller. - **Advertising rows are excluded by default.** They exist in the source's own result list; here they are dropped unless `includeSponsored` is on, are never charged for when dropped, and are never given an organic rank. When you do keep them, `domain` names the advertiser rather than the ad network. - **Duplicates are collapsed within a query and vertical.** Images collapse on the picture, not on the page: one page legitimately hosts dozens of distinct pictures, and collapsing those would throw most of a result set away. - **One bad input never ends the run.** An unrecognised vertical or an empty query line lands in `RUN_SUMMARY.rejectedInputs` and everything else proceeds. The run fails only if every query on every vertical failed. - **Public data only.** No credentials, and nothing that works around access controls. ### FAQ **Is this Google?** No, and that matters. It is DuckDuckGo's index, which is largely Bing-derived — the rows say so themselves in `syndicate`. Rankings, snippets and the domain mix all differ from Google's. If your work depends on Google's ordering specifically, use a Google SERP Actor instead. **Why does an image query return 400 rows and a news query 91?** Because those are the sizes of the underlying result sets. One image request carries roughly 95 rows and pages several times over; news is much thinner and runs out after a handful of requests. `maxResultsPerVertical` is what you use to make them comparable, and the two are priced separately so a wide image query does not cost more than the web run beside it. **Why is `snippet` null on images?** An image row has no text field of any kind — no caption, no description. The alternative would be to copy the picture's title into `snippet` so the column looks populated, which is worse than the honest null. **Do search operators work?** Yes. `site:`, `"quoted phrases"`, `-exclusions`, `intitle:` and `OR` were each tried against the live source and each narrowed the results as they should. Write them into `queries` exactly as you would type them. **How do I run 10,000 keywords?** Put them all in `queries` — the Actor takes a list, so it is one run and one start fee rather than one run per keyword. Keep `maxResultsPerVertical` low (10–20 is plenty for enrichment). On the `web` vertical **turn the proxy on**: the six-page web allowance is per run and shared by every query, so an unrotated run of 10,000 keywords returns web rows for roughly the first six and nothing for the rest. The news, images and videos verticals have no such limit and need no proxy at any query count. **When should I turn the proxy on?** Whenever a run asks the **web** vertical for more than about six pages in total — deep rank tracking, a large `maxResultsPerVertical`, or simply more than about six queries, since the allowance is per run and every query draws on it. Leave it off for a handful of web queries, and for news, images or videos at any depth and any query count. **Why do some rows have `isSponsored: true`?** Only because you set `includeSponsored`. They are advertising placements that the source returns inside its own result list. They carry `position: null` rather than a rank, and the organic rows keep the same 1, 2, 3… they would have had with the ads dropped, so a ranking built on `position` is unaffected by the flag. **Can I plug it into something else?** Yes. Apify API, the client libraries, webhooks, scheduled runs, dataset exports to JSON, CSV or Excel, or MCP. The output is structured JSON. ### API example ```bash curl -X POST "https://api.apify.com/v2/acts/arman-bd~duckduckgo-search-scraper/run-sync-get-dataset-items?token=YOUR_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "queries": ["lithium battery recycling"], "verticals": ["web", "news"], "maxResultsPerVertical": 25 }' ``` ### JavaScript example ```js import { ApifyClient } from 'apify-client'; const client = new ApifyClient({ token: 'YOUR_TOKEN' }); const run = await client.actor('arman-bd/duckduckgo-search-scraper').call({ queries: ['site:reuters.com openai', 'grid scale battery'], verticals: ['web', 'news', 'images', 'videos'], maxResultsPerVertical: 50, region: 'us-en', }); const { items } = await client.dataset(run.defaultDatasetId).listItems(); for (const r of items) { console.log(`${r.vertical} #${r.position} ${r.domain} — ${r.title}`); } ``` # Actor input Schema ## `queries` (type: `array`): What you would type into the search box, one per line. Operators work: "quoted phrase", site:reuters.com, intitle:changelog, -excluded, a OR b. Each query is searched independently on every vertical you select. ## `verticals` (type: `array`): Which surfaces to search. Every selected vertical runs for every query and returns the same flat record shape, so four verticals for one query is one table you can filter rather than four exports to join. Images return roughly 95 rows per request and videos roughly 58, so adding them multiplies the row count. ## `maxResultsPerVertical` (type: `integer`): Cap on results for each query on each vertical. Four verticals at 50 is up to 200 rows for a single query. This is the main cost control, since charging is per result. ## `region` (type: `string`): Locale for the news, images and videos verticals. On news it decides which country's outlets answer. On images and videos it reorders the results rather than translating them, so expect a different set in the same language. The web vertical is not region-targetable and ignores this entirely: web rows carry region: null so a row never claims a targeting it did not get. Only locales that were verified to return their own country's news are listed. ## `timeRange` (type: `string`): Restrict the web vertical to recently published pages. It measurably changes which results come back, but the web surface publishes no per-row date, so no row can show you which window it came from — publishedAt stays null on web either way. It does not apply to news, images or videos. ## `includeSponsored` (type: `boolean`): Advertising rows sit in the same list as ranked results upstream. Off by default they are excluded and never charged for; on, they are kept and flagged isSponsored: true with position: null. They are never numbered alongside organic results either way — the organic rows are ranked 1, 2, 3... whichever way you set this — because an ad counted as rank 3 corrupts every ranking built on the output. ## `proxyConfiguration` (type: `object`): Off by default, and a short web run does not need it. Turn it on for the web vertical whenever a run wants more than about six web pages in total: without rotation a run gets roughly six before the source stops answering, that does not clear inside the run, and the allowance belongs to the run rather than to each query, so a long query list spends it on the first few and the rest come back with no web rows. News, images and videos are unaffected either way, so an images-only or videos-only run never needs this at any depth or query count. ## Actor input object example ```json { "queries": [ "site:reuters.com openai", "\"large language model\" benchmark" ], "verticals": [ "web" ], "maxResultsPerVertical": 50, "region": "us-en", "timeRange": "any", "includeSponsored": false, "proxyConfiguration": { "useApifyProxy": false } } ``` # Actor output Schema ## `items` (type: `string`): Every result the run produced, one flat row per result. ## `runsummary` (type: `string`): The RUN\_SUMMARY record from the run's key-value store. # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "queries": [ "lithium battery recycling" ] }; // Run the Actor and wait for it to finish const run = await client.actor("arman-bd/duckduckgo-search-scraper").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "queries": ["lithium battery recycling"] } # Run the Actor and wait for it to finish run = client.actor("arman-bd/duckduckgo-search-scraper").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}") for item in client.dataset(run.default_dataset_id).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "queries": [ "lithium battery recycling" ] }' | apify call arman-bd/duckduckgo-search-scraper --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "type": "http", "url": "https://mcp.apify.com/?tools=fetch-actor-details,arman-bd/duckduckgo-search-scraper" } } } ``` The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer ` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations). ## OpenAPI specification Download the OpenAPI definition: https://api.apify.com/v2/actors/tkhyxzbdklJd00Qe7/builds/gdIKscPzHOxaAFiyX/openapi.json