# Duckduckgo Scraper (`scrapepilotapi/duckduckgo-scraper`) Actor - **URL**: https://apify.com/scrapepilotapi/duckduckgo-scraper.md - **Developed by:** [ScrapePilot](https://apify.com/scrapepilotapi) (community) - **Categories:** Automation, Lead generation, SEO tools - **Stats:** 1 total users, 0 monthly users, 0.0% runs succeeded, NaN bookmarks - **User rating**: No ratings yet ## Pricing from $3.99 / 1,000 results This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## DuckDuckGo Scraper A robust Apify Actor for scraping search results from DuckDuckGo across multiple search modes (web, videos, news, images). ### Why Choose Us? - **Multi-Mode Support**: Scrape web results, videos, news, and images from DuckDuckGo - **Intelligent Proxy Fallback**: Automatically switches from direct connection to datacenter to residential proxies if blocked - **Real-Time Data Saving**: Results are saved to the dataset as they're collected, so you can monitor progress live - **Reliable**: Built-in retry logic and error handling - **Structured Output**: Clean, organized data ready for analysis - **65+ Regions Supported**: Search from any region worldwide ### Key Features - ✅ Multiple search modes (default, videos, news, images) - ✅ Automatic proxy fallback mechanism - ✅ Real-time data collection and saving - ✅ Configurable result limits and pagination - ✅ 65+ region options with human-readable names - ✅ Safe search customization - ✅ Detailed logging for monitoring ### Input #### Example Input JSON ```json { "searchKeyword": "Python", "searchMode": "default", "region": "wt-wt", "safeSearch": "OFF", "maxItems": 20, "endPage": 1, "proxyConfiguration": { "useApifyProxy": false } } ```` #### Input Fields - **searchKeyword** (required): The keyword that you can search on DuckDuckGo - **searchMode** (required): What type of results you want to search on at DuckDuckGo - Options: `"default"`, `"images"`, `"news"`, `"videos"` - Default: `"default"` - **region** (required): Region that you want to get the results from DuckDuckGo - Options: 65+ regions including "All regions", "Argentina", "Australia", "Austria", "Belgium (fr)", "Belgium (nl)", "Brazil", "Bulgaria", "Canada (en)", "Canada (fr)", "Catalonia", "Chile", "China", "Colombia", "Croatia", "Czech Republic", "Denmark", "Estonia", "Finland", "France", "Germany", "Greece", "Hong Kong", "Hungary", "India (en)", "Indonesia (en)", "Ireland", "Israel (en)", "Italy", "Japan", "Korea", "Latvia", "Lithuania", "Malaysia (en)", "Mexico", "Netherlands", "New Zealand", "Norway", "Pakistan (en)", "Peru", "Philippines (en)", "Poland", "Portugal", "Romania", "Russia", "Saudi Arabia", "Singapore", "Slovakia", "Slovenia", "South Africa", "Spain (ca)", "Spain (es)", "Sweden", "Switzerland (de)", "Switzerland (fr)", "Taiwan", "Thailand (en)", "Turkey", "US (English)", "US (Spanish)", "Ukraine", "United Kingdom", "Vietnam (en)" and more - Default: `"wt-wt"` (All regions) - **safeSearch** (required): Safe mode when searching over DuckDuckGo - Options: `"OFF"`, `"MODERATE"`, `"STRICT"` - Default: `"OFF"` - **maxItems** (optional): Maximum number of listing items that you want as output. Default is all - Type: integer - Range: 1-1000 - Default: 20 - **endPage** (optional): The page number that you want to end with. By default there is no end page. This applies to all search requests individually. - Type: integer - Range: 0-100 - Default: 1 - Note: Set to 0 or leave empty for auto-calculation based on maxItems - **proxyConfiguration** (optional): Select proxies to be used by your crawler - Default: No proxy (with automatic fallback) ### Output Results are saved to the Apify dataset in real-time with the following structure: #### Default Mode Results ```json { "query": "Python", "searchMode": "default", "title": "Python Programming Language", "url": "https://www.python.org", "description": "Official Python website", "hostname": "www.python.org", "icon": "https://external-content.duckduckgo.com/ip3/www.python.org.ico" } ``` #### Images Mode Results ```json { "query": "Python", "searchMode": "images", "title": "Python Logo", "url": "https://...", "image": "https://...", "thumbnail": "https://...", "source": "example.com", "width": 1200, "height": 800 } ``` #### Video Mode Results ```json { "query": "Python", "searchMode": "videos", "url": "https://youtube.com/watch?v=...", "title": "Learn Python Tutorial", "description": "Complete Python course", "image": "https://...", "duration": "4:26:52", "published": "2018-07-11T18:00:42.0000000", "publishedOn": "YouTube", "publisher": "freeCodeCamp.org", "viewCount": 48169815 } ``` #### News Mode Results ```json { "query": "Python", "searchMode": "news", "title": "Python News Article", "url": "https://...", "excerpt": "Article excerpt...", "date": 1767079027, "relativeTime": "11 hours ago", "syndicate": "bing", "image": "https://...", "isOld": false } ``` ### 🚀 How to Use the Actor (via Apify Console) 1. Log in at https://console.apify.com and go to **Actors** 2. Find your actor (duckduckgo-scraper) and click it 3. Configure inputs: - Enter your search keyword - Select search mode (default, images, videos, or news) - Choose region from the dropdown (shows human-readable names) - Set safe search level - Configure max items and end page (optional) - Configure proxy if needed 4. Run the actor 5. Monitor logs in real time to see progress 6. Watch results appear in the **OUTPUT** tab as they're collected 7. Export results to JSON or CSV when complete ### Best Use Cases - **Market Research**: Gather information about competitors, products, or trends - **Content Discovery**: Find relevant videos, articles, and images - **SEO Analysis**: Research keywords and search results - **News Monitoring**: Track news articles on specific topics - **Data Collection**: Build datasets for machine learning or analysis - **Regional Research**: Get region-specific search results from 65+ countries ### Proxy Configuration The actor implements intelligent proxy fallback: 1. **Default**: Starts with no proxy (direct connection) 2. **First Fallback**: If DuckDuckGo rejects or blocks the request, automatically switches to datacenter proxy 3. **Second Fallback**: If datacenter fails, switches to residential proxy with 3 retries 4. **Persistence**: Once a fallback occurs, the actor sticks with that proxy for all remaining requests All proxy events are handled automatically. The actor will seamlessly switch proxies if needed without user intervention. ### Frequently Asked Questions **Q: Can I search multiple keywords?**\ A: The actor processes one keyword at a time. To search multiple keywords, run the actor multiple times or use Apify's scheduling feature. **Q: What happens if DuckDuckGo blocks my requests?**\ A: The actor automatically falls back to datacenter, then residential proxies with 3 retries. All proxy switching happens automatically. **Q: How many results can I get?**\ A: You can configure `maxItems` up to 1000. If not specified, the actor will collect all available results (up to 100 pages by default). **Q: Can I scrape images?**\ A: Yes! Set `searchMode` to `"images"` to scrape image results. **Q: Are the results saved in real-time?**\ A: Yes! Results are saved to the dataset as they're collected, so you can monitor progress in the Apify console output table in real-time. **Q: What regions are supported?**\ A: The actor supports 65+ regions worldwide, including all major countries and languages. Select from the dropdown in the Apify console to see all available options. **Q: What does "endPage" mean?**\ A: The page number to stop scraping at. If set to 0 or left empty, the actor will auto-calculate based on `maxItems`. If `maxItems` is also not set, it will scrape up to 100 pages. **Q: Can I see results while the actor is running?**\ A: Yes! Results are saved in real-time, so you can refresh the OUTPUT tab in Apify console to see results as they're collected. ### Support and Feedback 💬 For custom solutions or feature requests, contact us at dev.scraperengine@gmail.com ### Cautions - Data is collected only from publicly available sources - No data is taken from private or password-protected content - The end user is responsible for ensuring legal compliance (spam laws, privacy, data protection, etc.) - Respect DuckDuckGo's terms of service and rate limits - Use responsibly and in accordance with applicable laws and regulations # Actor input Schema ## `searchKeyword` (type: `string`): ✍️ What should DuckDuckGo look for? (e.g. “best pizza in Rome”, “react hooks tutorial”, a brand name…) ## `searchMode` (type: `string`): 📂 Pick the kind of results you want back — classic web links, 🖼️ images, 📰 headlines, or 🎬 videos. ## `region` (type: `string`): 🗺️ Results feel different per country — choose where DuckDuckGo should “search from” for the most relevant local mix. ## `safeSearch` (type: `string`): 👨‍👩‍👧 Filter how strict DuckDuckGo is about adult or sensitive content — OFF is widest, STRICT is most family-friendly. ## `maxItems` (type: `integer`): 🧮 Cap how many rows land in your dataset (1–1000). Lower = faster & cheaper; higher = deeper coverage. ## `endPage` (type: `integer`): 🔢 Stop after this page number (0–100). Handy when you only need the first few SERPs — applies per search / start URL. ## `proxyConfiguration` (type: `object`): 🚀 Route requests through datacenter or residential IPs to reduce blocks — turn on when you scale up or hit rate limits. ## Actor input object example ```json { "searchKeyword": "javascript", "searchMode": "default", "region": "wt-wt", "safeSearch": "OFF", "maxItems": 20, "endPage": 3, "proxyConfiguration": { "useApifyProxy": false } } ``` # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "proxyConfiguration": { "useApifyProxy": false } }; // Run the Actor and wait for it to finish const run = await client.actor("scrapepilotapi/duckduckgo-scraper").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "proxyConfiguration": { "useApifyProxy": False } } # Run the Actor and wait for it to finish run = client.actor("scrapepilotapi/duckduckgo-scraper").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "proxyConfiguration": { "useApifyProxy": false } }' | apify call scrapepilotapi/duckduckgo-scraper --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=scrapepilotapi/duckduckgo-scraper", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "Duckduckgo Scraper", "version": "0.1", "x-build-id": "OlaIwg1bV9X8vbl7s" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/scrapepilotapi~duckduckgo-scraper/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-scrapepilotapi-duckduckgo-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/scrapepilotapi~duckduckgo-scraper/runs": { "post": { "operationId": "runs-sync-scrapepilotapi-duckduckgo-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/scrapepilotapi~duckduckgo-scraper/run-sync": { "post": { "operationId": "run-sync-scrapepilotapi-duckduckgo-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "required": [ "searchKeyword", "searchMode", "region", "safeSearch" ], "properties": { "searchKeyword": { "title": "🔑 Search keyword", "type": "string", "description": "✍️ What should DuckDuckGo look for? (e.g. “best pizza in Rome”, “react hooks tutorial”, a brand name…)", "default": "javascript" }, "searchMode": { "title": "🎯 Search mode", "enum": [ "default", "images", "news", "videos" ], "type": "string", "description": "📂 Pick the kind of results you want back — classic web links, 🖼️ images, 📰 headlines, or 🎬 videos.", "default": "default" }, "region": { "title": "🌍 Region & language", "enum": [ "wt-wt", "ar-es", "au-en", "at-de", "be-fr", "be-nl", "br-pt", "bg-bg", "ca-en", "ca-fr", "ct-ca", "cl-es", "cn-zh", "co-es", "hr-hr", "cz-cs", "dk-da", "ee-et", "fi-fi", "fr-fr", "de-de", "gr-el", "hk-tzh", "hu-hu", "in-en", "id-en", "ie-en", "il-en", "it-it", "jp-jp", "kr-kr", "lv-lv", "lt-lt", "my-en", "mx-es", "nl-nl", "nz-en", "no-no", "pk-en", "pe-es", "ph-en", "pl-pl", "pt-pt", "ro-ro", "ru-ru", "xa-ar", "sg-en", "sk-sk", "sl-sl", "za-en", "es-ca", "es-es", "se-sv", "ch-de", "ch-fr", "tw-tzh", "th-en", "tr-tr", "us-en", "us-es", "ua-uk", "uk-en", "vn-en" ], "type": "string", "description": "🗺️ Results feel different per country — choose where DuckDuckGo should “search from” for the most relevant local mix.", "default": "wt-wt" }, "safeSearch": { "title": "🛡️ Safe search", "enum": [ "OFF", "MODERATE", "STRICT" ], "type": "string", "description": "👨‍👩‍👧 Filter how strict DuckDuckGo is about adult or sensitive content — OFF is widest, STRICT is most family-friendly.", "default": "OFF" }, "maxItems": { "title": "📊 Max results to save", "minimum": 1, "maximum": 1000, "type": "integer", "description": "🧮 Cap how many rows land in your dataset (1–1000). Lower = faster & cheaper; higher = deeper coverage.", "default": 20 }, "endPage": { "title": "📄 Last page to scrape", "minimum": 0, "maximum": 100, "type": "integer", "description": "🔢 Stop after this page number (0–100). Handy when you only need the first few SERPs — applies per search / start URL.", "default": 3 }, "proxyConfiguration": { "title": "🌐 Proxy configuration", "type": "object", "description": "🚀 Route requests through datacenter or residential IPs to reduce blocks — turn on when you scale up or hit rate limits." } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```