Duckduckgo Scraper

A robust Apify Actor for scraping search results from DuckDuckGo across multiple search modes (web, videos, news, images).

Why Choose Us?

• Multi-Mode Support: Scrape web results, videos, news, and images from DuckDuckGo
• Intelligent Proxy Fallback: Automatically switches from direct connection to datacenter to residential proxies if blocked
• Real-Time Data Saving: Results are saved to the dataset as they're collected, so you can monitor progress live
• Reliable: Built-in retry logic and error handling
• Structured Output: Clean, organized data ready for analysis
• 65+ Regions Supported: Search from any region worldwide

Key Features

• ✅ Multiple search modes (default, videos, news, images)
• ✅ Automatic proxy fallback mechanism
• ✅ Real-time data collection and saving
• ✅ Configurable result limits and pagination
• ✅ 65+ region options with human-readable names
• ✅ Safe search customization
• ✅ Detailed logging for monitoring

Input

Example Input JSON

{
  "searchKeyword": "Python",
  "searchMode": "default",
  "region": "wt-wt",
  "safeSearch": "OFF",
  "maxItems": 20,
  "endPage": 1,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}

Input Fields

• searchKeyword (required): The keyword that you can search on DuckDuckGo
• searchMode (required): What type of results you want to search on at DuckDuckGo
  • Options: "default", "images", "news", "videos"
  • Default: "default"
• region (required): Region that you want to get the results from DuckDuckGo
  • Options: 65+ regions including "All regions", "Argentina", "Australia", "Austria", "Belgium (fr)", "Belgium (nl)", "Brazil", "Bulgaria", "Canada (en)", "Canada (fr)", "Catalonia", "Chile", "China", "Colombia", "Croatia", "Czech Republic", "Denmark", "Estonia", "Finland", "France", "Germany", "Greece", "Hong Kong", "Hungary", "India (en)", "Indonesia (en)", "Ireland", "Israel (en)", "Italy", "Japan", "Korea", "Latvia", "Lithuania", "Malaysia (en)", "Mexico", "Netherlands", "New Zealand", "Norway", "Pakistan (en)", "Peru", "Philippines (en)", "Poland", "Portugal", "Romania", "Russia", "Saudi Arabia", "Singapore", "Slovakia", "Slovenia", "South Africa", "Spain (ca)", "Spain (es)", "Sweden", "Switzerland (de)", "Switzerland (fr)", "Taiwan", "Thailand (en)", "Turkey", "US (English)", "US (Spanish)", "Ukraine", "United Kingdom", "Vietnam (en)" and more
  • Default: "wt-wt" (All regions)
• safeSearch (required): Safe mode when searching over DuckDuckGo
  • Options: "OFF", "MODERATE", "STRICT"
  • Default: "OFF"
• maxItems (optional): Maximum number of listing items that you want as output. Default is all
  • Type: integer
  • Range: 1-1000
  • Default: 20
• endPage (optional): The page number that you want to end with. By default there is no end page. This applies to all search requests individually.
  • Type: integer
  • Range: 0-100
  • Default: 1
  • Note: Set to 0 or leave empty for auto-calculation based on maxItems
• proxyConfiguration (optional): Select proxies to be used by your crawler
  • Default: No proxy (with automatic fallback)

Output

Results are saved to the Apify dataset in real-time with the following structure:

Default Mode Results

{
  "query": "Python",
  "searchMode": "default",
  "title": "Python Programming Language",
  "url": "https://www.python.org",
  "description": "Official Python website",
  "hostname": "www.python.org",
  "icon": "https://external-content.duckduckgo.com/ip3/www.python.org.ico"
}

Images Mode Results

{
  "query": "Python",
  "searchMode": "images",
  "title": "Python Logo",
  "url": "https://...",
  "image": "https://...",
  "thumbnail": "https://...",
  "source": "example.com",
  "width": 1200,
  "height": 800
}

Video Mode Results

{
  "query": "Python",
  "searchMode": "videos",
  "url": "https://youtube.com/watch?v=...",
  "title": "Learn Python Tutorial",
  "description": "Complete Python course",
  "image": "https://...",
  "duration": "4:26:52",
  "published": "2018-07-11T18:00:42.0000000",
  "publishedOn": "YouTube",
  "publisher": "freeCodeCamp.org",
  "viewCount": 48169815
}

News Mode Results

{
  "query": "Python",
  "searchMode": "news",
  "title": "Python News Article",
  "url": "https://...",
  "excerpt": "Article excerpt...",
  "date": 1767079027,
  "relativeTime": "11 hours ago",
  "syndicate": "bing",
  "image": "https://...",
  "isOld": false
}

🚀 How to Use the Actor (via Apify Console)

1. Log in at https://console.apify.com and go to Actors
2. Find your actor (duckduckgo-scraper) and click it
3. Configure inputs:
   • Enter your search keyword
   • Select search mode (default, images, videos, or news)
   • Choose region from the dropdown (shows human-readable names)
   • Set safe search level
   • Configure max items and end page (optional)
   • Configure proxy if needed
4. Run the actor
5. Monitor logs in real time to see progress
6. Watch results appear in the OUTPUT tab as they're collected
7. Export results to JSON or CSV when complete

Best Use Cases

• Market Research: Gather information about competitors, products, or trends
• Content Discovery: Find relevant videos, articles, and images
• SEO Analysis: Research keywords and search results
• News Monitoring: Track news articles on specific topics
• Data Collection: Build datasets for machine learning or analysis
• Regional Research: Get region-specific search results from 65+ countries

Proxy Configuration

The actor implements intelligent proxy fallback:

1. Default: Starts with no proxy (direct connection)
2. First Fallback: If DuckDuckGo rejects or blocks the request, automatically switches to datacenter proxy
3. Second Fallback: If datacenter fails, switches to residential proxy with 3 retries
4. Persistence: Once a fallback occurs, the actor sticks with that proxy for all remaining requests

All proxy events are handled automatically. The actor will seamlessly switch proxies if needed without user intervention.

Frequently Asked Questions

Q: Can I search multiple keywords?
A: The actor processes one keyword at a time. To search multiple keywords, run the actor multiple times or use Apify's scheduling feature.

Q: What happens if DuckDuckGo blocks my requests?
A: The actor automatically falls back to datacenter, then residential proxies with 3 retries. All proxy switching happens automatically.

Q: How many results can I get?
A: You can configure maxItems up to 1000. If not specified, the actor will collect all available results (up to 100 pages by default).

Q: Can I scrape images?
A: Yes! Set searchMode to "images" to scrape image results.

Q: Are the results saved in real-time?
A: Yes! Results are saved to the dataset as they're collected, so you can monitor progress in the Apify console output table in real-time.

Q: What regions are supported?
A: The actor supports 65+ regions worldwide, including all major countries and languages. Select from the dropdown in the Apify console to see all available options.

Q: What does "endPage" mean?
A: The page number to stop scraping at. If set to 0 or left empty, the actor will auto-calculate based on maxItems. If maxItems is also not set, it will scrape up to 100 pages.

Q: Can I see results while the actor is running?
A: Yes! Results are saved in real-time, so you can refresh the OUTPUT tab in Apify console to see results as they're collected.

Support and Feedback

💬 For custom solutions or feature requests, contact us at dev.scraperengine@gmail.com

Cautions

• Data is collected only from publicly available sources
• No data is taken from private or password-protected content
• The end user is responsible for ensuring legal compliance (spam laws, privacy, data protection, etc.)
• Respect DuckDuckGo's terms of service and rate limits
• Use responsibly and in accordance with applicable laws and regulations