Duckduckgo Search Scraper

A powerful web scraping actor built with Apify SDK and Crawlee that extracts search results from DuckDuckGo in multiple search modes.

Features

• Multiple Search Modes: Default search, image search, news search, and video search
• Batch Processing: Process multiple search queries in a single run (comma-separated or array)
• Auto Pagination: Automatically navigate through multiple pages to collect more results
• SEO Data Extraction: Extract meta tags, structured data, and page insights
• Social Media Data: Extract Open Graph and Twitter Card meta tags
• Configurable Proxy: Optional Apify proxy integration for better scraping success
• Robust Extraction: Intelligent title and snippet extraction with fallback mechanisms
• Anti-Detection: Built-in measures to avoid being blocked
• Screenshot Support: Optional debugging screenshots for troubleshooting
• Clean Output: Results in a clean, structured array format

Search Modes

• Default: Standard web search results
• Images: Image search results with thumbnails
• News: News article search results
• Videos: Video search results

Input Parameters

• searchQuery (required): The search term to search for (can be comma-separated for multiple queries)

• multipleSearchQueries (optional): Array of search terms for batch processing

• searchMode (required): Which search mode to use (default, images, news, videos)

• maxResults: Maximum number of results to extract (default: 10)

• maxPages: Maximum number of pages to scrape (default: 1, max: 10)

• autoPagination: Automatically navigate to next pages (default: false)

• timeout: Page load timeout in seconds (default: 60)

• extractSEOData: Extract additional SEO information (default: false)

• extractSocialMedia: Extract social media meta tags (default: false)

• proxyConfiguration: Proxy configuration object

Output Format

The actor outputs a clean array of search results, each containing:

{
  "title": "Readable title of the result",
  "url": "URL of the result",
  "snippet": "Description or snippet text",
  "position": 1,
  "type": "default|images|news|videos",
  "page": 1
}

Enhanced Output (when SEO features enabled)

When extractSEOData or extractSocialMedia are enabled, additional fields are included:

{
  "title": "Readable title of the result",
  "url": "URL of the result",
  "snippet": "Description or snippet text",
  "position": 1,
  "type": "default|images|news|videos",
  "seoData": {
    "metaTitle": "Page title from meta tag",
    "metaDescription": "Meta description",
    "canonicalUrl": "Canonical URL",
    "structuredData": "JSON-LD structured data"
  },
  "socialMedia": {
    "openGraph": {
      "title": "Open Graph title",
      "description": "Open Graph description",
      "image": "Open Graph image URL"
    },
    "twitterCard": {
      "title": "Twitter Card title",
      "description": "Twitter Card description",
      "image": "Twitter Card image URL"
    }
  }
}

Legal

This actor is provided for educational and research purposes only. Users are responsible for:

• Complying with DuckDuckGo's Terms of Service
• Respecting robots.txt and website terms
• Following applicable laws and regulations
• Using the tool responsibly and ethically

The developers are not responsible for any misuse of this tool or violations of terms of service.

Usage Examples

Single Search Query

{
  "searchQuery": "artificial intelligence",
  "searchMode": "default",
  "maxResults": 10
}

Multiple Search Queries (comma-separated)

{
  "searchQuery": "machine learning, deep learning, neural networks",
  "searchMode": "default",
  "maxResults": 5
}

Multiple Search Queries (array)

{
  "multipleSearchQueries": ["SEO optimization", "content marketing", "link building"],
  "searchMode": "default",
  "maxResults": 5
}

SEO Data Extraction

{
  "searchQuery": "web development",
  "searchMode": "default",
  "maxResults": 5,
  "extractSEOData": true,
  "extractSocialMedia": true
}

Image Search with SEO

{
  "searchQuery": "nature photography",
  "searchMode": "images",
  "maxResults": 10,
  "extractSEOData": true
}

Multi-Page Search with Pagination

{
  "searchQuery": "web development tutorials",
  "searchMode": "default",
  "maxResults": 50,
  "maxPages": 5,
  "autoPagination": true
}

Batch Search with Pagination

{
  "multipleSearchQueries": ["SEO tips", "content marketing", "link building"],
  "searchMode": "default",
  "maxResults": 30,
  "maxPages": 3,
  "autoPagination": true
}