Email Scraper

A powerful Apify Actor designed to crawl websites and extract email addresses using advanced detection methods. Built on lightweight HTTP requests, it runs fast while still handling Cloudflare-protected emails, RTL Unicode obfuscation, and other common hiding techniques. Simply provide a list of starting URLs, configure crawl depth, and let it run.

🚀 Features

• Intelligent Email Discovery: Finds email addresses using multiple sophisticated detection methods, including:
  • Standard text pattern matching
  • Mailto links extraction
  • Cloudflare-protected emails
  • RTL (Right-to-Left) Unicode obfuscation
  • Common text obfuscation patterns
  • Multiple recipients in a single mailto: link
  • Cloudflare email-protection links (/cdn-cgi/l/email-protection#…)
• Email Filtering: Optionally drop automated/role addresses (noreply@, postmaster@, …), keep only emails on the same domain as the crawled site, or apply your own include/exclude regular expressions.
• Sitemap-Based Discovery: Reads the site's sitemap.xml (including nested and gzipped sitemaps) to reach contact/about pages directly, instead of relying on link-following alone.
• Keyword-Based Link Prioritization: High-value pages (contact, legal notice, privacy policy, about, team, …, in English and German) are crawled first so emails are found faster, even on large sites.
• Configurable Crawl Depth: Control how deep the crawler follows links from your starting URLs (0-10 levels).
• Domain-Focused or Broad Crawling: Choose to stay on the same domain or explore external links.
• Lightweight HTTP Crawling: Fast, efficient method using HTTP requests without the overhead of a browser.
• Anti-Detection Features: Built-in measures to avoid blocking, including user agent rotation, request delays, and robots.txt compliance.
• Proxy Support: Integrates seamlessly with Apify's proxy service for IP rotation and avoiding rate limits.
• Structured JSON Output: Delivers clean, well-structured data with full context about where and how each email was discovered.

📥 Input Parameters

Configure the actor's behavior using these fields in the Apify Console Input tab or via API:

Field	Type	Description	Default	Required
start_urls	Array	The URLs to start crawling from. The scraper will extract emails from these pages and follow links up to the specified depth.	[{ "url": "https://beispielbusiness.de/" }]	Yes
max_depth	Integer	Maximum depth of links to follow from start URLs. 0 = only start URLs, 1 = start URLs + one level of links, etc. Range: 0-10.	2	No
use_sitemap	Boolean	Discover pages from the site's sitemap.xml to reach contact/about pages directly, in addition to following links. Discovered pages still respect the per-domain and global page limits.	true	No
stay_on_domain	Boolean	Only follow links that stay on the same domain as each start URL. When enabled, the crawler won't visit external sites.	true	No
max_concurrent_pages	Integer	Maximum number of pages to process simultaneously. Higher values crawl faster but use more memory; lower values are gentler on target sites. The actor automatically caps this per start-URL domain so workers don't time out behind the per-domain request delay (see Performance Tips). Range: 1-100.	50	No
max_pages_per_domain	Integer	Maximum number of pages to crawl from each individual domain. Leave empty for unlimited. This limit applies separately to each domain.	200	No
max_requests_per_run	Integer	Maximum number of pages to crawl globally across all domains. Leave empty for unlimited.	Unlimited	No
request_delay_min	Number	Minimum delay in seconds between consecutive requests to the same domain. A random value between min and max is used per request; different domains are crawled in parallel. Range: 0-60.	1	No
request_delay_max	Number	Maximum delay in seconds between consecutive requests to the same domain. A random value between min and max is used per request. Range: 0-60.	3	No
respect_robots_txt	Boolean	Honor robots.txt directives including crawl delays and disallowed paths.	false	No
rotate_user_agents	Boolean	Use a pool of realistic user agents to appear as different users.	true	No
max_block_retries	Integer	How many more times a page is requested after the site refused it (HTTP 401, 403, 429 or a challenge page) or the proxy failed, each with a new session. A new session gets a new IP only from a rotating proxy; without one the same IP asks again. 0 = never.	2	No
priority_keywords	Array	Links whose URL path or link text contains one of these keywords as a whole word are crawled first, which helps reach contact, legal notice and about pages quickly. A keyword can be several words (about us); case and German umlauts don't matter (ueber-uns also matches "Über uns"). Leave empty to use the built-in defaults (contact, kontakt, impressum, datenschutz, privacy, about, team, …).	Built-in list	No
exclude_role_emails	Boolean	Skip automated/role addresses such as noreply@, no-reply@, postmaster@, mailer-daemon@, and bounce@.	false	No
same_domain_emails_only	Boolean	Keep only emails whose domain matches the website they were found on (e.g. crawling sap.com keeps name@sap.com but drops name@gmail.com).	false	No
include_email_patterns	Array	Regular expressions; if any are provided, only emails matching at least one pattern are kept. Leave empty to keep all. Example: @sap\.com$.	[]	No
exclude_email_patterns	Array	Regular expressions; emails matching any of these are dropped. Applied after the include filter. Example: @(example|test)\.com$.	[]	No
proxy_configuration	Object	Proxy settings to avoid being blocked. Apify Proxy is recommended for large crawls.	{}	No

📤 Output Data Structure

The actor outputs one record per unique email address found during the crawl.

Example Output

[
  {
    "email": "info@beispielbusiness.de",
    "found_on_url": "https://beispielbusiness.de/contact",
    "start_url": "https://beispielbusiness.de/",
    "extraction_method": "mailto_link",
    "depth": 1
  },
  {
    "email": "support@beispielbusiness.de",
    "found_on_url": "https://beispielbusiness.de/about",
    "start_url": "https://beispielbusiness.de/",
    "extraction_method": "text_standard",
    "depth": 1
  },
  {
    "email": "sales@beispielbusiness.de",
    "found_on_url": "https://beispielbusiness.de/impressum",
    "start_url": "https://beispielbusiness.de/",
    "extraction_method": "cloudflare_protected",
    "depth": 2
  }
]

📧 Extraction Methods Explained

The actor uses multiple sophisticated techniques to find email addresses, even when websites try to hide them from bots:

Method	Description
mailto_link	Email addresses found in standard mailto: links in the HTML.
text_standard	Email addresses found in plain text using standard pattern matching.
text_obfuscated	Email addresses that use common text obfuscation like "info [at] example [dot] com".
cloudflare_protected	Email addresses protected by Cloudflare's email obfuscation that are decoded from the page.
rtl_obfuscated	Email addresses hidden using Right-to-Left (RTL) Unicode characters to confuse simple scrapers.

💡 Performance Tips

• Find emails faster: The crawler automatically prioritizes links matching keywords like contact, impressum, or privacy. Keywords match whole words, so team matches /team but not /teamtraining. Add your own terms via priority_keywords to front-load the pages most likely to contain emails, including compound forms such as kontaktformular.
• For small sites: Keep the default settings for optimal speed.
• For large crawls: Use proxy rotation to avoid blocking and rate limits.
• Memory constraints: Set max_concurrent_pages to a lower value (2-5) if running on limited memory.
• Faster crawling: Increase max_concurrent_pages if you have sufficient resources. The effective ceiling per domain is roughly request_handler_timeout × 0.75 / request_delay_max (≈15 per domain at the defaults), because requests stacked deeper than this would time out while waiting in the per-domain rate limiter. To raise the cap on a single-domain crawl, lower request_delay_max; for multi-domain crawls the cap scales linearly with the number of start-URL domains.

🎯 Use Cases

• Lead Generation: Build targeted contact lists for sales and marketing outreach.
• Competitive Research: Discover contact information for companies in your industry.
• Data Enrichment: Enhance existing company databases with email addresses.
• Market Analysis: Gather communication channels for businesses in specific sectors or regions.
• Recruitment: Find contact emails for potential candidates or hiring managers.
• Partnership Development: Identify contact points for potential business partnerships.

🛠️ Maintainer

• Author: Dominic M. Quaiser
• Contact: mail@dominic-quaiser.io
• Website: dominic-quaiser.io

---

🔧 Troubleshooting

No Emails Found

• Check if the website contains any publicly visible emails
• Try increasing max_depth to crawl more pages
• Verify that stay_on_domain isn't preventing you from reaching contact pages on subdomains
• Check if the website might be blocking the scraper (try enabling proxies)

Actor Running Out of Memory

• Decrease max_concurrent_pages to process fewer pages simultaneously
• Use max_requests_per_run to limit the total crawl size
• Upgrade to a larger memory tier on Apify

A linked file does not cost its size: the crawler leaves a response that is no page at its first bytes, and reads at most 10 MB of a page.

Getting Blocked by Websites

• Enable proxy rotation via proxy_configuration
• Increase request_delay_min and request_delay_max
• Enable rotate_user_agents
• Consider enabling respect_robots_txt to honor crawl delays

Log Says "Reducing concurrency from X to Y"

This warning means your requested max_concurrent_pages exceeds what the per-domain rate limit can sustain — the actor lowered concurrency so requests don't time out while waiting for their delay slot. To restore the requested value, either lower request_delay_max or add more start-URL domains (the cap scales per domain).