# Website Contact Crawler (`competent_clarinet/website-contact-crawler`) Actor Crawls websites to extract emails, phones, and social links. - **URL**: https://apify.com/competent\_clarinet/website-contact-crawler.md - **Developed by:** [Man Mohit verma](https://apify.com/competent_clarinet) (community) - **Categories:** Lead generation, Automation - **Stats:** 5 total users, 2 monthly users, 100.0% runs succeeded, NaN bookmarks - **User rating**: No ratings yet ## Pricing Pay per usage This Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage, which gets cheaper the higher subscription plan you have. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-usage ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## Website Contact Crawler Python Apify Actor that crawls a list of start URLs, follows links up to a configurable depth, and extracts: - email addresses - phone numbers - Facebook, X/Twitter, WhatsApp, YouTube, Instagram, and LinkedIn links Each extracted contact is stored with: - `startingUrl` — seed URL for the crawl branch - `currentPage` — URL that was requested and crawled (e.g. `/pages/contact-us`) - `pageFetched` — final URL after HTTP redirects, where the HTML was parsed - `type` - `value` ### Output 1. **Default dataset** — one row per unique contact (standard Apify export as JSON/CSV). 2. **Key-Value Store** - `contacts.json` — full aggregated array of all contacts from the run. - `pages-scraped.json` — per seed URL, all HTML pages that were successfully scraped (`startingUrl` + `pagesScraped` array). ### Input - `startUrls`: list of seed URLs (JSON array; supports large lists such as ~1,000 sites) - `depthOfPages`: crawl depth from each seed URL - `defaultPhoneRegion`: default region for `phonenumbers` - `maxConcurrencyPerIp`: concurrent fetches per worker band (default 50) - `proxyPoolSize`: number of worker bands (default 10); total workers = `maxConcurrencyPerIp` × `proxyPoolSize` - `maxConcurrencyPerHost`: cap simultaneous requests per website host (default 5; set `0` to disable) - `dedupeScope`: `global` (one row per value) or `perStartingUrl` (same value allowed under different seeds) - `proxyConfiguration`: Apify Proxy or custom proxy settings (RESIDENTIAL recommended on Apify) - `additionalPaths` / `excludeKeywords`: add depth-1 paths and filter URLs ### Concurrency and proxy - **Worker bands**: `proxyPoolSize` × `maxConcurrencyPerIp` async workers (default 500) share a global crawl queue. - **Per-request IP rotation**: when Apify Proxy is enabled, every HTTP request uses a **new** residential proxy session (`session_id` is unique per fetch). Worker bands organize parallelism; they do **not** pin 10 fixed IPs. - **Per-host limit**: `maxConcurrencyPerHost` reduces hammering a single domain when many seeds or pages target the same host. - **Cost**: high concurrency with residential proxies can be expensive; lower `maxConcurrencyPerIp` or `proxyPoolSize` if you hit rate limits or budget limits. ### Notes - The crawler stays on the same host or subdomain family as the seed URL, and also follows links on other hosts seen in that crawl branch (common for Shopify: `*.myshopify.com` seed redirecting to a custom domain while HTML still links to `myshopify.com` pages). - Static assets, `mailto:`, `tel:`, `javascript:`, and fragment-only links are ignored for crawling. - `additionalPaths` are applied when the seed page at depth 0 is fetched, so they become depth-1 pages alongside links discovered from that page. `excludeKeywords` blocks matching URLs at every depth. - 429 responses trigger host-specific cooldowns and respect `Retry-After`. Lower concurrency if a site still rate limits heavily. - Local runs work without Apify Proxy credentials; on Apify, the actor uses the residential proxy pool when available. - Default run options: 2-hour timeout, 8 GB memory (see `.actor/actor.json`). Increase timeout for very large seed lists and depth. ### Local run ```bash python -m pip install -r requirements.txt python -m src ```` For local testing, put an `INPUT.json` file under `storage/key_value_stores/default/` or set `APIFY_LOCAL_STORAGE_DIR` to a folder with that structure. After a run, check `storage/datasets/default/` for dataset rows and `storage/key_value_stores/default/contacts.json` and `pages-scraped.json` for aggregated JSON files. ### Publish to Apify ```bash apify login apify push ``` Smoke-test with a few `startUrls` and `depthOfPages=1`, then scale up gradually before running ~1,000 seeds at full concurrency. # Actor input Schema ## `startUrls` (type: `array`): List of website URLs to start crawling from (supports hundreds or thousands of seeds as a JSON array). ## `depthOfPages` (type: `integer`): How many link hops to follow from each starting URL. ## `maxPagesPerStartUrl` (type: `integer`): Maximum number of pages to scrape per starting URL (including the start page itself). Crawling stops for a site when this limit or the depth limit is reached, whichever comes first. Set to 0 for no limit. ## `defaultPhoneRegion` (type: `string`): Default region used by phonenumbers for national numbers. ## `maxConcurrencyPerIp` (type: `integer`): Concurrent page fetches per worker band. With proxyPoolSize=10 and this set to 50, up to 500 pages can be fetched in parallel. Each fetch uses a new residential proxy session (per-request IP rotation), not a fixed IP pool. ## `proxyPoolSize` (type: `integer`): Number of parallel worker bands. Total workers = maxConcurrencyPerIp × proxyPoolSize. Does not reserve fixed IPs; each request still rotates to a new residential proxy session. ## `maxConcurrencyPerHost` (type: `integer`): Maximum simultaneous in-flight requests to the same website host. Helps avoid overwhelming a single domain when many seeds share infrastructure. Set to 0 to disable the limit. ## `dedupeScope` (type: `string`): global: keep one row per contact value across all seeds. perStartingUrl: allow the same value under different starting URLs. ## `additionalPaths` (type: `array`): Paths that are added for depth 1 from each start URL. ## `excludeKeywords` (type: `array`): Skip any URL containing one of these keywords at any depth. ## `proxyConfiguration` (type: `object`): Use Apify Proxy (RESIDENTIAL recommended) or your own proxy URLs. Each HTTP request opens a new proxy session for IP rotation. ## Actor input object example ```json { "startUrls": [ "https://apify.com" ], "depthOfPages": 1, "maxPagesPerStartUrl": 0, "defaultPhoneRegion": "US", "maxConcurrencyPerIp": 50, "proxyPoolSize": 10, "maxConcurrencyPerHost": 5, "dedupeScope": "global", "additionalPaths": [ "/about", "/about-us", "/contact", "/contact-us", "/pages/contact-us", "/pages/about-us", "/pages/privacy-policy", "/privacy-policy", "/pages/terms-and-conditions", "/terms-and-conditions", "/pages/sitemap", "/sitemap", "/pages/faq", "/faq", "/pages/our-story", "/our-story", "/pages/our-team", "/our-team" ], "excludeKeywords": [ "whatsapp", "wa.me", "facebook", "instagram", "linkedin", "youtube", "twitter", "x.com", "blog", "blogs", "news", "setting", "settings", "#", "collection", "collections", "product", "products", "account", "login", "logout", "signin", "signout", "cart", "checkout", "search", "compare", "wishlist" ], "proxyConfiguration": { "useApifyProxy": true, "apifyProxyGroups": [ "RESIDENTIAL" ] } } ``` # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "startUrls": [ "https://apify.com" ], "additionalPaths": [ "/about", "/about-us", "/contact", "/contact-us", "/pages/contact-us", "/pages/about-us", "/pages/privacy-policy", "/privacy-policy", "/pages/terms-and-conditions", "/terms-and-conditions", "/pages/sitemap", "/sitemap", "/pages/faq", "/faq", "/pages/our-story", "/our-story", "/pages/our-team", "/our-team" ], "excludeKeywords": [ "whatsapp", "wa.me", "facebook", "instagram", "linkedin", "youtube", "twitter", "x.com", "blog", "blogs", "news", "setting", "settings", "#", "collection", "collections", "product", "products", "account", "login", "logout", "signin", "signout", "cart", "checkout", "search", "compare", "wishlist" ], "proxyConfiguration": { "useApifyProxy": true, "apifyProxyGroups": [ "RESIDENTIAL" ] } }; // Run the Actor and wait for it to finish const run = await client.actor("competent_clarinet/website-contact-crawler").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "startUrls": ["https://apify.com"], "additionalPaths": [ "/about", "/about-us", "/contact", "/contact-us", "/pages/contact-us", "/pages/about-us", "/pages/privacy-policy", "/privacy-policy", "/pages/terms-and-conditions", "/terms-and-conditions", "/pages/sitemap", "/sitemap", "/pages/faq", "/faq", "/pages/our-story", "/our-story", "/pages/our-team", "/our-team", ], "excludeKeywords": [ "whatsapp", "wa.me", "facebook", "instagram", "linkedin", "youtube", "twitter", "x.com", "blog", "blogs", "news", "setting", "settings", "#", "collection", "collections", "product", "products", "account", "login", "logout", "signin", "signout", "cart", "checkout", "search", "compare", "wishlist", ], "proxyConfiguration": { "useApifyProxy": True, "apifyProxyGroups": ["RESIDENTIAL"], }, } # Run the Actor and wait for it to finish run = client.actor("competent_clarinet/website-contact-crawler").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "startUrls": [ "https://apify.com" ], "additionalPaths": [ "/about", "/about-us", "/contact", "/contact-us", "/pages/contact-us", "/pages/about-us", "/pages/privacy-policy", "/privacy-policy", "/pages/terms-and-conditions", "/terms-and-conditions", "/pages/sitemap", "/sitemap", "/pages/faq", "/faq", "/pages/our-story", "/our-story", "/pages/our-team", "/our-team" ], "excludeKeywords": [ "whatsapp", "wa.me", "facebook", "instagram", "linkedin", "youtube", "twitter", "x.com", "blog", "blogs", "news", "setting", "settings", "#", "collection", "collections", "product", "products", "account", "login", "logout", "signin", "signout", "cart", "checkout", "search", "compare", "wishlist" ], "proxyConfiguration": { "useApifyProxy": true, "apifyProxyGroups": [ "RESIDENTIAL" ] } }' | apify call competent_clarinet/website-contact-crawler --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=competent_clarinet/website-contact-crawler", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "Website Contact Crawler", "description": "Crawls websites to extract emails, phones, and social links.", "version": "0.1", "x-build-id": "iyhFGOTi1mYjqp3PQ" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/competent_clarinet~website-contact-crawler/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-competent_clarinet-website-contact-crawler", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/competent_clarinet~website-contact-crawler/runs": { "post": { "operationId": "runs-sync-competent_clarinet-website-contact-crawler", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/competent_clarinet~website-contact-crawler/run-sync": { "post": { "operationId": "run-sync-competent_clarinet-website-contact-crawler", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "required": [ "startUrls" ], "properties": { "startUrls": { "title": "Start URLs", "type": "array", "description": "List of website URLs to start crawling from (supports hundreds or thousands of seeds as a JSON array).", "items": { "type": "string" } }, "depthOfPages": { "title": "Depth of pages", "minimum": 0, "maximum": 10, "type": "integer", "description": "How many link hops to follow from each starting URL.", "default": 1 }, "maxPagesPerStartUrl": { "title": "Max pages per start URL", "minimum": 0, "type": "integer", "description": "Maximum number of pages to scrape per starting URL (including the start page itself). Crawling stops for a site when this limit or the depth limit is reached, whichever comes first. Set to 0 for no limit.", "default": 0 }, "defaultPhoneRegion": { "title": "Default phone region", "type": "string", "description": "Default region used by phonenumbers for national numbers.", "default": "US" }, "maxConcurrencyPerIp": { "title": "Max concurrency per worker band", "minimum": 1, "maximum": 200, "type": "integer", "description": "Concurrent page fetches per worker band. With proxyPoolSize=10 and this set to 50, up to 500 pages can be fetched in parallel. Each fetch uses a new residential proxy session (per-request IP rotation), not a fixed IP pool.", "default": 50 }, "proxyPoolSize": { "title": "Worker bands", "minimum": 1, "maximum": 100, "type": "integer", "description": "Number of parallel worker bands. Total workers = maxConcurrencyPerIp × proxyPoolSize. Does not reserve fixed IPs; each request still rotates to a new residential proxy session.", "default": 10 }, "maxConcurrencyPerHost": { "title": "Max concurrency per host", "minimum": 0, "maximum": 100, "type": "integer", "description": "Maximum simultaneous in-flight requests to the same website host. Helps avoid overwhelming a single domain when many seeds share infrastructure. Set to 0 to disable the limit.", "default": 5 }, "dedupeScope": { "title": "Contact dedupe scope", "enum": [ "global", "perStartingUrl" ], "type": "string", "description": "global: keep one row per contact value across all seeds. perStartingUrl: allow the same value under different starting URLs.", "default": "global" }, "additionalPaths": { "title": "Additional paths", "type": "array", "description": "Paths that are added for depth 1 from each start URL.", "items": { "type": "string" } }, "excludeKeywords": { "title": "Exclude keywords", "type": "array", "description": "Skip any URL containing one of these keywords at any depth.", "items": { "type": "string" } }, "proxyConfiguration": { "title": "Proxy configuration", "type": "object", "description": "Use Apify Proxy (RESIDENTIAL recommended) or your own proxy URLs. Each HTTP request opens a new proxy session for IP rotation." } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```