# SpyFu Domain Intelligence Scraper (`automation-lab/spyfu-domain-intelligence-scraper`) Actor Extract public SpyFu domain intelligence for SEO/PPC competitor research: country keyword stats, SEO summaries, and keyword samples. - **URL**: https://apify.com/automation-lab/spyfu-domain-intelligence-scraper.md - **Developed by:** [Stas Persiianenko](https://apify.com/automation-lab) (community) - **Categories:** Marketing - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing Pay per event This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## SpyFu Domain Intelligence Scraper Extract public SpyFu domain intelligence at scale for SEO and PPC competitor research. Add a list of domains and get structured JSON rows with domain validation, country keyword counts, SEO overview estimates, organic movement summaries, and sample organic keywords. ### What does SpyFu Domain Intelligence Scraper do? SpyFu Domain Intelligence Scraper turns SpyFu's public domain research views into a clean Apify dataset. It is designed for teams that need repeatable competitor checks rather than manual copy-paste from the SpyFu interface. The actor normalizes domains, checks whether SpyFu recognizes each domain, fetches country-level keyword counts, retrieves public SEO summary metrics, and optionally includes a small sample of most valuable organic keywords. ### Who is it for? SpyFu SEO and PPC teams - ๐Ÿง‘โ€๐Ÿ’ผ SEO agencies auditing client competitors in bulk - ๐Ÿ“ˆ PPC teams triaging competitor domains before deeper ad research - ๐Ÿงช Growth marketers building competitive-intelligence spreadsheets - ๐Ÿข B2B lead researchers qualifying company domains by search footprint - ๐Ÿงพ Consultants preparing recurring monthly SEO/PPC reports ### Why use this actor? Manual SpyFu checks are slow when you have dozens of domains. This actor gives you a reproducible Apify run, downloadable JSON/CSV/Excel outputs, API access, and dataset history for recurring workflows. ### Data you can extract | Field | Description | | --- | --- | | `domain` | Normalized domain submitted to the actor | | `queryFound` | Whether SpyFu recognized the domain query | | `cleansedDomain` | SpyFu-normalized domain | | `queryId` | SpyFu query identifier when visible | | `countryKeywordStats` | Keyword counts by available SpyFu country database | | `organicSummary` | Public SEO overview metrics when visible | | `organicMovementStats` | New/gained/lost organic movement summaries | | `organicKeywords` | Optional sample of public most valuable organic keywords | | `sourceUrl` | SpyFu page URL for manual follow-up | | `error` | Per-domain error details when a section is unavailable | ### How much does it cost to scrape SpyFu domain intelligence? This actor uses pay-per-event pricing. There is a small start event and one domain event per dataset row. You only pay for domains that are processed and saved to the dataset. | Run size | Charged events | Estimated FREE-tier price* | Typical use case | | --- | --- | ---: | --- | | 3 domains | 1 start + 3 domain events | ~$0.0066 | Default smoke test | | 25 domains | 1 start + 25 domain events | ~$0.0107 | Small competitor shortlist | | 50 domains | 1 start + 50 domain events | ~$0.0154 | Agency prospect batch | | 100 domains | 1 start + 100 domain events | ~$0.0247 | Realistic bulk enrichment run | | 1,000 domains | 1 start + 1,000 domain events | ~$0.1933 | Large recurring SEO/PPC report | \*Estimates use the current FREE-tier event prices configured for this actor: $0.006 per run start and $0.00018731 per processed domain. Higher Apify plan tiers receive lower per-domain event prices where available. Your final Apify bill can also include normal platform compute/proxy charges shown by Apify for your account. With a $5 free Apify credit budget, the domain-event portion supports roughly 26,000 processed domains before compute/proxy overhead. A more conservative planning estimate is about 200 realistic 100-domain runs at the event-price level, so first-time users can test several small batches before scaling. The default input is intentionally small so first runs are cheap. For large lists, start with a short sample, review output quality, and then scale up. ### How to use 1. Open the actor on Apify. 2. Add domains such as `apify.com`, `semrush.com`, or `example.com`. 3. Keep `US` as the country database or choose another supported country. 4. Choose whether to include organic keyword samples. 5. Run the actor. 6. Download results from the dataset as JSON, CSV, Excel, or via API. ### Input example ```json { "domains": ["apify.com", "semrush.com", "spyfu.com"], "countryCode": "US", "includeOrganicKeywords": true, "maxKeywordsPerDomain": 5, "requestDelayMs": 750 } ```` ### Output example ```json { "domain": "apify.com", "queryFound": true, "cleansedDomain": "apify.com", "countryCode": "US", "primaryCountry": "US", "countryKeywordStats": [{ "countryCode": "US", "keywords": 95130, "isPrimary": true }], "organicKeywords": [{ "keyword": "web scraper", "rank": 12 }], "sourceUrl": "https://www.spyfu.com/overview/domain?query=apify.com", "scrapedAt": "2026-07-01T00:00:00.000Z" } ``` ### Input fields #### Domains A required list of domains or URLs. URLs are normalized to hostnames and `www.` is removed. #### Country code SpyFu country database code. `US` is the default and usually the richest public dataset. #### Include organic keyword samples Enable this to include a bounded keyword sample for each domain. Disable it for faster domain validation runs. #### Max organic keywords per domain Controls the size of the nested `organicKeywords` array. Keep it low for quick competitive triage. #### Request delay Adds a delay between domains to reduce pressure on SpyFu public endpoints. ### Tips for best results - Use root domains rather than long URLs when possible. - Start with 5-10 domains before scaling to a large list. - Keep keyword samples small for monitoring workflows. - Use the `sourceUrl` field when a domain needs manual follow-up in SpyFu. - Review the `error` field for domains where one public section was unavailable. ### Integrations Use this actor with: - Google Sheets exports for agency reporting - Apify webhooks for recurring competitor monitoring - CRM enrichment flows where domains need an SEO footprint signal - Data warehouses that combine SpyFu output with Search Console or Ads data - BI dashboards that track competitor movement over time ### API usage #### Node.js ```js import { ApifyClient } from 'apify-client'; const client = new ApifyClient({ token: process.env.APIFY_TOKEN }); const run = await client.actor('automation-lab/spyfu-domain-intelligence-scraper').call({ domains: ['apify.com', 'spyfu.com'], countryCode: 'US' }); console.log(run.defaultDatasetId); ``` #### Python ```python from apify_client import ApifyClient client = ApifyClient('MY-APIFY-TOKEN') run = client.actor('automation-lab/spyfu-domain-intelligence-scraper').call(run_input={ 'domains': ['apify.com', 'spyfu.com'], 'countryCode': 'US', }) print(run['defaultDatasetId']) ``` #### cURL ```bash curl -X POST 'https://api.apify.com/v2/acts/automation-lab~spyfu-domain-intelligence-scraper/runs?token=MY-APIFY-TOKEN' \ -H 'Content-Type: application/json' \ -d '{"domains":["apify.com","spyfu.com"],"countryCode":"US"}' ``` ### MCP usage Connect Apify MCP to Claude Desktop or Claude Code and enable this actor with: ```bash claude mcp add apify-spyfu https://mcp.apify.com/?tools=automation-lab/spyfu-domain-intelligence-scraper ``` Claude Desktop JSON configuration example: ```json { "mcpServers": { "apify-spyfu": { "url": "https://mcp.apify.com/?tools=automation-lab/spyfu-domain-intelligence-scraper" } } } ``` Example prompts: - "Run the SpyFu Domain Intelligence Scraper for these 20 competitor domains and summarize primary countries." - "Compare organic keyword samples for apify.com and spyfu.com." - "Export SpyFu domain intelligence for my CRM enrichment list." ### Reliability notes SpyFu's browser app may throttle automated browser sessions. This actor uses public JSON endpoints and conservative delays instead of rendering the full browser app. If a section is unavailable, the actor saves a row with an `error` field rather than failing the whole run. ### Legality and ethical use This actor extracts data visible from public SpyFu endpoints. Use it responsibly, obey applicable laws, respect SpyFu's terms, and avoid excessive request rates. Do not use the actor to collect personal data or bypass access controls. ### FAQ #### Does this require a SpyFu account? No. The actor is built for public, unauthenticated SpyFu domain intelligence endpoints. #### Why are PPC ads or backlinks not always included? Some high-value sections are not reliably exposed through public unauthenticated endpoints. The actor focuses on sections it can retrieve consistently without login. #### Why do some domains have empty keyword samples? SpyFu may not have public data for the domain/country pair, or a public endpoint may return only aggregate stats. #### What should I do if I see errors? Increase `requestDelayMs`, reduce keyword samples, and rerun a small domain subset. Per-domain errors are saved in the dataset for debugging. ### Related scrapers Explore other Automation Lab actors at: - https://apify.com/automation-lab/google-search-scraper - https://apify.com/automation-lab/google-ads-transparency-scraper - https://apify.com/automation-lab/website-contact-finder ### Changelog - 0.1 โ€” Initial public SpyFu domain intelligence scraper with domain validation, country keyword stats, SEO summaries, and keyword samples. ### Support If a run fails or a field looks wrong, share the run URL and a short description of the expected domain data. Include the input domains and country code so the issue can be reproduced. # Actor input Schema ## `domains` (type: `array`): Root domains to check on SpyFu, for example apify.com or semrush.com. URLs are normalized to domains automatically. ## `countryCode` (type: `string`): SpyFu country database code. US is the most complete public default. ## `includeOrganicKeywords` (type: `boolean`): Fetch a lightweight sample of most valuable organic keywords visible in SpyFu public endpoints. ## `maxKeywordsPerDomain` (type: `integer`): Limit keyword samples per domain. Keep this low for fast agency triage runs. ## `requestDelayMs` (type: `integer`): Conservative delay between domains to respect SpyFu public endpoints. ## Actor input object example ```json { "domains": [ "apify.com", "example.com", "spyfu.com" ], "countryCode": "US", "includeOrganicKeywords": true, "maxKeywordsPerDomain": 5, "requestDelayMs": 750 } ``` # Actor output Schema ## `overview` (type: `string`): No description # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "domains": [ "apify.com", "example.com", "spyfu.com" ], "countryCode": "US", "includeOrganicKeywords": true, "maxKeywordsPerDomain": 5, "requestDelayMs": 750 }; // Run the Actor and wait for it to finish const run = await client.actor("automation-lab/spyfu-domain-intelligence-scraper").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`๐Ÿ’พ Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // ๐Ÿ“š Want to learn more ๐Ÿ“–? Go to โ†’ https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "domains": [ "apify.com", "example.com", "spyfu.com", ], "countryCode": "US", "includeOrganicKeywords": True, "maxKeywordsPerDomain": 5, "requestDelayMs": 750, } # Run the Actor and wait for it to finish run = client.actor("automation-lab/spyfu-domain-intelligence-scraper").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("๐Ÿ’พ Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # ๐Ÿ“š Want to learn more ๐Ÿ“–? Go to โ†’ https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "domains": [ "apify.com", "example.com", "spyfu.com" ], "countryCode": "US", "includeOrganicKeywords": true, "maxKeywordsPerDomain": 5, "requestDelayMs": 750 }' | apify call automation-lab/spyfu-domain-intelligence-scraper --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=automation-lab/spyfu-domain-intelligence-scraper", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "SpyFu Domain Intelligence Scraper", "description": "Extract public SpyFu domain intelligence for SEO/PPC competitor research: country keyword stats, SEO summaries, and keyword samples.", "version": "0.1", "x-build-id": "UuCDt8S05blzmJswu" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/automation-lab~spyfu-domain-intelligence-scraper/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-automation-lab-spyfu-domain-intelligence-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/automation-lab~spyfu-domain-intelligence-scraper/runs": { "post": { "operationId": "runs-sync-automation-lab-spyfu-domain-intelligence-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/automation-lab~spyfu-domain-intelligence-scraper/run-sync": { "post": { "operationId": "run-sync-automation-lab-spyfu-domain-intelligence-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "required": [ "domains" ], "properties": { "domains": { "title": "Domains", "type": "array", "description": "Root domains to check on SpyFu, for example apify.com or semrush.com. URLs are normalized to domains automatically.", "items": { "type": "string" } }, "countryCode": { "title": "Country / database", "enum": [ "US", "UK", "CA", "AU", "DE", "FR", "ES", "IT", "NL", "BR", "IN" ], "type": "string", "description": "SpyFu country database code. US is the most complete public default.", "default": "US" }, "includeOrganicKeywords": { "title": "Include organic keyword samples", "type": "boolean", "description": "Fetch a lightweight sample of most valuable organic keywords visible in SpyFu public endpoints.", "default": true }, "maxKeywordsPerDomain": { "title": "Max organic keywords per domain", "minimum": 0, "maximum": 50, "type": "integer", "description": "Limit keyword samples per domain. Keep this low for fast agency triage runs.", "default": 5 }, "requestDelayMs": { "title": "Delay between domains (ms)", "minimum": 0, "maximum": 10000, "type": "integer", "description": "Conservative delay between domains to respect SpyFu public endpoints.", "default": 750 } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```