SpyFu Domain Intelligence Scraper

Extract public SpyFu organic domain intelligence at scale for SEO competitor triage. Add a list of domains and get structured JSON rows with domain validation, country keyword counts, SEO overview estimates, organic movement summaries, and sample organic keywords.

What does SpyFu Domain Intelligence Scraper do?

SpyFu Domain Intelligence Scraper turns SpyFu's public domain research views into a clean Apify dataset. It is designed for teams that need repeatable competitor checks rather than manual copy-paste from the SpyFu interface.

The actor normalizes domains, checks whether SpyFu recognizes each domain, fetches country-level keyword counts, retrieves public SEO summary metrics, and optionally includes a small sample of most valuable organic keywords.

Who is it for? SpyFu SEO and PPC teams

• 🧑‍💼 SEO agencies auditing client competitors in bulk
• 📈 PPC teams triaging competitor domains before deeper ad research
• 🧪 Growth marketers building competitive-intelligence spreadsheets
• 🏢 B2B lead researchers qualifying company domains by search footprint
• 🧾 Consultants preparing recurring monthly SEO/PPC reports

Why use this actor?

Manual SpyFu checks are slow when you have dozens of domains. This actor gives you a reproducible Apify run, downloadable JSON/CSV/Excel outputs, API access, and dataset history for recurring workflows.

Data you can extract

Field	Description
domain	Normalized domain submitted to the actor
queryFound	Whether SpyFu recognized the domain query
cleansedDomain	SpyFu-normalized domain
queryId	SpyFu query identifier when visible
countryKeywordStats	Keyword counts by available SpyFu country database
organicSummary	Public SEO overview metrics when visible
organicMovementStats	New/gained/lost organic movement summaries
organicKeywords	Optional most valuable organic keywords sampled from the selected country database
sourceUrl	SpyFu page URL for manual follow-up
sectionErrors	Optional section names mapped to fetch errors; other sections remain usable
error	Hard domain-search failure; the row is diagnostic and does not incur a domain event

How much does it cost to scrape SpyFu domain intelligence?

This actor uses pay-per-event pricing. There is a small start event and one domain event per successfully analyzed domain. Hard domain-search failures still produce diagnostic rows but do not incur a domain event. Partial rows with a successful domain search are charged once.

Run size	Charged events	Typical use case
3 domains	1 start + 3 domain events	Default smoke test
25 domains	1 start + 25 domain events	Small competitor shortlist
50 domains	1 start + 50 domain events	Agency prospect batch
100 domains	1 start + 100 domain events	Realistic bulk enrichment run
1,000 domains	1 start + 1,000 domain events	Large recurring SEO/PPC report

Estimate an event charge as $0.006 per run start + successfully analyzed domain count × the current FREE-tier price of $0.00018731 per processed domain. Higher tiers use lower per-domain event prices: $0.00016288 (BRONZE), $0.00012704 (SILVER), $0.000097725 (GOLD), $0.00006515 (PLATINUM), and $0.000045605 (DIAMOND). Your final Apify bill can also include normal platform compute/proxy charges shown by Apify for your account.

With a $5 free Apify credit budget, the domain-event portion supports roughly 26,000 processed domains before compute/proxy overhead. A more conservative planning estimate is about 200 realistic 100-domain runs at the event-price level, so first-time users can test several small batches before scaling.

The default input is intentionally small so first runs are cheap. For large lists, start with a short sample, review output quality, and then scale up.

How to use

1. Open the actor on Apify.
2. Add domains such as apify.com, semrush.com, or example.com.
3. Keep US as the country database or choose another supported country.
4. Choose whether to include organic keyword samples.
5. Run the actor.
6. Download results from the dataset as JSON, CSV, Excel, or via API.

Input example

{
  "domains": ["apify.com", "semrush.com", "spyfu.com"],
  "countryCode": "US",
  "includeOrganicKeywords": true,
  "maxKeywordsPerDomain": 5,
  "requestDelayMs": 750
}

Output example

{
  "domain": "apify.com",
  "queryFound": true,
  "cleansedDomain": "apify.com",
  "countryCode": "US",
  "primaryCountry": "US",
  "countryKeywordStats": [{ "countryCode": "US", "keywords": 95130, "isPrimary": true }],
  "organicKeywords": [{ "keyword": "example search term", "rank": 12 }],
  "sourceUrl": "https://www.spyfu.com/overview/domain?query=apify.com",
  "scrapedAt": "2026-07-01T00:00:00.000Z"
}

Input fields

Domains

A required list of domains or URLs. URLs are normalized to hostnames and www. is removed.

Country code

Selects the SpyFu country database for organicKeywords. US is the default. countryKeywordStats lists available countries and organicSummary / organicMovementStats are aggregate endpoint results; do not interpret those sections as filtered to this selection.

Include organic keyword samples

Enable this to include a bounded keyword sample for each domain. Disable it for faster domain validation runs.

Max organic keywords per domain

Controls the size of the nested organicKeywords array. Keep it low for quick competitive triage.

Request delay

Adds a delay between domains (not after the final domain) to reduce pressure on SpyFu public endpoints.

Tips for best results

• Use root domains rather than long URLs when possible.
• Start with 5-10 domains before scaling to a large list.
• Keep keyword samples small for monitoring workflows.
• Use the sourceUrl field when a domain needs manual follow-up in SpyFu.
• Review sectionErrors for unavailable optional sections and error for hard domain-search failures.

Integrations

Use this actor with:

• Google Sheets exports for agency reporting
• Apify webhooks for recurring competitor monitoring
• CRM enrichment flows where domains need an SEO footprint signal
• Data warehouses that combine SpyFu output with Search Console or Ads data
• BI dashboards that track competitor movement over time

API usage

Node.js

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('automation-lab/spyfu-domain-intelligence-scraper').call({
  domains: ['apify.com', 'spyfu.com'],
  countryCode: 'US'
});
console.log(run.defaultDatasetId);

Python

from apify_client import ApifyClient

client = ApifyClient('MY-APIFY-TOKEN')
run = client.actor('automation-lab/spyfu-domain-intelligence-scraper').call(run_input={
    'domains': ['apify.com', 'spyfu.com'],
    'countryCode': 'US',
})
print(run['defaultDatasetId'])

cURL

curl -X POST 'https://api.apify.com/v2/acts/automation-lab~spyfu-domain-intelligence-scraper/runs?token=MY-APIFY-TOKEN' \
  -H 'Content-Type: application/json' \
  -d '{"domains":["apify.com","spyfu.com"],"countryCode":"US"}'

MCP usage

Connect Apify MCP to Claude Desktop or Claude Code and enable this actor with:

$claude mcp add --transport http apify-spyfu 'https://mcp.apify.com/?tools=automation-lab/spyfu-domain-intelligence-scraper'

Claude Desktop JSON configuration example:

{
  "mcpServers": {
    "apify-spyfu": {
      "url": "https://mcp.apify.com/?tools=automation-lab/spyfu-domain-intelligence-scraper"
    }
  }
}

For Cursor: open Settings → MCP, add the same HTTP URL as an MCP server, and authenticate with your Apify account. For VS Code: add that HTTP URL to your MCP server configuration and authenticate with your Apify account. Claude Desktop uses the JSON configuration above.

Example prompts:

• "Run the SpyFu Domain Intelligence Scraper for these 20 competitor domains and summarize primary countries."
• "Compare organic keyword samples for apify.com and spyfu.com."
• "Export SpyFu domain intelligence for my CRM enrichment list."

Reliability notes

SpyFu's browser app may throttle automated browser sessions. This actor uses public JSON endpoints and conservative delays instead of rendering the full browser app. If an optional section fails, the actor saves a partial row with its existing empty/null fallback and a named diagnostic in sectionErrors. A hard domain-search failure produces an error row without a domain event. Disabled keyword samples remain empty without a section error.

Legality and ethical use

This Actor uses no AI model or external AI provider. Submitted domains and selected country are sent to SpyFu public endpoints. No login, cookies, or Actor-specific persistent cache are used. Apify retains run input, dataset and logs according to your account storage settings; delete runs and datasets through Console or API. Do not submit sensitive or personal domains. SpyFu is the independent public data source; this Actor is not affiliated with or endorsed by SpyFu. This actor extracts data visible from public SpyFu endpoints. Use it responsibly, obey applicable laws, respect SpyFu's terms, and avoid excessive request rates. Do not use the actor to collect personal data or bypass access controls.

FAQ

Does this require a SpyFu account?

No. The actor is built for public, unauthenticated SpyFu domain intelligence endpoints.

Why are PPC ads or backlinks not always included?

Paid ads, paid keyword intelligence, backlinks, and competitor lists are not included. The actor focuses on public organic domain triage, not full SpyFu paid-intelligence extraction.

Why do some domains have empty keyword samples?

SpyFu may have no public data for the domain/country pair; sampling can also be disabled or set to zero. A failed keyword request is marked in sectionErrors.organicKeywords; an empty array without that marker is not a fetch failure.

What should I do if I see errors?

Increase requestDelayMs, reduce keyword samples, and rerun a small domain subset. Per-domain errors are saved in the dataset for debugging.

Related scrapers

Explore other Automation Lab actors at:

• https://apify.com/automation-lab/google-search-scraper
• https://apify.com/automation-lab/google-ads-transparency-scraper
• https://apify.com/automation-lab/website-contact-finder

Changelog

• 0.1 — Initial public SpyFu domain intelligence scraper with domain validation, country keyword stats, SEO summaries, and keyword samples.

Support

If a run fails or a field looks wrong, share the run URL and a short description of the expected domain data. Include the input domains and country code so the issue can be reproduced.