Pricing

from $2.00 / 1,000 results

Try for free

Go to Apify Store

Semrush Full SEO Scraper - Traffic, Authority, Backlinks

Try for free

All-in-one Semrush & Moz scraper — no login required. Domain authority, traffic analytics, backlinks, AI traffic (ChatGPT/Gemini/Claude), competitors, top 100 trending websites. 80+ fields per domain, 10 parallel workers, unique IP per request. Two modes: Domain Analysis & Top Websites Ranking.

Pricing

from $2.00 / 1,000 results

Rating

5.0

(1)

Developer

Raven

Actor stats

Bookmarked

Total users

Monthly active users

1.1 hours

Issues response

3 days ago

Last modified

Semrush Domain Overview Scraper — Authority, Organic, Paid, AI Search, Backlinks & Competitors

🦅 More from Raven ►	Ahrefs SEO Tools	Semrush Scraper	Similarweb Scraper

This scraper turns Semrush into a plug-in SEO data API. Drop in a domain — get the full Domain Overview dashboard back as one structured JSON record: Authority Score, organic & paid keywords with CPC and intent, top organic + paid competitors, AI Search visibility across ChatGPT, AI Overview, AI Mode, Gemini, the full backlinks profile with anchor breakdown, daily and monthly traffic trends, and an optional Moz Domain Analysis snapshot. Everything that's behind the Semrush login — without a Semrush subscription or API key.

Key features

Full Domain Overview in one record — every plate from the Semrush UI, flattened into one JSON row per domain
Authenticated Semrush data — no degraded "public preview" numbers; you get what a paying seat sees
AI Search visibility — per-LLM mentions and cited pages across ChatGPT / AI Overview / AI Mode / Gemini, plus top co-cited domains
Top 50 organic keywords — position, volume, CPC, KD, traffic, traffic share %, decoded intent, SERP features, landing URL
Top 50 paid ads with full ad copy — title, description, visible URL, position, volume, CPC
Organic + paid competitors — top 20 each plus total counts, with shared keywords and traffic
Multi-year daily backlink and traffic trends — global and per-country, branded vs non-branded split
Optional Moz Domain Analysis — DA, Spam Score, top pages, ranking distribution, 60-day linking-domain trend
Top Websites Ranking mode — top 100 trending websites by 90+ countries × 80+ industries
Up to 100 domains per run, parallel — configurable concurrency (default 5, max 50)
Flat KPI columns up front — render as plain table columns in the Apify dataset view; no dot.notation paths needed
Export-ready — CSV / JSON / Excel / API webhook out of the box

How it works

Pick a mode — Domain Overview or Top Websites Ranking
For Domain Overview: paste 1–100 domains (e.g. amazon.com, github.com)
Choose a regional database (Worldwide + 100 country options)
Optionally enable Include Moz Data (adds Moz DA, Spam Score, top pages, ranking distribution)
Set parallelism (default 5 — matches Apify's free Datacenter proxy slots; up to 50 on paid plans)
Click Start — results land in the dataset as one row per domain

Each domain triggers a single batched JSON-RPC call covering organic + paid + backlinks + AI search + competitors simultaneously, so cost scales linearly with domain count, not with how many sections you want back.

Primary use cases

User type	Challenge	Solution
SEO specialist	Need Authority Score, keyword and backlink profile for 50 clients weekly	Bulk-feed 50 domains, get full Domain Overview as one CSV — without 50 Semrush seats
Competitive analyst	"Who is actually beating us in organic and paid?"	Top 20 organic + 20 paid competitors per domain, with shared keywords and traffic
AI/LLM researcher	Need to track which brands LLMs cite, and which sources LLMs cite alongside them	Per-LLM mentions and cited pages across ChatGPT / AIO / AIM / Gemini + top co-cited domains
Performance marketer	"What ad copy is the competitor running, and on which keywords?"	Top 50 paid keywords per domain with full ad title, description, URL and CPC
Agency / data team	Need daily traffic time-series to feed dashboards	Multi-year daily resolution, per-country and global, branded vs non-branded split
Market researcher	Need the top 100 sites in a specific country × industry	Top Websites mode — 90+ countries × 80+ industries, with MoM/YoY change and traffic-source split

Input parameters

mode — domain (Domain Overview) or top_websites (Top Websites Ranking)
domains — 1–100 domains or URLs for Domain Overview mode (e.g. amazon.com, https://github.com/openai). URLs are normalized to domains before scraping. Legacy urls input is still supported.
database — Semrush regional database: worldwide or one of 100 country codes (us, de, uk, jp, br, …)
include_moz — adds Moz DA / Spam Score / top pages / ranking distribution (off by default; adds 5–10 s and Residential traffic per domain)
country_top_websites — for Top Websites mode: 90+ countries or global
industry_top_websites — for Top Websites mode: 80+ industries or all
concurrency — domains processed in parallel (default 5, max 50). Each parallel slot needs its own proxy IP.

Output data

Each domain returns one structured JSON record with two layers:

Flat top-level KPI fields — what Semrush UI shows as the big plates on the overview page. Render as plain table columns in the Apify dataset view.
Full nested data — every detail behind those KPIs: per-keyword arrays, daily/monthly trends, per-country splits, sample backlinks, AI-search per-LLM breakdown.

Flat KPI fields (table-friendly)

Field	Meaning
`domain` / `database`	Domain + selected region (`worldwide` or country code)
`authority_score`	Semrush Authority Score 0–100
`backlinks` / `referring_domains`	Total backlinks + total referring domains
`follow_backlinks` / `nofollow_backlinks`	Follow vs nofollow split
`organic_traffic` / `organic_keywords`	Estimated monthly organic visits (excluding SERP features) + organic keyword count
`total_traffic`	Organic + SERP-feature traffic — matches the big number on Semrush's "Organic Search Traffic" plate
`organic_traffic_cost_usd`	What that organic traffic would cost via paid ads
`paid_traffic` / `paid_keywords` / `paid_traffic_cost_usd`	Paid search KPIs
`ai_visibility` / `ai_visibility_benchmark`	AI visibility 0–100 + industry benchmark
`ai_mentions` / `ai_cited_pages`	Total mentions across LLMs + cited pages
`top_country` / `top_country_traffic`	Country with the most organic traffic (most useful in `worldwide` mode)
`organic_competitors_count` / `paid_competitors_count`	Total competitor counts
`moz_domain_authority` / `moz_spam_score`	Moz DA + Spam Score (when Moz is enabled)

Source rule for organic_* flat fields: for a specific country DB (us, de, uk, …) we use that country's summary directly. For worldwide we use the latest point of the global daily trend.

Nested sections (raw detail)

Section	What's inside
`authority`	Authority Score (0–100), backlinks total, referring domains, link power, naturalness, health, search-traffic factor
`backlinks_detail`	Follow / nofollow / images / forms / frames split, link types, top anchor texts with backlink + domain counts, sample referring URLs
`organic.country_summary`	Selected country: rank, organic traffic, keywords, traffic cost USD, branded vs non-branded split, SERP-feature share
*`organic.summary_by_country_`**	Per-country daily and monthly aggregates across all Semrush regional databases
`organic.top_keywords`	Top 50 organic keywords with position (and previous), volume, CPC USD, KD, traffic, traffic share %, cost, decoded intent (informational/navigational/commercial/transactional), SERP feature codes, landing URL
`organic.competitors`	Top organic competitors with competition level, common/organic/paid keywords, organic traffic, traffic cost, total traffic, SERP-feature traffic
`organic.competitors_total`	Total organic competitor count
*`organic.trend__daily/monthly`**	Daily and monthly time-series of organic + paid + serp-feature traffic, country and global, with branded/non-branded split
`paid.top_ads`	Top 50 paid keywords with full ad copy (title, description, visible URL), position, volume, CPC, KD, traffic, traffic cost
`paid.competitors`	Top paid competitors with shared keywords, ad traffic, ad spend
`paid.competitors_total`	Total paid competitor count
`ai_search.ai_visibility`	0–100 AI Visibility score plus industry benchmark
`ai_search.by_llm`	Per-LLM breakdown: ChatGPT, AI Overview, AI Mode, Gemini — mentions, self-mentions, cited pages
`ai_search.top_cited_sources`	Top 20 domains cited by AI together with this domain
`ai_search.by_country`	AI visibility and mention counts per country
`serp_features`	Distribution of organic positions across Google SERP features
`moz`	DA, Spam Score, linking root domains, ranking keywords count, top pages by PA, top linking domains by DA, search competitors with visibility, top ranking keywords with rank, ranking distribution (#1-3 → #41-50), discovered/lost linking domains over 60 days (optional — toggle in input)

Top Websites Ranking mode

Section	What's inside
Top 100 Websites	Domain, total visits, desktop/mobile visits, desktop/mobile %, MoM %, YoY %, primary traffic source — filterable by 90+ countries and 80+ industries

Output example (abbreviated)

{
  "domain": "amazon.com",
  "database": "worldwide",

  "authority_score": 100,
  "backlinks": 5867395493,
  "referring_domains": 4342366,
  "follow_backlinks": 2839704998,
  "nofollow_backlinks": 2574690164,
  "organic_traffic": 568288700,
  "total_traffic": 615432100,
  "organic_keywords": 99114924,
  "organic_traffic_cost_usd": 428258681,
  "paid_traffic": 14773778,
  "paid_keywords": 58366,
  "paid_traffic_cost_usd": 14284968,
  "ai_visibility": 67,
  "ai_visibility_benchmark": 66,
  "ai_mentions": 8189191,
  "ai_cited_pages": 7827618,
  "top_country": "us",
  "top_country_traffic": 642837782,
  "organic_competitors_count": 6007717,
  "paid_competitors_count": 7988,
  "moz_domain_authority": "100",
  "moz_spam_score": "1%",

  "is_root_domain": false,

  "authority": {
    "score": 100,
    "backlinks": 5867395493,
    "referring_domains": 4342366,
    "link_power": 6.99,
    "naturalness": 6,
    "health": 26,
    "search_traffic_factor": 6.99
  },

  "backlinks_detail": {
    "total_backlinks": 5867395493,
    "follow": 2839704998,
    "nofollow": 2574690164,
    "domains": 4342366,
    "top_anchors": [
      {"anchor": "<EmptyAnchor>", "backlinks": 849358331, "domains": 1094218},
      {"anchor": "amazon.com",     "backlinks":  35520664, "domains":  210383},
      {"anchor": "buy from amazon","backlinks":  23179702, "domains":   10176}
    ],
    "sample_backlinks": [
      {"anchor": "Apple iPad Air 11-Inch", "nofollow": true,
       "source_url": "https://bestreviews.com/", "target_url": "https://www.amazon.com/dp/B0DZ754XKT"}
    ]
  },

  "organic": {
    "country_summary": null,
    "top_keywords": [
      {
        "keyword": "amazon",
        "position": 1,
        "volume": 151000000,
        "cpc_usd": 0.03,
        "keyword_difficulty": 100,
        "traffic": 120800000,
        "traffic_share_pct": 18.79,
        "url": "https://www.amazon.com/",
        "intents": ["commercial"]
      }
    ],
    "competitors": [
      {"domain": "walmart.com", "competition_level": 0.52,
       "common_keywords": 9651380, "organic_traffic": 117649918}
    ],
    "competitors_total": 6007717,
    "trend_global_daily": [
      {"date": "20250507", "organic_traffic": 568288700, "organic_keywords": 99114924,
       "paid_traffic": 14773778, "branded_traffic": 207381624, "non_branded_traffic": 435456158}
    ],
    "summary_by_country_daily": [
      {"database": "us", "rank": 6, "organicTraffic": 568288700, "organicPositions": 99114924, "traffic": 642837782},
      {"database": "in", "rank": 167, "organicTraffic": 12054910, "organicPositions": 1413942, "traffic": 16008667}
    ]
  },

  "paid": {
    "top_ads": [
      {
        "keyword": "amazon",
        "title": "Amazon",
        "description": "Amazon® Official Site — Find deals and compare prices on amazon at Amazon.",
        "url": "https://www.amazon.com/",
        "position": 1,
        "volume": 151000000,
        "cpc_usd": 0.03
      }
    ],
    "competitors_total": 7988
  },

  "ai_search": {
    "ai_visibility": 67,
    "ai_visibility_benchmark": 66,
    "total_cited_pages": 7827618,
    "total_mentions": 8189191,
    "by_llm": [
      {"llm": "ChatGPT",     "mentions": 1388071, "cited_pages":  998414},
      {"llm": "AI Overview", "mentions": 1887053, "cited_pages": 2067040},
      {"llm": "AI Mode",     "mentions": 2851034, "cited_pages": 5640097},
      {"llm": "Gemini",      "mentions": 2063033, "cited_pages":   26249}
    ],
    "top_cited_sources": [
      {"domain": "youtube.com", "mentions": 1511374},
      {"domain": "reddit.com",  "mentions":  954001}
    ],
    "by_country": [
      {"country": "us", "visibility": 79, "mentions": 4872535},
      {"country": "uk", "visibility": 77, "mentions":  486610}
    ]
  },

  "moz": {
    "domain_authority": "100",
    "spam_score": "1%",
    "linking_root_domains": "5.5m",
    "ranking_keywords": "27.4m"
  }
}

The first 23 keys are the flat KPI block (what gets rendered as table columns in the Apify dataset view). Everything from is_root_domain downwards is the same data plus the nested detail (per-keyword arrays, history, per-country splits) for downstream pipelines.

What makes this scraper different

Feature	Public-page scrapers	This scraper
Authority Score and full backlink profile	Limited	Yes — full Domain Overview
Top keywords with CPC, KD, intent, SERP features	No	Yes — top 50 per domain
Paid ad copy (title + description + URL)	No	Yes — top 50 paid keywords
Organic + paid competitors with shared keywords	Partial	Yes — 20 each + total counts
AI Search per-LLM breakdown (ChatGPT / AIO / AIM / Gemini)	No	Yes
Top cited sources alongside the domain in AI answers	No	Yes
Daily + monthly traffic trends, country + global	3–6 months	Multi-year daily resolution
Top anchor breakdown + sample backlinks	No	Yes
Moz Domain Analysis (DA, Spam, top pages, ranking distribution, 60-day trend)	DA only	Yes — 10+ metrics
No Semrush subscription or API key needed	Varies	Yes
Async parallel processing (configurable, default 5)	Varies	Yes
Up to 100 domains per run	Varies	Yes

Frequently asked questions

How does this differ from running domains through the public Semrush homepage? The public preview shows only Authority Score and a clipped traffic estimate. This scraper returns the full authenticated Domain Overview — top 50 organic and paid keywords, full backlink profile, AI search per-LLM breakdown, competitors with shared-keyword counts, and multi-year daily trends.

Do I need a Semrush subscription or API key? No. The actor handles authentication and proxies internally — you only need an Apify account.

How do I find which competitors are stealing my organic traffic? Put your domain in domains, set the appropriate database (e.g. us), and read organic.competitors. You get up to 20 organic competitors per domain with competition level, common keywords, organic traffic, and total traffic — plus organic.competitors_total for the full count.

Can I see exactly what ad copy a competitor is running? Yes. paid.top_ads returns the top 50 paid keywords with full ad title, description, and visible URL alongside position, volume, and CPC.

How is AI Search visibility measured? ai_search.by_llm returns mentions and cited pages per LLM — ChatGPT, AI Overview, AI Mode, and Gemini — plus a 0–100 aggregate AI Visibility score and the industry benchmark. ai_search.top_cited_sources lists the 20 domains most often cited alongside yours in AI answers, so you see who shares your AI-result shelf-space.

What's the difference between organic_traffic and total_traffic? organic_traffic is the classic estimate of monthly visits from organic positions only. total_traffic includes SERP features (featured snippets, People Also Ask, AI Overview, etc.) and matches the big number Semrush shows on the "Organic Search Traffic" plate.

Why are some organic_* numbers blank for a country DB? Some smaller Semrush regional databases don't have a full country summary; the actor falls back to the global trend point for those. Use worldwide if you want the aggregated view.

How many domains can I process at once? Up to 100 domains per run, processed in parallel up to your configured concurrency (default 5, max 50). Each parallel slot uses its own proxy IP — keep concurrency at 5 on Apify's free tier.

Should I enable Moz data? Only if you specifically need Moz DA, Spam Score, top pages, or ranking distribution. Moz adds 5–10 seconds per domain and routes through Residential proxy. Off by default.

Who is this for

SEO specialists — Authority Score, full backlink profile, organic/paid keywords with CPC and intent — without paying for a Semrush seat
Competitive analysts — Top competitors per domain with shared keywords, traffic, and traffic cost — organic and paid side by side
AI/LLM researchers — Per-LLM mentions and cited pages across ChatGPT, AI Overview, AI Mode, Gemini — including which third-party domains co-rank in AI answers
Performance marketers — Top paid ads with full ad copy and CPC for any competitor — feed into ad-research pipelines
Agencies & data teams — Daily and monthly time-series for traffic, keywords, branded vs non-branded split, exportable as JSON / CSV / Excel from the Apify dataset

Pro tips

Pick the right database — country-specific DBs (us, de, uk, …) give a real country summary; worldwide aggregates the global trend. Don't mix them in one analysis.
Use summary_by_country_* — even in a single run you get a per-country breakdown across all Semrush regional databases. Great for "which country drives this domain's traffic" without re-running.
Read intents on keywords — informational vs commercial vs transactional changes how you treat a competitor's ranking. The decoded intent saves you from inferring it.
Use top_cited_sources for AI co-citation research — the domains most often cited alongside yours in AI answers are the natural targets for digital PR and link campaigns.
Keep concurrency ≤ 5 on free tier — Apify's free Datacenter pool gives you 5 slots. Higher concurrency starves itself on IP rotation.
Combine with the Top Websites mode — find the top 100 sites in your industry, then bulk-feed them back into Domain Overview mode to score them all.

Support

Having issues or need a custom feature? Open an issue in the Issues tab or contact: africanec@gmail.com