Semrush Domain Overview Scraper — Authority, Organic, Paid, AI Search, Backlinks & Competitors

🦅 More from Raven ►	Ahrefs SEO Tools	Semrush Scraper	Similarweb Scraper

Pulls the full Semrush Domain Overview dashboard for any domain — Authority Score, organic & paid keywords with CPC and intent, top organic + paid competitors, AI Search visibility across ChatGPT / AI Overview / AI Mode / Gemini, full backlinks profile with anchor breakdown, daily and monthly traffic trends, and an optional Moz Domain Analysis snapshot. Everything that's behind the Semrush login — without needing a Semrush subscription or API key.

What data do you get?

Each domain returns one structured JSON record. The result has two layers:

1. Flat top-level KPI fields — what Semrush UI shows as the big plates on the overview page. These render as plain table columns in the Apify dataset view (no dot.notation paths needed). Power-users still get the raw nested structures below.
2. Full nested data — every detail behind those KPIs: per-keyword breakdown, daily/monthly trends, per-country splits, sample backlinks, AI-search per-LLM breakdown, etc.

Flat top-level fields (table-friendly)

Field	Meaning
domain / database	Domain + selected region (worldwide or country code)
authority_score	Semrush Authority Score 0–100
backlinks / referring_domains	Total backlinks + total referring domains
follow_backlinks / nofollow_backlinks	Follow vs nofollow split
organic_traffic / organic_keywords	Estimated monthly organic visits (excluding SERP features) + organic keyword count
total_traffic	Organic + SERP-feature traffic — matches the big number on Semrush's "Organic Search Traffic" plate in the UI
organic_traffic_cost_usd	What that organic traffic would cost via paid ads
paid_traffic / paid_keywords / paid_traffic_cost_usd	Paid search KPIs
ai_visibility / ai_visibility_benchmark	AI visibility 0–100 + industry benchmark
ai_mentions / ai_cited_pages	Total mentions across LLMs + cited pages
top_country / top_country_traffic	Country with the most organic traffic (most useful in worldwide mode)
organic_competitors_count / paid_competitors_count	Total competitor counts
moz_domain_authority / moz_spam_score	Moz DA + Spam Score (when Moz is enabled)

Source rule for organic_* flat fields: for a specific country DB (us, de, uk, …) we use that country's summary directly. For worldwide we use the latest point of the global daily trend.

Nested sections (raw detail)

Section	What's inside
authority	Authority Score (0–100), backlinks total, referring domains, link power, naturalness, health, search-traffic factor
backlinks_detail	Follow / nofollow / images / forms / frames split, link types, top anchor texts with backlink + domain counts, sample referring URLs
organic.country_summary	Selected country: rank, organic traffic, keywords, traffic cost USD, branded vs non-branded split, SERP-feature share
organic.summary_by_country_*	Per-country daily and monthly aggregates across all Semrush regional databases
organic.top_keywords	Top 50 organic keywords with position (and previous), volume, CPC USD, KD, traffic, traffic share %, cost, decoded intent (informational/navigational/commercial/transactional), SERP feature codes, landing URL
organic.competitors	Top organic competitors with competition level, common/organic/paid keywords, organic traffic, traffic cost, total traffic, SERP-feature traffic
organic.competitors_total	Total organic competitor count
organic.trend_*_daily/monthly	Daily and monthly time-series of organic + paid + serp-feature traffic, country and global, with branded/non-branded split
paid.top_ads	Top 50 paid keywords with full ad copy (title, description, visible URL), position, volume, CPC, KD, traffic, traffic cost
paid.competitors	Top paid competitors with shared keywords, ad traffic, ad spend
paid.competitors_total	Total paid competitor count
ai_search.ai_visibility	0–100 AI Visibility score plus industry benchmark
ai_search.by_llm	Per-LLM breakdown: ChatGPT, AI Overview, AI Mode, Gemini — mentions, self-mentions, cited pages
ai_search.top_cited_sources	Top 20 domains cited by AI together with this domain
ai_search.by_country	AI visibility and mention counts per country
serp_features	Distribution of organic positions across Google SERP features
moz	DA, Spam Score, linking root domains, ranking keywords count, top pages by PA, top linking domains by DA, search competitors with visibility, top ranking keywords with rank, ranking distribution (#1-3 → #41-50), discovered/lost linking domains over 60 days (optional — toggle in input)

Top Websites Ranking mode

Section	What's inside
Top 100 Websites	Domain, total visits, desktop/mobile visits, desktop/mobile %, MoM %, YoY %, primary traffic source — filterable by 90+ countries and 80+ industries

How to use

Mode 1: Domain Overview (default)

1. Select mode "Domain Overview".
2. Enter one or more domains (e.g. amazon.com, github.com). Up to 100 per run, processed in parallel.
3. Choose a Database (region) — defaults to Worldwide (100 country options available).
4. Optionally enable Include Moz Data (off by default — adds Moz DA / Spam Score / top pages etc.).
5. Set Concurrency (default 5 — fits free Apify Datacenter proxy slots; raise it on paid plans up to 50).
6. Click Start.

Mode 2: Top Websites Ranking

1. Select mode "Top Websites Ranking".
2. Choose country (90+ options) and industry (80+ categories).
3. Click Start. Returns top 100 trending websites.

Output example (abbreviated)

{
  "domain": "amazon.com",
  "database": "worldwide",

  "authority_score": 100,
  "backlinks": 5867395493,
  "referring_domains": 4342366,
  "follow_backlinks": 2839704998,
  "nofollow_backlinks": 2574690164,
  "organic_traffic": 568288700,
  "total_traffic": 615432100,
  "organic_keywords": 99114924,
  "organic_traffic_cost_usd": 428258681,
  "paid_traffic": 14773778,
  "paid_keywords": 58366,
  "paid_traffic_cost_usd": 14284968,
  "ai_visibility": 67,
  "ai_visibility_benchmark": 66,
  "ai_mentions": 8189191,
  "ai_cited_pages": 7827618,
  "top_country": "us",
  "top_country_traffic": 642837782,
  "organic_competitors_count": 6007717,
  "paid_competitors_count": 7988,
  "moz_domain_authority": "100",
  "moz_spam_score": "1%",

  "is_root_domain": false,

  "authority": {
    "score": 100,
    "backlinks": 5867395493,
    "referring_domains": 4342366,
    "link_power": 6.99,
    "naturalness": 6,
    "health": 26,
    "search_traffic_factor": 6.99
  },

  "backlinks_detail": {
    "total_backlinks": 5867395493,
    "follow": 2839704998,
    "nofollow": 2574690164,
    "domains": 4342366,
    "top_anchors": [
      {"anchor": "<EmptyAnchor>", "backlinks": 849358331, "domains": 1094218},
      {"anchor": "amazon.com",     "backlinks":  35520664, "domains":  210383},
      {"anchor": "buy from amazon","backlinks":  23179702, "domains":   10176}
    ],
    "sample_backlinks": [
      {"anchor": "Apple iPad Air 11-Inch", "nofollow": true,
       "source_url": "https://bestreviews.com/", "target_url": "https://www.amazon.com/dp/B0DZ754XKT"}
    ]
  },

  "organic": {
    "country_summary": null,
    "top_keywords": [
      {
        "keyword": "amazon",
        "position": 1,
        "volume": 151000000,
        "cpc_usd": 0.03,
        "keyword_difficulty": 100,
        "traffic": 120800000,
        "traffic_share_pct": 18.79,
        "url": "https://www.amazon.com/",
        "intents": ["commercial"]
      }
    ],
    "competitors": [
      {"domain": "walmart.com", "competition_level": 0.52,
       "common_keywords": 9651380, "organic_traffic": 117649918}
    ],
    "competitors_total": 6007717,
    "trend_global_daily": [
      {"date": "20250507", "organic_traffic": 568288700, "organic_keywords": 99114924,
       "paid_traffic": 14773778, "branded_traffic": 207381624, "non_branded_traffic": 435456158}
    ],
    "summary_by_country_daily": [
      {"database": "us", "rank": 6, "organicTraffic": 568288700, "organicPositions": 99114924, "traffic": 642837782},
      {"database": "in", "rank": 167, "organicTraffic": 12054910, "organicPositions": 1413942, "traffic": 16008667}
    ]
  },

  "paid": {
    "top_ads": [
      {
        "keyword": "amazon",
        "title": "Amazon",
        "description": "Amazon® Official Site — Find deals and compare prices on amazon at Amazon.",
        "url": "https://www.amazon.com/",
        "position": 1,
        "volume": 151000000,
        "cpc_usd": 0.03
      }
    ],
    "competitors_total": 7988
  },

  "ai_search": {
    "ai_visibility": 67,
    "ai_visibility_benchmark": 66,
    "total_cited_pages": 7827618,
    "total_mentions": 8189191,
    "by_llm": [
      {"llm": "ChatGPT",     "mentions": 1388071, "cited_pages":  998414},
      {"llm": "AI Overview", "mentions": 1887053, "cited_pages": 2067040},
      {"llm": "AI Mode",     "mentions": 2851034, "cited_pages": 5640097},
      {"llm": "Gemini",      "mentions": 2063033, "cited_pages":   26249}
    ],
    "top_cited_sources": [
      {"domain": "youtube.com", "mentions": 1511374},
      {"domain": "reddit.com",  "mentions":  954001}
    ],
    "by_country": [
      {"country": "us", "visibility": 79, "mentions": 4872535},
      {"country": "uk", "visibility": 77, "mentions":  486610}
    ]
  },

  "moz": {
    "domain_authority": "100",
    "spam_score": "1%",
    "linking_root_domains": "5.5m",
    "ranking_keywords": "27.4m"
  }
}

The first 23 keys are the flat KPI block (what gets rendered as table columns in the Apify dataset view). Everything from is_root_domain downwards is the same data plus the nested detail (per-keyword arrays, history, per-country splits) for downstream pipelines.

Who is this for?

• SEO specialists — Authority Score, full backlink profile, organic/paid keywords with CPC and intent — without paying for a Semrush seat.
• Competitive analysts — Top competitors per domain with shared keywords, traffic, and traffic cost — organic and paid side by side.
• AI/LLM researchers — Per-LLM mentions and cited pages across ChatGPT, AI Overview, AI Mode, Gemini — including which third-party domains co-rank in AI answers.
• Performance marketers — Top paid ads with full ad copy and CPC for any competitor — feed into ad-research pipelines.
• Agencies & data teams — Daily and monthly time-series for traffic, keywords, branded vs non-branded split, exportable as JSON / CSV / Excel from the Apify dataset.

What makes this scraper different?

Feature	Public-page scrapers	This scraper
Authority Score and full backlink profile	Limited	Yes — full Domain Overview
Top keywords with CPC, KD, intent, SERP features	No	Yes — top 50 per domain
Paid ad copy (title + description + URL)	No	Yes — top 50 paid keywords
Organic + paid competitors with shared keywords	Partial	Yes — 20 each + total counts
AI Search per-LLM breakdown (ChatGPT / AIO / AIM / Gemini)	No	Yes
Top cited sources alongside the domain in AI answers	No	Yes
Daily + monthly traffic trends, country + global	3–6 months	Multi-year daily resolution
Top anchor breakdown + sample backlinks	No	Yes
Moz Domain Analysis (DA, Spam, top pages, ranking distribution, 60-day trend)	DA only	Yes — 10+ metrics
No Semrush subscription or API key needed	Varies	Yes
Async parallel processing (configurable, default 5)	Varies	Yes
Up to 100 domains per run	Varies	Yes

Support

Having issues or need a custom feature? Open an issue in the Issues tab or contact: africanec@gmail.com

Tags

Semrush Scraper, Domain Overview, Authority Score, Backlinks Profile, Organic Keywords, Paid Keywords, Ad Copy Scraper, Competitor Analysis, AI Search Visibility, ChatGPT Mentions, AI Overview Mentions, AI Mode, Gemini Mentions, Moz Domain Authority, Moz Spam Score, Top Cited Sources, Bulk SEO, Bulk Domain Analysis, Parallel Scraper, Trending Websites, Market Research Tool, Competitive Intelligence, Daily Traffic Trends, SERP Features Distribution, Branded Traffic Split