Semrush Scraper — Domain Overview, Keyword Research, AI Search, Backlinks, Competitors & Top Websites

This scraper turns Semrush into a plug-in SEO data API. Pick a mode and go:

• 🌐 Domain Overview — drop in a domain, get the Semrush Domain Overview dashboard as one structured JSON record: Authority Score, backlink and keyword totals, top organic keywords and paid ads, top organic + paid competitors, AI Search visibility across ChatGPT, AI Overview, AI Mode, Gemini, backlink type and anchor breakdown, daily and monthly traffic trends, and an optional Moz Domain Analysis snapshot (DA + Spam Score). Optionally add row-level lists nested in the same record: the top 10 organic keywords, up to 80 backlinks and up to 60 referring domains per domain.
• 🔑 Keyword Research — drop in a keyword, get volume, CPC, Keyword Difficulty, competition, search intent, a 12-month trend, volume across every country, related keywords, question keywords, topic clusters, ad copies and the live SERP.
• 🔍 SEO Audit (on-page) — drop in a page URL, get a full on-page audit (title, meta, H1–H6, internal/external links, images & missing alt, canonical, robots/sitemap, Open Graph, hreflang) plus Semrush organic context (authority, traffic, top keywords, competitors).
• 🏆 Top Websites Ranking — the top 100 trending websites for any of 90+ countries × 80+ industries.

The full Semrush dataset as structured JSON — no Semrush subscription or API key required.

---

Raven · SEO & web-intelligence suite   •  Ahrefs, Semrush, Similarweb, contact data
Semrush Scraper ➤ You are here	Ahrefs SEO Tools DR, backlinks, keywords, SERP	Similarweb Scraper Traffic, rankings, analytics	Website Contacts Emails, phones, socials

Key features

• Four modes, one Actor — 🌐 Domain Overview, 🔑 Keyword Research, 🔍 SEO Audit, 🏆 Top Websites — switch with a single mode selector
• Keyword Research in one record — volume, CPC, KD, competition, decoded intent, 12-month trend, per-country volume, related + question keywords, clusters, ad copies, and the full SERP
• On-page SEO Audit — title, meta, H1–H6, link & image checks, canonical, robots/sitemap, Open Graph & hreflang, enriched with Semrush organic keywords and competitors
• Domain Overview in one record — every plate from the Semrush UI, flattened into one JSON row per domain
• Logged-in Semrush data — the Domain Overview a signed-in user sees, not the clipped "public preview" numbers
• Organic keyword, backlink and referring-domain lists (optional) — top 10 organic keywords with position, volume, KD, CPC, URL, traffic %, intent; up to 80 backlinks (source → target URL, anchor, follow/nofollow, first/last seen, page AS); up to 60 referring domains — nested in the domain's record, still one result per domain
• AI Search visibility — per-LLM mentions and cited pages across ChatGPT / AI Overview / AI Mode / Gemini, plus top co-cited domains
• Top organic keywords and paid ads from the overview — the short lists the Domain Overview widgets show (usually 7 keywords and up to ~6 ads with full ad copy)
• Organic + paid competitors — the overview's top competitors (usually 6 each) plus total counts, with shared keywords and traffic
• Multi-year daily backlink and traffic trends — global and per-country, branded vs non-branded split
• Optional Moz Domain Analysis — DA, Spam Score, linking root domains, top pages by PA, top linking domains, 60-day linking-domain trend
• Top Websites Ranking mode — top 100 trending websites by 90+ countries × 80+ industries
• Up to 100 domains per run, parallel — configurable concurrency (default 5, max 50)
• Flat KPI columns up front — render as plain table columns in the Apify dataset view; no dot.notation paths needed
• Export-ready — CSV / JSON / Excel / API webhook out of the box

---

How it works

1. Pick a mode — 🌐 Domain Overview, 🔑 Keyword Research, or 🏆 Top Websites Ranking
2. Domain Overview: paste 1–100 domains (e.g. amazon.com, github.com); optionally enable Include Moz Data and the 📋 lists (organic keywords, backlinks, referring domains)
3. Keyword Research: paste 1–100 keywords (e.g. best running shoes, seo tools)
4. Choose a regional database (Worldwide + 100 country options; Keyword mode uses the country, Worldwide → United States)
5. Set parallelism (default 5; up to 50 for faster runs on larger inputs)
6. Click Start — results land in the dataset as one record per domain / keyword (the lists, when on, are nested inside the domain's record)

Domain Overview triggers a single batched JSON-RPC call covering organic + paid + backlinks + AI search + competitors. Keyword Research fans out the keyword's volume/CPC/KD/intent, related + question ideas, clusters, ad copies and SERP — all in a single pass. Cost scales with the number of records in the dataset: exactly one per domain or keyword — the lists don't add records.

---

Primary use cases

User type	Challenge	Solution
SEO specialist	Need Authority Score, keyword and backlink profile for 50 clients weekly	Bulk-feed 50 domains, get the Domain Overview plus top keywords, backlinks and referring domains in one run
Link builder	"Which sites link to my competitor, and which links are new?"	Backlinks list: strongest, newest and most recently seen links with anchor, follow/nofollow and first/last seen
Competitive analyst	"Who is actually beating us in organic and paid?"	Top organic + paid competitors per domain, with shared keywords and traffic
AI/LLM researcher	Need to track which brands LLMs cite, and which sources LLMs cite alongside them	Per-LLM mentions and cited pages across ChatGPT / AIO / AIM / Gemini + top co-cited domains
Performance marketer	"What ad copy is the competitor running, and on which keywords?"	The overview's top paid keywords with full ad title, description, URL and CPC
Agency / data team	Need daily traffic time-series to feed dashboards	Multi-year daily resolution, per-country and global, branded vs non-branded split
Market researcher	Need the top 100 sites in a specific country × industry	Top Websites mode — 90+ countries × 80+ industries, with MoM/YoY change and traffic-source split

---

Input parameters

• mode — domain (🌐 Domain Overview), keyword (🔑 Keyword Research), seo_audit (🔍 SEO Audit) or top_websites (🏆 Top Websites Ranking)
• domains — 1–100 domains or URLs. Domain Overview normalizes them to domains; SEO Audit keeps the full URL and audits that exact page (e.g. https://stripe.com/pricing). Legacy urls input is still supported.
• keywords — 1–100 keywords/phrases for Keyword Research mode (e.g. best running shoes, seo tools)
• database — Semrush regional database: worldwide or one of 100 country codes (us, de, uk, jp, br, …). Keyword mode uses the chosen country; worldwide falls back to United States.
• include_moz — fills moz_domain_authority / moz_spam_score and the nested moz object (top pages, top linking domains, 60-day trend). Off by default — leave it off and those columns stay empty; adds 5–10 s per domain
• include_organic_keywords — Domain mode: add the domain's top 10 organic keywords to its record (organic_keywords_list). Off by default
• include_backlinks + backlinks_limit (1–80, default 10) — Domain mode: add backlinks to the domain's record (backlinks_list). Off by default
• include_referring_domains + referring_domains_limit (1–60, default 10) — Domain mode: add referring domains to the domain's record (referring_domains_list). Off by default
• country_top_websites — for Top Websites mode: 90+ countries or global
• industry_top_websites — for Top Websites mode: 80+ industries or all
• concurrency — domains/keywords processed in parallel (default 5, max 50).

---

Output data

Each domain returns exactly one structured JSON record (with the optional lists nested in it — see the lists section) with two layers:

1. Flat top-level KPI fields — what Semrush UI shows as the big plates on the overview page. Render as plain table columns in the Apify dataset view.
2. Full nested data — every detail behind those KPIs: per-keyword arrays, daily/monthly trends, per-country splits, sample backlinks, AI-search per-LLM breakdown.

Flat KPI fields (table-friendly)

Field	Meaning
domain / database	Domain + selected region (worldwide or country code)
authority_score	Semrush Authority Score 0–100
backlinks / referring_domains	Total backlinks + total referring domains
follow_backlinks / nofollow_backlinks	Follow vs nofollow split
organic_traffic / organic_keywords	Estimated monthly organic visits (excluding SERP features) + organic keyword count
total_traffic	Organic + SERP-feature traffic — matches the big number on Semrush's "Organic Search Traffic" plate
organic_traffic_cost_usd	What that organic traffic would cost via paid ads
paid_traffic / paid_keywords / paid_traffic_cost_usd	Paid search KPIs
ai_visibility / ai_visibility_benchmark	AI visibility 0–100 + industry benchmark
ai_mentions / ai_cited_pages	Total mentions across LLMs + cited pages
top_country / top_country_traffic	Country with the most organic traffic (most useful in worldwide mode)
organic_competitors_count / paid_competitors_count	Total competitor counts
moz_domain_authority / moz_spam_score	Moz DA + Spam Score (when Moz is enabled)

Source rule for organic_* flat fields: for a specific country DB (us, de, uk, …) we use that country's summary directly. For worldwide we use the latest point of the global daily trend.

Nested sections (raw detail)

Section	What's inside
authority	Authority Score (0–100), backlinks total, referring domains, link power, naturalness, health, search-traffic factor
backlinks_detail	Follow / nofollow / images / forms / frames split, link types, top 5 anchor texts with backlink + domain counts, 5 sample backlinks (turn on the backlinks list for more)
organic.country_summary	Selected country: rank, organic traffic, keywords, traffic cost USD, branded vs non-branded split, SERP-feature share
organic.summary_by_country_*	Per-country daily and monthly aggregates across all Semrush regional databases
organic.top_keywords	The overview widget's top organic keywords (usually 7) with position (and previous), volume, CPC USD, KD, traffic, traffic share %, cost, decoded intent (informational/navigational/commercial/transactional), SERP feature codes, landing URL — turn on the organic keywords list to get the top 10 in organic_keywords_list
organic.competitors	Top organic competitors (usually 6) with competition level, common/organic/paid keywords, organic traffic, traffic cost, total traffic, SERP-feature traffic
organic.competitors_total	Total organic competitor count
organic.trend_*_daily/monthly	Daily and monthly time-series of organic + paid + serp-feature traffic, country and global, with branded/non-branded split
paid.top_ads	The overview's top paid keywords (up to ~6) with full ad copy (title, description, visible URL), position, volume, CPC, KD, traffic, traffic cost
paid.competitors	Top paid competitors (usually 6) with shared keywords, ad traffic, ad spend
paid.competitors_total	Total paid competitor count
ai_search.ai_visibility	0–100 AI Visibility score plus industry benchmark
ai_search.by_llm	Per-LLM breakdown: ChatGPT, AI Overview, AI Mode, Gemini — mentions, self-mentions, cited pages
ai_search.top_cited_sources	Top 10 domains cited by AI together with this domain
ai_search.by_country	AI visibility and mention counts per country
serp_features	Distribution of organic positions across Google SERP features
moz	DA, Spam Score, linking root domains, ranking keywords count, top pages by PA, top linking domains by DA, discovered/lost linking domains over 60 days (optional — toggle in input)

Every domain record carries "type": "domain_overview".

Organic keywords, backlinks & referring domains (Domain mode lists)

Switch on include_organic_keywords, include_backlinks and/or include_referring_domains and each domain's record also carries those lists — still exactly one dataset item per domain. Only the lists you switch on are added:

Field	What's inside
organic_keywords_list	One object per keyword the domain ranks for: rank, keyword, position, previous_position, url, volume, keyword_difficulty, cpc_usd, traffic, traffic_share_pct, intents, serp_features_codes, volume_trend_12mo, last_updated
backlinks_list	One object per link pointing to the domain: source_url, source_title, target_url, anchor, nofollow, sponsored, ugc, link_type, page_authority_score, source_domain_authority_score, first_seen, last_seen, is_new, is_lost, selection
referring_domains_list	One object per domain linking to it: referring_domain, backlinks_count, domain_authority_score, follow, first_seen, last_seen, referring_domain_country, referring_domain_category, selection
*_list_count / *_list_total	Rows you got / rows Semrush has, e.g. backlinks_list_count: 80, backlinks_list_total: 2356171
organic_keywords_list_database	The country database the keywords come from
lists_note	null when the lists are complete; otherwise the reason in plain English (e.g. the run ran out of Semrush quota halfway through the domain, or the domain has no keywords in the chosen country)

The Output tab has a ready table for each list — Organic Keywords, Backlinks, Referring Domains — that unrolls the nested rows: one table row per keyword / backlink / referring domain, with its domain next to it.

How many rows you get — honestly. The Actor reads Semrush through signed-in free Semrush accounts, and Semrush shows those accounts only the first 10 rows of each report — there is no page 2.

List	Rows per domain	Example: backlinko.com has in Semrush
Organic keywords	top 10 by traffic share (the report can't be paged, re-sorted or filtered)	66,756 keywords (US)
Backlinks	up to 80 (backlinks_limit, default 10)	~2.4M backlinks
Referring domains	up to 60 (referring_domains_limit, default 10)	~77.6K referring domains

Backlink and referring-domain reports can be re-sorted, so for more than 10 rows the Actor asks Semrush for several sort orders — strongest linking page, strongest linking domain, newest, most recently seen, oldest, … — and merges them without duplicates. The selection field says which sort order surfaced each row. A row that turns up in two sort orders is kept once, so you can get a little less than the maximum: in our test run backlinko.com came back with 10 keywords, 80 backlinks and 58 referring domains, and the small stoneware.com with 57 backlinks and 46 referring domains (and no US keywords). These are the top rows of each list, not the complete list — for millions of backlinks you need a paid Semrush API plan.

Every 10 rows is one more Semrush request; the Actor spreads them across its account pool and, when an account runs out of daily quota halfway through a domain, carries on with the next account and merges the rows into the same record. If the lists still can't be finished, you get the domain's one record with every row collected so far and the reason in lists_note — never a second record for the same domain.

Organic keywords use the 🌍 database country; with worldwide the Actor picks each domain's top country (top_country of the overview — e.g. si for a Slovenian site, which has no US keywords) and records it in organic_keywords_list_database. Backlinks and referring domains are not country-specific.

Cost: the lists add nothing to the bill. A domain is one result — lists off, at their defaults, or all at maximum (10 + 80 + 60 rows). They cost time instead: a domain with every list at maximum makes about 16 extra Semrush requests.

{
  "domain": "backlinko.com",
  "database": "us",
  "authority_score": 55,
  "backlinks": 2356171,
  "referring_domains": 77601,
  "organic_traffic": 137422,
  "organic_keywords": 65188,
  "type": "domain_overview",

  "organic_keywords_list": [
    {
      "rank": 1,
      "keyword": "seo tools",
      "position": 6,
      "previous_position": 6,
      "url": "https://backlinko.com/best-free-seo-tools",
      "volume": 1500000,
      "keyword_difficulty": 95,
      "cpc_usd": 9.57,
      "traffic": 36000,
      "traffic_share_pct": 25.1,
      "intents": [],
      "serp_features_codes": [9, 21, 36, 43, 52],
      "last_updated": "2026-10-18"
    }
  ],
  "organic_keywords_list_count": 10,
  "organic_keywords_list_total": 66756,
  "organic_keywords_list_database": "us",

  "backlinks_list": [
    {
      "source_url": "https://www.athena.vc/blog/startup-curriculum",
      "source_title": "Athena VC Blog - Startup Curriculum",
      "target_url": "https://backlinko.com/hub/seo",
      "anchor": "SEO",
      "nofollow": false,
      "link_type": "text",
      "page_authority_score": 89,
      "source_domain_authority_score": 12,
      "first_seen": "2026-03-04",
      "last_seen": "2026-09-29",
      "is_lost": false,
      "selection": "highest_page_authority"
    }
  ],
  "backlinks_list_count": 80,
  "backlinks_list_total": 2356171,

  "referring_domains_list": [
    {
      "referring_domain": "semrush.com",
      "backlinks_count": 1159448,
      "domain_authority_score": 88,
      "follow": true,
      "first_seen": "2024-04-25",
      "last_seen": "2026-10-18",
      "referring_domain_country": "us",
      "referring_domain_category": "advertising_and_marketing",
      "selection": "most_backlinks"
    }
  ],
  "referring_domains_list_count": 58,
  "referring_domains_list_total": 77601,

  "lists_note": null
}

(Abbreviated: the record also has every Domain Overview field described above, and each list has all its rows — here 10, 80 and 58.)

Keyword Research mode (mode = keyword)

Each keyword returns one structured JSON record: a flat KPI block up front, full nested research data behind it.

Flat KPI fields (table-friendly)

Field	Meaning
keyword / database	Keyword + country database
volume	Average monthly searches in the selected country
cpc_usd	Average Google Ads cost-per-click
competition	Paid competition density 0–1
keyword_difficulty	How hard it is to rank organically (0–100)
referring_domains_median	Median referring domains of current top pages
organic_results_count	Number of organic results Google returns
intents	Decoded intent(s): informational / navigational / commercial / transactional
keyword_ideas_total / keyword_ideas_total_volume	Size of the related-ideas pool and its total volume
related_keywords_count / questions_count / serp_results_count	Counts of what's returned in the nested arrays

Nested sections

Section	What's inside
volume_trend_12mo	12-point relative monthly volume trend (0–100)
volume_by_country	Search volume for the same keyword across every Semrush country DB (top 40 by volume)
related_keywords	Broad-match keyword ideas with volume and difficulty
questions	Question-form keyword ideas with volume and difficulty
clusters	Topic clusters with per-cluster intent distribution
ads	Google Ads creatives for the keyword — title, description, display URL, landing URL, domain
serp	Live SERP — position, domain, URL, result kind, SERP feature codes

{
  "keyword": "best running shoes",
  "database": "us",
  "volume": 60500,
  "cpc_usd": 1.1,
  "competition": 1,
  "keyword_difficulty": 49,
  "referring_domains_median": 11,
  "organic_results_count": 753000000,
  "intents": ["commercial"],
  "keyword_ideas_total": 31402,
  "keyword_ideas_total_volume": 1003570,
  "volume_trend_12mo": [44, 44, 44, 36, 44, 44, 44, 36, 44, 44, 44, 100],
  "volume_by_country": [
    {"country": "us", "volume": 60500},
    {"country": "in", "volume": 40500},
    {"country": "uk", "volume": 12100}
  ],
  "related_keywords": [
    {"keyword": "best running shoes for men", "volume": 33100, "keyword_difficulty": 45}
  ],
  "questions": [
    {"keyword": "what are the best running shoes", "volume": 2400, "keyword_difficulty": 42}
  ],
  "clusters": [
    {"name": "best shoes to run long distance", "intents": {"commercial": 1, "informational": 0, "navigational": 0, "transactional": 0}}
  ],
  "ads": [
    {"title": "Shop Trail Performance Shoe", "description": "Engineered for performance.", "domain": "weekends.world"}
  ],
  "serp": [
    {"position": 1, "domain": "runrepeat.com", "url": "https://runrepeat.com/catalog/running-shoes", "result_kind": 50, "serp_features_codes": []}
  ]
}

SEO Audit mode (mode = seo_audit)

Each URL returns one record: on-page SEO fetched directly from the page, plus Semrush organic context for the domain.

Group	Fields
Response	status_code, final_url, response_time_ms, page_size_bytes
Meta	title + title_length, meta_description + meta_description_length, meta_robots, canonical, lang
Headings	h1 (array) + h1_count, h2_count, full headings H1–H6
Content & links	word_count, internal_links, external_links, images_count, images_missing_alt, structured_data_blocks
Social / i18n	open_graph, twitter_card, hreflang
Crawlability	has_robots_txt, has_sitemap, sitemaps
Semrush context	authority_score, organic_traffic, organic_keywords_count, backlinks, referring_domains, plus semrush.top_organic_keywords and semrush.organic_competitors

{
  "url": "https://stripe.com/pricing",
  "final_url": "https://stripe.com/en-cz/pricing",
  "domain": "stripe.com",
  "status_code": 200,
  "response_time_ms": 906,
  "page_size_bytes": 898028,
  "title": "Pricing & Fees | Stripe",
  "title_length": 23,
  "meta_description": "Find Stripe fees and pricing information...",
  "meta_description_length": 138,
  "canonical": "https://stripe.com/en-cz/pricing",
  "lang": "en-CZ",
  "h1_count": 1,
  "word_count": 4205,
  "internal_links": 301,
  "external_links": 115,
  "images_count": 56,
  "images_missing_alt": 10,
  "has_robots_txt": true,
  "has_sitemap": true,
  "authority_score": 79,
  "organic_traffic": 9512300,
  "organic_keywords_count": 1640000,
  "backlinks": 156797276,
  "semrush": {
    "top_organic_keywords": [
      {"keyword": "stripe", "position": 1, "volume": 550000, "cpc_usd": 5.18, "keyword_difficulty": 83}
    ],
    "organic_competitors": [{"domain": "paypal.com"}]
  }
}

Top Websites Ranking mode

Section	What's inside
Top 100 Websites	Domain, total visits, desktop/mobile visits, desktop/mobile %, MoM %, YoY %, primary traffic source — filterable by 90+ countries and 80+ industries

---

Output example (abbreviated)

{
  "domain": "amazon.com",
  "database": "worldwide",

  "authority_score": 100,
  "backlinks": 5867395493,
  "referring_domains": 4342366,
  "follow_backlinks": 2839704998,
  "nofollow_backlinks": 2574690164,
  "organic_traffic": 568288700,
  "total_traffic": 615432100,
  "organic_keywords": 99114924,
  "organic_traffic_cost_usd": 428258681,
  "paid_traffic": 14773778,
  "paid_keywords": 58366,
  "paid_traffic_cost_usd": 14284968,
  "ai_visibility": 67,
  "ai_visibility_benchmark": 66,
  "ai_mentions": 8189191,
  "ai_cited_pages": 7827618,
  "top_country": "us",
  "top_country_traffic": 642837782,
  "organic_competitors_count": 6007717,
  "paid_competitors_count": 7988,
  "moz_domain_authority": "100",
  "moz_spam_score": "1%",

  "is_root_domain": false,

  "authority": {
    "score": 100,
    "backlinks": 5867395493,
    "referring_domains": 4342366,
    "link_power": 6.99,
    "naturalness": 6,
    "health": 26,
    "search_traffic_factor": 6.99
  },

  "backlinks_detail": {
    "total_backlinks": 5867395493,
    "follow": 2839704998,
    "nofollow": 2574690164,
    "domains": 4342366,
    "top_anchors": [
      {"anchor": "<EmptyAnchor>", "backlinks": 849358331, "domains": 1094218},
      {"anchor": "amazon.com",     "backlinks":  35520664, "domains":  210383},
      {"anchor": "buy from amazon","backlinks":  23179702, "domains":   10176}
    ],
    "sample_backlinks": [
      {"anchor": "Apple iPad Air 11-Inch", "nofollow": true,
       "source_url": "https://bestreviews.com/", "target_url": "https://www.amazon.com/dp/B0DZ754XKT"}
    ]
  },

  "organic": {
    "country_summary": null,
    "top_keywords": [
      {
        "keyword": "amazon",
        "position": 1,
        "volume": 151000000,
        "cpc_usd": 0.03,
        "keyword_difficulty": 100,
        "traffic": 120800000,
        "traffic_share_pct": 18.79,
        "url": "https://www.amazon.com/",
        "intents": ["commercial"]
      }
    ],
    "competitors": [
      {"domain": "walmart.com", "competition_level": 0.52,
       "common_keywords": 9651380, "organic_traffic": 117649918}
    ],
    "competitors_total": 6007717,
    "trend_global_daily": [
      {"date": "20250507", "organic_traffic": 568288700, "organic_keywords": 99114924,
       "paid_traffic": 14773778, "branded_traffic": 207381624, "non_branded_traffic": 435456158}
    ],
    "summary_by_country_daily": [
      {"database": "us", "rank": 6, "organicTraffic": 568288700, "organicPositions": 99114924, "traffic": 642837782},
      {"database": "in", "rank": 167, "organicTraffic": 12054910, "organicPositions": 1413942, "traffic": 16008667}
    ]
  },

  "paid": {
    "top_ads": [
      {
        "keyword": "amazon",
        "title": "Amazon",
        "description": "Amazon® Official Site — Find deals and compare prices on amazon at Amazon.",
        "url": "https://www.amazon.com/",
        "position": 1,
        "volume": 151000000,
        "cpc_usd": 0.03
      }
    ],
    "competitors_total": 7988
  },

  "ai_search": {
    "ai_visibility": 67,
    "ai_visibility_benchmark": 66,
    "total_cited_pages": 7827618,
    "total_mentions": 8189191,
    "by_llm": [
      {"llm": "ChatGPT",     "mentions": 1388071, "cited_pages":  998414},
      {"llm": "AI Overview", "mentions": 1887053, "cited_pages": 2067040},
      {"llm": "AI Mode",     "mentions": 2851034, "cited_pages": 5640097},
      {"llm": "Gemini",      "mentions": 2063033, "cited_pages":   26249}
    ],
    "top_cited_sources": [
      {"domain": "youtube.com", "mentions": 1511374},
      {"domain": "reddit.com",  "mentions":  954001}
    ],
    "by_country": [
      {"country": "us", "visibility": 79, "mentions": 4872535},
      {"country": "uk", "visibility": 77, "mentions":  486610}
    ]
  },

  "moz": {
    "domain_authority": "100",
    "spam_score": "1%",
    "linking_root_domains": "5.5m",
    "ranking_keywords": "27.4m"
  }
}

The first 23 keys are the flat KPI block (what gets rendered as table columns in the Apify dataset view). Everything from is_root_domain downwards is the same data plus the nested detail (per-keyword arrays, history, per-country splits) for downstream pipelines.

---

What makes this scraper different

Feature	Public-page scrapers	This scraper
Authority Score and backlink profile summary	Limited	Yes — full Domain Overview
Top keywords with position, URL, CPC, KD, intent, SERP features	No	Yes — top 10 per domain, in the domain's record
Backlinks and referring domains lists	No	Yes — up to 80 backlinks + 60 referring domains per domain
Paid ad copy (title + description + URL)	No	Yes — the overview's top paid keywords
Organic + paid competitors with shared keywords	Partial	Yes — top competitors + total counts
AI Search per-LLM breakdown (ChatGPT / AIO / AIM / Gemini)	No	Yes
Top cited sources alongside the domain in AI answers	No	Yes
Daily + monthly traffic trends, country + global	3–6 months	Multi-year daily resolution
Top anchor breakdown + sample backlinks	No	Yes
Moz Domain Analysis (DA, Spam, linking root domains, top pages, top linking domains, 60-day trend)	DA only	Yes — 7 metric groups
No Semrush subscription or API key needed	Varies	Yes
Async parallel processing (configurable, default 5)	Varies	Yes
Up to 100 domains per run	Varies	Yes

---

Frequently asked questions

How does this differ from running domains through the public Semrush homepage? The public preview shows only Authority Score and a clipped traffic estimate. This scraper returns the Domain Overview a signed-in user sees — keyword and backlink totals, top organic keywords and paid ads, backlink type and anchor breakdown, AI search per-LLM breakdown, competitors with shared-keyword counts, and multi-year daily trends — plus, if you switch them on, the top 10 organic keywords, up to 80 backlinks and up to 60 referring domains in the same record.

Can I get ALL keywords or backlinks of a domain (e.g. all 66,000 keywords or 2.4M backlinks)? No — and no scraper built on Semrush's web app without a paid Semrush plan can. This Actor reads Semrush through free signed-in Semrush accounts, and Semrush shows those accounts only the first 10 rows of a report: there is no page 2, and the organic positions report cannot be re-sorted or filtered. That is why the organic keywords list is capped at the top 10 by traffic. Backlink and referring-domain reports can be re-sorted, so the Actor asks for several sort orders (strongest page, strongest domain, newest, most recently seen, oldest, …) and merges them without duplicates — that gets you up to 80 backlinks and 60 referring domains per domain. Each list comes with *_list_count (rows you got) and *_list_total (rows Semrush has), and the run log prints the same "rows obtained / total in Semrush" for every domain.

Why did I get fewer backlinks than backlinks_limit? Either the domain has fewer backlinks than that, or several sort orders returned the same links (they are de-duplicated). Small sites often have all their backlinks in the first 10. If a list was cut short for another reason — say the run ran out of Semrush quota halfway through the domain — lists_note says so; the domain still comes as its one record with every row collected so far.

How do I get the backlinks as a flat table (CSV / Excel)? Open the Backlinks view in the Output tab (also Organic Keywords and Referring Domains): it unrolls the nested list into one row per backlink with the domain next to it, and exports like any other view. Via the API, add ?fields=domain,backlinks_list&unwind=backlinks_list to the dataset items URL.

Do I need a Semrush subscription or API key? No. You only need an Apify account — just provide your domains or keywords and run.

What does Keyword Research mode return? Set mode to keyword and paste keywords. Each keyword returns volume, CPC, Keyword Difficulty, competition, decoded search intent, a 12-month volume trend, the same keyword's volume across every country, related and question keywords, topic clusters, the ad copies running on it, and the live SERP (position, domain, URL). It's the full Semrush Keyword Overview as one JSON row.

Can I get Traffic Analytics (visits, bounce rate, time on site)? Not in this Actor. Semrush's Traffic & Market and Site Audit data are out of scope for this scraper. Domain Overview already includes organic, paid and SERP-feature traffic estimates.

How do I find which competitors are stealing my organic traffic? Put your domain in domains, set the appropriate database (e.g. us), and read organic.competitors. You get the top organic competitors from the overview (usually 6) with competition level, common keywords, organic traffic, and total traffic — plus organic.competitors_total for the full count.

Can I see exactly what ad copy a competitor is running? Yes. paid.top_ads returns the overview's top paid keywords (up to ~6) with full ad title, description, and visible URL alongside position, volume, and CPC.

How is AI Search visibility measured? ai_search.by_llm returns mentions and cited pages per LLM — ChatGPT, AI Overview, AI Mode, and Gemini — plus a 0–100 aggregate AI Visibility score and the industry benchmark. ai_search.top_cited_sources lists the 10 domains most often cited alongside yours in AI answers, so you see who shares your AI-result shelf-space.

What's the difference between organic_traffic and total_traffic? organic_traffic is the classic estimate of monthly visits from organic positions only. total_traffic includes SERP features (featured snippets, People Also Ask, AI Overview, etc.) and matches the big number Semrush shows on the "Organic Search Traffic" plate.

Why are some organic_* numbers blank for a country DB? Some smaller Semrush regional databases don't have a full country summary; the actor falls back to the global trend point for those. Use worldwide if you want the aggregated view.

How many domains can I process at once? Up to 100 domains per run, processed in parallel up to your configured concurrency (default 5, max 50).

Should I enable Moz data? Only if you specifically need Moz DA, Spam Score, top pages or top linking domains. Moz adds 5–10 seconds per domain. It is off by default, so moz_domain_authority and moz_spam_score stay empty until you switch it on.

---

Who is this for

• SEO specialists — Authority Score, backlink profile, top keywords, backlinks and referring domains with CPC, intent and anchors — all in structured JSON
• Competitive analysts — Top competitors per domain with shared keywords, traffic, and traffic cost — organic and paid side by side
• AI/LLM researchers — Per-LLM mentions and cited pages across ChatGPT, AI Overview, AI Mode, Gemini — including which third-party domains co-rank in AI answers
• Performance marketers — Top paid ads with full ad copy and CPC for any competitor — feed into ad-research pipelines
• Agencies & data teams — Daily and monthly time-series for traffic, keywords, branded vs non-branded split, exportable as JSON / CSV / Excel from the Apify dataset

---

Pro tips

• Pick the right database — country-specific DBs (us, de, uk, …) give a real country summary; worldwide aggregates the global trend. Don't mix them in one analysis.
• Use summary_by_country_* — even in a single run you get a per-country breakdown across all Semrush regional databases. Great for "which country drives this domain's traffic" without re-running.
• Read intents on keywords — informational vs commercial vs transactional changes how you treat a competitor's ranking. The decoded intent saves you from inferring it.
• Use top_cited_sources for AI co-citation research — the domains most often cited alongside yours in AI answers are the natural targets for digital PR and link campaigns.
• Lists are free, but not instant — the lists don't add results to your bill, but every 10 backlinks or referring domains is one more Semrush request. Start with the default 10 and raise backlinks_limit / referring_domains_limit for the domains you dig into.
• Raise concurrency for large inputs — more parallel workers finish 50–100 domains/keywords faster; start at 5 and increase if your run is large.
• Combine with the Top Websites mode — find the top 100 sites in your industry, then bulk-feed them back into Domain Overview mode to score them all.

---

Support

Having issues or need a custom feature? Open an issue in the Issues tab or contact: africanec@gmail.com

---

Tags

Semrush Scraper, Domain Overview, Authority Score, Backlinks Profile, Organic Keywords, Paid Keywords, Ad Copy Scraper, Competitor Analysis, AI Search Visibility, ChatGPT Mentions, AI Overview Mentions, AI Mode, Gemini Mentions, Moz Domain Authority, Moz Spam Score, Top Cited Sources, Bulk SEO, Bulk Domain Analysis, Parallel Scraper, Trending Websites, Market Research Tool, Competitive Intelligence, Daily Traffic Trends, SERP Features Distribution, Branded Traffic Split