Semrush Scraper — Domain Overview, Keyword Research, AI Search, Backlinks, Competitors & Top Websites

🦅 More from Raven ►	Ahrefs SEO Tools	Semrush Scraper	Similarweb Scraper

This scraper turns Semrush into a plug-in SEO data API. Pick a mode and go:

• 🌐 Domain Overview — drop in a domain, get the full Domain Overview dashboard as one structured JSON record: Authority Score, organic & paid keywords with CPC and intent, top organic + paid competitors, AI Search visibility across ChatGPT, AI Overview, AI Mode, Gemini, the full backlinks profile with anchor breakdown, daily and monthly traffic trends, and an optional Moz Domain Analysis snapshot.
• 🔑 Keyword Research — drop in a keyword, get volume, CPC, Keyword Difficulty, competition, search intent, a 12-month trend, volume across every country, related keywords, question keywords, topic clusters, ad copies and the live SERP.
• 🔍 SEO Audit (on-page) — drop in a page URL, get a full on-page audit (title, meta, H1–H6, internal/external links, images & missing alt, canonical, robots/sitemap, Open Graph, hreflang) plus Semrush organic context (authority, traffic, top keywords, competitors).
• 🏆 Top Websites Ranking — the top 100 trending websites for any of 90+ countries × 80+ industries.

Everything that's behind the Semrush login — without a Semrush subscription or API key.

---

Key features

• Four modes, one Actor — 🌐 Domain Overview, 🔑 Keyword Research, 🔍 SEO Audit, 🏆 Top Websites — switch with a single mode selector
• Keyword Research in one record — volume, CPC, KD, competition, decoded intent, 12-month trend, per-country volume, related + question keywords, clusters, ad copies, and the full SERP
• On-page SEO Audit — title, meta, H1–H6, link & image checks, canonical, robots/sitemap, Open Graph & hreflang, enriched with Semrush organic keywords and competitors
• Full Domain Overview in one record — every plate from the Semrush UI, flattened into one JSON row per domain
• Authenticated Semrush data — no degraded "public preview" numbers; you get what a paying seat sees
• AI Search visibility — per-LLM mentions and cited pages across ChatGPT / AI Overview / AI Mode / Gemini, plus top co-cited domains
• Top 50 organic keywords — position, volume, CPC, KD, traffic, traffic share %, decoded intent, SERP features, landing URL
• Top 50 paid ads with full ad copy — title, description, visible URL, position, volume, CPC
• Organic + paid competitors — top 20 each plus total counts, with shared keywords and traffic
• Multi-year daily backlink and traffic trends — global and per-country, branded vs non-branded split
• Optional Moz Domain Analysis — DA, Spam Score, top pages, ranking distribution, 60-day linking-domain trend
• Top Websites Ranking mode — top 100 trending websites by 90+ countries × 80+ industries
• Up to 100 domains per run, parallel — configurable concurrency (default 5, max 50)
• Flat KPI columns up front — render as plain table columns in the Apify dataset view; no dot.notation paths needed
• Export-ready — CSV / JSON / Excel / API webhook out of the box

---

How it works

1. Pick a mode — 🌐 Domain Overview, 🔑 Keyword Research, or 🏆 Top Websites Ranking
2. Domain Overview: paste 1–100 domains (e.g. amazon.com, github.com); optionally enable Include Moz Data
3. Keyword Research: paste 1–100 keywords (e.g. best running shoes, seo tools)
4. Choose a regional database (Worldwide + 100 country options; Keyword mode uses the country, Worldwide → United States)
5. Set parallelism (default 5 — matches Apify's free Datacenter proxy slots; up to 50 on paid plans)
6. Click Start — results land in the dataset as one row per domain / keyword

Domain Overview triggers a single batched JSON-RPC call covering organic + paid + backlinks + AI search + competitors. Keyword Research fans out the keyword's volume/CPC/KD/intent, related + question ideas, clusters, ad copies and SERP — all behind your authenticated session. Cost scales linearly with the number of items, not with how many sections you want back.

---

Primary use cases

User type	Challenge	Solution
SEO specialist	Need Authority Score, keyword and backlink profile for 50 clients weekly	Bulk-feed 50 domains, get full Domain Overview as one CSV — without 50 Semrush seats
Competitive analyst	"Who is actually beating us in organic and paid?"	Top 20 organic + 20 paid competitors per domain, with shared keywords and traffic
AI/LLM researcher	Need to track which brands LLMs cite, and which sources LLMs cite alongside them	Per-LLM mentions and cited pages across ChatGPT / AIO / AIM / Gemini + top co-cited domains
Performance marketer	"What ad copy is the competitor running, and on which keywords?"	Top 50 paid keywords per domain with full ad title, description, URL and CPC
Agency / data team	Need daily traffic time-series to feed dashboards	Multi-year daily resolution, per-country and global, branded vs non-branded split
Market researcher	Need the top 100 sites in a specific country × industry	Top Websites mode — 90+ countries × 80+ industries, with MoM/YoY change and traffic-source split

---

Input parameters

• mode — domain (🌐 Domain Overview), keyword (🔑 Keyword Research), seo_audit (🔍 SEO Audit) or top_websites (🏆 Top Websites Ranking)
• domains — 1–100 domains or URLs. Domain Overview normalizes them to domains; SEO Audit keeps the full URL and audits that exact page (e.g. https://stripe.com/pricing). Legacy urls input is still supported.
• keywords — 1–100 keywords/phrases for Keyword Research mode (e.g. best running shoes, seo tools)
• database — Semrush regional database: worldwide or one of 100 country codes (us, de, uk, jp, br, …). Keyword mode uses the chosen country; worldwide falls back to United States.
• include_moz — adds Moz DA / Spam Score / top pages / ranking distribution (off by default; adds 5–10 s and Residential traffic per domain)
• country_top_websites — for Top Websites mode: 90+ countries or global
• industry_top_websites — for Top Websites mode: 80+ industries or all
• concurrency — domains/keywords processed in parallel (default 5, max 50). Each parallel slot needs its own proxy IP.

---

Output data

Each domain returns one structured JSON record with two layers:

1. Flat top-level KPI fields — what Semrush UI shows as the big plates on the overview page. Render as plain table columns in the Apify dataset view.
2. Full nested data — every detail behind those KPIs: per-keyword arrays, daily/monthly trends, per-country splits, sample backlinks, AI-search per-LLM breakdown.

Flat KPI fields (table-friendly)

Field	Meaning
domain / database	Domain + selected region (worldwide or country code)
authority_score	Semrush Authority Score 0–100
backlinks / referring_domains	Total backlinks + total referring domains
follow_backlinks / nofollow_backlinks	Follow vs nofollow split
organic_traffic / organic_keywords	Estimated monthly organic visits (excluding SERP features) + organic keyword count
total_traffic	Organic + SERP-feature traffic — matches the big number on Semrush's "Organic Search Traffic" plate
organic_traffic_cost_usd	What that organic traffic would cost via paid ads
paid_traffic / paid_keywords / paid_traffic_cost_usd	Paid search KPIs
ai_visibility / ai_visibility_benchmark	AI visibility 0–100 + industry benchmark
ai_mentions / ai_cited_pages	Total mentions across LLMs + cited pages
top_country / top_country_traffic	Country with the most organic traffic (most useful in worldwide mode)
organic_competitors_count / paid_competitors_count	Total competitor counts
moz_domain_authority / moz_spam_score	Moz DA + Spam Score (when Moz is enabled)

Source rule for organic_* flat fields: for a specific country DB (us, de, uk, …) we use that country's summary directly. For worldwide we use the latest point of the global daily trend.

Nested sections (raw detail)

Section	What's inside
authority	Authority Score (0–100), backlinks total, referring domains, link power, naturalness, health, search-traffic factor
backlinks_detail	Follow / nofollow / images / forms / frames split, link types, top anchor texts with backlink + domain counts, sample referring URLs
organic.country_summary	Selected country: rank, organic traffic, keywords, traffic cost USD, branded vs non-branded split, SERP-feature share
organic.summary_by_country_*	Per-country daily and monthly aggregates across all Semrush regional databases
organic.top_keywords	Top 50 organic keywords with position (and previous), volume, CPC USD, KD, traffic, traffic share %, cost, decoded intent (informational/navigational/commercial/transactional), SERP feature codes, landing URL
organic.competitors	Top organic competitors with competition level, common/organic/paid keywords, organic traffic, traffic cost, total traffic, SERP-feature traffic
organic.competitors_total	Total organic competitor count
organic.trend_*_daily/monthly	Daily and monthly time-series of organic + paid + serp-feature traffic, country and global, with branded/non-branded split
paid.top_ads	Top 50 paid keywords with full ad copy (title, description, visible URL), position, volume, CPC, KD, traffic, traffic cost
paid.competitors	Top paid competitors with shared keywords, ad traffic, ad spend
paid.competitors_total	Total paid competitor count
ai_search.ai_visibility	0–100 AI Visibility score plus industry benchmark
ai_search.by_llm	Per-LLM breakdown: ChatGPT, AI Overview, AI Mode, Gemini — mentions, self-mentions, cited pages
ai_search.top_cited_sources	Top 20 domains cited by AI together with this domain
ai_search.by_country	AI visibility and mention counts per country
serp_features	Distribution of organic positions across Google SERP features
moz	DA, Spam Score, linking root domains, ranking keywords count, top pages by PA, top linking domains by DA, search competitors with visibility, top ranking keywords with rank, ranking distribution (#1-3 → #41-50), discovered/lost linking domains over 60 days (optional — toggle in input)

Keyword Research mode (mode = keyword)

Each keyword returns one structured JSON record: a flat KPI block up front, full nested research data behind it.

Flat KPI fields (table-friendly)

Field	Meaning
keyword / database	Keyword + country database
volume	Average monthly searches in the selected country
cpc_usd	Average Google Ads cost-per-click
competition	Paid competition density 0–1
keyword_difficulty	How hard it is to rank organically (0–100)
referring_domains_median	Median referring domains of current top pages
organic_results_count	Number of organic results Google returns
intents	Decoded intent(s): informational / navigational / commercial / transactional
keyword_ideas_total / keyword_ideas_total_volume	Size of the related-ideas pool and its total volume
related_keywords_count / questions_count / serp_results_count	Counts of what's returned in the nested arrays

Nested sections

Section	What's inside
volume_trend_12mo	12-point relative monthly volume trend (0–100)
volume_by_country	Search volume for the same keyword across every Semrush country DB (top 40 by volume)
related_keywords	Broad-match keyword ideas with volume and difficulty
questions	Question-form keyword ideas with volume and difficulty
clusters	Topic clusters with per-cluster intent distribution
ads	Google Ads creatives for the keyword — title, description, display URL, landing URL, domain
serp	Live SERP — position, domain, URL, result kind, SERP feature codes

{
  "keyword": "best running shoes",
  "database": "us",
  "volume": 60500,
  "cpc_usd": 1.1,
  "competition": 1,
  "keyword_difficulty": 49,
  "referring_domains_median": 11,
  "organic_results_count": 753000000,
  "intents": ["commercial"],
  "keyword_ideas_total": 31402,
  "keyword_ideas_total_volume": 1003570,
  "volume_trend_12mo": [44, 44, 44, 36, 44, 44, 44, 36, 44, 44, 44, 100],
  "volume_by_country": [
    {"country": "us", "volume": 60500},
    {"country": "in", "volume": 40500},
    {"country": "uk", "volume": 12100}
  ],
  "related_keywords": [
    {"keyword": "best running shoes for men", "volume": 33100, "keyword_difficulty": 45}
  ],
  "questions": [
    {"keyword": "what are the best running shoes", "volume": 2400, "keyword_difficulty": 42}
  ],
  "clusters": [
    {"name": "best shoes to run long distance", "intents": {"commercial": 1, "informational": 0, "navigational": 0, "transactional": 0}}
  ],
  "ads": [
    {"title": "Shop Trail Performance Shoe", "description": "Engineered for performance.", "domain": "weekends.world"}
  ],
  "serp": [
    {"position": 1, "domain": "runrepeat.com", "url": "https://runrepeat.com/catalog/running-shoes", "result_kind": 50, "serp_features_codes": []}
  ]
}

SEO Audit mode (mode = seo_audit)

Each URL returns one record: on-page SEO fetched directly from the page, plus authenticated Semrush organic context for the domain.

Group	Fields
Response	status_code, final_url, response_time_ms, page_size_bytes
Meta	title + title_length, meta_description + meta_description_length, meta_robots, canonical, lang
Headings	h1 (array) + h1_count, h2_count, full headings H1–H6
Content & links	word_count, internal_links, external_links, images_count, images_missing_alt, structured_data_blocks
Social / i18n	open_graph, twitter_card, hreflang
Crawlability	has_robots_txt, has_sitemap, sitemaps
Semrush context	authority_score, organic_traffic, organic_keywords_count, backlinks, referring_domains, plus semrush.top_organic_keywords and semrush.organic_competitors

{
  "url": "https://stripe.com/pricing",
  "final_url": "https://stripe.com/en-cz/pricing",
  "domain": "stripe.com",
  "status_code": 200,
  "response_time_ms": 906,
  "page_size_bytes": 898028,
  "title": "Pricing & Fees | Stripe",
  "title_length": 23,
  "meta_description": "Find Stripe fees and pricing information...",
  "meta_description_length": 138,
  "canonical": "https://stripe.com/en-cz/pricing",
  "lang": "en-CZ",
  "h1_count": 1,
  "word_count": 4205,
  "internal_links": 301,
  "external_links": 115,
  "images_count": 56,
  "images_missing_alt": 10,
  "has_robots_txt": true,
  "has_sitemap": true,
  "authority_score": 79,
  "organic_traffic": 9512300,
  "organic_keywords_count": 1640000,
  "backlinks": 156797276,
  "semrush": {
    "top_organic_keywords": [
      {"keyword": "stripe", "position": 1, "volume": 550000, "cpc_usd": 5.18, "keyword_difficulty": 83}
    ],
    "organic_competitors": [{"domain": "paypal.com"}]
  }
}

Top Websites Ranking mode

Section	What's inside
Top 100 Websites	Domain, total visits, desktop/mobile visits, desktop/mobile %, MoM %, YoY %, primary traffic source — filterable by 90+ countries and 80+ industries

---

Output example (abbreviated)

{
  "domain": "amazon.com",
  "database": "worldwide",

  "authority_score": 100,
  "backlinks": 5867395493,
  "referring_domains": 4342366,
  "follow_backlinks": 2839704998,
  "nofollow_backlinks": 2574690164,
  "organic_traffic": 568288700,
  "total_traffic": 615432100,
  "organic_keywords": 99114924,
  "organic_traffic_cost_usd": 428258681,
  "paid_traffic": 14773778,
  "paid_keywords": 58366,
  "paid_traffic_cost_usd": 14284968,
  "ai_visibility": 67,
  "ai_visibility_benchmark": 66,
  "ai_mentions": 8189191,
  "ai_cited_pages": 7827618,
  "top_country": "us",
  "top_country_traffic": 642837782,
  "organic_competitors_count": 6007717,
  "paid_competitors_count": 7988,
  "moz_domain_authority": "100",
  "moz_spam_score": "1%",

  "is_root_domain": false,

  "authority": {
    "score": 100,
    "backlinks": 5867395493,
    "referring_domains": 4342366,
    "link_power": 6.99,
    "naturalness": 6,
    "health": 26,
    "search_traffic_factor": 6.99
  },

  "backlinks_detail": {
    "total_backlinks": 5867395493,
    "follow": 2839704998,
    "nofollow": 2574690164,
    "domains": 4342366,
    "top_anchors": [
      {"anchor": "<EmptyAnchor>", "backlinks": 849358331, "domains": 1094218},
      {"anchor": "amazon.com",     "backlinks":  35520664, "domains":  210383},
      {"anchor": "buy from amazon","backlinks":  23179702, "domains":   10176}
    ],
    "sample_backlinks": [
      {"anchor": "Apple iPad Air 11-Inch", "nofollow": true,
       "source_url": "https://bestreviews.com/", "target_url": "https://www.amazon.com/dp/B0DZ754XKT"}
    ]
  },

  "organic": {
    "country_summary": null,
    "top_keywords": [
      {
        "keyword": "amazon",
        "position": 1,
        "volume": 151000000,
        "cpc_usd": 0.03,
        "keyword_difficulty": 100,
        "traffic": 120800000,
        "traffic_share_pct": 18.79,
        "url": "https://www.amazon.com/",
        "intents": ["commercial"]
      }
    ],
    "competitors": [
      {"domain": "walmart.com", "competition_level": 0.52,
       "common_keywords": 9651380, "organic_traffic": 117649918}
    ],
    "competitors_total": 6007717,
    "trend_global_daily": [
      {"date": "20250507", "organic_traffic": 568288700, "organic_keywords": 99114924,
       "paid_traffic": 14773778, "branded_traffic": 207381624, "non_branded_traffic": 435456158}
    ],
    "summary_by_country_daily": [
      {"database": "us", "rank": 6, "organicTraffic": 568288700, "organicPositions": 99114924, "traffic": 642837782},
      {"database": "in", "rank": 167, "organicTraffic": 12054910, "organicPositions": 1413942, "traffic": 16008667}
    ]
  },

  "paid": {
    "top_ads": [
      {
        "keyword": "amazon",
        "title": "Amazon",
        "description": "Amazon® Official Site — Find deals and compare prices on amazon at Amazon.",
        "url": "https://www.amazon.com/",
        "position": 1,
        "volume": 151000000,
        "cpc_usd": 0.03
      }
    ],
    "competitors_total": 7988
  },

  "ai_search": {
    "ai_visibility": 67,
    "ai_visibility_benchmark": 66,
    "total_cited_pages": 7827618,
    "total_mentions": 8189191,
    "by_llm": [
      {"llm": "ChatGPT",     "mentions": 1388071, "cited_pages":  998414},
      {"llm": "AI Overview", "mentions": 1887053, "cited_pages": 2067040},
      {"llm": "AI Mode",     "mentions": 2851034, "cited_pages": 5640097},
      {"llm": "Gemini",      "mentions": 2063033, "cited_pages":   26249}
    ],
    "top_cited_sources": [
      {"domain": "youtube.com", "mentions": 1511374},
      {"domain": "reddit.com",  "mentions":  954001}
    ],
    "by_country": [
      {"country": "us", "visibility": 79, "mentions": 4872535},
      {"country": "uk", "visibility": 77, "mentions":  486610}
    ]
  },

  "moz": {
    "domain_authority": "100",
    "spam_score": "1%",
    "linking_root_domains": "5.5m",
    "ranking_keywords": "27.4m"
  }
}

The first 23 keys are the flat KPI block (what gets rendered as table columns in the Apify dataset view). Everything from is_root_domain downwards is the same data plus the nested detail (per-keyword arrays, history, per-country splits) for downstream pipelines.

---

What makes this scraper different

Feature	Public-page scrapers	This scraper
Authority Score and full backlink profile	Limited	Yes — full Domain Overview
Top keywords with CPC, KD, intent, SERP features	No	Yes — top 50 per domain
Paid ad copy (title + description + URL)	No	Yes — top 50 paid keywords
Organic + paid competitors with shared keywords	Partial	Yes — 20 each + total counts
AI Search per-LLM breakdown (ChatGPT / AIO / AIM / Gemini)	No	Yes
Top cited sources alongside the domain in AI answers	No	Yes
Daily + monthly traffic trends, country + global	3–6 months	Multi-year daily resolution
Top anchor breakdown + sample backlinks	No	Yes
Moz Domain Analysis (DA, Spam, top pages, ranking distribution, 60-day trend)	DA only	Yes — 10+ metrics
No Semrush subscription or API key needed	Varies	Yes
Async parallel processing (configurable, default 5)	Varies	Yes
Up to 100 domains per run	Varies	Yes

---

Frequently asked questions

How does this differ from running domains through the public Semrush homepage? The public preview shows only Authority Score and a clipped traffic estimate. This scraper returns the full authenticated Domain Overview — top 50 organic and paid keywords, full backlink profile, AI search per-LLM breakdown, competitors with shared-keyword counts, and multi-year daily trends.

Do I need a Semrush subscription or API key? No. The actor handles authentication and proxies internally — you only need an Apify account.

What does Keyword Research mode return? Set mode to keyword and paste keywords. Each keyword returns volume, CPC, Keyword Difficulty, competition, decoded search intent, a 12-month volume trend, the same keyword's volume across every country, related and question keywords, topic clusters, the ad copies running on it, and the live SERP (position, domain, URL). It's the full Semrush Keyword Overview as one JSON row.

Can I get Traffic Analytics (visits, bounce rate, time on site)? Not in this Actor. Semrush's Traffic & Market and Site Audit are paid-tier products that aren't exposed to the authenticated free-account pool this Actor uses, so they're intentionally out of scope. Domain Overview already includes organic, paid and SERP-feature traffic estimates.

How do I find which competitors are stealing my organic traffic? Put your domain in domains, set the appropriate database (e.g. us), and read organic.competitors. You get up to 20 organic competitors per domain with competition level, common keywords, organic traffic, and total traffic — plus organic.competitors_total for the full count.

Can I see exactly what ad copy a competitor is running? Yes. paid.top_ads returns the top 50 paid keywords with full ad title, description, and visible URL alongside position, volume, and CPC.

How is AI Search visibility measured? ai_search.by_llm returns mentions and cited pages per LLM — ChatGPT, AI Overview, AI Mode, and Gemini — plus a 0–100 aggregate AI Visibility score and the industry benchmark. ai_search.top_cited_sources lists the 20 domains most often cited alongside yours in AI answers, so you see who shares your AI-result shelf-space.

What's the difference between organic_traffic and total_traffic? organic_traffic is the classic estimate of monthly visits from organic positions only. total_traffic includes SERP features (featured snippets, People Also Ask, AI Overview, etc.) and matches the big number Semrush shows on the "Organic Search Traffic" plate.

Why are some organic_* numbers blank for a country DB? Some smaller Semrush regional databases don't have a full country summary; the actor falls back to the global trend point for those. Use worldwide if you want the aggregated view.

How many domains can I process at once? Up to 100 domains per run, processed in parallel up to your configured concurrency (default 5, max 50). Each parallel slot uses its own proxy IP — keep concurrency at 5 on Apify's free tier.

Should I enable Moz data? Only if you specifically need Moz DA, Spam Score, top pages, or ranking distribution. Moz adds 5–10 seconds per domain and routes through Residential proxy. Off by default.

---

Who is this for

• SEO specialists — Authority Score, full backlink profile, organic/paid keywords with CPC and intent — without paying for a Semrush seat
• Competitive analysts — Top competitors per domain with shared keywords, traffic, and traffic cost — organic and paid side by side
• AI/LLM researchers — Per-LLM mentions and cited pages across ChatGPT, AI Overview, AI Mode, Gemini — including which third-party domains co-rank in AI answers
• Performance marketers — Top paid ads with full ad copy and CPC for any competitor — feed into ad-research pipelines
• Agencies & data teams — Daily and monthly time-series for traffic, keywords, branded vs non-branded split, exportable as JSON / CSV / Excel from the Apify dataset

---

Pro tips

• Pick the right database — country-specific DBs (us, de, uk, …) give a real country summary; worldwide aggregates the global trend. Don't mix them in one analysis.
• Use summary_by_country_* — even in a single run you get a per-country breakdown across all Semrush regional databases. Great for "which country drives this domain's traffic" without re-running.
• Read intents on keywords — informational vs commercial vs transactional changes how you treat a competitor's ranking. The decoded intent saves you from inferring it.
• Use top_cited_sources for AI co-citation research — the domains most often cited alongside yours in AI answers are the natural targets for digital PR and link campaigns.
• Keep concurrency ≤ 5 on free tier — Apify's free Datacenter pool gives you 5 slots. Higher concurrency starves itself on IP rotation.
• Combine with the Top Websites mode — find the top 100 sites in your industry, then bulk-feed them back into Domain Overview mode to score them all.

---

Support

Having issues or need a custom feature? Open an issue in the Issues tab or contact: africanec@gmail.com

---

Tags

Semrush Scraper, Domain Overview, Authority Score, Backlinks Profile, Organic Keywords, Paid Keywords, Ad Copy Scraper, Competitor Analysis, AI Search Visibility, ChatGPT Mentions, AI Overview Mentions, AI Mode, Gemini Mentions, Moz Domain Authority, Moz Spam Score, Top Cited Sources, Bulk SEO, Bulk Domain Analysis, Parallel Scraper, Trending Websites, Market Research Tool, Competitive Intelligence, Daily Traffic Trends, SERP Features Distribution, Branded Traffic Split