Semrush Scraper

Turn any domain into structured Semrush website-overview data — Authority Score, traffic estimates, global/country/category ranks, backlinks, referring domains, top competitors, top organic keywords, and even AI/LLM traffic. Paste one domain or a thousand; each returns a single clean, export-ready row.

Two modes in one Actor: Domain overview (look up any list of domains) and Top websites (the ranked leaders for a country + industry, with traffic and month/year growth).

No Semrush seat, no API key, no login. This Actor reads Semrush's own public website-overview pages and hands you the numbers as JSON, CSV, or Excel.

How it works

✨ Why use this scraper?

• Full domain overview in one row — Authority Score, traffic, ranks, backlinks, engagement, competitors, keywords and AI traffic, flattened into table-friendly columns plus nested detail.
• Bulk by design — feed 1 or 1,000 domains in a single run; results stream in parallel, one row per domain.
• No Semrush account needed — reads only public website-overview data, so there is no seat cost, API quota, or login to manage.
• AI-search visibility included — see how much traffic ChatGPT, Gemini, Grok and other LLMs send to a domain.
• Flexible input — bare domains, full site URLs, or Semrush overview URLs all work; www. and paths are stripped automatically.
• Export anywhere — JSON, CSV, Excel, XML or the Apify API, straight out of the dataset.

🎯 Use cases

Audience	What they get
SEO specialists & agencies	Bulk Authority Score, traffic and backlink profiles for a whole client roster without buying extra Semrush seats.
Competitive analysts	Top organic + paid competitors per domain, with estimated visits and relevance, to map a market.
Content & keyword teams	Top organic keywords with search volume, CPC, position and decoded intent for any domain.
AI / LLM researchers	Per-LLM traffic share (ChatGPT, Gemini, Grok…) to track which brands AI assistants send traffic to.
Growth & BD teams	Traffic, rank and engagement signals to qualify prospects, partners or acquisition targets at scale.
Data teams & dashboards	Monthly history arrays and per-country/per-device splits to feed BI tools and trend charts.

📥 Supported inputs

Mode 1 — Domain overview (mode: "domains"): supply a list of domains.

Input form	Example	Result
Bare domain	stripe.com	Overview for stripe.com
Full site URL	https://stripe.com/pricing	Host extracted → stripe.com
Subdomain / path	blog.hubspot.com/marketing	Host extracted → blog.hubspot.com
Semrush overview URL	https://www.semrush.com/website/stripe.com/overview/	Domain parsed out → stripe.com

Mode 2 — Top websites (mode: "top_websites"): supply a country + industry and get the ranked leaders.

Field	Example	Result
Country	us or united-states	Ranking scoped to that country
Industry	finance, retail, all	Ranking scoped to that industry
Limit	100	Top N domains (max 100)
Enrich	true	Each ranked domain also gets a full overview scrape

Not supported: keyword-only research (volume/CPC for a bare keyword with no domain), position-tracking projects, or any data that Semrush gates behind a logged-in seat. Domains with no Semrush coverage return no row (they are logged and skipped, never billed).

🔄 How it works

1. Normalise each input into a clean registrable domain (strip scheme, www., path, query; dedupe).
2. Fetch the domain's public Semrush overview page — a single HTTP request per domain, run in parallel with a global sliding window.
3. Decode the metrics embedded in the page's rendered state.
4. Flatten the headline KPIs into table columns and keep the rich sections (history, per-country, per-device, journey, competitors, keywords, AI traffic) as nested fields.
5. Push exactly one row per domain to the dataset. A domain Semrush has no data for is skipped with a clear log line.

⚙️ Input parameters

Field	Type	Default	Description
mode	string	domains	domains (look up domains) or top_websites (ranking by country + industry).
domains	array	["stripe.com", …]	(domains mode) Domains or URLs to analyse. One row per unique domain.
topWebsitesCountry	string	us	(top websites) 2-letter code (us, gb) or full slug (united-states).
topWebsitesIndustry	string	all	(top websites) Industry slug (all, finance, retail, news…).
topWebsitesLimit	integer	100	(top websites) How many ranked domains to return (1–100).
enrichTopWebsites	boolean	false	(top websites) Also run a full overview scrape per ranked domain.
includeHistory	boolean	true	Month-by-month history for visits, organic/paid, backlinks, referring domains.
includeTrafficByCountry	boolean	true	Per-country traffic with device split and share.
includeTrafficByDevice	boolean	true	Desktop vs mobile visit history.
includeTrafficJourney	boolean	true	Inbound / outbound referral hosts.
includeCompetitors	boolean	true	Top organic + paid competitor domains.
includeKeywords	boolean	true	Top organic keywords (volume, CPC, position, intent).
includeAiTraffic	boolean	true	AI / LLM traffic sources and share.
includeSiteAudit	boolean	false	On-page SEO audit block (usually empty unless already crawled).
maxItems	integer	1000	Hard cap on domains processed per run.
maxConcurrency	integer	10	Domains fetched in parallel.
proxy	object	Apify Residential	Proxy used to reach Semrush. Residential recommended.

The include* toggles trim the output only — every section arrives in the same single request, so turning them off does not change the price. Use them to keep rows lean when you only need headline KPIs.

📊 Output overview

Each dataset row is one domain. The top of the row is a flat block of headline KPIs (Authority Score, ranks, visits, organic/paid traffic, engagement, backlinks, device split, top competitor, AI traffic share) that render as plain columns in the Apify table view. Below that sit the nested detail sections — monthly history, per-country and per-device breakdowns, the traffic journey, the full competitor list, top keywords, and AI-traffic sources — each gated by its include* toggle.

📦 Output sample

{
  "type": "domain_overview",
  "domain": "stripe.com",
  "url": "https://www.semrush.com/website/stripe.com/overview/",
  "dataMonth": "2026-05-15",
  "dateModified": "2026-06-12",
  "country": "US",
  "categories": ["Computer Software and Development", "Finance", "Information Technology", "Online Services"],
  "authorityScore": 79,
  "googlePenaltyRisk": "Low",
  "globalRank": 167,
  "countryRank": 165,
  "categoryRank": 6,
  "categoryRankName": "Finance",
  "visits": 190046716,
  "visitsChange": 0.1381,
  "organicTraffic": 5485936,
  "paidTraffic": 311438,
  "bounceRate": 0.6053,
  "pagesPerVisit": 2.8708,
  "avgVisitDurationSec": 508,
  "backlinks": 181169732,
  "referringDomains": 609490,
  "desktopShare": 0.4972,
  "mobileShare": 0.5028,
  "topCountry": "United States",
  "topCountryTraffic": 53060565,
  "competitorsCount": 3,
  "topCompetitor": "gocardless.com",
  "aiTrafficShare": 0.0608,
  "competitors": [
    { "name": "gocardless.com", "relevance": 0.16, "visits": 2688810 }
  ],
  "topKeywords": [
    { "keyword": "stripe", "position": 1, "searchVolume": 550000, "cpc": 5.18, "traffic": 0.2294 }
  ],
  "trafficByCountry": [
    { "countryName": "United States", "traffic": 53060565, "trafficShare": 0.2792, "desktopShare": 0.4296, "mobileShare": 0.5704 }
  ],
  "trafficAi": [
    { "id": "grok.com", "name": null, "traffic": { "value": 5518189, "share": 0.03663 } }
  ],
  "scrapedAt": "2026-07-10T22:44:21.115Z"
}

🗂 Key output fields

Field	Type	Description
Identity
domain	string	Registrable domain the row describes.
country	string	Primary country for the traffic data.
categories	string[]	Semrush business categories for the domain.
dataMonth / dateModified	string	Reference month of the data / date Semrush last updated it.
Authority & rank
authorityScore	number	Semrush Authority Score (0–100).
googlePenaltyRisk	string	Backlink penalty-risk label (e.g. Low).
globalRank / countryRank / categoryRank	number	Semrush traffic ranks.
categoryRankName	string	The category the categoryRank applies to.
Traffic & engagement
visits / visitsChange	number	Monthly visits and month-over-month change.
organicTraffic / paidTraffic	number	Estimated organic and paid search traffic.
bounceRate / pagesPerVisit / avgVisitDurationSec	number	Engagement metrics.
desktopShare / mobileShare	number	Device split of visits.
topCountry / topCountryTraffic	string / number	Largest traffic country and its volume.
Backlinks
backlinks / referringDomains	number	Total backlinks and unique referring domains.
Competition & keywords
competitorsCount / topCompetitor	number / string	Competitor count and the closest one.
competitors[]	object[]	Competitor domains with relevance + estimated visits.
topKeywords[]	object[]	Top organic keywords: volume, CPC, position, intent, traffic share.
AI traffic
aiTrafficShare	number	Combined share of traffic from AI/LLM sources.
trafficAi[]	object[]	Per-LLM traffic (ChatGPT, Gemini, Grok…) with share and history.
History & breakdowns
visitsHistory / searchTrafficHistory / backlinksHistory / referringDomainsHistory	object[]	Monthly time series.
trafficByCountry[]	object[]	Per-country traffic with device split.
trafficByDeviceHistory[]	object[]	Desktop vs mobile visits over time.
trafficJourney	object	Inbound and outbound referral hosts.
scrapedAt	string	ISO timestamp of the scrape.

🏆 Top Websites output

In top_websites mode each row is one ranked domain. With enrichTopWebsites off (default), rows are lightweight ranking records; with it on, each row is a full domain-overview (as above) plus rank, industry and rankingCountry.

{
  "type": "top_website",
  "rank": 1,
  "domain": "yahoo.com",
  "country": "us",
  "industry": "finance",
  "dataMonth": "2026-05-01",
  "totalTraffic": 1108419488,
  "desktopTraffic": 458469356,
  "desktopShare": 0.4136,
  "mobileTraffic": 649950132,
  "mobileShare": 0.5864,
  "momChange": 0.0654,
  "yoyChange": -0.1345,
  "topSource": "direct"
}

Field	Type	Description
rank	number	Position in the country × industry ranking (1 = highest traffic).
domain	string	The ranked domain.
totalTraffic	number	Estimated monthly visits.
desktopTraffic / mobileTraffic	number	Device-split visit estimates.
desktopShare / mobileShare	number	Device split as a 0–1 fraction.
momChange / yoyChange	number	Month-over-month and year-over-year change (0–1 fraction, can be negative).
topSource	string	Largest traffic source (e.g. direct, search).

❓ FAQ

Do I need a Semrush subscription or API key? No. This Actor reads Semrush's public website-overview pages, so there is nothing to log into and no API quota to manage.

How accurate are the numbers? They are Semrush's own publicly published estimates for each domain — the same figures shown on the public overview page. They are modelled traffic estimates, not server-side analytics, and update roughly monthly (dataMonth tells you the reference period).

What happens for a domain Semrush doesn't cover? Semrush returns a 404 for domains it has no data on. The Actor detects this immediately, logs it as "not found — no coverage", and moves on. No row is pushed and you are not billed for it.

Can I get keyword volume for a keyword that isn't tied to a domain? Not in this Actor — standalone keyword research is gated behind a Semrush login. You do get each domain's top organic keywords (with volume, CPC, position and intent) as part of its overview row.

Why is competitors sometimes empty? Semrush controls whether the competitors block is exposed on the public overview page, and it toggles this per data snapshot — some months it's fully populated, some months it's withheld (the page returns it hidden). When Semrush exposes it you get the full competitor list; when it doesn't, competitors is an empty array. Every other section (traffic, ranks, backlinks, keywords, AI traffic, per-country/device) is unaffected.

How do I get the top websites in my industry? Set mode to top_websites, pick a topWebsitesCountry (e.g. us) and topWebsitesIndustry (e.g. finance, or all). You get up to 100 ranked domains with traffic, device split and month/year growth in a single request. Turn on enrichTopWebsites to also pull a full overview for each ranked domain.

Why is siteAudit empty? The on-page SEO audit only has data when Semrush has already crawled the domain for a project. It is off by default; enable includeSiteAudit if you specifically want that block.

💬 Support

Found a domain that parses oddly or a field you'd like added? Open an issue on the Actor's Issues tab in the Apify Console and include the exact input you ran. Feature requests for extra fields are welcome.

🛠 Additional services

Need this wired into a pipeline, scheduled, or combined with other data sources? The same team builds custom scrapers and enrichment flows across job boards, marketplaces, real-estate, and SEO/analytics platforms — reach out via the Apify Store profile.

🔎 Explore more scrapers

Looking at the wider SEO/analytics stack? Compare domains across tools by pairing this with Similarweb, Ahrefs, Moz and Ubersuggest scrapers for a triangulated view of authority and traffic.

---

⚠️ Disclaimer

This Actor is an independent tool and is not affiliated with, endorsed by, or sponsored by Semrush Inc. or any of its subsidiaries or affiliates. "Semrush" and all related trademarks are the property of their respective owners.

The scraper accesses only publicly available Semrush website-overview pages — no authenticated endpoints, no paid-seat features, and no content behind a Semrush login. All metrics are Semrush's own publicly published estimates. Users are responsible for ensuring their use complies with Semrush's Terms of Service, applicable data-protection law (GDPR, CCPA, etc.), and any contractual obligations of their own organisation.

---

SEO Keywords

semrush scraper, scrape semrush, semrush api, semrush data export, apify semrush, website traffic scraper, domain authority score scraper, seo data scraper, backlink data scraper, referring domains scraper, competitor analysis scraper, organic keywords scraper, keyword volume cpc scraper, traffic analytics scraper, ai traffic scraper, chatgpt traffic scraper, domain overview api, bulk domain seo data, seo rank tracker data, top websites ranking scraper, trending websites scraper, top sites by industry, semrush alternative scraper, similarweb alternative scraper, marketing intelligence data