Semrush Scraper - Authority Score, Traffic & Competitors

What does Semrush Scraper do?

Semrush Scraper reads Semrush's public website data for any list of domains: Authority Score, monthly visits and their change, pages per visit, time on site, bounce rate, desktop / mobile split, worldwide / country / industry rank, organic and paid search traffic, backlinks, referring domains, Google penalty risk, the top 5 countries, the top 5 organic keywords (position, volume, CPC, intent), competitors, traffic sources and destinations, AI-assistant traffic and 3-6 months of history. It also returns competitor rows (every website on a domain's Semrush competitors page, with its own metrics) and Semrush's top-websites rankings (the 100 most visited websites of 240 countries x 111 industries). One row per website, competitor or ranking entry. SEO agencies, investors, lead-scoring and market-research teams use it as a Semrush API alternative.

• 🏅 Authority: Semrush Authority Score, backlinks and referring domains with the month's change, Google penalty risk.
• 🚦 Traffic: exact monthly visits (not "232.54M"), engagement, desktop / mobile, organic and paid search traffic.
• 🌍 Countries and keywords: the 5 countries and 5 organic keywords that bring the most traffic.
• 🥊 Competitors: the closest competitors on every row; competitor rows add each competitor's Authority Score, visits, ranks and engagement.
• 🔀 Traffic journey and AI traffic: where visitors come from and go next, and how much ChatGPT, Claude, Grok and other AI sites send.
• 🏆 Top websites by country and industry, with exact visits, month-over-month and year-over-year change.
• 🔑 No Semrush account, no API key, no subscription; domains Semrush does not publish are listed and never charged.
• 🔔 Monitor mode, schedules, API, MCP, Make / Zapier / n8n.

What data can you get from Semrush?

🏅 Authority Score authority_score	🔗 Backlinks backlinks	🌐 Referring domains referring_domains
🚦 Monthly visits visits	🔎 Organic traffic organic_traffic	💵 Paid traffic paid_traffic
🌍 World rank global_rank	🗺️ Country rank country_rank	📂 Industry rank category_rank
⏱️ Time on site time_on_site_seconds	📄 Pages per visit pages_per_visit	↩️ Bounce rate bounce_rate_percent
📱 Desktop / mobile desktop_visits, mobile_visits	🌍 Top 5 countries top_countries	🔑 Top 5 keywords top_keywords
🥊 Competitors competitors	🔀 Traffic sources traffic_sources	🤖 AI traffic ai_traffic
📅 History search_traffic_history, backlinks_history	⚠️ Penalty risk google_penalty_risk	🏷️ Brand, industries brand_name, categories
🧩 Competitor rows competitor_of, position	🏆 Top-websites rows rank, visits_change_yoy_percent	🔔 Changes since last run change_since_last_run

How to scrape Semrush with Semrush Scraper

1. Create a free Apify account (it comes with free monthly credit).
2. Open Semrush Scraper and type your Domains or URLs, one per line (stripe.com).
3. Tick Competitor rows for each domain's competitors, or pick Top websites: countries and industries. All of them can run together.
4. Set Max rows, and the filters if you want them.
5. Click Start, then download the rows as JSON, CSV or Excel, or read them through the API.

Input

Parameter	Type	Description
🌐 Websites
domains	array	Domains, website URLs or Semrush pages (semrush.com/website/stripe.com/overview/, .../competitors/, semrush.com/trending-websites/us/finance, any language host). www., case and paths are normalised.
urlsFromFile	string	A pasted list (text, CSV, JSON) or a public .csv / .json / .txt link, up to 2 MiB.
allTrackedDomains	boolean	Every website Semrush publishes a page for (about 250,000, from its sitemaps), up to Max rows. Default false.
domainSuffixes, excludeKeywords	array	Keep only .de, .co.uk...; skip domains containing a word. Free, before any page is read.
includeCompetitors	boolean	One competitor row per website on each domain's competitors page (charged per row). The website row also gets its rank changes in places and Semrush's description. Default false.
competitorsOnly	boolean	Only the competitor rows, no website row. Default false.
maxCompetitorsPerDomain	integer	Competitor rows per domain (Semrush lists 1-10). Default 20 (all).
includeHistory	boolean	Visits (3 months), device, search traffic, backlinks, referring domains and AI sources (6 months each). Default true, free.
includeTopCountries, includeTopKeywords, includeTrafficSources, includeAiTraffic	boolean	Leave a section out of the rows. Default true, free.
🏆 Top websites
locations	array	global (World), us, gb, de... (240 countries). Empty = no rankings.
topWebsitesCategories	array	all, finance, automotive, online-services... (111 industries).
topWebsitesLimit	integer	Rows per ranking, up to 100. Default 100.
enrichTopWebsites	boolean	Add each ranked website's full website data to its row (one top-website + one website event).
🔎 Filters (before the charge)
minAuthorityScore, maxAuthorityScore, minVisits, maxVisits, minOrganicTraffic, onlyCategories, onlyCountries		Rows that miss them are dropped free.
🔔 Monitoring
monitorMode, monitorOutput, monitorStoreName		Only new or changed rows since the last run with the same name.
⚙️ Run options
maxItems	integer	Rows for the whole run, all kinds together (0 = no limit). Default 300.
maxConcurrency	integer	Pages in parallel. Default 8.
proxyConfiguration	object	Optional; by default the actor reads directly and climbs to Apify datacenter, then residential, only when a route is refused.

Rival field names work too: urls, targets, websites, startUrls, queries, country_top_websites, industry_top_websites, topWebsitesCountry, topWebsitesIndustry, rankingCountry, rankingIndustry, includeCompetitorDetails, competitorsPerDomain, includeMonthlyHistory, onlyWithTrafficEstimate, filterCountries, filterCategories, onlyNew, onlyChanged, stateKey, maxResults.

Input JSON

{
    "domains": ["stripe.com", "github.com", "https://www.notion.so/product"],
    "includeCompetitors": true,
    "locations": ["us", "de"],
    "topWebsitesCategories": ["finance"],
    "topWebsitesLimit": 100
}

Give the bare domain, not a page

Semrush keeps one page per registrable domain. https://www.stripe.com/pricing becomes stripe.com; a subdomain without its own page (docs.github.com) falls back once to github.com and the row says resolved_from.

Big lists: give the run more memory

At the default 256 MB a run reads about 2.4 websites a second (300 websites in 124 s). At 1,024 MB the same 300 took 32 s and cost the same ($0.0034 of platform usage either way), because the work is CPU-bound and CPU grows with memory. For 1,000+ domains, set Memory to 1,024 MB in the run options.

Output

A real website row (stripe.com with includeCompetitors, run 2026-10-28; arrays cut to their first entry):

{
  "type": "domain_overview",
  "domain": "stripe.com",
  "url": "https://www.semrush.com/website/stripe.com/overview/",
  "main_country": "US",
  "categories": ["Computer Software and Development", "Finance", "Information Technology", "Online Services"],
  "data_month": "2026-09",
  "authority_score": 79,
  "global_rank": 145,
  "global_rank_trend": "up",
  "global_rank_change_places": 12,
  "country_rank": 145,
  "country_rank_country": "US",
  "category_rank": 3,
  "category_rank_name": "Finance",
  "visits": 232542245,
  "visits_change_percent": 11.79,
  "pages_per_visit": 2.812,
  "time_on_site_seconds": 498,
  "bounce_rate_percent": 61.53,
  "desktop_visits": 125511872,
  "mobile_visits": 107030373,
  "organic_traffic": 4963905,
  "organic_traffic_change_percent": -11.1,
  "paid_traffic": 192150,
  "total_search_traffic": 5156055,
  "backlinks": 141005083,
  "backlinks_change_percent": -3.87,
  "referring_domains": 732469,
  "google_penalty_risk": "Low",
  "top_country": "US",
  "top_country_traffic": 59382926,
  "top_countries": [{ "country": "US", "country_name": "United States", "visits": 59382926, "share_percent": 25.54, "desktop_share_percent": 37.37, "mobile_share_percent": 62.63 }],
  "keywords_database": "US",
  "top_keywords": [{ "keyword": "stripe login", "position": 1, "search_volume": 135000, "cpc": 14.46, "traffic_percent": 7.08, "intents": [] }],
  "competitors": [{ "domain": "gocardless.com", "relevance": 0.14, "visits": 3032397 }],
  "traffic_sources": [{ "source": "Direct", "source_type": "Direct", "visits": 54586835, "previous_visits": 46022598, "share_percent": 43.49 }],
  "ai_traffic": [{ "source": "chatgpt.com", "name": "ChatGPT", "visits": 1268580, "share_percent": 0.7 }],
  "search_traffic_history": [{ "month": "2026-09", "organic": 4963905, "paid": 192150, "total": 5156055 }],
  "competitor_count": 3
}

A competitor row and a top-websites row (same run):

{ "type": "competitor", "competitor_of": "stripe.com", "position": 1, "domain": "gocardless.com", "relevance": 0.14,
  "authority_score": 58, "visits": 3032397, "visits_change_percent": 16.98, "global_rank": 19485, "global_rank_change_places": 1656,
  "country_rank": 954, "country_rank_country": "GB", "category_rank_name": "Accounting and Auditing",
  "bounce_rate_percent": 44.72, "pages_per_visit": 4.1327, "time_on_site_seconds": 303, "data_month": "2026-09" }
{ "type": "top_website", "ranking_country_name": "World", "ranking_industry_name": "All Industries", "rank": 1, "domain": "google.com",
  "visits": 108899227055, "desktop_share_percent": 35.85, "mobile_share_percent": 64.15,
  "visits_change_mom_percent": 3, "visits_change_yoy_percent": 1.62, "top_traffic_source": "direct", "data_month": "2026-09" }

Fields

Fill measured on 2026-10-28: 75 websites from a 99-domain buyer list (24 had no public Semrush page), 1,000 websites from Semrush's own list (mostly small sites), 162 competitor rows of 30 domains, 1,000 top-websites rows of 10 countries:

Field	Buyer list	Small sites	Meaning
authority_score, backlinks, referring_domains, google_penalty_risk	100%	99.6%	authority and links
visits, engagement, top_countries, global_rank	100%	86.5%	traffic and ranks
organic_traffic, paid_traffic, search_traffic_history	100%	65.2%	search traffic
top_keywords	100%	45.5%	5 keywords
competitors	94.7%	37.8%	closest competitors
categories, category_rank	94.7%, 80%	39.1%, 0.9%	industries
ai_traffic	97.3%	22.9%	AI sources
brand_name	62.7%	0.1%	only well-known brands
description, *_rank_change_places (with includeCompetitors)	100% on 30 of 30		from the competitors page
competitor rows: Authority Score, visits, ranks, engagement / relevance	100% / 54.9%		relevance only for the competitors the website page shows
top-websites rows: every field / visits_change_yoy_percent	100% / 96.8%

type tells the three row kinds apart; columns that do not apply to a row are absent or null. Percent fields are percent (61.53 = 61.53%).

Use cases

• 🧲 Lead scoring: add visits, Authority Score, industry and main country to a CRM list of company domains.
• 📊 Competitor benchmarking: line up competitors' traffic, engagement, ranks and backlinks month by month.
• 🔗 Link building: score outreach lists by Authority Score, referring domains and penalty risk.
• 💼 Domain and website due diligence: traffic, channels and history before you buy or invest.
• 🗺️ Market maps: the 100 most visited websites of a country and industry, with full data per site.
• 🤖 AI traffic reporting: see how much ChatGPT, Claude and other AI sites send to each website.

How much does it cost to scrape Semrush?

Pay per row, no start fee:

Event	Price
Website row	$0.0009 ($0.90 per 1,000)
Top-websites row	$0.0009
Competitor row	$0.0013

Run	Rows	Price
100 websites	100	$0.09
30 websites + their competitors	30 + 162	$0.24
Top 100 of 10 countries	1,000	$0.90
1,000 websites	1,000	$0.90

A domain without a public Semrush page, a filtered row and an unchanged monitor row cost nothing. Cap any run with Maximum cost per run; it stops at the cap and says so.

Integrations, API, scheduling & MCP

• Schedule it monthly (Semrush refreshes these pages once a month) with Monitor mode on, and only changed websites come back, with change_since_last_run.
• Webhooks send the finished dataset to your app; connect Make, Zapier, n8n, Google Sheets or Slack from the Integrations tab.
• API: run it from Python or JavaScript with apify-client, or call the REST endpoint.
• MCP: add it to Claude, Cursor or any MCP client through the Apify MCP server.

More SEO and website scrapers

• Ahrefs Scraper - Domain Rating, organic traffic, top websites and AI visibility from Ahrefs.
• SimilarWeb Scraper - total website traffic, rank and competitors from SimilarWeb.
• Website Technology Scraper - tech stack, CMS and analytics of a website.

More from TopApi.

Limits & empty runs

• Coverage: Semrush publishes a public page for about 250,000 websites. On a 99-domain buyer list 75 had one; some well-known sites have none (zapier.com, mazda.de on 2026-10-28). Those come back in the run summary, free.
• Not in the rows: keyword research for your own keywords (volume, keyword difficulty, ideas, SERP features), full organic keyword and backlink lists, keyword counts, traffic cost and AI visibility scores. Semrush shows those only to signed-in users or behind a reCAPTCHA check on its free tools.
• Rank changes in places are on competitor rows and on website rows with includeCompetitors; every website row has *_rank_trend (up / down / same).
• Estimates: every figure is Semrush's monthly estimate for data_month; Semrush updates the pages once a month.
• Empty runs say why in the status message and in RUN_SUMMARY (not published, filtered out, unchanged since the last run, or not reachable). If Semrush refuses every request on every route, the run ends FAILED and nothing is charged.