Semrush Scraper

Scrape Semrush public domain overviews and top website rankings into a structured dataset. Export Semrush data to CSV, JSON, or Excel for traffic benchmarking, backlink checks, competitor research, and ranking reports. This Actor offers a practical Semrush API alternative for the metrics visible on public website pages; it does not require a site account.

A domain overview returns one record per domain. A top websites run returns one record per ranked website in a country and industry. Choose both to collect overview records first and then add the selected ranking table to the same dataset. Every record includes recordType, domain, sourceUrl, and dataDate, so mixed exports remain easy to filter.

What data can you extract from Semrush?

The domain overview contains public traffic, search, backlink, engagement, and competitor samples. Ranking records contain visits, device share, growth, and the leading traffic source. The site publishes estimates; treat the numbers as directional measures rather than first-party analytics.

Field	Description	Example
recordType	Identifies a domain or ranking row.	domain
domain	Normalized website domain.	archive.org
sourceUrl	Public source page.	https://www.semrush.com/website/archive.org/overview/
dataDate	Date attached to the source metrics.	2026-08-15
updatedAt	Date the overview was updated.	2026-09-18
brandName	Public website label.	Internet Archive
country	Country attached to the overview summary.	US
categories	Public category labels.	Online Services
globalRank	Worldwide rank for a domain.	147
countryRank	Rank within the overview country.	136
categoryRank	Rank within a category, when shown.	6
authorityScore	Public 0–100 authority score.	91
backlinks	Estimated number of backlinks.	1887667482
referringDomains	Estimated distinct referring domains.	1269407
googlePenaltyRisk	Public risk label.	Low
organicTraffic	Estimated organic search traffic.	22553842
paidTraffic	Estimated paid search traffic.	0
totalSearchTraffic	Organic plus paid search traffic.	22553842
monthlyVisits	Estimated visits for the latest available month.	219290927
monthlyVisitsChangePercent	Change in overview visits, in percent.	-1.72
desktopVisits	Estimated desktop visits.	102636143
mobileVisits	Estimated mobile visits.	116654784
pagesPerVisit	Mean pages viewed per visit.	7.4943
visitDurationSeconds	Mean visit duration in seconds.	622
bounceRatePercent	Share of visits that bounce, in percent.	47.7
topCountry	First country in the public traffic table.	US
topCountryTraffic	Visits estimated for that country.	61576421
topOrganicKeywords	Public sample of organic keywords, positions, search volume, CPC, and traffic share.	Array of keyword objects
organicCompetitors	Public sample of competing domains, relevance, and visits.	Array of domain objects
trafficByCountry	Country-level visit and device mix estimates.	Array of country objects
searchTrafficHistory	Historical organic and paid search traffic.	Array of dated points
visitHistory	Historical monthly visits.	Array of dated points
backlinksHistory	Historical backlink totals.	Array of dated points
referringDomainsHistory	Historical referring domain totals.	Array of dated points
assistantReferrals	Public referral source estimates and histories.	Array of source objects
rank	Position in the chosen top websites table.	1
rankingCountry	Country slug used for the ranking.	global
rankingIndustry	Industry slug used for the ranking.	all
desktopSharePercent	Percentage of ranking visits from desktop.	31.22
mobileSharePercent	Percentage of ranking visits from mobile.	68.78
momChangePercent	Month-over-month visit change in percent.	-0.12
yoyChangePercent	Year-over-year visit change in percent.	0.71
topTrafficSource	Largest traffic source category.	direct

Fields belong to their respective record types. A domain row has overview fields and public sample arrays; a ranking row has rank, ranking filters, device shares, growth measures, and traffic source. Common fields make it possible to combine both in one export without confusing the two measurements. A blank value means the public page omitted that measure. The Actor never fills missing fields with guessed values.

How to scrape Semrush

1. Set mode to domain, topWebsites, or both.
2. For domain overviews, paste one or more website domains or full URLs into domains. The Actor uses each URL's host and removes a leading www..
3. For top websites, choose rankingCountry and rankingIndustry. Use the slugs in public ranking page URLs, such as global and all.
4. Set maxItems if you need a strict output cap, then start the Actor. The dataset is available in JSON, CSV, and spreadsheet formats when the run finishes.

The both mode processes the requested domains before the ranking table. Set maxItems above the number of requested domains if you want ranking rows as well. If your cap is reached during the domain phase, the ranking page is not requested.

Input

Name	Type	Default	Description
mode	String	domain	Select domain overviews, top websites, or both.
domains	String array	Example prefill	Domains or URLs for domain and both. Maximum 100 entries. Duplicate normalized domains are collected once.
rankingCountry	String	global	Country slug used in the public top websites URL.
rankingIndustry	String	all	Industry slug used in the public top websites URL.
maxItems	Integer	200	Maximum total records. Set 0 for no Actor cap.
proxyConfiguration	Object	Default proxy	Optional proxy configuration for public page requests.

A realistic input for a combined report is:

{
  "mode": "both",
  "domains": ["wikipedia.org", "archive.org"],
  "rankingCountry": "global",
  "rankingIndustry": "all",
  "maxItems": 200
}

For a ranking-only run, set mode to topWebsites and leave domains empty. For a domain-only run, the ranking filters have no effect. Enter an industry or country slug exactly as it appears in a public ranking URL; the Actor checks that the page returned the requested filters and stops if it did not.

Output example

These selected fields come from the archive.org domain record in a successful 94-record golden run on September 27, 2026. The full item also includes keyword, competitor, country, history, and referral arrays.

{
  "recordType": "domain",
  "domain": "archive.org",
  "sourceUrl": "https://www.semrush.com/website/archive.org/overview/",
  "dataDate": "2026-08-15",
  "updatedAt": "2026-09-18",
  "brandName": "Internet Archive",
  "country": "US",
  "categories": ["Online Services"],
  "globalRank": 147,
  "countryRank": 136,
  "authorityScore": 91,
  "backlinks": 1887667482,
  "referringDomains": 1269407,
  "organicTraffic": 22553842,
  "paidTraffic": 0,
  "totalSearchTraffic": 22553842,
  "monthlyVisits": 219290927,
  "desktopVisits": 102636143,
  "mobileVisits": 116654784,
  "pagesPerVisit": 7.4943,
  "visitDurationSeconds": 622,
  "topCountry": "US",
  "topCountryTraffic": 61576421
}

The same run returned two domain records and 92 ranking records for global/all. The count is the number of entries exposed on that public page at run time; it is not a fixed promise of 100 records. Ranking pages can expose a different number of rows for another country or industry.

Use cases

• Competitive benchmarking: Compare monthly visits, organic search traffic, paid search traffic, authority, and referring domains for a short list of sites. Keep source dates beside each metric so reports do not mix periods silently.
• Prospect review: Screen domains for traffic scale, backlink profile, and engagement before a manual review. The public overview offers a quick first pass when a full private research account is unnecessary.
• Market landscape reports: Export the top websites for a country and industry, then sort by month-over-month or year-over-year visit change to find growing sites.
• Channel planning: Compare the leading traffic source in rankings with the search traffic estimates on a domain overview. The measures come from different public sections, so retain recordType and dataDate when joining them.
• Periodic snapshots: Schedule repeated runs to capture changes in public authority, visits, traffic mix, and rankings. Store each run's date because the site's underlying month may update on a different schedule from your run.
• Content research: Inspect the small public keyword and competitor samples attached to domain overviews. These samples identify topics worth checking manually; they are not complete keyword exports.

How much does it cost to scrape Semrush?

The current result event price is $0.001 per record, or $1 per 1,000 records. The charge follows delivered dataset items, and the Actor's platform compute is included in that event price for the user. A combined run that returns two domain rows and 92 ranking rows delivers 94 records, so its result charge is $0.094. A batch of 50 successful domain overviews would cost $0.05 in result events. Ten monthly ranking snapshots with 90 rows each would deliver 900 records and cost $0.90 in result events.

The number of public ranking entries varies. A domain that has no public overview does not produce a result event. maxItems prevents additional result events after the chosen cap. Account-level billing and the current event price should be checked on the Actor's pricing panel before a larger run. The examples above describe result events and do not promise a particular site response or a fixed ranking size.

Tips for best results

Use a plain domain for overviews, such as wikipedia.org. Full URLs also work; the path and query are removed because the overview describes the domain. Keep domain batches focused if you need to investigate a failed site individually. Invalid domains are rejected before requests begin, and repeated domains in the same input are deduplicated.

For rankings, copy the country and industry slugs from a public top websites page. global/all is a broad starting point. An industry table in a specific country can be more useful when the goal is to compare direct peers. The Actor reads the table's embedded state from one page; there is no page-by-page pagination of deeper rankings. Use separate runs for separate country and industry combinations.

Use both when you want a compact benchmark: collect several domain overviews, then add one ranking table. Filter the exported dataset by recordType before comparing columns. monthlyVisits appears in both row types, but domain and ranking pages may have different dates or estimation methods. Preserve dataDate and sourceUrl in downstream reports.

Limits

The Actor uses publicly available website pages. A public domain overview shows a limited keyword sample and a limited competitor sample; it does not expose the full private research dataset. The tested overview showed five organic keywords and three competitors for each sampled domain. Public backlinks are aggregate counts and short histories, not a list of every linking page. The public page also does not provide the leader's paid keyword list, ad copy, full keyword difficulty research, or complete country-specific domain dashboard.

Anonymous calls to the keyword research endpoint returned an operation-forbidden response during development. Therefore this Actor has no keyword research mode and does not turn an input keyword into volume, difficulty, related questions, or a search results page. The public overview's keyword sample is tied to a domain. An on-page audit of arbitrary third-party URLs is also outside this Actor's scope.

The public domain overview's country value is chosen by the source. Adding a country query to the tested overview URL did not change its country. rankingCountry filters only top websites. Some sites have no public overview and return a missing page. Other pages can temporarily fail or change their embedded data format. The Actor logs a domain failure and continues with the next domain. A run with no records fails rather than returning an apparently successful empty export.

Rankings are limited to the entries embedded in the selected public page. In the tested worldwide table there were 92 entries; a tested country-and-industry table had 99. Ranking positions are assigned in the displayed order. Estimates and historical series can change as the source refreshes data. Proxy retries handle access interruptions where possible, but they cannot provide data that the public page does not expose.

Use with AI agents (MCP)

The input schema names each parameter and explains which modes use it. An agent can create a run, wait for completion, and read the default dataset. Two example prompts:

Collect public domain overview metrics for wikipedia.org and archive.org. Return monthly visits, organic traffic, authority score, referring domains, and the data date.

Get the global top websites across all industries. Show the ten rows with the largest positive month-over-month visit change, including rank, visits, and source date.

Ask for recordType and dataDate in mixed-mode summaries. That keeps domain overviews separate from ranking rows and makes estimates traceable to the public source month.

FAQ

Is the data public? Yes. The Actor reads pages accessible without a site login. Follow the site's terms and applicable law, and use exported data responsibly.

Do I need a site subscription or key? No subscription or key is required for the public modes documented here. Private keyword research and full dashboard data are not included.

Do I need to configure proxies? The default configuration uses a proxy automatically. Override it only when your setup requires a different routing choice. Access can vary by region and over time.

How fast is a run? A ranking table takes one public page request. Domain mode makes one overview request per unique domain, with retries on access problems. The observed golden run completed with 94 records, but request time and site behavior vary.

Can I schedule and monitor runs? Yes. Schedule the Actor and inspect each run's status and dataset. Keep dataDate because several runs may refer to the same source month.

Can I export to a spreadsheet? Yes. Export the dataset as CSV or a spreadsheet file, or read the JSON records through the dataset API. Nested sample arrays remain easiest to inspect in JSON.

Why is a field empty? The source may not publish that measure for the domain or ranking. For example, a category rank may be absent. The Actor leaves missing values empty and does not estimate them.

Integrations

Read the dataset through the platform API, attach a webhook to a finished run, or feed exported rows to a spreadsheet, reporting workflow, or automation service. Filter by recordType before loading mixed-mode rows into a fixed-column report. Scheduling supports recurring snapshots; keep the run date and dataDate together to distinguish collection time from the source period.

Support

Open an issue on the Issues tab; we reply within 24h and add fields on request.