Semrush Scraper: Domain Traffic, Authority & Top Websites

📈 What is Semrush Scraper?

Semrush Scraper extracts the domain overview of any website from Semrush: Authority Score, monthly visits, global and country rank, organic and paid search traffic, backlinks and referring domains, top keywords, competitors, traffic by country and device, traffic sources, and visits sent by AI assistants (ChatGPT, Gemini, Perplexity…), with 6 months of history. It also lists the top websites of any country and industry, as ranked by Semrush. It works as a Semrush API alternative: no Semrush subscription and no API units, run it on a schedule, call it from your code, or plug it into Make, Zapier or n8n.

Type a list of domains (stripe.com, notion.so), or pick a country and an industry, click Start, and download the data in JSON, CSV or Excel. No login, nothing to set up.

📋 What data can you extract from Semrush?

One row per domain or per ranked website, 65 fields:

Category	What you get
🌐 Domain	domain, brand, main country, industry categories, month of the data
🏆 Authority and rank	Semrush Authority Score, global rank, country rank, industry rank — 79, #145 worldwide
👥 Traffic	monthly visits and their change, pages per visit, visit duration, bounce rate, desktop vs mobile
🔍 Search traffic	organic and paid traffic with their change, the 5 keywords that bring the most visits (position, volume, CPC)
🔗 Backlinks	backlinks, referring domains, their change, Google penalty risk
🤖 AI assistant traffic	visits sent by ChatGPT, Gemini, Perplexity, Grok… per assistant, with history
🧭 Traffic sources	where visitors come from (direct, Google, social…) and where they go next
🌍 Countries	the 5 countries sending the most visits, desktop and mobile share for each
🥊 Competitors	the 3 closest competitors by shared audience — and, as an option, their own authority, visits and ranks
📅 History	6 months of visits, search traffic, devices, backlinks and referring domains
📊 Top websites	rank in a country × industry ranking, visits, month-over-month and year-over-year change, top traffic source

Every field, with an example, is listed in the Output section below.

✅ Why use Semrush Scraper?

• 🧾 The widest domain row: authority, traffic, search, backlinks, keywords, competitors, countries, sources and AI traffic in one row, with 6-month history — flat columns, ready for a spreadsheet.
• 🤖 AI assistant traffic: how many visits ChatGPT, Gemini, Perplexity or Grok send to a site, month by month.
• 📊 Top websites of 239 countries + worldwide × 110 industries: the Semrush ranking, about 100 websites per list, with the full overview of each one as an option.
• 🎯 Paste anything: a domain, www. or a full URL — the Actor reads the root domain and keeps what you typed.
• 💸 Pay only for data: a domain Semrush has no data for comes back as a free row that says so.
• 🔔 Monitoring built in: tick Only new rows, schedule the Actor, and each run returns (and charges) only the rows it has never delivered before.
• 🔌 API, scheduling, monitoring, integrations (Make, Zapier, n8n, Google Sheets…) and JSON/CSV/Excel export via the Apify platform.

🚀 How to scrape Semrush

1. Create a free Apify account.
2. Open Semrush Scraper and type your Domains (e.g. stripe.com), one per line.
3. Or pick a Top websites: country and one or more industries to get the Semrush ranking — or paste Semrush pages into Semrush URLs.
4. Set Max rows (1000 by default, 0 = no limit), tick the options you want, then click Start.
5. Download the dataset in JSON, CSV, Excel or via API.

💰 How much does it cost to scrape Semrush?

This Actor uses pay per event pricing: you pay per domain overview delivered, per top website row, and per competitor detailed when that option is on, plus a small fee per run start. A domain Semrush has no data for is delivered free. Higher Apify plans pay less per row. Platform usage (compute, proxy) is included in the price.

⚙️ Input

{
    "domains": ["stripe.com", "notion.so", "leboncoin.fr"],
    "maxItems": 100
}

The top websites of two countries in two industries, each with its full overview and its competitors:

{
    "location": "us",
    "locations": ["fr"],
    "topWebsitesCategories": ["finance", "banking"],
    "maxItemsPerQuery": 20,
    "enrichTopWebsites": true,
    "includeCompetitorDetails": true
}

Or with Semrush pages copied from your browser:

{
    "startUrls": [
        { "url": "https://www.semrush.com/website/stripe.com/overview/" },
        { "url": "https://www.semrush.com/trending-websites/us/finance" }
    ],
    "maxItems": 500
}

Field	Notes
domains	Domains to analyse: stripe.com, www.stripe.com or a full URL (https://stripe.com/pricing). The root domain is read (Semrush publishes one overview per root domain); what you typed is kept in inputDomain.
startUrls	Semrush pages: a domain page (https://www.semrush.com/website/stripe.com/overview/) or a top websites page (https://www.semrush.com/trending-websites/us/finance).
location, locations	Country of the top websites ranking: us, fr, de… or global for worldwide. Each country is crossed with each industry (max 500 country × category rankings per run, plus the ranking pages you paste in startUrls).
topWebsitesCategories	Industries of the ranking: finance, banking, apparel-and-fashion… or all for every website (default).
enrichTopWebsites	Also read the full domain overview of every ranked website: it is delivered as a domain row that keeps its rank.
includeCompetitorDetails	Add the authority, visits, ranks and engagement of up to 5 competitors of each domain (competitorDetails, one more page per domain, charged per competitor delivered: a domain with 3 competitors listed is charged 3, never 5).
includeHistory	Keep the monthly history columns (default on). Turn it off for lighter rows.
maxItems	Stop after this many rows for the whole run (0 = unlimited).
maxItemsPerQuery	Cap for EACH ranking (country × industry): its top N. 0 = no per-list cap.
excludeKeywords	Skip the domains whose name contains one of these words: casino, porn.
minAuthorityScore, minMonthlyVisits	Keep only the domains above these figures (0 = no filter). Visits are the worldwide visits of the website; for a ranking row without enrichTopWebsites, its visits in the ranking's country. A domain without data is always kept, so you see it was not found.
onlyNew, stateKey, resetState	Monitoring: only the rows never delivered under this memory key; resetState forgets the memory.
Advanced	proxyConfiguration (Apify proxy by default, included in the price; the residential proxy is not available), maxConcurrency, maxRequestsPerMinute, minRequestIntervalMs, maxRequestRetries, debugLog.

📦 Output

One domain row, from a real run on stripe.com (shortened: lists cut to their first entry, some columns left out):

{
    "type": "domain",
    "id": "stripe.com",
    "url": "https://www.semrush.com/website/stripe.com/overview/",
    "domain": "stripe.com",
    "inputDomain": "https://stripe.com/pricing",
    "error": null,
    "found": true,
    "mainCountry": "US",
    "categories": ["Computer Software and Development", "Finance", "Information Technology", "Online Services"],
    "dataMonth": "2026-09-01",
    "globalRank": 145,
    "countryRank": 145,
    "countryRankCountry": "United States",
    "categoryRank": 3,
    "categoryRankName": "Finance",
    "authorityScore": 79,
    "visits": 232542245,
    "visitsChangePct": 11.79,
    "pagesPerVisit": 2.812,
    "avgVisitDurationSec": 498,
    "bounceRate": 0.6153,
    "desktopShare": 0.5397,
    "mobileShare": 0.4603,
    "organicTraffic": 4963905,
    "organicTrafficChangePct": -11.1,
    "paidTraffic": 192150,
    "backlinks": 141005083,
    "referringDomains": 732469,
    "googlePenaltyRisk": "Low",
    "topKeywords": [{ "keyword": "stripe login", "position": 1, "searchVolume": 135000, "cpc": 14.46, "trafficShare": 0.0708, "intents": ["commercial"] }],
    "competitors": [{ "domain": "gocardless.com", "relevance": 0.14, "visits": 3032397 }],
    "topCountry": "US",
    "topCountryVisits": 59382926,
    "topTrafficSource": "Direct",
    "aiTrafficVisits": 6801579,
    "visitsHistory": [{ "month": "2026-09-01", "value": 232542245 }],
    "scrapedAt": "2026-10-15T14:00:00.000Z"
}

You can download the dataset in various formats such as JSON, HTML, CSV or Excel.

Rows are saved as soon as they are read, so with several domains, several rankings or enrichTopWebsites they are not in the order you typed or in the order of the Semrush ranking: sort by rank (and rankingCountry) to get the ranking back.

All 65 fields

Fields	What you get
type, id, url	domain or topWebsite; stripe.com (or us/finance/stripe.com for a ranked website); the Semrush page read
domain, inputDomain, found, error	root domain, what you typed, false + the reason when Semrush has no data (free row)
brandName, mainCountry, categories, categorySlugs	Domain: brand, US, Finance, finance
dataMonth, semrushUpdatedAt, backlinksDataMonth	month of the data, always its first day (2026-09-01), last update 2026-10-11, month of the backlinks
globalRank, globalRankTrend, countryRank, countryRankCountry, categoryRank, categoryRankName	Authority and rank: 145, 1 (going up), 145, United States, 3, Finance
authorityScore	Semrush Authority Score, 0-100 — 79
visits, visitsChangePct, pagesPerVisit, avgVisitDurationSec, bounceRate	Traffic: 232542245, 11.79 %, 2.812, 498 s, 0.6153. Worldwide figures for a domain row; a ranking row without enrichTopWebsites gets the ranking country's visits from the ranking
desktopVisits, mobileVisits, desktopShare, mobileShare	desktop and mobile visits, shares from 0 to 1
organicTraffic, organicTrafficChangePct, paidTraffic, paidTrafficChangePct	Search traffic and its change in %
backlinks, backlinksChangePct, referringDomains, referringDomainsChangePct, googlePenaltyRisk	Backlinks: 141005083, -3.87, 732469, 4.94, Low
topKeywordsCountry, topKeywords	the 5 keywords bringing most organic traffic: keyword, position, volume, CPC, share, intents by name (commercial, informational, navigational, transactional)
competitors, competitorDetails	Competitors: 3 domains with audience overlap and visits; with includeCompetitorDetails, up to 5 competitors (the Semrush competitors page lists 5) with their authority, visits, ranks (with the country of the country rank), pages per visit, duration, bounce rate
topCountry, topCountryVisits, trafficByCountry	Countries: US, 59382926, the 5 top countries with their desktop / mobile share
topTrafficSource, trafficSources, trafficDestinations	Traffic sources: Direct; where visitors come from and where they go next, with share and visits — desktop visits only (Semrush's traffic journey does not count mobile), so share is a share of desktopVisits, not of visits. The host is cut from Semrush's channel: google.com + channel: "Google organic" (channel is null when there is none). Case is kept as Semrush writes it (Direct on a domain row, direct on a ranking row)
aiTraffic, aiTrafficVisits	AI assistant traffic: per assistant (visits, share, monthly history), and the sum of the assistants listed
visitsHistory, searchTrafficHistory, deviceHistory, backlinksHistory, referringDomainsHistory	History: 6 months, one entry per month: { month, value } for visitsHistory, backlinksHistory and referringDomainsHistory, { month, organic, paid, total } for searchTrafficHistory, { month, visits, desktopVisits, mobileVisits } for deviceHistory (all empty with includeHistory off, as is the history of each aiTraffic assistant)
rank, rankingCountry, rankingCategory, rankingCategoryName	Top websites: 1, us, finance, Finance
rankingVisits, visitsMomChangePct, visitsYoyChangePct, rankingTotalVisits	the ranking country's visits of the website (on every ranking row, with or without enrichTopWebsites), its change vs last month and last year in % (2 decimals), visits of all the websites of the ranking
scrapedAt	ISO timestamp

💡 Tips

How to get more results

A ranking holds about 100 websites: to go wider, add countries in locations and industries in topWebsitesCategories — every country is crossed with every industry. Set maxItems to 0 to remove the overall cap.

How to reduce costs

The price is per row, so the levers are maxItems, maxItemsPerQuery, the filters and onlyNew for recurring runs (you never pay twice for the same row). enrichTopWebsites and includeCompetitorDetails are off by default: tick them only when you need the full overview of every ranked website or the figures of each competitor.

Several lists in one run

Fill locations and topWebsitesCategories: the Actor reads one ranking per country × industry (3 countries × 4 industries = 12 lists, up to 500 per run). The single location field is added to the list. Set maxItemsPerQuery to give every list its own cap: without it the first lists can use up the whole maxItems budget. Domains and rankings can be mixed in the same run.

Monitoring: only the new rows

Tick Only new rows (onlyNew) and schedule the Actor. The first run returns everything; each later run skips the rows already delivered: they are not saved and not charged. The memory lives in a named key-value store of your account (semrush-seo-scraper-seen, up to 150,000 rows per key) and is only updated with rows that really reached the dataset, so a failed run never hides anything. On a ranking with a cap (maxItemsPerQuery, or maxItems), the schedule watches that top N: each run returns the newcomers of the top N only, never the websites below it — a run where the top N has not changed returns nothing and charges nothing. Give each schedule its own stateKey, and tick resetState once to start over. Semrush updates its data once a month: a monthly schedule with resetState gives a fresh snapshot each month.

🔌 Integrations and API

Call the Actor via the Apify API, the JavaScript or Python clients, or connect it with integrations and webhooks (Make, Zapier, n8n, Google Sheets, Slack, Airtable…). The dataset can be fetched as JSON or CSV from any tool.

🤖 Use with AI agents (MCP)

AI agents (Claude, ChatGPT, Cursor…) can find and run this Actor through the Apify MCP server, billed to their Apify account like any run. It returns one row per domain overview or per ranked website from Semrush. Actor id: nice_dev/semrush-seo-scraper; MCP server with this Actor only: https://mcp.apify.com/?tools=fetch-actor-details,nice_dev/semrush-seo-scraper.

Smallest input, for a cheap first call:

{
    "domains": ["stripe.com"],
    "maxItems": 10
}

Key output fields: domain, authorityScore, visits, globalRank, organicTraffic, backlinks, topKeywords, competitors, aiTraffic.

Cost: pay per row delivered, see the pricing section above. Cap each call with maxItems and, through the API, with the run option maxTotalChargeUsd.

❓ FAQ

Is it legal to scrape Semrush?

The Actor only reads what Semrush shows publicly to any anonymous visitor on its free website pages. It logs in to nothing and solves no captcha. The data are figures about websites, not about people. You are responsible for using the data in compliance with Semrush's Terms of Use and applicable law. This Actor is not affiliated with Semrush.

Does it need a login, a Semrush account or a proxy?

No login and no Semrush subscription. The proxy is included in the price: leave the default setting (the residential proxy is not available). A request the site turns away is retried at once on a new proxy session, up to 10 times on top of the retries (without a proxy, after a pause of 5 seconds, doubled at each retry up to 150 seconds).

Why does my domain come back with found: false?

Semrush publishes no overview for that domain (too little traffic, or too new). A subdomain you paste is read as its root domain (docs.stripe.com → stripe.com), so it never ends up here. The row tells you so in error and is not charged.

Which country are the keywords from?

The 5 top keywords are those of the domain's main country (topKeywordsCountry: US for stripe.com, FR for leboncoin.fr), as Semrush shows them; the country cannot be chosen.

Is the data safe to open in Excel or to show on a web page?

The texts are Semrush's own (domain names, keywords, category names), copied as they are. A keyword can begin with -, +, = or @: Excel and Google Sheets may read such a cell of a CSV file as a formula. The Actor leaves the text as it is, so that the JSON and the API give the real value: when you open a CSV, import these columns as text. On a web page, escape every field like any text from another site.

Known limitations

• Semrush shows the latest month only: older rankings and other keyword countries are not available.
• The free pages give 5 top keywords, 3 competitors and 5 countries per domain, not the full lists.
• onlyNew remembers row ids, not their content: a domain whose figures changed is not returned again (use resetState for a new snapshot).
• Two runs sharing the same stateKey at the same time may both return the same new row.

A run that reaches its timeout stops itself about 45 seconds before it: no new page is asked, what it read is saved and, with onlyNew, remembered, and the run ends Succeeded with "Stopped before the run's timeout". Resurrect it to go on from there, or give the next run a longer timeout (Run options).

A run the platform stops without warning (out of memory)

• Resurrect it: it goes on from where it stood at most a minute before the stop. What it had read since is read again, and the rows already saved are skipped: none is delivered or charged twice, and maxItems still counts them.
• With onlyNew, the memory is saved after each write (at most once every 2 seconds): resurrect the stopped run and the rows it had saved meanwhile join the memory; leave it stopped for good, and the next run may return the rows of its last 2 seconds once more.

Something doesn't work?

The last line of the log counts the rows saved and filtered out, and the requests that failed after every retry. Those requests are listed, with the reason, in the FAILED_REQUESTS record of the run's key-value store. A run that saved nothing and had failed requests fails, and its last message gives the cause.

If Semrush changes its pages, you are told instead of paying for blank rows. If the first 20 rows read all lack their domain, visits or Authority Score, the run saves nothing more, stops and fails, and its last message names the missing field: at most those first rows are charged. A run of fewer rows (a scheduled onlyNew run, a small Max rows) is checked at its end, from 3 rows: its rows are already saved, and the run fails with the missing field instead of ending green.

🛟 Support

Open an issue in the Issues tab with a link to your run: the run log and the FAILED_REQUESTS record of the key-value store show exactly which URLs failed and why.