📈 Semrush Scraper

🚀 Export Semrush website analytics in seconds. 48 fields per domain, no Semrush login, no API key, roughly half a second per domain.

This Actor reads Semrush's public Website Traffic Checker pages and turns them into clean, flat dataset rows. Feed it a list of domains and you get back authority score, global and country rank, monthly visits, engagement, organic and paid search traffic, backlinks and referring domains, the top ranking keywords, the closest competitors, traffic split by country and device, inbound and outbound referral traffic, and the slice of traffic arriving from AI assistants. Everything comes from semrush.com/website/{domain}/overview/, the pages Semrush publishes to logged-out visitors, so no subscription is involved on either side.

Each row carries 48 fields, including 5 top keywords with position, search volume, CPC and search intent, 3 competitors, 5 countries, 5 AI platforms, 5 inbound and 5 outbound referral sources, and six months of history for visits, search traffic, backlinks and referring domains. On a 36 domain research sample, 31 domains returned Semrush's complete block set and 5 returned a partial one. Every row says which it is.

🎯 Target Audience	💡 Primary Use Cases
SEO agencies and consultants	Bulk competitor and client audits
Growth and demand-gen teams	Market sizing, channel mix, share of search
Sales and lead-gen teams	Scoring and enriching domain lists before outreach
Investors and M&A analysts	Traffic due diligence across a portfolio or pipeline
Content and PR teams	Finding which sites actually pull organic and referral traffic
Data teams and AI builders	Feeding SEO features into models, dashboards and warehouses

📋 What the Semrush Scraper does

1. Takes a list of domains, or full Semrush website URLs, and normalises them. AMAZON.COM, https://www.amazon.com/dp/B01, and amazon.com all collapse to the same target, and duplicates are dropped before anything is fetched.
2. Fetches the public Semrush overview page for each domain and reads the structured payload the page ships with, rather than scraping rendered text. Numbers arrive as numbers.
3. Optionally fetches the Semrush competitors page as well, which upgrades each of the 3 rivals from a name and a visit count into a full profile with authority score, global rank, country rank, bounce rate, pages per visit and average visit duration.
4. Converts every ratio into a human percentage. 0.4068 becomes 40.68, -0.0251 becomes -2.51.
5. Decodes keyword intent codes into Commercial, Informational, Navigational and Transactional, and splits Semrush's internal google.com__Google organic referral encoding into a clean source plus a channel.
6. Labels every row Full or Partial and writes "Not Disclosed" wherever Semrush withholds a block, so a gap in the data is never confused with a gap in the scraper.

💡 Why it matters: a Semrush seat is priced per user and its exports are capped. This Actor turns the same public surface into a bulk pipeline you can point at a thousand domains, schedule nightly, and pipe straight into a sheet, a warehouse or a model.

🎬 Full Demo (🚧 Coming soon)

📊 Output

One row per domain, 48 fields. Arrays stay as arrays in JSON and flatten cleanly in CSV and Excel.

Field	Type	Description
🖼 imageUrl	string	Domain favicon as served by Semrush
🌐 domain	string	The analysed domain
🔗 url	string	The Semrush page the row came from
🏷 brandName	string	Brand name Semrush associates with the domain
✅ dataCompleteness	string	Full or Partial
📂 categories / primaryCategory	array / string	Semrush industry categories
🌍 globalRank	number	Global traffic rank
📍 countryRank / rankCountry / rankCountryCode	number / string	Rank in its strongest country
🏆 categoryRank / categoryRankName	number / string	Rank inside its category
💪 authorityScore	number	Semrush Authority Score, 0-100
⚠️ googlePenaltyRisk	string	Semrush penalty-risk label
👥 monthlyVisits	number	Visits in the snapshot month
📊 monthlyVisitsChangePercent	number	Month-over-month change
📄 pagesPerVisit	number	Average pages per visit
⏱ avgVisitDurationSeconds	number	Average session length
🚪 bounceRatePercent	number	Bounce rate
🖥 desktopVisits / desktopSharePercent	number	Desktop split
📱 mobileVisits / mobileSharePercent	number	Mobile split
🌱 organicSearchTraffic / organicTrafficChangePercent	number	Organic search traffic
💰 paidSearchTraffic / paidTrafficChangePercent	number	Paid search traffic
🔗 backlinks / backlinksChangePercent	number	Total backlinks
🌐 referringDomains / referringDomainsChangePercent	number	Unique referring domains
🤖 aiTrafficTotal / aiTrafficSharePercent	number	Traffic arriving from AI assistants
🧠 aiPlatforms	array	Per platform: name, referrer, visits, share
🔑 topKeywords	array	5 keywords: term, position, volume, CPC, traffic share, intent
⚔️ topCompetitors	array	3 rivals, optionally with full metrics
🗺 trafficByCountry	array	5 countries: visits, share, desktop and mobile split
⬅️ trafficSourcesIn	array	5 inbound sources with channel and previous month
➡️ trafficSourcesOut	array	5 outbound destinations
📈 visitsHistory	array	6 months of visits, desktop and mobile
📉 searchTrafficHistory	array	6 months of organic, paid and total
🔗 backlinksHistory	array	6 months of backlink counts
🌐 referringDomainsHistory	array	6 months of referring domains
📅 dataDate / lastUpdated	string	Snapshot date and Semrush publish date
🕒 scrapedAt	string	ISO timestamp of the run
❌ error	string	Populated only on failed rows

Three real records from a live run:

{
  "domain": "amazon.com",
  "brandName": "Amazon",
  "dataCompleteness": "Full",
  "primaryCategory": "Online Services",
  "globalRank": 12,
  "countryRank": 5,
  "authorityScore": 100,
  "monthlyVisits": 2418264166,
  "bounceRatePercent": 40.68,
  "pagesPerVisit": 7.07,
  "avgVisitDurationSeconds": 690,
  "organicSearchTraffic": 674663550,
  "paidSearchTraffic": 23636433,
  "backlinks": 5871314878,
  "referringDomains": 4454552,
  "aiTrafficTotal": 3086498,
  "aiTrafficSharePercent": 0.13,
  "topKeywords": [
    { "keyword": "amazon", "position": 1, "searchVolume": 151000000, "cpcUsd": 0.03, "trafficSharePercent": 22, "intent": "Navigational" }
  ],
  "topCompetitors": [
    { "domain": "ebay.com", "relevancePercent": 51, "monthlyVisits": 556736632, "authorityScore": 100, "globalRank": 60, "countryRank": 23, "bounceRatePercent": 43.89, "pagesPerVisit": 5.97, "avgVisitDurationSeconds": 733 }
  ],
  "trafficByCountry": [
    { "countryCode": "US", "country": "United States", "visits": 1963851803, "trafficSharePercent": 81.21, "desktopSharePercent": 43.01, "mobileSharePercent": 56.99 }
  ],
  "dataDate": "2026-07-15"
}

{
  "domain": "stripe.com",
  "dataCompleteness": "Full",
  "primaryCategory": "Computer Software and Development",
  "globalRank": 164,
  "authorityScore": 79,
  "monthlyVisits": 203227314,
  "bounceRatePercent": 60.72,
  "pagesPerVisit": 2.82,
  "avgVisitDurationSeconds": 498,
  "organicSearchTraffic": 5667468,
  "backlinks": 151822916,
  "referringDomains": 664614,
  "aiTrafficTotal": 7665861,
  "aiTrafficSharePercent": 4.78,
  "aiPlatforms": [
    { "platform": "grok.com", "referrer": "grok.com", "monthlyVisits": 3049041 },
    { "platform": "ChatGPT", "referrer": "chatgpt.com", "monthlyVisits": 798467 }
  ],
  "topKeywords": [
    { "keyword": "stripe", "position": 1, "searchVolume": 673000, "cpcUsd": 2.99, "trafficSharePercent": 27.23, "intent": "Navigational" }
  ],
  "topCompetitors": [
    { "domain": "gocardless.com", "relevancePercent": 16, "monthlyVisits": 2631134, "authorityScore": 63, "globalRank": 19708, "bounceRatePercent": 51.55 }
  ],
  "dataDate": "2026-07-15"
}

{
  "domain": "nytimes.com",
  "brandName": "The New York Times",
  "dataCompleteness": "Full",
  "primaryCategory": "Newspapers",
  "globalRank": 66,
  "countryRank": 28,
  "authorityScore": 100,
  "monthlyVisits": 493008613,
  "bounceRatePercent": 57.09,
  "organicSearchTraffic": 178429595,
  "backlinks": 666074840,
  "referringDomains": 1795806,
  "aiTrafficTotal": 242773,
  "topKeywords": [
    { "keyword": "wordle", "position": 1, "searchVolume": 55600000, "cpcUsd": 0.99, "trafficSharePercent": 13.52, "intent": "Informational" }
  ],
  "trafficSourcesIn": [
    { "source": "Direct", "channel": "Direct", "visits": 102763996, "trafficSharePercent": 57.12, "previousVisits": 104694613 },
    { "source": "google.com", "channel": "Google organic", "visits": 36474601, "trafficSharePercent": 20.27, "previousVisits": 35539195 }
  ],
  "dataDate": "2026-07-15"
}

✨ Why choose this Actor

• No Semrush account needed. Not yours, not ours. The Actor only reads pages Semrush publishes to anonymous visitors.
• Honest about gaps. Semrush publishes its blocks independently, and for a minority of domains it withholds the visits, backlink and keyword blocks. Every row is stamped Full or Partial, and withheld numbers read "Not Disclosed" rather than an empty cell that looks like a bug.
• No nulls, ever. Sentinels everywhere, [] for empty arrays, booleans as Yes and No. CSV and Excel exports stay clean.
• Numbers, not strings. The Actor reads the page's own structured payload, so 2418264166 arrives as an integer instead of "2.4B" that you then have to parse back.
• Percentages already computed. Bounce rate, traffic shares, device splits and month-over-month deltas come out as readable percentages.
• Search intent decoded. Keyword intent arrives as Commercial, Informational, Navigational or Transactional, not as a raw integer code.
• Competitor enrichment on demand. One toggle upgrades each rival from a name into a full profile with authority score, ranks and engagement.
• Six months of history in every row. Visits, organic and paid search, backlinks and referring domains, so you can see direction, not just a single month.
• Fast and cheap to run. Roughly half a second per domain, one request each, no proxy and no browser.

📈 How it compares to alternatives

	This Actor	Minimal Semrush scrapers	Full-suite Semrush scrapers	A Semrush seat
Fields per domain	48	4-8	80+	Full product
Semrush login required	No	No	No	Yes, paid
Keyword intent decoded	Yes	No	Varies	Yes
Referral traffic in and out	Yes	No	Varies	Yes
AI assistant traffic	Yes	No	Yes	Yes
History depth	6 months	None	Varies	Years
Data gaps labelled per row	Yes	No	No	Not applicable
Keywords per domain	5	0	5	Unlimited
Competitors per domain	3	0	3	Unlimited
Typical price per 1,000 rows	$4.50	$1.80-$2.00	$15.00-$27.50	Per seat, per month

The honest read: if you only need an authority score and a visit count, a minimal scraper is cheaper. If you need Moz data and trending-website lists on top of Semrush, a full-suite scraper carries more. This Actor sits between them, at a price closer to the cheap end, and it is the one that tells you when Semrush withheld a block instead of handing you a silently empty column.

Ceilings you should know before you buy, all of them limits of Semrush's public pages rather than of this Actor: 5 keywords, 3 competitors, 5 countries, 5 AI platforms and 5 inbound plus 5 outbound referral sources per domain, six months of history, and a monthly snapshot rather than live data. Deeper cuts of any of these live behind a Semrush login.

🚀 How to use

1. Create a free Apify account. New accounts come with $5 in free credit, which is enough to try this Actor on roughly a thousand domains. Sign up here.
2. Open the Semrush Scraper and go to the Input tab.
3. Paste your domains into Domains, one per line. Bare domains such as amazon.com are ideal, but full URLs work too. You can also paste Semrush page URLs into Semrush URLs instead, or use both fields together.
4. Set Max Items to the number of rows you want back.
5. Turn on Include full competitor metrics if you want each rival enriched with authority score, ranks and engagement. It costs one extra request per domain.
6. Click Start and watch the log. Each domain reports its visits and authority score as it lands.
7. Open the Storage tab and download the dataset as CSV, JSON, Excel or XML, or pull it from the API.

{
  "domains": ["amazon.com", "stripe.com", "nytimes.com"],
  "maxItems": 100,
  "includeCompetitorMetrics": true
}

💼 Business use cases

Competitive benchmarking at portfolio scale

Feed in your domain plus every competitor you track and get one comparable row each. Authority score, monthly visits, organic versus paid split, backlink counts and engagement all sit side by side, so a quarterly benchmark deck becomes a scheduled run and a pivot table instead of a week of tab-switching. The six-month history in each row means you can chart direction without waiting two quarters to accumulate your own snapshots.

Lead scoring and sales prioritisation

Sales lists arrive as domains. Enrich them before anyone picks up the phone: a prospect with 2 million monthly visits, an authority score of 79 and rising organic traffic is a different conversation from one with 4,000 visits and a falling backlink count. trafficByCountry tells you which territory to route the lead to, and primaryCategory gives you a segment without anyone hand-tagging the list.

Investment and acquisition due diligence

Traffic quality is the fastest cheap signal on an online business. Run the target and its three closest competitors, then read the ratios: how much of the traffic is organic versus paid, how much is branded navigational search, whether backlinks and referring domains are growing or shrinking, and how dependent the site is on a single referral source. trafficSourcesIn exposes concentration risk that a headline visit count hides.

Tracking the shift to AI search

aiPlatforms breaks out how much traffic arrives from ChatGPT, Gemini, Claude, Perplexity and the rest, per domain and per month. Run it monthly across your category and you get an early read on whether AI assistants are starting to send meaningful volume in your market, and which of your competitors is capturing it first. In the sample above that share ranged from 0.05% for a news site to 4.78% for a developer-facing SaaS, a spread you can only see by measuring it.

🔌 Automating Semrush Scraper

• Make and Zapier: trigger a run from a new row in a CRM or a sheet, then push the enriched fields back onto the record.
• Slack: post a weekly digest of the domains whose authority score or organic traffic moved most.
• Airbyte: sync the dataset into BigQuery, Snowflake or Postgres and join it to your own pipeline data.
• GitHub Actions: call the Actor from a workflow and commit the CSV so your competitive data is versioned alongside your site.
• Google Drive: drop each run's export into a shared folder so the whole team reads the same numbers.
• Apify Scheduler: run it monthly, right after Semrush publishes a new snapshot, and let the history build itself.

🌟 Beyond business use cases

• Research. Study how organic, paid, direct and referral traffic mix differs across industries, or track how quickly AI-assistant referrals grow across a fixed sample of sites.
• Personal. Check how your own blog or side project ranks, which keywords actually bring it traffic, and which sites link to it.
• Non-profit. Benchmark your reach against peer organisations without paying for a seat you would use twice a year.
• Experimentation. Wire the dataset into a notebook or an LLM and ask it to cluster domains by traffic profile, or to draft a competitive summary from the raw fields.

🤖 Ask an AI assistant about this scraper

Paste this into ChatGPT, Claude or Gemini to get help shaping a run:

I am using the Semrush Scraper on Apify. It returns 48 fields per domain from Semrush's public website pages, including authority score, global and country rank, monthly visits, bounce rate, pages per visit, organic and paid search traffic, backlinks, referring domains, 5 top keywords with intent, 3 competitors, traffic by country, inbound and outbound referral sources, AI-assistant traffic, and six months of history. Help me design a run for [describe your goal] and tell me which fields to use.

❓ Frequently Asked Questions

🔑 Do I need a Semrush account or an API key?

No. The Actor reads only the pages Semrush publishes to logged-out visitors. You never enter a Semrush credential, and no Semrush subscription is used on our side either.

📊 How accurate is the data?

It is exactly what Semrush shows on its own public pages, passed through without adjustment. Semrush's traffic figures are modelled estimates rather than measured analytics, so treat them as a comparable index across sites, not as a substitute for a site's own analytics.

🗓 How fresh is it?

Semrush publishes a monthly snapshot. Each row carries dataDate for the snapshot the numbers describe and lastUpdated for when Semrush published it, so you always know the age of what you are reading.

🚧 Why do some rows say "Partial"?

Semrush publishes each block of a page independently, and for a minority of domains it withholds the visits, backlink, keyword and competitor blocks while still publishing rank, categories, search traffic and the device split. Those rows are labelled Partial and the missing numbers read "Not Disclosed". On a 36 domain research sample, 31 were Full and 5 were Partial. It is deterministic per domain, so a domain that is partial today will be partial on the next run.

🔢 Why only 5 keywords and 3 competitors?

That is the whole list Semrush publishes without a login. There is no pagination to follow and no hidden parameter that returns more. Anything deeper requires a Semrush subscription.

🌍 Can I get keyword data for a specific country?

Not from this surface. Semrush serves one keyword database per domain, its strongest country, and the country query parameters on those pages were tested and change nothing in the response. trafficByCountry still gives you the geographic split of visits.

⚔️ What does "Include full competitor metrics" add?

Without it, each of the 3 competitors comes back as a domain, a relevance score and a visit count. With it, the Actor fetches Semrush's competitors page as well and adds authority score, global rank, country rank, bounce rate, pages per visit and average visit duration to each rival. It costs one extra request per domain.

🤖 What counts as AI traffic?

Visits that arrive at the domain from an AI assistant or AI-powered tool, broken out per platform with its referrer host, monthly visits and share. ChatGPT, Gemini, Claude and Perplexity appear regularly, alongside smaller AI products that vary by industry.

🌐 Do I need a proxy?

No. Semrush serves these pages without blocking, and the Actor runs with proxies off by default. A proxy option is exposed for very large domain lists, but most runs never need it.

📥 What export formats are available?

CSV, JSON, Excel and XML from the Storage tab, plus the Apify API and the JavaScript and Python clients for programmatic access.

⚡ How fast is it, and does it scale?

Roughly half a second per domain with a single request each. A thousand domains is a matter of minutes, and maxItems accepts up to 1,000,000.

🔁 What happens to an invalid or unknown domain?

Entries that are not domains are skipped with a warning before any request is made. A domain Semrush has no page for produces an error row naming the domain, and it is never charged as a result.

🔌 Integrate with any app

Every run's dataset is available through the Apify API, the JavaScript client and the Python client, and through Apify's integrations with Make, Zapier, Airbyte, Slack, Google Drive, GitHub and webhooks. Schedule it, chain it after another Actor, or call it directly from your own code.

🔗 Recommended Actors

• Similarweb Traffic Scraper - traffic and engagement from a second source, ideal for cross-checking the numbers here.
• Ahrefs Free Tools Scraper - domain authority, backlinks and keyword difficulty from Ahrefs' free tools.
• RDAP Domain Lookup Scraper - registration, expiry and nameserver records for the same domain list.
• Hugging Face Model Scraper - model metadata and download counts for AI market research.
• AWS Marketplace Scraper - vendor and product listings for B2B software mapping.

💡 Pro Tip: browse the complete ParseForge collection.

🆘 Need Help? Open our contact form

⚠️ Disclaimer: This is an independent tool and is not affiliated with, endorsed by, or sponsored by Semrush. It collects only publicly available data from pages Semrush serves to anonymous visitors. Semrush is a trademark of its respective owner.