Semrush Scraper - Traffic, Authority Score, Backlinks & Keywords

Turn any list of domains into a clean competitive-intelligence table. This actor reads Semrush's public website-overview pages and returns the whole page as structured data - monthly visits, Authority Score, global/country/category rank, organic vs paid search traffic, backlink profile, top organic keywords with CPC, competitors, per-country traffic with device splits, and AI-assistant referral traffic.

No Semrush login. No API key. No browser. Keyless HTTP against pages Semrush publishes for search engines.

Why this Semrush scraper?

Data	Typical Semrush scrapers	This actor
Monthly visits + change %	✅	✅
Authority Score	✅	✅
Global / country rank	partial	✅ + trend (up/down/flat)
Category rank + category name	❌	✅
Pages per visit, time on site, bounce rate	partial	✅
Organic vs paid search traffic, each with change %	❌	✅
Backlinks + referring domains, each with change %	❌	✅
6-month history — visits, organic/paid, backlinks, referring domains	❌	✅
Top organic keywords with position, volume, CPC, traffic share	❌	✅
Organic competitors with relevance + their visits	partial	✅
Traffic by country with desktop/mobile share per country	❌	✅
Desktop vs mobile visits + 6-month device history	❌	✅
AI-assistant referral traffic (ChatGPT, Perplexity…)	❌	✅
Referrals in / out (who sends traffic here, where it goes)	❌	✅
Google penalty risk	❌	✅

53 fields per domain. Uncovered domains are reported honestly and never charged.

Use cases

• Competitor benchmarking - drop in 50 competitor domains, get one comparable table.
• Lead qualification - score prospects by real traffic, Authority Score and backlink profile before you pitch.
• SEO due diligence - check a site's organic trend and backlink growth before an acquisition.
• Agency reporting - refresh a client + competitor set on a schedule and push to Sheets.
• AI-visibility tracking - see how much traffic ChatGPT and friends actually send a domain.

How to use

1. Add your domains under Domains (one per line) - or paste/link a whole list under Import domains from a file.
2. Leave everything else at its default and hit Start.
3. Export as JSON, CSV or Excel, or pull it from the API.

Input

{
    "domains": ["booking.com", "stripe.com"],
    "includeKeywords": true,
    "includeCompetitors": true,
    "includeAiTraffic": true,
    "outputFields": [],
    "maxItems": 100
}

• domains - domains to analyse. Full URLs and Semrush overview links are accepted and reduced to the registrable domain.
• startUrls - paste Semrush overview URLs directly. Also accepts the Console's Link remote text file option.
• urlsFromFile - bulk import: paste the whole list, or give one link to a .txt/.csv, Google Sheet, Drive share or key-value-store record. Blank lines, # comments, CRLF, a UTF-8 BOM, quotes and extra CSV columns all parse; duplicates are analysed once.
• includeHistory / includeKeywords / includeCompetitors / includeCountries / includeAiTraffic / includeDevices / includeReferrals - toggle each section. All come from the same page load, so turning one off makes the record leaner, not cheaper.
• outputFields (pick-list) - tick the fields you want and every record is trimmed to exactly those. Nothing to memorise - the dropdown lists all 53 with plain-English labels and their exact key.
• maxItems - cap per run.
• proxyConfiguration - Apify datacenter by default; Semrush serves these pages cleanly to it.

Output

One record per domain (type: "website"):

{
    "type": "website",
    "domain": "stripe.com",
    "url": "https://www.semrush.com/website/stripe.com/overview/",
    "country": "US",
    "categories": ["Finance"],
    "authorityScore": 79,
    "googlePenaltyRisk": "Low",
    "globalRank": 163,
    "globalRankTrend": "up",
    "countryRank": 164,
    "countryRankName": "United States",
    "categoryRank": 6,
    "categoryRankName": "Finance",
    "visits": 179937086,
    "visitsChangePercent": -0.0757,
    "pagesPerVisit": 2.7837,
    "timeOnSiteSec": 484,
    "bounceRate": 0.6082,
    "organicTraffic": 96442310,
    "organicChangePercent": 0.0312,
    "paidTraffic": 1204882,
    "backlinks": 224518903,
    "backlinksChangePercent": 0.0141,
    "referringDomains": 633826,
    "desktopVisits": 70856331,
    "mobileVisits": 109080755,
    "desktopShare": 0.3938,
    "mobileShare": 0.6062,
    "topKeywords": [
        { "keyword": "stripe", "position": 1, "searchVolume": 1830000, "cpc": 3.11, "trafficShare": 0.0421 }
    ],
    "competitors": [{ "domain": "gocardless.com", "relevance": 0.58, "visits": 3204118 }],
    "trafficByCountry": [
        { "country": "US", "countryName": "United States", "traffic": 52207086, "trafficShare": 0.2901, "desktopShare": 0.3938, "mobileShare": 0.6062 }
    ],
    "aiTraffic": [{ "source": "ChatGPT", "traffic": 227075, "share": 0.0005 }],
    "referralsIn": [{ "host": "github.com", "share": 0.11, "visits": 402118 }],
    "visitsHistory": [{ "date": "2026-06-01", "visits": 179937086 }],
    "dataDate": "2026-06-15",
    "scrapedAt": "2026-08-09T19:44:00.000Z"
}

What to expect (field coverage)

Measured over a deliberately varied sample (mega-sites, SaaS, retail, news, a small niche tool):

Field	Coverage
domain, visits, authorityScore, globalRank, countryRank, bounceRate, pagesPerVisit, timeOnSiteSec	100%
organicTraffic, paidTraffic, backlinks, referringDomains + their change %	100%
topKeywords, trafficByCountry, aiTraffic, referralsIn/referralsOut, all history series	100%
competitors	~85% - Semrush omits the set when too few organic keywords overlap
paidChangePercent	~85% - no delta exists when the domain ran no paid search last month
categoryRank, categoryRankName	~57% - only for domains Semrush has classified
brandName, favicon	~57% - only where Semrush holds a brand entity

Not available, and we say so rather than shipping an empty column: Site Audit metrics (site health, crawled pages, thematic reports) require a logged-in Semrush project - the public page ships them as zeros, so they are deliberately excluded from the record.

Domains Semrush does not cover (below its traffic threshold) end the run with a clear message naming them, and are never charged.

Automate & schedule

Pair with Apify Schedules to refresh a competitor set weekly and push straight into Google Sheets, Slack or your warehouse via Apify integrations.

Use with AI assistants (MCP)

Expose this actor as a tool to Claude, ChatGPT or any MCP client via the Apify MCP server and ask for competitive data in plain language.

Agent-ready: autonomous payments (x402 & Skyfire)

This actor is agent-ready — AI agents can discover it, run it, and pay for it autonomously, with no Apify account and no human in the loop. It uses pay-per-event pricing and limited permissions, so it qualifies for Apify's agentic-payment standards:

• x402 — an open, HTTP-native payment protocol. Agents pay per run in USDC on the Base network directly through the Apify MCP server — no account, no API key.
• Skyfire — agent-to-service payments for fully autonomous AI-agent workflows.

Building an AI agent, MCP tool, or autonomous data pipeline? This scraper is ready to plug in and pay as it goes.

Pricing

Pay per event: $0.006 per website overview. Domains Semrush does not cover cost nothing.

FAQ

Does this need a Semrush subscription? No. It reads only the public overview pages Semrush publishes for search engines.

Why is a domain "not covered"? Semrush publishes an overview only above a traffic threshold. Use the registrable domain (example.com, not a sub-path), and expect very small or brand-new sites to be absent.

Are the numbers Semrush's own? Yes - they are Semrush's published estimates, read from the page as-is. No modelling or re-estimation.

Can I import a big list? Yes - paste it, link it, or use the Start-URLs Link remote text file option. A file that cannot be read says so and charges nothing.

Need help?

Open an issue on the actor's Issues tab - feature requests genuinely do get built.

---

Not affiliated with, endorsed by, or sponsored by Semrush. "Semrush" is a trademark of its respective owner and is used here only to describe what this tool reads. Scrapes public pages only.