SEMrush Scraper — Website Traffic, Authority Score & SEO Metrics (with AI Traffic)

Check domain authority, website traffic, backlinks, top keywords and competitors for any domain — no SEMrush subscription required. This scraper reads SEMrush's public website directory and returns 30+ clean, structured SEO metrics per domain in seconds.

Unlike browser-based scrapers, this Actor is HTTP-only: no headless Chrome, no rendering wait. A typical domain finishes in 3–6 seconds and large batches run in parallel.

⚡ Why this SEMrush scraper?

• 🚀 Fast & lightweight — pure HTTP extraction, ~3–6s per domain (browser scrapers need 35–60s)
• 🤖 AI traffic data — visits referred by ChatGPT, Gemini and other AI assistants, the metric every GEO / AI-visibility report needs in 2026
• 📈 6-month history — traffic history AND backlink history, not just a snapshot
• 💰 Keyword CPC + search intent included with every top keyword
• 🔗 Top referral sources with share and visits (including Direct share)
• 🌍 Per-country device split (desktop vs mobile) on top of the global split
• 🆓 No SEMrush account, no API key, no login — public data only

📊 Data you get per domain

Group	Fields
Authority & rank	domain_authority (0–100), global_rank, country_rank + country, google_penalty_risk, industry_category
Traffic & engagement	monthly_visits, monthly_visits_change_percent, bounce_rate, pages_per_visit, avg_visit_duration_seconds
Search traffic	organic_search_visits, organic_traffic_change_percent, paid_search_visits, paid_traffic_change_percent
Sources	direct_visits_percent, top_traffic_sources[] (source, share %, visits)
🤖 AI visibility	ai_traffic[] — visits from ChatGPT, Gemini & other AI assistants with share %
Backlinks	total_backlinks, backlinks_change_percent, referring_domains, referring_domains_change_percent
Keywords	organic_keywords[] — keyword, position, search volume, CPC, intent, traffic share
Competitors	competitors[] — domain, relevance score, monthly visits
Geography & devices	top_countries[] with traffic %, visits and per-country desktop/mobile split; overall desktop_traffic_percent / mobile_traffic_percent
History	visits_history[] and backlinks_history[] (up to 6 months)

🔧 Input

{
    "domains": ["amazon.com", "shopify.com", "zapier.com"],
    "maxConcurrency": 5,
    "proxyConfiguration": { "useApifyProxy": true }
}

• domains (required) — up to 500 domains per run. Full URLs are fine; they are normalized to root domains automatically.
• maxConcurrency — parallel fetches (default 5, max 10).
• proxyConfiguration — Apify datacenter proxies (free) work well by default.

📤 Output example

{
    "domain": "make.com",
    "data_month": "2026-06",
    "domain_authority": 57,
    "global_rank": 6414,
    "country_rank": 6856,
    "country": "United States",
    "industry_category": "Computer Software and Development",
    "monthly_visits": 8367047,
    "monthly_visits_change_percent": -8.69,
    "bounce_rate": 39.46,
    "pages_per_visit": 7.05,
    "avg_visit_duration_seconds": 772,
    "organic_search_visits": 637690,
    "paid_search_visits": 124775,
    "direct_visits_percent": 84.28,
    "total_backlinks": 2050718,
    "referring_domains": 38512,
    "ai_traffic": [
        { "source": "chatgpt.com", "name": "ChatGPT", "visits": 63665, "share_percent": 0.77 },
        { "source": "gemini.google.com", "name": "Gemini", "visits": 21643, "share_percent": 0.26 }
    ],
    "organic_keywords": [
        { "keyword": "make ai", "position": 1, "search_volume": 4400, "cpc": 3.15, "intents": "2" }
    ],
    "competitors": [
        { "domain": "n8n.io", "relevance": 0.08, "monthly_visits": 6864336 }
    ],
    "top_countries": [
        { "country": "United States", "traffic_percent": 19.36, "visits": 1619950, "desktop_share_percent": 48.96 }
    ],
    "visits_history": [
        { "date": "2026-06-01", "visits": 8367047 },
        { "date": "2026-05-01", "visits": 9163168 }
    ]
}

💡 Use cases

• Competitor analysis — compare authority score, traffic trends, keyword overlap and traffic sources across your whole market in one run
• SEO audits & client reporting — drop live domain KPIs into audit headers and monthly reports
• AI visibility / GEO tracking — monitor how much traffic competitors already get from ChatGPT and Gemini
• Domain due diligence — vet domains before buying: authority, backlink trend, penalty risk, traffic history
• Link prospecting & lead scoring — qualify outreach lists by authority score and referring domains
• Market research — map industry leaders, geography and device mix in any niche

❓ FAQ

Do I need a SEMrush account or API key? No. The Actor reads SEMrush's public website directory pages only.

How fresh is the data? SEMrush refreshes its public directory roughly monthly. Each result includes data_month so you always know which month you're looking at.

Why are some fields null? Low-traffic or brand-new domains often have no public data — SEMrush simply hasn't profiled them. You still get an item with the domain and whatever is available.

Why does a big site return an error? A few domains (e.g. zapier.com) have opted out of SEMrush's public directory. Those return a clear error item — "Domain not found in the SEMrush public website directory" — instead of misleading nulls.

Can I check subdomains? Root domains work best (example.com). Subdomains only return data when SEMrush publishes a profile for them.

How fast is it? What does it cost? ~3–6 seconds per domain with default settings. Because there is no browser, compute cost per domain is a fraction of Playwright-based scrapers.

Integrations? Works with the Apify API, scheduler, webhooks and all Apify integrations (Make, Zapier, Google Sheets, Slack). Example API call:

curl -X POST "https://api.apify.com/v2/acts/<ACTOR_ID>/run-sync-get-dataset-items?token=<YOUR_TOKEN>" \
  -H "Content-Type: application/json" \
  -d '{"domains": ["amazon.com"]}'

⚖️ Disclaimer

This Actor is not affiliated with, endorsed by, or sponsored by Semrush Inc. "SEMrush" is a trademark of its respective owner, used here only to describe the data source. The Actor extracts publicly accessible data only. You are responsible for how you use the data and for compliance with applicable laws and terms of service.