SEMrush Scraper - Domain Authority, Traffic & SEO Data

Fast, reliable extraction of domain authority, traffic data, rankings, and competitor insights from SEMrush. No SEMrush account required. Perfect for SEO tracking, competitor monitoring, and market research.

Features

Extract 20+ SEO metrics from any domain in under 20 seconds:

Traffic Metrics

• Monthly visits - Total website traffic
• Bounce rate - Visitor engagement percentage
• Pages per visit - Average pages viewed per session
• Visit duration - Time spent on site (in seconds)
• Traffic change - Month-over-month growth/decline percentage

Traffic Sources

• Organic search visits - Traffic from search engines
• Paid search visits - Traffic from paid ads
• Direct traffic % - Percentage of direct visitors
• Desktop traffic % - Desktop vs mobile breakdown
• Mobile traffic % - Mobile device traffic share

Rankings & Authority

• Domain authority - SEMrush authority score (0-100)
• Global rank - Worldwide ranking
• Country rank - Ranking in primary country
• Industry category - Business category classification

Backlinks & SEO

• Total backlinks - Number of incoming links
• Referring domains - Unique domains linking to site

Competitive Intelligence

• Top 5 competitors - Similar websites by audience
• Top 5 organic keywords - Best performing search terms with position & volume
• Top 5 countries - Geographic traffic distribution with percentages

Input

{
  "domains": [
    "amazon.com",
    "apple.com",
    "make.com"
  ],
  "maxConcurrency": 3,
  "navigationTimeout": 60,
  "pageLoadDelay": 5000,
  "proxyConfiguration": {
    "useApifyProxy": true,
    "apifyProxyGroups": ["RESIDENTIAL"]
  }
}

Parameters

• domains (required) - Array of domain names to analyze (e.g., ["example.com"])
• maxConcurrency (optional) - Number of pages to scrape simultaneously (default: 3, max: 10)
• navigationTimeout (optional) - Maximum time to wait for page navigation in seconds (default: 60, max: 300). Increase if you get timeout errors.
• pageLoadDelay (optional) - Additional wait time for dynamic content in milliseconds (default: 5000, max: 15000)
• proxyConfiguration (optional) - Proxy settings (residential proxies recommended)

Output

{
  "domain": "make.com",
  "scrapedAt": "2025-11-10T20:43:27.750Z",
  "domain_authority": 55,
  "monthly_visits": 8300000,
  "bounce_rate": 37.22,
  "pages_per_visit": 6.68,
  "avg_visit_duration_seconds": 770,
  "traffic_change_percent": -2.91,
  "organic_search_visits": 657960,
  "paid_search_visits": 123320,
  "direct_visits_percent": 79.72,
  "desktop_traffic_percent": 76.92,
  "mobile_traffic_percent": 23.08,
  "global_rank": 6757,
  "country_rank": 7264,
  "country": "United States",
  "industry_category": "Computer Software and Development",
  "total_backlinks": 2370000,
  "referring_domains": 26590,
  "competitors": [
    {"domain": "n8n.io"},
    {"domain": "zapier.com"},
    {"domain": "pipedream.com"}
  ],
  "organic_keywords": [
    {
      "keyword": "make",
      "position": 1,
      "search_volume": 74000
    }
  ],
  "top_countries": [
    {
      "country": "United States",
      "traffic_percent": 16.96,
      "visits": 1410000
    }
  ]
}

Use Cases

1. Competitor Analysis

Monitor your competitors' traffic, keywords, and backlink profiles to identify opportunities.

2. SEO Monitoring

Track domain authority, organic traffic, and keyword rankings over time.

3. Market Research

Discover industry leaders, traffic patterns, and top-performing keywords in your niche.

4. Lead Generation

Identify high-traffic websites in your target industry for partnership opportunities.

5. Portfolio Tracking

Monitor multiple websites' performance metrics from a single dashboard.

Performance

• Speed: ~35-40 seconds per domain
• Success rate: 95%+ on most domains
• Concurrency: Process up to 10 domains simultaneously
• Data freshness: SEMrush updates monthly (scrapes latest available data)

Notes

• No SEMrush account required - Scrapes publicly available data
• Residential proxies recommended for high-volume scraping
• Rate limiting: Respect SEMrush's infrastructure (max 10 concurrent requests)
• Some metrics may be unavailable for low-traffic domains
• Data is as fresh as SEMrush's latest update (typically monthly)

Technical Details

• Runtime: Node.js with Playwright
• Browser: Chromium (headless mode)
• Rendering: Full JavaScript execution for dynamic content
• Extraction: Regex pattern matching on rendered page text
• Proxy support: Apify Proxy (Residential, Datacenter) or custom proxies

Support

Having issues? Check these common solutions:

1. Timeout errors: Increase the navigationTimeout parameter (e.g., 120 seconds) or reduce maxConcurrency
2. Slow loading sites: Increase the pageLoadDelay parameter (e.g., 8000ms) to allow more time for dynamic content
3. Missing data: Some domains may have limited public data on SEMrush
4. Proxy blocks: Switch to residential proxies for better reliability
5. Rate limiting: Reduce maxConcurrency (e.g., set to 1 or 2) to slow down request rate

For bugs or feature requests, please contact support.

Example Integrations

Zapier/Make.com

Trigger analysis when competitors launch new pages or campaigns.

Google Sheets

Export data automatically to spreadsheets for reporting.

Slack/Discord

Get alerts when competitors' traffic changes significantly.

Custom Dashboards

Feed data into Tableau, Power BI, or custom analytics platforms.

---

No SEMrush subscription required. Get started analyzing domains today!