Semrush Scraper: Authority Score, Backlinks & Traffic

What does Semrush Scraper do?

It turns a domain into Semrush's Authority Score, backlink count, referring domains, and an estimated monthly organic traffic figure, the same numbers Semrush's own free Website Authority Checker tool shows a visitor for free. Give it hubspot.com and you get a structured JSON record: the score, twelve months of trend history, the domain's top pages by backlinks, and its top competing domains with their own scores.

No Semrush account, no login, no paid Semrush Data API subscription. This reads exactly what Semrush's own public, unauthenticated marketing page shows any visitor, and turns it into a dataset you can export, join, or feed into an agent.

✅ No Semrush account | ✅ No login | ✅ No Semrush subscription | ✅ MCP-ready for AI agents

Guide and FAQs: Semrush Authority Score Scraper on themineworks.com.

Coverage: read this before buying

This is important, so it is stated plainly: this actor reads Semrush's free preview tool, not the paid Semrush platform. It cannot return anything from Semrush's paid dashboard, Data API, or Pro, Guru, or Business plans. Full keyword lists, complete backlink exports, position tracking, keyword-gap analysis, and traffic-by-country breakdowns all live behind a Semrush subscription and are genuinely out of scope here. If you need the full Semrush suite, you need a Semrush account.

What you get instead is exactly what semrush.com/free-tools/website-authority-checker/ shows an anonymous visitor, structured as JSON:

• Authority Score (1 to 100) and its recent trend
• Total backlinks and referring domains, each with a recent trend
• An estimated monthly organic traffic figure
• Up to 13 months of historical Authority Score, backlink, and referring-domain data points
• Up to 5 pages on the domain with the most raw backlinks, and up to 5 with the most unique referring domains
• Up to 5 competing domains, each with its own Authority Score and a shared-keyword count
• A link-power, natural-profile, and organic-traffic sub-score with a qualitative health label

There is no separate organic-keyword-count field for the target domain itself. Semrush's free tool does not expose one. If your workflow needs the actual keyword list, keyword volumes, or SERP positions, this is the wrong tool; those require a Semrush subscription.

Does this log in to a real Semrush account?

No. There is no Semrush username, password, or account session anywhere in this actor. What it does use is its own anonymous browser session, a fresh cookie jar and a fresh residential exit IP, created for each domain and thrown away right after, purely so the request looks like an ordinary visitor rather than a shared script. That session cookie belongs to nobody's Semrush account; it is issued anonymously by the page itself, the same way a cookie is issued to any first-time visitor who has never signed up for anything. Plain HTTP requests with a copied cookie are rejected outright, so the browser session exists to satisfy the page's bot check, not to hold anyone's login.

How does it read a protected free tool without an account?

Semrush's free Website Authority Checker is served from a public, no-login marketing page, but the JSON endpoint behind it is not casually scriptable. Plain HTTP requests, curl, a bare fetch, got, are rejected outright with 403 Access denied, even with realistic headers and a warm cookie, so a real browser is required, full stop. A real browser alone is not enough either: the endpoint also requires a live Google reCAPTCHA v3 token, and Google scores every request 0 to 1 for risk before Semrush returns data. That score leans heavily on network-level signals (IP reputation) as well as in-browser behavior.

So this actor drives a real, headed Chromium browser (Crawlee's PlaywrightCrawler) over an Apify residential proxy. For each domain it opens the tool page in a fresh session, types the domain into the real input field, clicks the real "Check Authority" button with human-paced timing, and reads the result back through the browser's own network listener, the same interaction a person would perform, not a re-implemented API call.

The free tool is also rate-limited to 3 checks per day per source IP. A fresh cookie jar from the same IP does not reset that limit; only a new IP does. That is why this actor requests a brand-new residential proxy session, and therefore a fresh exit IP, for every single domain rather than reusing one session across a batch, so a run checking 20 domains is not racing to exhaust one IP's daily allowance.

What is the known reliability risk?

Read this before running a large batch. Semrush protects this endpoint with Google reCAPTCHA v3, which scores every request on a continuous, mostly IP-reputation-driven risk model rather than a one-time puzzle. In live testing, requests routed through proxy infrastructure, both residential sessions and direct egress, have scored in the "suspicious" range and been rejected, across independently configured attempts covering different stealth settings, memory tiers, proxy types, and interaction timing. The same flow succeeds reliably from a browser on a clean, non-shared-proxy network. This is stated plainly rather than claiming a reliability that has not been observed: treat this as an actor whose success rate depends on the proxy IP's reputation at the moment you run it, not a guaranteed-to-work scraper. Blocked and rejected checks are never charged, so a low-scoring run costs nothing beyond the run itself, but budget for the possibility that a given run delivers fewer domains than requested.

How much does it cost to check a domain's Authority Score?

Pay per domain actually delivered with real data. The price is tiered by your Apify plan.

Apify plan	Price per domain	Price per 1,000
Free	$0.0150	$15.00
Bronze	$0.0126	$12.60
Silver	$0.01065	$10.65
Gold and above	$0.0090	$9.00

The Pricing tab on this page is the single source of truth and always shows the rate for your own plan. If this table and the Pricing tab ever disagree, the Pricing tab is right.

What a real job costs. Apify's Free plan includes $5 of usage credit every month, roughly 333 domains checked at no cost to you, budget permitting for the reliability risk above. A 100-domain competitor sweep costs about $1.50 on Free, or about $0.90 on Gold.

What is never charged. A blocked, rate-limited, or failed check costs nothing. The charge event fires only once real Authority Score data is in your dataset.

🧾 What input does it take?

{
  "domains": ["hubspot.com", "salesforce.com"],
  "maxDomains": 20,
  "proxyConfiguration": {
    "useApifyProxy": true,
    "apifyProxyGroups": ["RESIDENTIAL"]
  }
}

Input	Required	What it does
domains	Yes	One or more domains to check. Protocol and www. are stripped automatically.
maxDomains	No	Safety cap on how many domains from the list are actually checked in one run. Default 20, maximum 100.
proxyConfiguration	No	RESIDENTIAL proxy is required, the target blocks non-browser traffic and rate-limits per IP. Defaults to RESIDENTIAL.

📤 What data do you get back?

This is a real record, captured from a live, successful run against hubspot.com:

[
  {
    "domain": "hubspot.com",
    "data_as_of": "2026-08-01",
    "authority_score": 79,
    "authority_score_change_pct": 0,
    "backlinks": 79228382,
    "backlinks_change_pct": -7.24,
    "referring_domains": 421085,
    "referring_domains_change_pct": -3.22,
    "organic_traffic_monthly": 248376,
    "organic_traffic_change_pct": 73.58,
    "health_label": "Very good",
    "link_power_score": 6.99,
    "natural_profile_score": 6,
    "organic_traffic_score": 5.68,
    "top_pages_by_backlinks": [
      { "url": "https://meetings.hubspot.com/andrea873", "referring_domains": 3, "backlinks": 1660245 },
      { "url": "https://community.hubspot.com/t5/user/viewprofilepage/user-id/1027138", "referring_domains": 187, "backlinks": 1456533 },
      { "url": "https://meetings.hubspot.com/linda67", "referring_domains": 1, "backlinks": 1359173 }
    ],
    "most_linked_pages": [
      { "url": "https://www.hubspot.com/", "referring_domains": 66453, "backlinks": 849049 },
      { "url": "https://www.hubspot.com/marketing-statistics", "referring_domains": 20323, "backlinks": 174048 },
      { "url": "https://legal.hubspot.com/privacy-policy", "referring_domains": 18866, "backlinks": 699633 }
    ],
    "top_competitors": [
      { "domain": "salesforce.com", "authority_score": 84, "common_keywords": 25208 },
      { "domain": "mailchimp.com", "authority_score": 76, "common_keywords": 14625 },
      { "domain": "zapier.com", "authority_score": 76, "common_keywords": 14405 }
    ],
    "authority_score_history": [
      { "date": "2026-08-01", "value": 79 },
      { "date": "2026-07-01", "value": 79 },
      { "date": "2026-06-01", "value": 77 },
      { "date": "2026-05-01", "value": 77 },
      { "date": "2026-04-01", "value": 76 },
      { "date": "2026-03-01", "value": 76 },
      { "date": "2026-02-01", "value": 76 }
    ],
    "checked_at": "2026-08-19T19:31:36.017Z"
  }
]

One full record is shown here rather than padding the sample with domains this actor could not currently verify live, given the reliability risk documented above; every field below is genuinely returned by the tool.

Field	Description
🌐 domain	The domain that was checked
📅 data_as_of	Date of Semrush's underlying data snapshot
🏆 authority_score	Authority Score, 1 to 100
📈 authority_score_change_pct	Change vs. the prior period, as a percentage
🔗 backlinks	Total backlinks pointing to the domain
📈 backlinks_change_pct	Backlink count change vs. the prior period
🌍 referring_domains	Number of unique referring domains
📈 referring_domains_change_pct	Referring domain count change vs. the prior period
🚦 organic_traffic_monthly	Estimated monthly organic search traffic
📈 organic_traffic_change_pct	Organic traffic change vs. the prior period
🏷️ health_label	Semrush's qualitative label for the backlink profile's health
💪 link_power_score	Sub-score: strength and quality of the backlink profile
🌱 natural_profile_score	Sub-score: how natural, non-manipulated, the backlink profile looks
🚦 organic_traffic_score	Sub-score: organic traffic's contribution to the Authority Score
📄 top_pages_by_backlinks	Up to 5 pages with the most raw backlinks
📄 most_linked_pages	Up to 5 pages with the most unique referring domains
⚔️ top_competitors	Up to 5 competing domains with their own Authority Score and shared-keyword count
📊 authority_score_history	Up to 13 months of historical Authority Score snapshots
🕒 checked_at	ISO timestamp when this record was captured

A field the tool does not return for a given domain is omitted rather than sent as null, so downstream parsing stays clean.

What are the limitations?

Worth knowing before you buy, so there are no surprises:

• Success rate depends on proxy IP reputation, not just your input. See "What is the known reliability risk?" above. A blocked check is never charged, but budget for a run that delivers fewer domains than requested.
• This is the free tool, not the Semrush platform. No full keyword list, no keyword-gap analysis, no SERP position tracking, no traffic-by-country breakdown. See "Coverage" above.
• 3 checks per source IP per day. This actor works around the limit with a fresh residential exit IP per domain, but the underlying quota is Semrush's, not something this actor can raise.
• Authority Score refreshes roughly every two weeks. Running this actor hourly or daily against the same domain reads the same underlying snapshot; weekly or monthly is the cadence that matches how often the number actually moves.

💼 What can you use it for?

Quick competitive snapshots. Pull Authority Score, backlinks, and traffic estimates for a shortlist of competitors before deciding which one deserves a deeper, paid audit.

Prospect and vendor screening. Before a guest-post pitch, link exchange, partnership, or vendor evaluation, get an instant read on a domain's authority and backlink health.

Lightweight SEO monitoring. Track your own domain's Authority Score and backlink trend over time without paying for a full SEO suite seat.

Enrichment for lead and account lists. Attach a domain-authority signal to a list of company websites as a cheap qualification layer.

🚀 How do I get started?

1. Open the actor and enter one or more domains in domains.
2. Set maxDomains to control cost, default 20.
3. Leave proxyConfiguration on RESIDENTIAL, it is required for this target.
4. Click Start.
5. Download as JSON, CSV, or Excel, or pull the dataset via API or MCP.

🔁 Can I run it on a schedule?

Yes, with Apify's built-in Schedules. No code and no cron server of your own. Authority Score and backlink counts move slowly, and Semrush refreshes them roughly every two weeks, so weekly or monthly is the cadence that makes sense here.

1. Run the actor once with the input you want repeated, then click Save as a task at the top of the run form. This keeps your exact input attached for every future run.
2. In the Apify Console, go to Schedules in the left sidebar, then Create new.
3. Name it, set your timezone, and pick a frequency: a preset (hourly, daily, weekly) or a cron expression.
4. Under Actors or tasks to run, add the task you saved in step 1.
5. Save. It then runs unattended, billed the same pay-per-domain way as a manual run. Nothing is charged just for the schedule existing.

Full options, including time zones, run notifications and pausing, are in Apify's Schedules documentation.

Is reading Semrush's free tool this way legal?

This actor reads only a public, unauthenticated page that Semrush publishes for any visitor to use for free. It does not log in to Semrush, does not hold or use anyone's Semrush account credentials, and does not access anything behind Semrush's paid dashboard or API.

That said, this is general information and not legal advice. You remain responsible for how you use the data, including your obligations under GDPR, CCPA and similar laws if you join this data to personal data elsewhere in your pipeline, and for complying with Semrush's own terms of service for its free tool.

FAQ

Do I need a Semrush account or API key? No. This actor reads Semrush's public, unauthenticated Website Authority Checker, the same tool anyone can use at semrush.com/free-tools/website-authority-checker/ without signing up.

Does the actor log in to Semrush? No. See "Does this log in to a real Semrush account?" above. It uses its own anonymous, disposable browser session, not a Semrush user login.

Is this the same as Semrush's full Domain Overview or Data API? No. Semrush's main analytics dashboard requires an account for any unauthenticated visitor, it shows a "Register to get 10 free requests" wall with no data behind it. This actor does not touch that surface or Semrush's paid Data API. It only returns what the free Website Authority Checker exposes.

Why does it need a residential proxy? The free tool's API blocks plain HTTP outright and rate-limits to 3 checks per day per source IP. A residential proxy with a fresh session per domain is what makes checking more than 3 domains in one run possible at all.

Why might a run deliver fewer domains than I requested? Google's reCAPTCHA v3 check scores every request for risk, and a low score gets rejected before any data returns. See "What is the known reliability risk?" above. A rejected check is never charged.

How often does the underlying data update? Semrush states it refreshes Authority Scores for all domains roughly every two weeks, pulled from its own backlink and traffic databases.

Can I use Semrush Scraper through an MCP server? Yes. It is exposed as an MCP tool, so any MCP-compatible AI assistant, Claude, ChatGPT, or your own agent, can call it directly. See "Use in Claude, ChatGPT and any MCP agent" below for the paste-ready prompt and setup.

🤖 Use in Claude, ChatGPT and any MCP agent

Hosted MCP endpoint, no install, OAuth on first connect:

https://mcp.apify.com/?tools=themineworks/semrush-scraper

Claude Desktop / Cursor config (token auth):

{
  "mcpServers": {
    "semrush-scraper": {
      "url": "https://mcp.apify.com/?tools=themineworks/semrush-scraper",
      "headers": { "Authorization": "Bearer YOUR_APIFY_TOKEN" }
    }
  }
}

Ask your agent: "What is the Authority Score and backlink count for hubspot.com?"

Copy this into your AI assistant

Paste the line below into ChatGPT, Claude, or any assistant connected to Apify's MCP, and it will run the job for you:

Use the Apify actor themineworks/semrush-scraper to check the Authority Score, backlinks, and referring domains for hubspot.com and salesforce.com. Return the results as a table.

Or call it programmatically with the Apify client:

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: 'YOUR_APIFY_TOKEN' });

const run = await client.actor('themineworks/semrush-scraper').call({
  domains: ['hubspot.com', 'salesforce.com'],
  maxDomains: 20,
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items);

🛠️ Related actors

Pair this with the rest of the themineworks competitive-intelligence suite for a fuller picture beyond one free-tool snapshot:

• Similarweb Scraper: traffic estimates, global and country rank, and traffic sources for the same domain.
• Google Ads Transparency Scraper: every ad a competitor is currently running.
• Backlink Building Agent: guest-post and resource-link prospects, to build the backlink profile this actor scores.
• Crunchbase Scraper: company funding, investors, and firmographics.
• AmbitionBox Scraper: company ratings, reviews, and salary data.

Typical flow: use this actor for a fast, free authority and traffic read on a shortlist of domains, then move only the domains worth deeper research into a full paid SEO tool.

Found a bug or have a feature request? Open an issue on the actor's Apify Console page.