Semrush Domain Overview Scraper

Returns: one row per domain with authorityScore, backlinks, referringDomains, organicTraffic, monthlyVisits, globalRank, trafficSources and aiTraffic. Input: {"domains": ["github.com", "bbc.com"], "maxItems": 10} Best for: competitor benchmarking, prospect list enrichment, monthly SEO and traffic monitoring.

{
    "id": "bbc.com",
    "domain": "bbc.com",
    "authorityScore": 100,
    "referringDomains": 859231,
    "monthlyVisits": 354237748,
    "categoryRank": 9,
    "googlePenaltyRisk": "Low",
    "monthlyVisitsChangePercent": -0.1236
}

What does Semrush Domain Overview Scraper do?

Semrush Domain Overview Scraper reads the public Semrush website overview page for each domain you supply and returns the numbers shown there as clean, structured records. You give it a list of domains or URLs, and for every domain it returns the authority score, backlinks, referring domains, estimated organic and paid search traffic, top organic keywords, organic competitors, traffic by country, the desktop and mobile split, and the six-month history of backlinks and traffic.

It is built for SEO teams, agencies, analysts and sales researchers who want a quick benchmark of many websites without opening each page by hand. Typical uses include comparing a client against competitors, enriching a list of prospect domains with authority and traffic figures, tracking how well-known sites change month to month, and feeding domain metrics into spreadsheets or dashboards.

The Actor only uses pages that anyone can open without signing in. It does not use the paid Semrush API and it does not log in to anything. Because of that, the data is the limited public view: a handful of top keywords, a few competitors and six months of history rather than the full lists you see in a paid account. Only well-known domains have a public overview page. For smaller sites Semrush returns no page, and the Actor skips them with a warning instead of producing an empty record.

Each record carries the exact page it came from in the sourceUrl field and a stable id, which is the normalized domain. Duplicated domains in your input are collapsed, so you never pay twice for the same one.

What data can you get?

Every record contains the fields below. Fields the page does not show for a particular domain are returned as null.

Field	Type	Description
id	string	Stable record id, the normalized domain
domain	string	Domain the overview describes
sourceUrl	string	Semrush overview page the data came from
brandName	string	Brand name shown by Semrush
categories	array	Category names Semrush assigns to the site
globalRank	integer	Global traffic rank
countryRank	integer	Traffic rank in the site's top country
authorityScore	integer	Authority Score from 0 to 100
backlinks	integer	Total backlinks
referringDomains	integer	Number of referring domains
organicTraffic	integer	Estimated monthly organic search traffic
paidTraffic	integer	Estimated monthly paid search traffic
monthlyVisits	integer	Estimated monthly visits
pagesPerVisit	number	Average pages per visit
timeOnSiteSeconds	number	Average time on site in seconds
bounceRate	number	Bounce rate as a fraction between 0 and 1
backlinksHistory	array	Monthly backlinks, newest first
trafficHistory	array	Monthly organic, paid and total traffic
topKeywords	array	Top organic keywords with position, search volume, CPC and traffic share
competitors	array	Organic competitors with name, relevance and visits
trafficByCountry	array	Top countries with traffic, share, desktop share and mobile share
trafficByDevice	object	Latest desktop and mobile visits and shares, plus monthly history
dataDate	string	Date of the Semrush data snapshot
countryCode	string	Country the Semrush data refers to
categoryRank	integer	Rank within the main category
categoryRankName	string	Category the category rank refers to
googlePenaltyRisk	string	Backlink Google penalty risk level shown by Semrush
backlinksChangePercent	number	Month-over-month backlinks change as a fraction
referringDomainsChangePercent	number	Month-over-month referring domains change as a fraction
organicTrafficChangePercent	number	Month-over-month organic traffic change as a fraction
paidTrafficChangePercent	number	Month-over-month paid traffic change as a fraction
monthlyVisitsChangePercent	number	Month-over-month visits change as a fraction
referringDomainsHistory	array	Monthly referring domain counts
visitsHistory	array	Monthly total visits
trafficSources	array	Where visitors come from, with host, source, channel, share, visits and previous visits
outgoingTraffic	array	Sites visitors go to next, with host, share and visits
aiTraffic	array	Visits from AI assistants such as ChatGPT and Claude, with share and history
favicon	string	Favicon URL listed by Semrush

How to use Semrush Domain Overview Scraper

1. Open the Actor in Apify Console and go to the Input tab.
2. Paste the domains you want into the Domains list. You can use bare domains such as amazon.com or full URLs such as https://www.github.com/. The www. prefix and any path are removed for you.
3. Set Max items to the number of records you want. The Actor stops as soon as that many records have been returned.
4. Leave the proxy setting on its default unless you have a reason to change it. Requests go out directly, and if the site answers with a block or rate limit, the Actor retries with backoff on a fresh Apify Proxy session. Challenge detection checks the page title and short visible text, ignoring scripts and styles. Retries for each target are bounded, and no backoff is added after the final attempt.
5. Click Start. When the run finishes, open the Output tab to browse the table, or download the data as JSON, CSV or Excel.

The Actor makes about one request per second, with two requests in flight at most, so a list of 50 domains takes roughly a minute. That pace is deliberate: it keeps the load on the target site small and makes blocks unlikely.

If a domain has no public overview page, the log shows a warning with the reason and the run carries on with the next domain. If none of your domains has data, or every supplied target is skipped as an unrecognized domain, the run finishes successfully with zero items and a status message suggesting well-known domains. Unexpected errors are caught separately for each domain, so other targets can still return data. If the input is not valid at all, for example an empty domain list, the run fails right away with a clear message.

Input

Field	Type	Required	Description
domains	array of strings	Yes	Domains or URLs to look up, for example ["amazon.com", "github.com"]
maxItems	integer	No	Maximum number of records to return, from 1 to 100000. Default 50
proxyConfiguration	object	No	Proxy settings. Direct by default, with an automatic Apify Proxy fallback when blocked

Example input:

{
    "domains": ["amazon.com", "github.com", "bbc.com"],
    "maxItems": 3
}

Output

Each domain becomes one item in the dataset. A shortened example record:

{
    "id": "github.com",
    "domain": "github.com",
    "sourceUrl": "https://www.semrush.com/website/github.com/overview/",
    "brandName": "GitHub",
    "globalRank": 28,
    "authorityScore": 100,
    "backlinks": 5335220083,
    "referringDomains": 4115198,
    "organicTraffic": 70689927,
    "paidTraffic": 24372,
    "monthlyVisits": 1212551945,
    "backlinksHistory": [
        { "displayDate": "2026-08-01", "backlinks": 5335220083 },
        { "displayDate": "2026-07-01", "backlinks": 7724668284 }
    ],
    "topKeywords": [
        { "keyword": "github", "intents": "2", "position": 1, "searchVolume": 1220000, "cpc": 3.29, "traffic": 0.0547 }
    ],
    "competitors": [
        { "name": "stackoverflow.com", "slug": "stackoverflow.com", "relevance": 0.23, "visits": 25915762 }
    ],
    "dataDate": "2026-08-15",
    "categoryRank": 1,
    "googlePenaltyRisk": "Low",
    "monthlyVisitsChangePercent": 0.0648,
    "trafficSources": [
        { "host": "Direct", "share": 0.619, "visits": 574779767, "previousVisits": 538970535 }
    ],
    "aiTraffic": [
        { "id": "chatgpt.com", "name": "ChatGPT", "visits": 7665006, "share": 0.00678 }
    ]
}

Fields added in the latest version (category rank, penalty risk, month-over-month changes, traffic sources, outgoing traffic and AI traffic) are null when the page does not show them for a domain.

The numbers are the estimates Semrush shows publicly. They are not exact counts, and they change every month.

How much does it cost?

The Actor is charged per result: you pay for each domain record that is returned, and nothing for domains that have no data or for runs that fail. See the Pricing tab of the Actor page for the current rate. Apify's free plan includes monthly platform credit, which is enough to try the Actor on a few dozen domains.

Because charging is per result, Max items is also your spending cap. Set it to the number of domains you actually need.

Use with AI agents (MCP)

You can call this Actor from any MCP-compatible assistant through the Apify MCP server. Add the following to your client configuration:

{
    "mcpServers": {
        "apify": {
            "url": "https://mcp.apify.com?tools=axiomworks/semrush-domain-overview-scraper"
        }
    }
}

Example prompts you can give your assistant:

• "Get the Semrush authority score and referring domains for stackoverflow.com and shopify.com."
• "Compare organic traffic for walmart.com and target.com and list their top keywords."
• "For these ten domains, return the competitors Semrush lists and the monthly backlinks history."

Integrations and API

You can run the Actor from code with the Apify API clients.

Python

from apify_client import ApifyClient

client = ApifyClient("<YOUR_APIFY_TOKEN>")
run = client.actor("axiomworks/semrush-domain-overview-scraper").call(
    run_input={"domains": ["amazon.com", "github.com"], "maxItems": 2}
)
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item["domain"], item["authorityScore"], item["backlinks"])

JavaScript

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: '<YOUR_APIFY_TOKEN>' });
const run = await client.actor('axiomworks/semrush-domain-overview-scraper').call({
    domains: ['amazon.com', 'github.com'],
    maxItems: 2,
});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => process.stdout.write(`${item.domain} ${item.authorityScore}\n`));

cURL

curl -X POST "https://api.apify.com/v2/acts/axiomworks~semrush-domain-overview-scraper/run-sync-get-dataset-items" \
  -H "Authorization: Bearer $APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"domains": ["amazon.com", "github.com"], "maxItems": 2}'

Apify also lets you schedule runs, send results to Google Sheets, Slack or a webhook, and connect the dataset to tools such as Zapier or Make. This is useful for a monthly refresh of a fixed list of domains.

FAQ

Why did a domain return no record? Semrush publishes overview pages only for sites with enough data. Smaller or very new sites return a not-found page. The Actor logs a warning and moves on, and you are not charged for it.

Why are there only a few keywords and competitors? The public page shows a short preview: the top keywords, three competitors and six months of history. The full lists need a paid Semrush account, which this Actor does not use.

How fresh is the data? Semrush updates the public pages about once a month. The dataDate field tells you the date of the snapshot a record was built from.

Does it need a proxy? Usually not. Requests go out directly. If the site answers with a 403 or 429, the Actor retries with backoff and falls back to Apify Proxy on a fresh session.

Can I pass URLs instead of domains? Yes. The Actor keeps only the host name, drops www. and removes duplicates.

Does the Actor return the full backlink list? No. It returns the summary figures shown on the public overview page, such as total backlinks, referring domains and authority score.

How many domains can I run at once? Up to the Max items limit. The Actor processes about one domain per second, so large lists simply take longer.

Is it legal to scrape Semrush?

The Actor reads only pages that are open to everyone, with no login and no access to paid or private data. It makes a small number of polite requests and does not collect personal data. Whether scraping is appropriate in your case still depends on your jurisdiction and your use of the data. Website terms of service may restrict automated access, so review them and take legal advice where you need it. This Actor is an independent tool and is not affiliated with or endorsed by Semrush.

Feedback

If a field is missing, a domain you expected is skipped or the page layout changes and the data stops parsing, open an issue from the Actor page's Issues tab with the domain and the date of the run. Bug reports with a concrete example are fixed fastest.