Semrush Domain Overview Scraper avatar

Semrush Domain Overview Scraper

Pricing

from $0.05 / 1,000 item extracteds

Go to Apify Store
Semrush Domain Overview Scraper

Semrush Domain Overview Scraper

Scrape public Semrush domain overview metrics for traffic, rank, authority, backlinks, keywords, countries, and competitor monitoring.

Pricing

from $0.05 / 1,000 item extracteds

Rating

0.0

(0)

Developer

Fetch Cat

Fetch Cat

Maintained by Community

Actor stats

0

Bookmarked

13

Total users

6

Monthly active users

3 days ago

Last modified

Features

Limited permissions

Share

Collect public Semrush domain overview snapshots for traffic, rankings, authority and backlink research. Give Semrush Domain Overview Scraper a list of domains, then export the available summary fields as JSON, CSV or Excel. Coverage varies by domain, and missing metrics stay null. This is not a Semrush account export or its official API.

Quick start

  1. Open Input and add one or more domains or URLs.
  2. Set a small maximum, then click Start.
  3. Open the dataset and review each row’s status before comparing metrics.
{
"domains": ["openai.com", "github.com", "stripe.com"],
"maxItems": 3
}

Prefer a video guide? This short Apify walkthrough shows you how to set up and run an Actor. It uses a different Actor, but the setup steps are the same.

What you get

  • Domain snapshots: available traffic, rank, authority, referring-domain and backlink summaries.
  • Keywords and traffic sources: the top organic keywords with position, search volume and CPC, where visitors come from and go next, and how many visits AI assistants such as ChatGPT send.
  • Recent trends: monthly visits (desktop and mobile), organic and paid search traffic, backlinks and referring domains for recent months.
  • Context: reporting period, page metadata, country/category breakdowns, Google penalty risk and competitor summaries with their visits when present.
  • Status per domain: distinguish useful metrics from unavailable pages, blocked requests and invalid inputs.
  • Structured exports: numeric metric objects with original values and change percentages when available.

Who is it for?

  • SEO teams collecting public domain snapshots for client reports.
  • Growth teams comparing the available summaries of competing websites.
  • Agencies and analysts building spreadsheets or dashboards from domain lists.

Input settings

Setting and API keyWhat it does
Domains or URLs (domains)Required list of 1–100 domains or URLs.
Maximum domains (maxItems)Limits unique valid domains, default 10, maximum 100.
Maximum concurrency (maxConcurrency)Accepted legacy field. Processing is sequential and this value has no effect.
Proxy configuration (proxyConfiguration)Existing proxy settings. See field help for defaults and options.
Include raw HTML snippet (includeRawHtmlSnippet)Adds the first 2,000 characters of the response to fetched-page rows. Default false.
Save debug artifacts (saveDebugArtifacts)Saves the run counters in RUN-SUMMARY. Default false.

Domains are lowercased, a leading www. is removed and duplicate normalized domains are processed once. Invalid inputs may add diagnostic rows beyond maxItems.

Input recipes

For a competitor snapshot, use:

{"domains":["notion.so","clickup.com","asana.com","monday.com"],"maxItems":4}

Output data

The following is a historical successful output excerpt, not a current measurement or a promise of field coverage.

{
"input": "openai.com",
"domain": "openai.com",
"url": "https://www.semrush.com/website/openai.com/overview/",
"status": "success",
"reportingPeriod": "2026-06-01",
"visits": { "raw": "508002719", "value": 508002719, "diffPercent": -0.0945 },
"authorityScore": { "raw": "99", "value": 99, "diffPercent": null },
"referringDomains": { "raw": "423708", "value": 423708, "diffPercent": 0.0496 },
"backlinks": { "raw": "64639384", "value": 64639384, "diffPercent": 0.015 },
"topCountries": [{ "country": "United States", "share": 0.1397, "visits": 70981896, "desktopShare": 0.6019, "mobileShare": 0.3981 }],
"googlePenaltyRisk": "Low",
"topOrganicKeywords": [{ "keyword": "stripe", "position": 1, "searchVolume": 673000, "cpc": 2.89, "trafficShare": 0.3147, "intents": "1" }],
"trafficSources": [{ "source": "Google organic", "domain": "google.com", "share": 0.0319, "visits": 2660087, "previousVisits": 2611151 }],
"aiAssistantTraffic": [{ "source": "chatgpt.com", "name": "ChatGPT", "visits": 703479, "share": 0.0081, "history": [{ "date": "2026-10-01", "visits": 703479 }] }],
"searchTrafficHistory": [{ "date": "2026-10-15", "organic": 4972550, "paid": 98486, "total": 5071036 }],
"error": null
}
FieldDescription
input, domain, urlOriginal input, normalized domain, and public Semrush overview URL.
status, errorsuccess, no_public_metrics, not_found, blocked, failed, or invalid_input.
fetchedAt, reportingPeriodCollection time and visible Semrush reporting period when available.
title, descriptionPublic page metadata.
globalRank, countryRank, categoryRankRanking metrics when present.
countryRankName, categoryRankNameCountry and category the ranks refer to.
visits, pagesPerVisit, avgVisitDuration, bounceRateTraffic and engagement summaries.
organicSearchTraffic, paidSearchTraffic, organicKeywordsPublic search summaries. organicKeywords counts available keyword-series entries, not all keywords ranking for the domain.
topOrganicKeywords, keywordsDatabaseUp to 10 top organic keywords shown on the overview (keyword, Google position, monthly search volume, CPC in USD, share of organic traffic, Semrush intent codes) and the regional database they come from.
authorityScore, referringDomains, backlinksAuthority and backlink summary counts.
googlePenaltyRiskSemrush penalty-risk label for the backlink profile, for example Low.
trafficHistoryRecent monthly visits with desktop and mobile visits.
searchTrafficHistoryRecent monthly organic, paid and total search traffic.
backlinksHistoryRecent monthly backlinks and referring-domain counts.
trafficSources, trafficDestinationsUp to 10 top sources of visits (Direct, Google organic, referring sites) and up to 10 sites visitors go to next, with share, visits and previous-period visits.
aiAssistantTrafficUp to 10 AI assistants and AI sites sending visits, such as ChatGPT or Claude, with visits, share and monthly history.
topCountries, topCategories, competitorsArrays of public breakdowns when present. Countries include desktop and mobile shares; competitors include their estimated visits.

Metric objects include raw, normalized numeric value when reliable, and diffPercent when Semrush exposes a change percentage.

organicKeywords is the number of available keyword-series entries, not a complete keyword inventory. Device-share metrics are nullable. Optional rawHtmlSnippet contains response text, not a complete saved page. OUTPUT in the key-value store contains dataset and run counters. If the run reaches its time budget, PENDING_DOMAINS lists unprocessed domains.

Download your results

Open your run's dataset, choose Export and select JSON, CSV or Excel. JSON keeps nested fields together. CSV and Excel are useful when you want to review the records in a spreadsheet.

See Apify's download and export guide for the available formats and field options.

Pricing

This Actor uses pay-per-event pricing. The start event is a one-time run charge.

The item event charges for each successful metric-bearing domain overview record. Diagnostic rows with invalid_input, not_found, blocked, failed or no_public_metrics are not charged as successful results. A start charge can still apply when no useful metrics are returned.

See the live Pricing tab for current rates and discounts.

Tips for best results

  • Start with a few domains. Check coverage before submitting a larger list.
  • Compare the reporting periods. Public summaries may refer to different months.
  • Keep status columns in exports. Null metrics and diagnostic records are not zero traffic.
  • Use bare domains where possible. URL paths are discarded during normalization.

Limits and caveats

  • Public overview summaries only. No login, private dashboards, full backlink lists, full keyword exports, regional database selection or SEO audit mode. Keyword, source and AI-traffic lists hold what the public overview shows (often five entries).
  • Availability depends on Semrush. Pages can be missing, blocked or lack usable metrics.
  • maxConcurrency does not change execution speed. This Actor processes domains sequentially.
  • Per-domain errors are saved as status rows and processing continues. Invalid top-level input or infrastructure/storage failures can still fail a run.
  • topCountries, topCategories, competitors, topOrganicKeywords, trafficSources, trafficDestinations and aiAssistantTraffic each contain at most ten entries. Empty lists mean the overview did not show that section.
  • If the run's maximum cost or timeout is too low, the Actor stops early, keeps the rows already saved and explains the stop in its status message and the OUTPUT and RUN_SUMMARY records.
  • No guarantee of complete metrics, current-month data or agreement with a paid Semrush account.

Automation and integrations

Save your settings for next time

Once you have an input that works for you, save it as an Apify task. This keeps your settings together so you can use them again without filling in the form from scratch.

Run on a schedule

Need fresh results regularly? You can schedule the Actor or a saved task to run at a time that suits you.

Each scheduled run collects the data again. If you want to compare results or highlight changes, add that step in your own workflow.

Send results to your other tools

Use n8n, Make or Zapier to connect this scraper to the rest of your workflow. For example, you could pass its results to an AI assistant or add records to a spreadsheet. Follow the Apify integration guides for your chosen tool.

For a custom connection, an Apify webhook can notify your app when a run finishes or fails. Your app can then fetch the results or handle the failure.

API usage

Want to collect this data from your own app or automation? The examples below send input to the scraper and show how to collect its results.

The Node.js and Python examples wait for the run to finish before reading its results. The cURL example starts the run and returns its details. Wait for that run to finish, then use its defaultDatasetId to fetch the results.

For setup instructions, see Apify's JavaScript client guide, Python client guide and Actor runs API reference.

cURL

curl -X POST 'https://api.apify.com/v2/acts/fetch_cat~semrush-domain-overview-scraper/runs' \
-H "Authorization: Bearer $APIFY_TOKEN" \
-H 'Content-Type: application/json' \
-d '{"domains":["openai.com","github.com"],"maxItems":2}'

Node.js

import { ApifyClient } from 'apify-client';
const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('fetch_cat/semrush-domain-overview-scraper').call({
domains: ['openai.com', 'github.com'], maxItems: 2
});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items);

Python

import os
from apify_client import ApifyClient
client = ApifyClient(os.environ['APIFY_TOKEN'])
run = client.actor('fetch_cat/semrush-domain-overview-scraper').call(
run_input={'domains': ['openai.com', 'github.com'], 'maxItems': 2})
items = client.dataset(run['defaultDatasetId']).list_items().items
print(items)

MCP and AI agents

You can ask your AI assistant to collect data with Semrush Domain Overview Scraper and help you work with the results. The scraper returns the data, and your assistant can then summarize it, compare records or prepare notes.

Connect your assistant through the Apify MCP server. MCP gives your assistant a way to find this scraper, run it and read the results. Follow the official setup and sign-in guide for your app.

Easier setup with an Apify plugin

If you use a supported coding assistant, the official Apify plugin can set up the connection for you. It connects your assistant to Apify and includes instructions for common tasks. Follow the guide for your particular app.

Use Apify AI in Console

Prefer to keep everything in Apify? Open Apify AI in Console and ask it to find Fetch Cat's Semrush Domain Overview Scraper. You can describe the data you want and ask questions about the results.

Apify AI is currently in beta. It asks for confirmation before starting an Actor, and normal run charges apply.

Choose your assistant

Connect just this scraper

To keep your setup focused on this scraper, add this server configuration to a compatible MCP client:

{
"mcpServers": {
"apify": {
"url": "https://mcp.apify.com?tools=fetch_cat/semrush-domain-overview-scraper"
}
}
}

Sign in to Apify when your client asks. If your client uses a Bearer token instead, follow its setup guide and keep the token in its secure configuration. Configuration formats vary by app, so use the linked guide if your client needs a different format.

Use Apify's general server

Already connected to the general Apify MCP server at https://mcp.apify.com? You can use it to find this scraper without adding another server. Ask your assistant to use Semrush Domain Overview Scraper by Fetch Cat, or give it the identifier fetch_cat/semrush-domain-overview-scraper.

Your assistant can check the available settings and retrieve the results from the returned run. If the run is still working, it should follow that run rather than start it again. Both connection options use Apify's hosted service.

Example requests

  • “Get the public Semrush summaries for OpenAI and GitHub, then compare their available traffic figures.”
  • “Collect the available authority and backlink summaries for Notion, ClickUp and Asana and put the results in a comparison table.”
  • “Fetch the public overview for Stripe and explain which metrics are missing from the returned record.”

Normal Apify charges apply. Check the live Pricing tab before asking your assistant to run a larger batch.

Pay with x402

If your AI agent has a funded wallet, it can pay for Apify usage through x402. This experimental option lets an agent use Apify without a conventional account.

Apify AGI provides a prepaid API token with a spending cap after payment. Your agent can use that token to run this scraper through the Apify API or an authenticated MCP connection.

Some MCP setups also support paying for each request directly. That is a different payment option, so check the official guide for its availability.

Follow the official guide for wallet setup, supported payment options and current terms. Apify handles the payments. Keep wallet keys and payment tokens in your agent's secure configuration.

Other ways for agents to pay

Apify also supports MPP for prepaid access through AGI and Skyfire for eligible Actors. If your agent already uses one of these, see the agentic payment guide to choose the right setup.

FAQ

Do I need a Semrush account?

No. The scraper reads public overview pages without your Semrush credentials.

Does this replace the Semrush API?

It is an alternative for collecting available public summaries, not a replacement for paid account datasets or official API coverage.

Can I export the results or use an AI assistant?

Yes. Export the dataset or use the API and MCP instructions above. The assistant, not the scraper, performs any additional comparison or summarization.

Why are metrics missing?

Some domains lack public coverage, and pages can be unavailable or blocked. Inspect status, error and the reporting period. no_public_metrics means the response lacked the principal metrics this scraper uses to identify a useful overview.

What are the alternatives?

Use Semrush’s own tools for account-level reports. For public traffic comparisons from another source, see Similarweb Traffic Scraper below.

Support and feedback

For custom additions or integrations, email hello@fetchcat.dev.

Found an error or have a feature request? Open an issue with the run URL, a non-sensitive input example and what you expected.

Need help sharing it? See Apify’s guide to sharing a run when reporting an issue. Sharing a run can give the developer access to its input, logs and results, so use a run that contains no sensitive data. Please leave out API tokens, passwords, cookies, private documents and personal data. Use a public sample URL or a redacted example instead.

Tried this scraper? We'd love to hear how it worked for you. Leave a review to help others choose the right tool and help us improve it. Your feedback means a lot to us.

Made with care for curious people, AI agents and the possibilities in data. ♥