Semrush Domain Overview Scraper
Pricing
from $0.05 / 1,000 item extracteds
Semrush Domain Overview Scraper
Scrape public Semrush domain overview metrics for traffic, rank, authority, backlinks, keywords, countries, and competitor monitoring.
Pricing
from $0.05 / 1,000 item extracteds
Rating
0.0
(0)
Developer
Fetch Cat
Maintained by CommunityActor stats
0
Bookmarked
13
Total users
6
Monthly active users
3 days ago
Last modified
Categories
Features
Share
Collect public Semrush domain overview snapshots for traffic, rankings, authority and backlink research. Give Semrush Domain Overview Scraper a list of domains, then export the available summary fields as JSON, CSV or Excel. Coverage varies by domain, and missing metrics stay null. This is not a Semrush account export or its official API.
Quick start
- Open Input and add one or more domains or URLs.
- Set a small maximum, then click Start.
- Open the dataset and review each row’s
statusbefore comparing metrics.
{"domains": ["openai.com", "github.com", "stripe.com"],"maxItems": 3}
Prefer a video guide? This short Apify walkthrough shows you how to set up and run an Actor. It uses a different Actor, but the setup steps are the same.
What you get
- Domain snapshots: available traffic, rank, authority, referring-domain and backlink summaries.
- Keywords and traffic sources: the top organic keywords with position, search volume and CPC, where visitors come from and go next, and how many visits AI assistants such as ChatGPT send.
- Recent trends: monthly visits (desktop and mobile), organic and paid search traffic, backlinks and referring domains for recent months.
- Context: reporting period, page metadata, country/category breakdowns, Google penalty risk and competitor summaries with their visits when present.
- Status per domain: distinguish useful metrics from unavailable pages, blocked requests and invalid inputs.
- Structured exports: numeric metric objects with original values and change percentages when available.
Who is it for?
- SEO teams collecting public domain snapshots for client reports.
- Growth teams comparing the available summaries of competing websites.
- Agencies and analysts building spreadsheets or dashboards from domain lists.
Input settings
| Setting and API key | What it does |
|---|---|
Domains or URLs (domains) | Required list of 1–100 domains or URLs. |
Maximum domains (maxItems) | Limits unique valid domains, default 10, maximum 100. |
Maximum concurrency (maxConcurrency) | Accepted legacy field. Processing is sequential and this value has no effect. |
Proxy configuration (proxyConfiguration) | Existing proxy settings. See field help for defaults and options. |
Include raw HTML snippet (includeRawHtmlSnippet) | Adds the first 2,000 characters of the response to fetched-page rows. Default false. |
Save debug artifacts (saveDebugArtifacts) | Saves the run counters in RUN-SUMMARY. Default false. |
Domains are lowercased, a leading www. is removed and duplicate normalized domains are processed once. Invalid inputs may add diagnostic rows beyond maxItems.
Input recipes
For a competitor snapshot, use:
{"domains":["notion.so","clickup.com","asana.com","monday.com"],"maxItems":4}
Output data
The following is a historical successful output excerpt, not a current measurement or a promise of field coverage.
{"input": "openai.com","domain": "openai.com","url": "https://www.semrush.com/website/openai.com/overview/","status": "success","reportingPeriod": "2026-06-01","visits": { "raw": "508002719", "value": 508002719, "diffPercent": -0.0945 },"authorityScore": { "raw": "99", "value": 99, "diffPercent": null },"referringDomains": { "raw": "423708", "value": 423708, "diffPercent": 0.0496 },"backlinks": { "raw": "64639384", "value": 64639384, "diffPercent": 0.015 },"topCountries": [{ "country": "United States", "share": 0.1397, "visits": 70981896, "desktopShare": 0.6019, "mobileShare": 0.3981 }],"googlePenaltyRisk": "Low","topOrganicKeywords": [{ "keyword": "stripe", "position": 1, "searchVolume": 673000, "cpc": 2.89, "trafficShare": 0.3147, "intents": "1" }],"trafficSources": [{ "source": "Google organic", "domain": "google.com", "share": 0.0319, "visits": 2660087, "previousVisits": 2611151 }],"aiAssistantTraffic": [{ "source": "chatgpt.com", "name": "ChatGPT", "visits": 703479, "share": 0.0081, "history": [{ "date": "2026-10-01", "visits": 703479 }] }],"searchTrafficHistory": [{ "date": "2026-10-15", "organic": 4972550, "paid": 98486, "total": 5071036 }],"error": null}
| Field | Description |
|---|---|
input, domain, url | Original input, normalized domain, and public Semrush overview URL. |
status, error | success, no_public_metrics, not_found, blocked, failed, or invalid_input. |
fetchedAt, reportingPeriod | Collection time and visible Semrush reporting period when available. |
title, description | Public page metadata. |
globalRank, countryRank, categoryRank | Ranking metrics when present. |
countryRankName, categoryRankName | Country and category the ranks refer to. |
visits, pagesPerVisit, avgVisitDuration, bounceRate | Traffic and engagement summaries. |
organicSearchTraffic, paidSearchTraffic, organicKeywords | Public search summaries. organicKeywords counts available keyword-series entries, not all keywords ranking for the domain. |
topOrganicKeywords, keywordsDatabase | Up to 10 top organic keywords shown on the overview (keyword, Google position, monthly search volume, CPC in USD, share of organic traffic, Semrush intent codes) and the regional database they come from. |
authorityScore, referringDomains, backlinks | Authority and backlink summary counts. |
googlePenaltyRisk | Semrush penalty-risk label for the backlink profile, for example Low. |
trafficHistory | Recent monthly visits with desktop and mobile visits. |
searchTrafficHistory | Recent monthly organic, paid and total search traffic. |
backlinksHistory | Recent monthly backlinks and referring-domain counts. |
trafficSources, trafficDestinations | Up to 10 top sources of visits (Direct, Google organic, referring sites) and up to 10 sites visitors go to next, with share, visits and previous-period visits. |
aiAssistantTraffic | Up to 10 AI assistants and AI sites sending visits, such as ChatGPT or Claude, with visits, share and monthly history. |
topCountries, topCategories, competitors | Arrays of public breakdowns when present. Countries include desktop and mobile shares; competitors include their estimated visits. |
Metric objects include raw, normalized numeric value when reliable, and diffPercent when Semrush exposes a change percentage.
organicKeywords is the number of available keyword-series entries, not a complete keyword inventory. Device-share metrics are nullable. Optional rawHtmlSnippet contains response text, not a complete saved page. OUTPUT in the key-value store contains dataset and run counters. If the run reaches its time budget, PENDING_DOMAINS lists unprocessed domains.
Download your results
Open your run's dataset, choose Export and select JSON, CSV or Excel. JSON keeps nested fields together. CSV and Excel are useful when you want to review the records in a spreadsheet.
See Apify's download and export guide for the available formats and field options.
Pricing
This Actor uses pay-per-event pricing. The start event is a one-time run charge.
The item event charges for each successful metric-bearing domain overview record. Diagnostic rows with invalid_input, not_found, blocked, failed or no_public_metrics are not charged as successful results. A start charge can still apply when no useful metrics are returned.
See the live Pricing tab for current rates and discounts.
Tips for best results
- Start with a few domains. Check coverage before submitting a larger list.
- Compare the reporting periods. Public summaries may refer to different months.
- Keep status columns in exports. Null metrics and diagnostic records are not zero traffic.
- Use bare domains where possible. URL paths are discarded during normalization.
Limits and caveats
- Public overview summaries only. No login, private dashboards, full backlink lists, full keyword exports, regional database selection or SEO audit mode. Keyword, source and AI-traffic lists hold what the public overview shows (often five entries).
- Availability depends on Semrush. Pages can be missing, blocked or lack usable metrics.
maxConcurrencydoes not change execution speed. This Actor processes domains sequentially.- Per-domain errors are saved as status rows and processing continues. Invalid top-level input or infrastructure/storage failures can still fail a run.
topCountries,topCategories,competitors,topOrganicKeywords,trafficSources,trafficDestinationsandaiAssistantTrafficeach contain at most ten entries. Empty lists mean the overview did not show that section.- If the run's maximum cost or timeout is too low, the Actor stops early, keeps the rows already saved and explains the stop in its status message and the
OUTPUTandRUN_SUMMARYrecords. - No guarantee of complete metrics, current-month data or agreement with a paid Semrush account.
Automation and integrations
Save your settings for next time
Once you have an input that works for you, save it as an Apify task. This keeps your settings together so you can use them again without filling in the form from scratch.
Run on a schedule
Need fresh results regularly? You can schedule the Actor or a saved task to run at a time that suits you.
Each scheduled run collects the data again. If you want to compare results or highlight changes, add that step in your own workflow.
Send results to your other tools
Use n8n, Make or Zapier to connect this scraper to the rest of your workflow. For example, you could pass its results to an AI assistant or add records to a spreadsheet. Follow the Apify integration guides for your chosen tool.
For a custom connection, an Apify webhook can notify your app when a run finishes or fails. Your app can then fetch the results or handle the failure.
API usage
Want to collect this data from your own app or automation? The examples below send input to the scraper and show how to collect its results.
The Node.js and Python examples wait for the run to finish before reading its results. The cURL example starts the run and returns its details. Wait for that run to finish, then use its defaultDatasetId to fetch the results.
For setup instructions, see Apify's JavaScript client guide, Python client guide and Actor runs API reference.
cURL
curl -X POST 'https://api.apify.com/v2/acts/fetch_cat~semrush-domain-overview-scraper/runs' \-H "Authorization: Bearer $APIFY_TOKEN" \-H 'Content-Type: application/json' \-d '{"domains":["openai.com","github.com"],"maxItems":2}'
Node.js
import { ApifyClient } from 'apify-client';const client = new ApifyClient({ token: process.env.APIFY_TOKEN });const run = await client.actor('fetch_cat/semrush-domain-overview-scraper').call({domains: ['openai.com', 'github.com'], maxItems: 2});const { items } = await client.dataset(run.defaultDatasetId).listItems();console.log(items);
Python
import osfrom apify_client import ApifyClientclient = ApifyClient(os.environ['APIFY_TOKEN'])run = client.actor('fetch_cat/semrush-domain-overview-scraper').call(run_input={'domains': ['openai.com', 'github.com'], 'maxItems': 2})items = client.dataset(run['defaultDatasetId']).list_items().itemsprint(items)
MCP and AI agents
You can ask your AI assistant to collect data with Semrush Domain Overview Scraper and help you work with the results. The scraper returns the data, and your assistant can then summarize it, compare records or prepare notes.
Connect your assistant through the Apify MCP server. MCP gives your assistant a way to find this scraper, run it and read the results. Follow the official setup and sign-in guide for your app.
Easier setup with an Apify plugin
If you use a supported coding assistant, the official Apify plugin can set up the connection for you. It connects your assistant to Apify and includes instructions for common tasks. Follow the guide for your particular app.
Use Apify AI in Console
Prefer to keep everything in Apify? Open Apify AI in Console and ask it to find Fetch Cat's Semrush Domain Overview Scraper. You can describe the data you want and ask questions about the results.
Apify AI is currently in beta. It asks for confirmation before starting an Actor, and normal run charges apply.
Choose your assistant
- Claude and Claude Code (Anthropic): Apify setup guide.
- ChatGPT and Codex (OpenAI): Apify setup guides.
- Grok Bot (xAI): Apify plugin guide.
- Gemini CLI (Google): MCP server setup guide. Use the server URL below with the HTTP configuration supported by Gemini CLI.
Connect just this scraper
To keep your setup focused on this scraper, add this server configuration to a compatible MCP client:
{"mcpServers": {"apify": {"url": "https://mcp.apify.com?tools=fetch_cat/semrush-domain-overview-scraper"}}}
Sign in to Apify when your client asks. If your client uses a Bearer token instead, follow its setup guide and keep the token in its secure configuration. Configuration formats vary by app, so use the linked guide if your client needs a different format.
Use Apify's general server
Already connected to the general Apify MCP server at https://mcp.apify.com? You can use it to find this scraper without adding another server. Ask your assistant to use Semrush Domain Overview Scraper by Fetch Cat, or give it the identifier fetch_cat/semrush-domain-overview-scraper.
Your assistant can check the available settings and retrieve the results from the returned run. If the run is still working, it should follow that run rather than start it again. Both connection options use Apify's hosted service.
Example requests
- “Get the public Semrush summaries for OpenAI and GitHub, then compare their available traffic figures.”
- “Collect the available authority and backlink summaries for Notion, ClickUp and Asana and put the results in a comparison table.”
- “Fetch the public overview for Stripe and explain which metrics are missing from the returned record.”
Normal Apify charges apply. Check the live Pricing tab before asking your assistant to run a larger batch.
Pay with x402
If your AI agent has a funded wallet, it can pay for Apify usage through x402. This experimental option lets an agent use Apify without a conventional account.
Apify AGI provides a prepaid API token with a spending cap after payment. Your agent can use that token to run this scraper through the Apify API or an authenticated MCP connection.
Some MCP setups also support paying for each request directly. That is a different payment option, so check the official guide for its availability.
Follow the official guide for wallet setup, supported payment options and current terms. Apify handles the payments. Keep wallet keys and payment tokens in your agent's secure configuration.
Other ways for agents to pay
Apify also supports MPP for prepaid access through AGI and Skyfire for eligible Actors. If your agent already uses one of these, see the agentic payment guide to choose the right setup.
FAQ
Do I need a Semrush account?
No. The scraper reads public overview pages without your Semrush credentials.
Does this replace the Semrush API?
It is an alternative for collecting available public summaries, not a replacement for paid account datasets or official API coverage.
Can I export the results or use an AI assistant?
Yes. Export the dataset or use the API and MCP instructions above. The assistant, not the scraper, performs any additional comparison or summarization.
Why are metrics missing?
Some domains lack public coverage, and pages can be unavailable or blocked. Inspect status, error and the reporting period. no_public_metrics means the response lacked the principal metrics this scraper uses to identify a useful overview.
What are the alternatives?
Use Semrush’s own tools for account-level reports. For public traffic comparisons from another source, see Similarweb Traffic Scraper below.
Related Actors
- Similarweb Traffic Scraper — public traffic summaries from another source.
- Google Keyword Research Scraper — adjacent keyword research.
- Bing Search Results Scraper — Collect ranked Bing organic results with titles, URLs and snippets for public web research. This uses Bing rather than Google.
- Google Maps Lead Finder — local-business research.
- Website Content Crawler Lite — website text for follow-up research.
Support and feedback
For custom additions or integrations, email hello@fetchcat.dev.
Found an error or have a feature request? Open an issue with the run URL, a non-sensitive input example and what you expected.
Need help sharing it? See Apify’s guide to sharing a run when reporting an issue. Sharing a run can give the developer access to its input, logs and results, so use a run that contains no sensitive data. Please leave out API tokens, passwords, cookies, private documents and personal data. Use a public sample URL or a redacted example instead.
Tried this scraper? We'd love to hear how it worked for you. Leave a review to help others choose the right tool and help us improve it. Your feedback means a lot to us.
Made with care for curious people, AI agents and the possibilities in data. ♥