Semrush Website Scraper

A website's whole free Semrush profile as one spreadsheet row. Each row carries the global, country and industry rank with the movement on each, the authority score, monthly visits, pages per visit, time on site and bounce rate, the desktop-versus-mobile split with six months of history, the top five countries by traffic, organic and paid search traffic with six months of history, the top five organic keywords with their position, search volume and cost per click, backlinks and referring domains with their histories, the rivals Semrush names, the top five sources visitors arrive from and destinations they leave for, and referral traffic from each individual AI assistant.

That last one is the panel worth knowing about. ChatGPT, Claude, Gemini, Perplexity, Grok and coding agents each get their own traffic figure, their own share and their own six months of history, and it is on the free page.

Accepted input

domains is required and takes one website per line, either as a domain such as stripe.com or as any link to the site. A link is reduced to its host.

Semrush keys these pages on the registrable domain. www. is stripped because www.stripe.com is a 404 on Semrush while stripe.com is a 200, so removing it turns a broken request into a working one. Any other subdomain — blog.stripe.com, support.stripe.com — genuinely has no page of its own and is skipped with a line in the run log rather than being quietly rewritten to its parent, which would answer a question you did not ask with numbers for a different site.

maxItems defaults to 100 and caps how many websites the run reads. One website is one request and one row here, so it is a cap on the list rather than on a page of results. 0 means no limit.

There is no country input, and its absence is deliberate. The same page fetched from US, GB, DE and IN exits came back identical within two bytes, with the same database named in all four, and Semrush's own db and database parameters were ignored. A parameter that appears to select a market and does not is worse than a missing one.

{
  "domains": ["stripe.com", "gocardless.com", "https://adyen.com/pricing"],
  "maxItems": 100
}

Response fields

{
  "requested_domain": "stripe.com",
  "domain": "stripe.com",
  "url": "https://www.semrush.com/website/stripe.com/overview/",
  "as_of": "2026-07-15",
  "updated_at": "2026-08-12",
  "database": "US",
  "global_rank": 322,
  "global_rank_change_places": -1,
  "country_rank": 149,
  "country_code": "US",
  "country_name": "United States",
  "category_rank": 4,
  "category_name": "Finance",
  "category_slug": "finance",
  "authority_score": 92,
  "visits": 203227314,
  "visits_change_percent": 0.1294,
  "pages_per_visit": 4.6,
  "time_on_site_seconds": 498,
  "bounce_rate": 0.6072,
  "organic_search_traffic": 41208312,
  "paid_search_traffic": 182004,
  "backlinks": 151822916,
  "referring_domains": 289110,
  "google_penalty_risk": "Low",
  "ai_other_traffic": 149279270,
  "ai_other_share": 0.93069,
  "top_keywords": [
    { "keyword": "stripe", "position": 1, "search_volume": 673000, "cpc": 2.99 }
  ],
  "ai_traffic": [
    { "source": "chatgpt.com", "name": "ChatGPT", "traffic": 798467, "share": 0.00498 }
  ],
  "unavailable_panels": ["siteAudit"]
}

The example is trimmed; the full row also carries categories, visits_history, device_history, top_countries, search_traffic_history, backlinks_history, referring_domains_history, competitors, traffic_incoming, traffic_outgoing and related_domains.

Fields Semrush states no figure for are returned as null rather than omitted, so every row has the same shape.

null is never rendered as zero, and that distinction is the point. Semrush genuinely has no current traffic estimate for some websites — openai.com and doordash.com among them — and a zero in that place would be a traffic figure nobody measured. unavailable_panels names the sections Semrush itself flagged for that domain, under its own names, and the run log says so in words when the headline visits is missing.

bounce_rate is a fraction: 0.6072 is 60.72%. So is visits_change_percent: 0.1294 is +12.94%. The rank movements are the exception — they are stated in positions and named _change_places for it, and a negative number there is an improvement because the rank number fell.

top_keywords and the other subtrees are republished in Semrush's own shape. There is no measured field list to rename them into, and renaming fields nobody has seen populated publishes a permanently-null column and calls it a schema. The keyword intents code inside is republished undecoded for the same reason.

traffic_incoming and traffic_outgoing each carry a domain field alongside the source label, and it is non-null exactly when that source has a Semrush page of its own. So a traffic-journey row feeds straight back into another run of this Actor without parsing a label.

Behaviour on partial results

A website Semrush has no page for is skipped with a line in the run log and the rest of the list still runs — losing nineteen good rows to one domain that has no page is the wrong trade. An entry that is not a domain at all is skipped before any request is made, because noticing costs nothing and a request that was never going to resolve is worse than a message naming the shape.

A website with a page but no current traffic estimate is a successful row. The ranks, the authority score, the backlinks and often six months of device history are all still there; visits is null and unavailable_panels says which sections Semrush marked empty. That is stable across requests — openai.com returned the identical set as request 1 and again as request 24 of one walk — so it is a gap in Semrush's index rather than something a retry fixes.

Frequently asked questions

Are these live numbers? No, and nothing here pretends otherwise. They are monthly estimates. Every row carries the month it describes in as_of, and two runs a week apart return the same figures. Running this daily buys nothing; running it monthly is the shape of the data.

Is there a view limit, and does it start returning zeroes? No. That failure is real on some traffic-analytics sites — they meter anonymous profile views and then serve the same page with every metric zeroed — so it was tested for directly here. Sixty consecutive fetches on one identity with no delay returned sixty of sixty, and every repeated domain matched its first reading exactly: stripe.com at 203,227,314 visits on request 1 and again on request 40. Zero anomalies in sixty.

Why can I not choose a country database? Because there is nothing for the choice to select. The same page fetched from US, GB, DE and IN residential exits came back byte-identical within two bytes, with database reading US in all four, and both of Semrush's own database parameters were ignored. The database field on every row states what the figures are actually measured in.

Why did blog.example.com come back with nothing? Because Semrush publishes a page per registrable domain and that subdomain has none. It is not rewritten to example.com, because that would return numbers for the whole site under the name of the blog. Pass the registrable domain if that is what you meant.

Why are there only five countries and five keywords? Because five is what the free page carries. There is no parameter that returns more, and one that appeared to would be a lie in the input form.

Is a Semrush account or API key required? No. No account, session cookie or key of any kind is supplied to the Actor or needed by it. Everything it returns is what the overview page shows an anonymous visitor — open the url on any row and you see the same numbers.

Related Actors

Semrush Competitors Scraper takes the same website list and returns each rival with the subject's own metric set, so the two line up in one table. The competitors panel on a row here gives a name, a similarity score and a visits figure that is frequently null; that Actor gives every rival visits, authority score, bounce rate, pages per visit, time on site and all three ranks.

Semrush Top Websites goes the other way: instead of looking up a website you already know, it returns the hundred most visited websites for a country and an industry. Every domain it produces is a valid input here.