Samrush Bulk Website Stats Scraper

Collect public Semrush website overview statistics for a list of domains in one Apify run. The implementation accepts domain names or website URLs, checks each unique domain once, and returns dated metric records with source URLs. It uses the public website overview page without a Semrush account, API key or paid subscription.

Development status: this package is implemented and tested locally. The owner has confirmed permission for the source and intended use. The final cloud acceptance matrix and measured competitive pricing are pending. “Free” describes the Semrush source; the proposed Apify Actor pricing is Pay per event plus platform usage. No pricing has been activated.

Best-fit workflow

Use this Actor when you need to compare a website shortlist using public monthly traffic estimates and authority metrics. Avoid it when the job needs logged-in reports or domains without public coverage. For example, an agency can combine 100 customer and competitor domains, remove duplicates, and export the successful records into a reporting spreadsheet. Keep the reporting month beside each metric: retrieval today does not make the underlying estimate today's traffic.

When you need a small Authority Score and backlink shortlist, use the existing Semrush Authority Score Scraper. This bulk package adds engagement, rank, source-date and available public table fields. It is a public-overview collector, with coverage limited to domains Semrush exposes on those pages.

Input and first run

Add domains to the Input tab, choose your domain limit, and start a run after deployment. Review OUTPUT.outcome and RUN_SUMMARY before consuming the Dataset.

{
  "domains": ["google.com", "github.com", "wikipedia.org"],
  "maxDomains": 100,
  "includeDetails": true,
  "requestIntervalSeconds": 2,
  "requestTimeoutSeconds": 30,
  "maxRetries": 2
}

Input	Behavior
domains	Domain strings or HTTP(S) website URLs.
startUrls	Website URL entries, combined with domains.
maxDomains	First 1–1,000 unique valid domains; default 100.
includeDetails	Include public table samples; default true.
requestIntervalSeconds	1–30 seconds between domain checks; default 2.
requestTimeoutSeconds	5–60 seconds per HTTP attempt; default 30.
maxRetries	0–3 retries for transient failures; default 2.

At least one valid domain is required. URLs become hostnames: paths and query parameters do not select page-level traffic. www is removed; other subdomains remain separate. Credentials, IP addresses, unsupported schemes and nonstandard ports are rejected. Invalid and duplicate entries are counted without storing the original URL or its query parameters.

Website stats output

Each successful domain produces one validated Dataset record. The following is a shortened example from a public page retrieved on October 14, 2026; its metrics cover September 2026.

{
  "domain": "google.com",
  "reportMonth": "2026-09",
  "sourceUpdatedAt": "2026-10-11",
  "visits": 108899227055,
  "authorityScore": 100,
  "pagesPerVisit": 6.6669,
  "averageVisitDurationSeconds": 801,
  "bounceRatePercent": 32.18,
  "backlinks": 47653661167,
  "referringDomains": 64203372,
  "globalRank": 1,
  "sourceUrl": "https://www.semrush.com/website/google.com/overview/"
}

Field group	Included data
Traffic	Visits, monthly change, organic and paid search visit estimates.
Engagement	Pages per visit, average duration in seconds, bounce rate in percent.
Authority	Authority Score, backlinks, referring domains and public penalty-risk label.
Ranking	Global, country and category rank, with their country/category labels.
Detail samples	Available country shares, top organic keywords, competitor samples, categories, visits and device histories.
Provenance	Domain, source URL, retrieval time, reporting month and per-section source dates.

Percentage fields use 0–100 units; monthly changes may be negative. Zero remains zero. Unavailable scalar fields are null and named in missingMetrics. Detail arrays are empty when unavailable and omitted when includeDetails is false. Keyword and country tables are limited public samples, with a maximum of 20 entries per array. Different sections may use different reporting dates; inspect sourceDates and keywordDatabase before combining them. Export the Dataset as JSON, CSV or Excel using Apify's export controls.

Proposed Pay per event pricing

The private first-test price is $0.40 per 1,000 records on Free/Starter, $0.35 on Scale and $0.30 on Business, plus $0.00005 startup scaled by memory. Apify platform usage is billed separately. These event fees are below the verified narrow public-overview competitors, but total customer cost must include usage and can be higher for slow or low-yield jobs. The previous $5-per-1,000 proposal was rejected. Final launch pricing and worst-case costs remain under validation. Check the current Pricing tab after private deployment. For example, 100 Starter records have $0.04 in result fees, plus startup and measured usage.

Set the run's maxTotalChargeUsd to bound event fees and use maxDomains to bound requests. Collection stops before more requests when the SDK reports no remaining result-event capacity. Every plan receives the same field set and collection behavior.

API and agent workflow

After private deployment, use the Apify API with your token in the Authorization header:

curl --request POST \
  'https://api.apify.com/v2/acts/khadinakbar~semrush-bulk-website-stats-scraper/runs' \
  --header "Authorization: Bearer $APIFY_TOKEN" \
  --header 'Content-Type: application/json' \
  --data '{"domains":["google.com","github.com"],"maxDomains":2}'

Wait for the run to finish, read OUTPUT, then fetch items from its defaultDatasetId. The output schema links to the Dataset and both terminal records. Hosted MCP discovery and execution remain untested until deployment.

AI agent prompt:

Check the public Semrush website stats for these domains, return the report month and source URL with each result, and list unavailable domains separately from upstream failures. Limit the run to 100 unique domains and report any partial outcome.

Limits, retries and recovery

The Actor uses one worker with a delay between requests. It honors short Retry-After waits, applies bounded transient retries, and stops the batch on access denial or exhausted rate limits. Some domains have no public overview. Locked traffic-cost, advertising-spend and premium reports are excluded. These are Semrush estimates, with the source's coverage and update cycle.

COMPLETE means the requested bounded work finished; PARTIAL preserves useful rows alongside gaps, caps or interruption. VALID_EMPTY means no public data was available. INVALID_INPUT describes a fixable request or insufficient event budget. UPSTREAM_FAILED and CONFIG_ERROR fail the run when no useful data can be confirmed.

Progress is checkpointed per domain. A resumed run with matching input skips completed domains. A pending dataset write is treated as uncertain and stops further billing until reconciled; the implementation does not promise exactly-once delivery across interrupted writes. Start a fresh run for a fresh snapshot. Automatic restart recovery, cloud billing and hosted MCP still require live acceptance tests.

Builder's note and access requirements

The public overview embeds structured Astro data. Parsing that payload preserves exact numbers and distinguishes a real zero from a missing value. Sections marked unavailable are excluded, and schema validation occurs before any billable write. This avoids substituting rounded display labels or creating empty billable rows.

Semrush's Terms of Service currently prohibit scraping its content. The owner has confirmed permission covering automated collection and the intended data use. Callers are responsible for their own authorized use. This package does not grant those rights. Semrush is a trademark of its owner. This independent Actor is not affiliated with, associated with, or endorsed by Semrush.

Verification and freshness

The parser was checked against a public overview retrieved on 2026-10-14 and a sanitized metric fixture. Local tests also cover synthetic zero/no-data, locked sections, mismatched domains, schema drift, request limits, retries, deduplication and ambiguous writes. These local checks establish implementation behavior; final-build cloud run, charge, restart and MCP evidence remains pending.

For parser changes or missing coverage, use the Actor's Issues tab after deployment and include the domain, source month and run ID.