Similarweb Scraper — traffic, competitors, AI visibility & monitoring

Turn a list of domains into decisions, not a pile of nested JSON. Get website traffic, growth, ranks, acquisition channels, keywords, AI referrals, competitors, WHOIS, company signals, and homepage technologies — then compare sites, monitor changes, or build a competitive market map in one run.

No Similarweb login or browser session is required. Existing API integrations using base_data, similar_sites, aitdk, or all remain compatible.

🚀 Pick your goal

Workflow	Best for	What you receive	Billing event per valid result
base_data	Fast website overview	Traffic history, growth, ranks, engagement, channels, geography, keywords, AI visibility, screenshot	base-data
similar_sites	Competitor discovery	Similar websites, similarity grades, estimated visits, categories, tags, related apps	similar-sites
aitdk	Domain due diligence	WHOIS/RDAP, domain age, expiry, homepage keyword density, company metadata, social links, detected technologies	whois-keywords
all	Complete profile	All three sources merged into one row per domain	all-in-one
compare	Side-by-side benchmark	Actual traffic shares, leaders, ranks, engagement position, and gaps versus the first domain	base-data
monitor	Recurring intelligence	Persistent baseline plus material traffic, rank, channel, keyword, competitor, and AI-platform changes	base-data or all-in-one
market_map	Competitive landscape	Seed sites, auto-discovered competitors, enriched traffic rows, market shares, and a separate relationship dataset	base-data

Failed lookups are written to a separate Errors dataset and are not charged as result events.

✨ What makes the output comfortable

• Flat columns where they matter: latestMonthlyVisits, trafficChangePercent, bounceRatePercent, organicSearchPercent, and dozens more export cleanly to CSV and Excel.
• Original objects remain: existing consumers can still use engagement, estimatedMonthlyVisits, trafficSources, countryRank, and other legacy fields.
• Purpose-built table views: Overview, Traffic, Channels, Keywords & AI, Competitors, Domain & Technology, Comparison, Monitoring, and Market Map.
• Clean failures: no error rows mixed into paid result counts or downstream exports.
• Clear finish state: every run writes an OUTPUT JSON summary with exact result, error, filter, duration, and charged-event counts.
• Honest enrichment: unavailable values stay empty. The Actor does not invent demographics, traffic, technologies, or company data.

📊 Website overview fields

The base_data workflow returns the source-compatible objects plus useful derived fields.

Identity and rank

domain, websiteUrl, similarwebUrl, siteName, title, description, category, snapshotDate, screenshot, globalRank, countryRank, countryCode, countryRankValue, categoryRank, categoryRankValue, marketPosition

Traffic and engagement

engagement, estimatedMonthlyVisits, trafficHistory, latestMonthlyVisits, previousMonthlyVisits, trafficChangeMoM, trafficChangePercent, threeMonthGrowthPercent, trafficTrend, bounceRatePercent, pagesPerVisit, timeOnSiteSeconds

Acquisition channels

trafficSources, directPercent, organicSearchPercent, paidSearchPercent, organicSocialPercent, paidSocialPercent, referralsPercent, mailPercent, displayAdsPercent, affiliatePercent, aiTrafficPercent, topTrafficChannel, topTrafficChannelPercent, organicPaidRatio, trafficConcentrationScore, acquisitionDiversityScore

Geography, search, AI, and competitors

topCountryShares, topCountryCode, topCountrySharePercent, topKeywords, topKeyword, topKeywordVolume, aiTraffic, aiChatbotDistribution, aiTotalVisits, estimatedAiVisits, topAiPlatform, topAiPrompts, competitors, competitorCount, topCompetitor

Quality signals

dataCompletenessScore, riskFlags, isSmall, isDataFromGoogleAds, status

riskFlags are deterministic signals such as traffic_decline, single_channel_dependence, high_bounce_rate, domain_expiring_soon, and limited_public_data. They are not an opaque AI score.

🧭 Competitor discovery fields

The similar_sites workflow returns:

domain, title, description, category, categoryRank, totalVisits, thumbnail, screenshot, favicon, tags, similarSites, relatedApps, status

Each similarSites item can contain:

{
  "site": "peer-site.test",
  "description": "Example competitor website",
  "category": "Business_and_Consumer_Services",
  "similarityRank": 1,
  "topCountryRank": 2400,
  "totalVisits": 1850000,
  "grade": 0.91,
  "thumbnail": "https://example.test/example-preview.png"
}

All domains and URLs in examples are fictional or reserved .test values.

🔍 Domain, company & technology fields

The aitdk workflow combines RDAP and the public homepage without adding an extra homepage request for each feature.

• whois: registrar, status, nameservers, DNSSEC, registration, expiration, and last-change dates
• domainAgeDays, daysUntilDomainExpiration
• keywordDensity: page title, visible token count, top tokens, counts, and frequencies
• websiteIntelligence: page title, meta description, language, canonical URL, page image, company metadata, public social profiles, generator, and technology signals
• companyName, companyLogo
• technologies, technologyCount

Technology detection is based on visible homepage signatures and is labelled as detected or declared; it is not presented as a guaranteed full technology stack. A company logo is returned only when structured organization data explicitly identifies it as a logo — a generic Open Graph hero image is not mislabeled.

⚔️ Compare websites

Put the primary website first. The Actor fetches every site in parallel and calculates a real comparison rather than returning unrelated bulk rows.

{
  "domains": ["primary-site.test", "competitor-a.test", "competitor-b.test"],
  "mode": "compare",
  "maxConcurrency": 10
}

Comparison fields include:

primaryDomain, compareRank, trafficSharePercent, trafficGapVsPrimaryPercent, globalRankGapVsPrimary, growthRank, engagementRank, competitivePositionScore, winnerMetrics

Example row:

{
  "domain": "competitor-a.test",
  "mode": "compare",
  "primaryDomain": "primary-site.test",
  "compareRank": 2,
  "trafficSharePercent": 31.42,
  "trafficGapVsPrimaryPercent": -38.7,
  "growthRank": 1,
  "engagementRank": 3,
  "winnerMetrics": ["growth"]
}

🔔 Monitor changes

Monitoring uses a persistent named store. Run the same domains with the same monitorKey to compare against the previous successful check.

{
  "domains": ["primary-site.test", "competitor-a.test"],
  "mode": "monitor",
  "monitorKey": "weekly-competitors",
  "monitorDepth": "base_data",
  "changeThreshold": "medium",
  "onlyChanges": false
}

The first run returns changeStatus: "baseline". Later runs return changed or unchanged and can include:

checkedAt, previousCheckedAt, changeSeverity, changedMetrics, recommendedAction

Sensitivity presets:

• low: catches smaller movements
• medium: recommended default
• high: reports only major movements

Set monitorDepth to all to track the complete profile. Set onlyChanges to true to omit unchanged rows from the main dataset; each valid monitoring check is still charged because the source was checked and the baseline was evaluated.

🕸️ Build a market map

Market Map starts with one or more seed domains, discovers related sites, deduplicates them, enriches the strongest candidates, and calculates share inside the returned landscape.

{
  "domains": ["seed-site.test"],
  "mode": "market_map",
  "marketMapMaxCompetitors": 10,
  "maxConcurrency": 10
}

The main dataset contains site rows with:

marketRole, sourceSeedDomains, similarityGrade, similarityRank, marketTrafficSharePercent, marketRank, emergingCompetitor

The separate Market relationships dataset contains one edge per discovered connection:

{
  "seedDomain": "seed-site.test",
  "competitorDomain": "peer-site.test",
  "similarityRank": 1,
  "grade": 0.91,
  "competitorVisits": 1850000,
  "competitorGlobalRank": 43210,
  "trafficChangePercent": 8.4,
  "marketTrafficSharePercent": 17.6,
  "category": "Business_and_Consumer_Services"
}

📥 Input reference

Field	Type	Default	Notes
domains	array of strings	required	Domains or full URLs; normalized, deduplicated, and validated
mode	string	base_data	One of the seven workflows above
maxDomains	integer	10000	Safety limit for unexpectedly large pasted lists
monitorKey	string	my-monitor	Persistent monitor name; use the same value on later runs
monitorDepth	string	base_data	base_data or all
changeThreshold	string	medium	low, medium, or high
onlyChanges	boolean	false	Hide unchanged monitor rows after evaluation
marketMapMaxCompetitors	integer	10	Up to 100 unique discovered sites
maxConcurrency	integer	10	1–50 websites processed in parallel
proxyConfiguration	object	Apify Residential	Residential proxy is recommended for reliable traffic data

📦 Outputs and table views

Output	Contains
Results	Successful and partial rows in the default dataset
Traffic & engagement	Flat traffic history, growth, ranks, and engagement
Acquisition channels	Source percentages, concentration, and diversity
Keywords & AI visibility	Keywords, AI share, chatbot distribution, and available prompts
Competitors	Similar websites, direct competitors, tags, and apps
Domain & technology	WHOIS, company metadata, logos, social links, keywords, technologies
Comparison	Calculated Compare workflow fields
Monitoring changes	Baselines and material change details
Market sites	Enriched seed and competitor rows
Market relationships	Seed-to-competitor edge list in a separate dataset
Errors — not charged	Failed lookups with source-aware reasons
Run summary	OUTPUT JSON with counts, timing, filters, and workflow details

📤 Website overview example

{
  "domain": "demo-site.test",
  "mode": "base_data",
  "snapshotDate": "2026-07-01T00:00:00+00:00",
  "globalRank": 18420,
  "estimatedMonthlyVisits": {
    "2026-05-01": 220000,
    "2026-06-01": 238000,
    "2026-07-01": 245000
  },
  "latestMonthlyVisits": 245000,
  "previousMonthlyVisits": 238000,
  "trafficChangePercent": 2.94,
  "trafficTrend": "stable",
  "bounceRatePercent": 41.2,
  "pagesPerVisit": 3.84,
  "timeOnSiteSeconds": 176.5,
  "directPercent": 31.0,
  "organicSearchPercent": 46.0,
  "aiTrafficPercent": 2.1,
  "topTrafficChannel": "Organic search",
  "topCountryCode": "US",
  "topKeyword": "demo analytics",
  "topAiPlatform": "chatgpt.com",
  "dataCompletenessScore": 92,
  "riskFlags": [],
  "status": "ok"
}

Traffic values are estimates tied to the source snapshot date, not real-time analytics or exact first-party measurements.

🔌 API examples

Start a run:

curl -X POST "https://api.apify.com/v2/acts/trakk~similarweb-scraper/runs?token=<YOUR_APIFY_TOKEN>" \
  -H "Content-Type: application/json" \
  -d '{"domains":["demo-site.test"],"mode":"base_data","maxConcurrency":10}'

Run synchronously and receive dataset items:

curl -X POST "https://api.apify.com/v2/acts/trakk~similarweb-scraper/run-sync-get-dataset-items?token=<YOUR_APIFY_TOKEN>" \
  -H "Content-Type: application/json" \
  -d '{"domains":["primary-site.test","competitor-a.test"],"mode":"compare"}'

For larger jobs, start an asynchronous run and consume the output URLs returned with the run. Dataset items can be exported as JSON, JSONL, CSV, Excel, XML, HTML, or RSS.

⚙️ Reliability and scale

• Async collection and configurable concurrency
• Parallel source collection in all mode
• TLS browser impersonation without a heavyweight browser
• Fresh proxy-session rotation on throttling and anti-bot responses
• Hedged retries for slow requests
• Charge-limit handling before additional paid results
• Compact milestone logs for large runs, detailed per-domain logs for lists up to 50
• Honest ok, partial, and separate uncharged error outcomes

A 512 MB run is sufficient for every workflow. Actual duration depends on workflow depth, number of domains, proxy health, and source response time.

❓ FAQ

Do old inputs still work?

Yes. base_data, similar_sites, aitdk, and all keep their existing values and legacy nested fields. The new flat fields are additive.

Why is a field empty?

Public coverage varies by domain, size, source, and snapshot. Optional fields remain null, empty arrays, or empty objects instead of being guessed.

Are failed domains mixed into my export?

No. Valid and partial results go to the default dataset. Failures go to Errors — not charged with a short reason such as not_found, no_data, or source_unavailable.

Does onlyChanges make unchanged monitoring checks free?

No. It filters unchanged rows from the result dataset after the Actor checks the source and compares the saved baseline. The run summary tells you exactly how many checks were filtered.

Is technology detection definitive?

No. It reports signatures present on the public homepage. Server-side and hidden technologies may not be visible.

Is the example data real?

No. Examples use reserved .test domains and fictional values.