🔍 Spy on any website in seconds: traffic, rankings, top keywords, AI traffic share (ChatGPT/Claude/Gemini), competitors, similar sites & WHOIS — all from Similarweb. No login or API key. Bulk parallel scrape, captcha-resilient. Export to JSON/CSV/Excel. SEO, lead gen, research.
All notable changes to this Actor are documented here. Public builds use
Apify build revisions such as 0.4.19 and 0.3.15.
[0.4.81] - 2026-07-25
Fields the sources were returning but the Actor discarded
Added aiTrafficSources: every AI assistant Similarweb names for the domain,
with its rank and share. The five aiTrafficShare* columns are a fixed shape
over a variable ranked list, so assistants such as Grok and DeepSeek had
nowhere to go, and the share chart only carries the top few, which is why a
named column could read as null while the assistant was clearly ranked. The
old columns are unchanged for anyone already reading them.
Added aiTrafficVisits, the absolute number of AI-referred visits, and
aiReferralTraffic, which the source reports separately from genAiTraffic.
Added similarityRank and similarityScore to each entry in
similar_sites. The list was flat, giving no sign that the first competitor
is often far closer than the twentieth.
Added related_apps: mobile apps associated with the site, with platform,
store ranking and link.
Added searchOrganicTraffic, searchPaidTraffic, socialOrganicTraffic and
socialPaidTraffic. Both sources report search and social split by organic
versus paid, and the Actor was summing them away, hiding how much of a site's
search or social presence is bought. searchTraffic and socialTraffic keep
their combined meaning.
Made both base_data sources emit the same field set. The website fallback
was missing the new AI keys entirely rather than returning them empty, so a
record could change shape depending on which source answered.
Added whois_name_server_details with the IPv4 and IPv6 glue records the
registry publishes for each name server, and whois_unicode_name for the
readable form of an internationalised domain. All three WHOIS paths - RDAP,
the plain WHOIS fallback and the no-result case - now return one field set.
Added whois_abuse_email and whois_abuse_phone. Registrant, administrative
and technical contacts are redacted by the registries, but ICANN requires the
registrar abuse contact to stay public and it was present on every domain
checked. Registries nest it inside the registrar entity, which is why a
glance at the top-level roles suggests no contact data survives.
Added whois_transfer_date, whois_port43 and whois_dnssec_ds_records.
DNSSEC was reduced to a single boolean; the delegation signer records carry
the key tag, algorithm and digest.
These came from checking the parser against the RDAP object model in RFC 9083
rather than against sampled responses. Sampling three registries had suggested
the source was fully exploited; the specification named four members, five
secureDNS members and eleven entity roles that sampling had not surfaced.
Console views
Added a 🪞 Similar sites view. The mode had no view of its own, so its
results were rendered through Overview, which shows five mostly empty columns
and none of the competitor list, similarity scores or related apps.
Removed aiTrafficShareCopilot from the AI view, where it was empty on every
row of a 500-domain sample: the source publishes shares only for its top few
assistants. The field itself is unchanged, and aiTrafficSources now covers
every assistant including the ones without a column.
Added aiReferralTraffic to the AI view, and the readable domain plus name
server addresses to the AITDK view.
Every column in every view is now a declared field that carries data on real
runs. One field stays permanently empty and is deliberately left out of the
views: top_country in similar_sites, because the upstream payload no longer
contains the key it was read from.
Checked and deliberately left alone: Competitors.TopSimilarityCompetitors is
empty on every domain sampled, GlobalCategoryRank is always null, Countries
is a static 252-entry country lookup identical for every domain, and the unused
RDAP keys are protocol metadata. Registrant, admin and technical contacts are
absent from RDAP responses themselves.
[0.4.80] - 2026-07-25
Faster aitdk runs
Capped the request deadline for follow-up homepage candidates at 4 seconds
while the first candidate keeps the full deadline. Tracing a 50-domain sample
showed 76% of the time went into attempts that ran to the full deadline, and
35 of 39 successes came from the very first candidate, so the later ones were
long shots against hosts that had already stopped answering.
Made RDAP rate limiting per registry instead of global. Politeness is owed to
each RDAP server, and a single shared limit made unrelated registries queue
behind each other. The per-server limit keeps the same conservative defaults
(3 concurrent requests, 0.35 s apart) while removing the queueing.
Measured over four runs of each build against the same 50-domain sample, since
a single run of this mode varies by up to a fifth depending on how the sampled
sites happen to respond: median wall time fell from 105.6 s to 68.9 s, a 35%
reduction, and the two ranges do not overlap (102.3-107.2 s before, 63.0-89.4 s
after). Keyword coverage was unchanged at 11-13 of 50 domains missing, and
WHOIS stayed at 50/50.
Raising worker concurrency was measured as well and does not help this mode:
10, 20 and 25 workers all landed within the same band, because wall time is set
by a dozen domains grinding the full candidate ladder against hosts that refuse
or never answer. Lifting the RDAP limits outright was also tried and produced
RDAP timeouts, which the per-server limit avoids.
[0.4.79] - 2026-07-25
Keyword density now works outside English
Replaced the ASCII-only keyword tokenizer. It matched [a-zA-Z]{3,} only, so
Cyrillic, Greek, Arabic, Hebrew, Chinese and Japanese pages produced no
keywords at all, and accented Latin words were dropped outright — cafe
counted, café did not. Tokens are now any run of Unicode letters.
Added character-bigram counting for Han and Katakana, which are written
without spaces and therefore cannot be split by a word pattern. Hiragana is
treated as a separator, since bigramming grammatical particles buries the
real terms.
This also fixes silent quality loss rather than only outright failures: a
Japanese page previously scraped past the 30-word threshold on stray English
boilerplate, producing keyword density made of words like "home" and
"copyright". The same page now yields 838 real terms.
Added two last-resort homepage candidates for aitdk, tried only after every
other option failed: one that ignores an invalid TLS certificate, and one
over plain HTTP for hosts that never answer on 443. Certificate validation is
unchanged everywhere else; these requests only read public page text.
On a fixed 50-domain sample, domains ending without keyword density dropped
from 19 to 12. The remainder are refusals and dead hosts rather than parsing
gaps: 3 x HTTP 403, 3 x too little text on the page, 2 x transport failure, and
one each of empty response, HTTP 429, timeout and an unclassified error. The
extra candidates make an aitdk run about 18% longer, since they are attempted
for exactly the domains that would otherwise return nothing.
[0.4.78] - 2026-07-25
Honest reporting for domains Similarweb barely covers
Added a dataCoverage field to every base_data item: full when the
domain is ranked, small_site when it sits below Similarweb's ranking
threshold but descriptive data or visit estimates are still returned, and
no_data when the source answers with every metric null or zero.
Reported those rows in a new sourceCoverage block (ranked, smallSites,
noData) kept deliberately separate from dataQuality. Thin coverage is a
property of Similarweb's data, not a scraping shortfall, so these rows stay
counted as complete results: they already carry every value the source
provides. dataQuality.partialItems continues to mean only what the Actor
itself could not retrieve.
Added the coverage column to the Console overview so a thin row is visible
without opening the record.
On a 500-domain sample spread across the whole Majestic rank range, the split
was 301 full, 120 small_site and 79 no_data. The small_site rows are
not empty: they keep title, description, category, monthly visit estimates and
screenshots, and the existing isSmall flag already matched every unranked row
exactly. The no_data rows cannot be filled from the second source either -
Similarweb returns the same empty record for them.
[0.4.76] - 2026-07-25
Sticky proxy exits instead of a new session per request
Replaced the "fresh Apify Proxy session on every HTTP call" behaviour with a
pool of sticky exits. Each slot owns one AsyncSession pinned to a single
proxy session id, so consecutive requests reuse the established tunnel.
Profiling put the Chrome-fingerprint TLS handshake at ~96% of per-domain CPU,
which was the binding constraint on the 512 MB default run.
Kept IP rotation, but driven by evidence rather than by every request: an
exit is burned and replaced on a block signal (202 / 403 / 429 / proxy
connect / timeout) or after proxy_slot_max_uses requests (default 50).
Sized the pool at twice the worker count so a hedged request always lands on
a different exit instead of waiting for the worker holding the primary one.
Released the domain-pinned exit when aitdk finishes a domain; without it
the pool drained after one pinned slot per concurrent worker.
Measured on the same 500-domain list, base_data, concurrency 10: a run that
took 83.4 s now takes 21.4 s (5.99 -> 23.35 domains/s), with retry attempts
down from 18 to 6 and the proxy-layer error classes gone entirely. Saved rows
and failure count were unchanged at 500/500 and 0. similar_sites over 50
domains went from 7.1 s to 3.1 s with an identical 14 complete / 36 partial
split. aitdk was unchanged in both speed and quality, since that mode fetches
RDAP and the first homepage candidates without a proxy.
Re-runnable list of missing domains
Added a FAILED_DOMAINS key-value store record with the complete list of
domains that produced no dataset item, plus a domainsText form that can be
pasted straight back into the Domains field. OUTPUT.dataQualityfailedResultsSample stays capped at 50 entries, so on bulk runs the full
list previously existed only in the run log and users had to repeat the whole
batch to recover a handful of domains.
Reported input lines rejected during validation as invalidEntries in the
same record and as dataQuality.invalidEntries in OUTPUT, kept out of the
re-run list because they would fail again.
Wrote the record on every terminating path, including empty / invalid input
runs, so count: 0 is a reliable "nothing is missing" signal.
Named the missing-domain count in the terminal run status message and the run
summary log instead of leaving a partially complete batch looking identical
to a complete one.
[0.4.75] - 2026-07-14
Input compatibility
Added support for copied/API inputs that pass domains through websites,
websiteUrls, or website_urls. These aliases now resolve the same way as
domains and urls instead of finishing with zero processed domains.
[0.4.74] - 2026-06-29
Expanded traffic-source output
Added displayAdsTraffic and genAiTraffic to base_data output.
Added both fields to the Apify Dataset schema and the Traffic sources table
view so users can see the full traffic-source breakdown returned by the
current Similarweb payload.
[0.4.73] - 2026-06-29
Traffic-source parsing
Fixed base_data traffic-source fields for the current Similarweb extension
payload. SearchOrganic + SearchPaid now populate searchTraffic,
SocialOrganic + SocialPaid populate socialTraffic, and Affiliate
populates paidReferralsTraffic.
[0.4.72] - 2026-06-26
Input prefill visibility
Added an explicit datasetMode prefill value so Apify Console's JSON input
view shows "datasetMode": "base_data" together with the 20 prefilled
domains.
[0.4.71] - 2026-06-26
Dataset selector conflict handling
Fixed mixed old/new inputs where the visible datasetMode is
similar_sites but a stale legacy datasets=base_data value is also
present. The explicit public selector now wins in that case.
Kept the intended API compatibility path where a legacy datasets value can
still override Apify's injected public default datasetMode=base_data.
[0.4.70] - 2026-06-25
Build archive cleanup
Kept local tmp_*.json test inputs out of future Apify upload archives.
No runtime behavior changed.
[0.4.69] - 2026-06-25
Base-data default restored
Restored base_data as the public input default now that the clean HTTP
Similarweb data endpoint is stable again.
Expanded the public input prefill to 20 common domains so first-time users
can run a meaningful bulk base_data test immediately.
Updated README examples to match the public input default.
[0.4.68] - 2026-06-25
Base-data clean HTTP source
Switched base_data back to the Similarweb extension JSON endpoint with
extension-style HTTP headers instead of relying on the captcha-gated website
page.
Treats temporary 403 / 429 responses from the data endpoint as retryable
source issues so proxy session rotation can recover before the run finishes.
Added a generic AITDK app-manifest metadata candidate as a clean HTTP
fallback for blocked homepages.
[0.4.64] - 2026-06-25
Base-data source refresh
Added a parser for Similarweb's current website SSR payload
(window.__APP_DATA__) and made base_data try
https://www.similarweb.com/website/{domain}/ before the legacy extension
endpoint.
Kept the base-data contract strict: if Similarweb returns an anti-bot
challenge or a payload for a different domain, no guessed/fallback item is
saved as base_data.
Verified in Apify cloud that the new parser is ready, but the public
no-browser HTTP path can still return challenge_202; similar_sites
remains unaffected.
[0.4.60] - 2026-06-25
Base-data zero-result cost reduction
Stopped retrying the public Similarweb base-data endpoint after HTTP 403.
This endpoint currently returns a CloudFront refusal for tested domains, so
repeated proxy attempts only make zero-result runs slower and more expensive.
Kept the result contract unchanged: no fallback data is saved as base_data
unless the base-data source itself returns a usable payload.
Fixed copied/API inputs where Apify injects the public datasetMode default
(base_data) alongside a legacy dataset or datasets value. A supported
legacy value such as similar_sites or aitdk now wins over the injected
default instead of accidentally running base_data.
Changed the public input default to similar_sites so first-time UI runs use
a currently available source instead of the public base-data endpoint that is
returning HTTP 403.
[0.4.59] - 2026-06-25
Zero-result run cost guardrail
Added an optional actor-start pay-per-event charge hook. When this event is
configured in the Actor pricing tab, empty or invalid-input runs can cover
their startup/platform cost instead of relying only on per-result charges.
Stopped before source requests when the user's max charge limit does not
allow the start event to be charged.
Kept the hook backward-compatible: if the actor-start event is not yet
configured, the Actor continues normally without calling the charge API.
[0.4.58] - 2026-06-21
Documentation alignment
Updated README input examples to use the current public datasetMode field;
the older datasets key remains accepted as a legacy/API alias.
[0.4.57] - 2026-06-21
Safer copied-input handling
Moved the public dataset selector to datasetMode while keeping datasets
as a flexible legacy/API input, so copied JSON with a foreign datasets
array no longer fails schema validation before the Actor starts.
Stopped declaring flexible copied-input aliases as hidden schema fields; the
Actor still parses them, but Apify no longer rejects valid copied shapes such
as startUrl: { "url": "https://example.com" }.
Accepted top-level requestsFromUrl values in addition to request-list
objects inside startUrls, requests, or requestListSources.
[0.4.56] - 2026-06-21
Broader copied-input compatibility
Accepted more common copied Actor input aliases: single url, domain,
website, websiteUrl, startUrl, requests, and requestListSources.
Added support for standard Apify request-list sources with requestsFromUrl,
so inputs copied from Web Scraper or Website Content Crawler can resolve
remote URL lists instead of finishing with zero usable domains.
Accepted the common camelCase proxyConfiguration field and hidden dataset
aliases such as dataset / selectedDataset.
[0.4.55] - 2026-06-21
Tightened async cancellation handling
Kept the per-source stream guardrail from 0.4.54, but let forced task
cancellation and other system-level interruptions propagate normally.
[0.4.54] - 2026-06-21
Hardened run completion paths
Guarded OUTPUT and status-message writes so temporary Apify metadata API
issues do not turn an otherwise completed run into a failed run.
Isolated per-domain source stream exceptions so one unexpected source error is
recorded as unavailable data for that domain instead of bubbling out of the
worker pool.
Converted proxy URL generation failures into retryable source errors and made
HTTP client shutdown tolerant of close-time transport issues.
[0.4.53] - 2026-06-21
Hardened API input guardrails
Sanitized hidden proxy_configuration values before creating an Apify proxy
configuration, so malformed API inputs such as numeric proxy groups no longer
fail the whole run before a summary can be written.
Added a safe proxy fallback path: if a custom proxy configuration is rejected,
the Actor falls back to the default Residential proxy and, if that is
unavailable, finishes the run with normal per-domain source handling.
Clamped hidden retry, timeout, RDAP and concurrency settings to bounded values
so malformed API inputs cannot accidentally stretch a run to the platform
timeout.
Switched domain scheduling to a bounded worker pool, keeping large input lists
from creating every domain task at once.
[0.4.52] - 2026-06-17
Clearer malformed input summary
Inputs that include a domain field but no usable domain values now report
that no usable domains were found, instead of saying the domain input was
entirely missing.
[0.4.51] - 2026-06-17
Non-failing empty input handling
Empty domain input and inputs where every provided domain is invalid now
finish with a structured OUTPUT summary and zero dataset items instead of
marking the whole Apify run as failed.
Legacy API inputs that disable every old include flag now fall back to the
single default base_data mode instead of failing dataset selection.
[0.4.50] - 2026-06-16
Explicit input aliases
Updated the public input schema to accept the domain formats the Actor code
supports: the UI domains list, domainsText for newline/comma-separated
API input, legacy urls, and Apify-style startUrls request objects.
Kept dataset selection strict in the public schema: only base_data,
similar_sites, and aitdk are accepted, so old multi-dataset values do
not silently enable extra modes.
[0.4.49] - 2026-06-16
Corrected input guardrails
Invalid custom domain_timeout_seconds values are now ignored so the normal
per-mode timeout is used instead of accidentally shortening a domain to a
one-second budget.
Verified the public schema still rejects unsupported dataset names before a
container starts; the Actor continues to expose one selected dataset per run.
[0.4.48] - 2026-06-16
Safer public API inputs
Accepted common API shapes inside domains / legacy urls, including
newline/comma-separated strings and objects with url, domain, or
website fields.
Guarded hidden numeric and proxy input fields so malformed API values are
handled defensively instead of crashing the run.
[0.4.47] - 2026-06-16
Compact homepage metadata
Added a compact homepage metadata threshold for title/meta descriptions with
at least 8 usable words. This keeps concise homepage descriptions usable
without lowering the threshold for manifests or JSON bundle sources.
Improved anti-bot diagnostics for final URLs that redirect to captcha or
challenge pages, and for pages titled like Are you a robot?.
[0.4.46] - 2026-06-16
Direct-first AITDK homepage checks
Reordered AITDK homepage checks to try direct root and www homepage
requests before residential proxy requests. Many public homepages are less
likely to return anti-bot pages when they are fetched directly.
Kept residential proxy root and www checks as the immediate fallback, using
one sticky proxy session across both proxy homepage candidates.
[0.4.45] - 2026-06-16
Leaner AITDK homepage variants
Narrowed generic homepage variants to language query forms that keep the
final path on /, such as /?lang=en, /?hl=en, and /?locale=en_US.
Removed generic path variants such as /home and /en-US after volume
testing showed they often redirect into site sections or error pages instead
of the requested homepage.
[0.4.44] - 2026-06-16
Safer AITDK homepage variants
Added direct same-site homepage variant candidates for language and canonical
homepage paths such as /?lang=en, /?hl=en, /home, and /en-US.
Variants are accepted only when the final URL still points to the requested
domain homepage shape. Redirects into unrelated sections such as questions,
flights, premium pages, app configs, or sitemaps are rejected.
Removed keyword-candidate backoff delays. AITDK now moves between homepage
candidates immediately instead of waiting between different source checks.
Improved anti-bot detection for Just a moment, Bot or Not, and network
security block pages while keeping content-rich homepage pages usable.
[0.4.43] - 2026-06-16
Actor memory default removed
Removed the Actor-level defaultMemoryMbytes setting. Runs now use Apify's
platform default unless the user explicitly chooses another memory value.
Validation runs are performed with an explicit 512 MB memory setting so
network/source behavior is tested under the lowest practical resource
profile instead of being hidden by higher CPU allocation.
[0.4.42] - 2026-06-16
Explicit AITDK run memory default
Set the Actor's default memory to a fixed 2048 MB. This keeps the public
default explicit and predictable, avoids the previous input-size memory
formula, and performed better than 1024 MB in the 80-domain AITDK benchmark
while staying cheaper than 4096 MB.
[0.4.41] - 2026-06-16
Safer AITDK homepage fallbacks
Removed the generic app-state JSON fallback from AITDK keyword extraction.
Hydration state such as __NEXT_DATA__ can contain share URLs, tracking
parameters, build metadata, and client state that are not reliable homepage
content. AITDK now keeps only visible homepage text, title/meta fields,
JSON-LD structured data, and same-site Web App Manifest sources.
Replaced the dynamic memory expression with a fixed 1024 MB default run
memory. Bulk users can still raise run memory manually, but the Actor no
longer hides a 4096 MB allocation behind an input-size formula.
[0.4.39] - 2026-06-16
Generic AITDK homepage app-data fallbacks
Added dynamic default memory for Apify runs: small inputs start at 1024 MB,
larger domain batches scale up automatically, and 4096 MB is used for large
AITDK batches. This avoids the slow 256 MB default that can make network-heavy
runs hold proxy sessions much longer.
Added a generic app-state JSON fallback for thin homepages. When the visible
homepage body is only an app shell, AITDK can now use official same-page
embedded JSON such as __NEXT_DATA__, Nuxt data, Remix context, Apollo state,
or initial/preloaded state before marking keyword density unavailable.
App-state parsing is intentionally limited to semantic SEO/content fields
such as title, description, headline, summary, keywords, and tags. Generic
client state, share URLs, tracking parameters, build ids, and chunks are not
counted as keywords.
Added linked Web App Manifest discovery from the homepage HTML. AITDK now
tries same-site <link rel="manifest"> URLs before the standard manifest
paths, so hashed or custom manifest routes can be used without site-specific
rules.
Manifest parsing continues to ignore routes, chunks, URLs, icons, colors,
and other technical fields so keyword density is based on human-readable
homepage metadata only.
[0.4.38] - 2026-06-16
Internal AITDK validation build
Tested broad app-state extraction on the 80-domain benchmark and rejected the
broad variant because it could count client/share tracking state on some
sites. The public follow-up build keeps only the stricter semantic extractor.
[0.4.35] - 2026-06-16
Generic AITDK homepage metadata fallbacks
Added a generic same-page JSON-LD fallback for thin homepages. When the
visible homepage body is only an app shell, AITDK can now use official
application/ld+json WebSite/Organization metadata before marking keyword
density unavailable.
Added a fast generic same-domain Web App Manifest fallback for thin app-shell
homepages: /manifest.json and /site.webmanifest on the requested host and
www host. It is used only after an accessible homepage returns too little
text, and accepted only when it contains enough human-readable words.
Generic manifest checks use a short per-URL deadline so unavailable manifest
routes cannot slow a domain down for long.
Manifest JSON payloads are deduplicated across bare and www hosts before
keyword density is calculated.
AITDK homepage HTML checks can inspect the response body on HTTP 202/403
before discarding it. Real WAF/interstitial bodies are still rejected by the
visible-text block detector, but valid metadata returned with a non-200 edge
status can now be used.
[0.4.32] - 2026-06-16
Cleaner AITDK homepage sources
AITDK block-page detection now checks visible page text instead of raw HTML.
Valid homepages that load captcha-related helper scripts are no longer
discarded as captcha pages.
AITDK homepage fetches now keep one sticky proxy session per domain and then
try the same homepage directly when the proxy route is blocked. This reduces
missing keyword density caused by proxy-only WAF responses.
Thin SPA homepages can now use their official title/meta description as the
keyword source instead of being marked missing when the visible body is only
an app shell.
Added verified same-site metadata candidates for proxy-sensitive public
pages: PayPal's canonical US homepage, Booking's homepage component
metadata, ESPN's public active-sports homepage data endpoint, and X's app
manifest.
Captcha detection no longer treats normal Google reCAPTCHA form disclaimers
as an anti-bot page. Valid landing pages such as Netflix are kept when the
actual homepage content is visible.
Added Adidas' canonical US homepage as a clean same-site fallback, with both
direct and residential-proxy attempts when the root homepage is blocked by
the upstream edge.
Added stricter detection for real visible WAF/ad-block/interstitial pages so
blocked pages stay partial instead of becoming misleading keyword density.
[0.4.31] - 2026-06-14
Timeout cleanup logs
Hedged HTTP child tasks are now always cancelled and drained when a
per-domain timeout cancels the parent request. This removes scary
Task exception was never retrieved messages after an intentional timeout.
[0.4.30] - 2026-06-14
Timeout guardrails
Added a hard deadline around each low-level HTTP request so a stuck proxy,
TLS handshake, or HTTP client call is converted into a retryable source
timeout instead of holding the Actor run open.
Added a total deadline for hedged proxy requests. The backup proxy attempt
can still recover slow sessions, but the hedge loop can no longer wait
indefinitely.
Added a per-domain processing timeout so one stuck domain is saved as a
failed domain and the rest of the run can finish normally.
[0.4.29] - 2026-06-14
AITDK source quality checks
Added an official WHOIS fallback discovered through IANA for TLDs where a
registry RDAP route is missing or unavailable. The fallback maps into the
existing WHOIS/RDAP fields, so dataset shape stays unchanged.
AITDK keyword-density now requires enough usable homepage text before it is
saved as complete data. Very small source payloads and error-shell pages are
returned as partial keyword data instead of misleading top keywords.
Retry summaries now label these cases as thin_payload, making run output
clearer without treating low-content source responses as network failures.
[0.4.28] - 2026-06-14
Emoji schema labels restored
Restored UTF-8 emoji labels in the public input form and dataset views after
verifying the local README and schema files are stored as UTF-8.
Dataset tabs now use the visual labels shown in the README:
📊 Overview, 🚦 Traffic sources, 💫 Engagement,
🤖 AI traffic share, and 🆔 AITDK (WHOIS + keywords).
[0.4.27] - 2026-06-14
Booking metadata cloud route
Switched the Booking.com homepage component metadata candidate from the
www.booking.com/page/v0/chunk-metadata route to Booking's
accommodations.booking.com/orca/chunk-metadata route. It returns the same
homepage component metadata locally and avoids the www.booking.com WAF
challenge seen in Apify cloud.
[0.4.26] - 2026-06-14
Verified homepage component endpoints
Replaced the Booking.com weekend-deals GraphQL candidate with verified
CAPLA homepage component metadata endpoints loaded by Booking.com's own
homepage. The Actor now uses hero/search and trust/USP homepage copy instead
of unrelated deal-card data.
Booking.com homepage metadata is fetched with normal JSON GET requests and
no Residential proxy session, reducing retries for this hard-blocked
homepage pattern.
JSON keyword extraction now strips template placeholders such as
{num_reviews} before counting tokens.
[0.4.25] - 2026-06-14
Booking homepage endpoint routing
Booking.com's homepage data endpoint is now fetched directly instead of
through Residential proxy sessions. The endpoint is not a Similarweb source,
and direct routing avoids proxy-side empty responses observed in Apify runs.
[0.4.24] - 2026-06-14
Homepage endpoint candidates
Added a verified homepage-candidate layer for AITDK keyword extraction.
Standard domains still use only the bare-domain and www. homepages.
Added an Amazon mobile homepage candidate (/gp/aw/h.html) for Amazon
domains when the root homepage is blocked by an upstream challenge.
Added a Booking.com homepage data endpoint candidate for the public landing
page's weekend-deals component when the root homepage is blocked.
JSON homepage data is parsed from human-readable values only; technical
keys, IDs, image URLs, prices, and GraphQL error payloads are ignored.
[0.4.23] - 2026-06-14
Source summary labels
Kept the original homepage anti-bot reason when switching from bare-domain
homepage to the www. homepage variant, so run summaries group those source
checks as anti-bot challenges instead of generic source issues.
[0.4.22] - 2026-06-14
Homepage variant retry
AITDK homepage keyword extraction now tries both the bare domain homepage
and the www. homepage variant before marking hard-blocked homepage
keywords as unavailable. This keeps the fix within the homepage contract
without falling back to other same-domain pages.
[0.4.21] - 2026-06-14
AITDK partial log cleanup
Cleaned the AITDK progress and summary log wording for partial homepage
keyword results.
[0.4.20] - 2026-06-14
Homepage-only AITDK keywords
Removed the official sitemap keyword fallback from the default AITDK result.
keyword_density now contains homepage text only, so blocked homepages are
returned as partial AITDK items instead of mixing in keywords from other
same-domain pages.
Kept fast proxy recovery for transient homepage network issues, but stop
early on hard homepage anti-bot responses after a fresh proxy session has
already been tried.
Removed the redundant keyword-source column from the AITDK table view.
[0.4.19] - 2026-06-14
Clean schema labels
Removed emoji and corrupted Unicode symbols from Apify input and dataset
schema labels. Console tabs and field titles now use plain ASCII labels such
as Overview, Traffic sources, Engagement, AI traffic share, and
AITDK (WHOIS + keywords).
[0.4.18] - 2026-06-14
Faster fallback decisions
Stopped guessing default sitemap paths when a domain's robots.txt is
available but does not declare any Sitemap: directives. This avoids slow
timeouts on hard-block domains that publish no official sitemap entry.
Added early retry stop for blocked AITDK homepage keyword checks when the
official sitemap fallback is also unavailable.
[0.4.17] - 2026-06-14
Official keyword fallback
Added a generic AITDK keyword fallback for blocked homepages. When the
homepage is unavailable because of an upstream challenge, block, rate limit,
or empty text, the Actor now checks official domain-published sources:
robots.txt sitemap directives, sitemap indexes, gzip sitemaps, and a small
bounded set of same-domain crawlable pages.
Added keyword_density_source and keyword_density_source_urls to AITDK
output so users can see whether keyword density came from the homepage or
from official sitemap pages.
Added English/no-language sitemap preference and common web UI stopwords to
keep fallback keyword density cleaner and avoid navigation/cookie/footer
terms dominating the result.
[0.4.16] - 2026-06-14
Output tab default
Reordered the Actor output schema so Dataset items opens first in Apify
Console. The structured Run summary remains available as the second output
option.
[0.4.15] - 2026-06-14
Cleaner run output
Added a structured OUTPUT record in the default key-value store. Each run
now saves a machine-readable summary with processed domains, saved items,
data-quality counts, retry/source-availability counters, and samples of
partial or unsaved domains.
Added an Apify key-value store schema and linked the OUTPUT summary from
the Actor output schema, so the run summary is available from the Output tab
and API alongside the dataset items.
Reduced normal log noise by moving per-request HTTP traces and retry details
to debug logs. Standard logs now show high-level progress, saved results, and
a calm final summary instead of warning/error lines for expected upstream
source limitations.
[0.4.14] - 2026-06-14
Run diagnostics
Added retry diagnostics to the run summary. Large runs now report retry
counts grouped by stream and cause, such as timeout, proxy tunnel failure,
TLS error, HTTP 202 challenge, HTTP 403, HTTP 429, empty payload, or parse
failure.
Added partial-data diagnostics to the run summary for empty SimilarSites
lists, missing AITDK WHOIS/RDAP data, and missing AITDK keyword-density
data. This makes it easier to separate transient network recovery from
external hard-blocks or unavailable source data.
[0.4.13] - 2026-06-14
RDAP completeness for subdomains
AITDK now queries RDAP for the registrable domain instead of the full
hostname. Inputs such as cloud.google.com now use WHOIS/RDAP data for
google.com, while homepage keyword-density still uses the requested host.
Added direct RDAP registry endpoints for .io, .so, and .us to avoid
unnecessary fallback through rdap.org for those TLDs.
RDAP registrar parsing now also checks nested entity data, improving
registrar extraction for registries that omit a top-level registrar vCard.
[0.4.12] - 2026-06-14
HTML fetch hardening
Added hedged HTML fetching for similar_sites, matching the faster proxy
recovery already used by base_data.
Added an AITDK keyword-density retry budget with fresh proxy sessions. A
transient homepage block, timeout, or tunnel failure no longer leaves
keyword_density empty after a single failed request.
AITDK keyword-density retries now alternate between the bare domain and
www. homepage variant when the input domain does not already start with
www..
[0.4.11] - 2026-06-14
RDAP source routing
AITDK now resolves the official RDAP server for each TLD from IANA bootstrap
data and queries the registry endpoint directly instead of relying on
rdap.org for every domain.
RDAP still falls back to rdap.org through hedged residential proxy sessions
if the registry endpoint is unavailable, preserving a backup path for WHOIS
completeness.
[0.4.10] - 2026-06-14
WHOIS completeness on bulk AITDK runs
Added an RDAP-only concurrency and request-start limiter. Large AITDK runs no
longer send a burst of direct WHOIS/RDAP requests at once, reducing RDAP 429
rate-limit responses without slowing base_data or similar_sites.
Added an RDAP-specific retry budget before AITDK accepts a partial result.
A successful homepage keyword-density fetch no longer prevents the Actor from
retrying a transient WHOIS/RDAP failure.
[0.4.9] - 2026-06-14
Bulk-run network hardening
AITDK now fetches RDAP and homepage keyword-density data in parallel, so a
slow WHOIS lookup no longer delays the homepage request for the same domain.
RDAP lookups now use a direct connection instead of Apify Residential proxy.
This reduces proxy bandwidth and avoids unnecessary RDAP timeouts on larger
runs.
[0.4.8] - 2026-06-05
Faster proxy rotation
Added hedged proxy fetching for base_data. If the first Similarweb API
request does not answer quickly, the Actor starts a second request with a
fresh proxy session and uses the first successful response.
Fast proxy failures now trigger a fresh proxy session inside the same
fetch attempt, reducing first-run latency without exposing extra inputs.
[0.4.7] - 2026-06-05
Faster first-run defaults
Reduced the hidden request timeout from 30 seconds to 10 seconds so bad
residential proxy routes are discarded faster during one-domain test runs.
Reduced the hidden default retry budget from 5 to 4 attempts per selected
mode. The Actor still rotates to a fresh proxy session on every retry.
Shortened retry backoff delays because proxy rotation matters more than
waiting when an exit IP returns a block or tunnel failure.
[0.4.6] - 2026-06-05
Input form simplification
Removed the extra output-format selector from the public input form. Each
run now writes one dataset item per processed domain for the selected
dataset mode.
Removed the alternate per-stream output path from runtime handling. Old
JSON/API fields for that selector are ignored.
Updated README, input schema, output schema, and dataset field descriptions
so Store users only see the two choices that matter: domains and dataset
mode.
[0.4.4] - 2026-06-05
Release history cleanup
Rewrote the changelog in Apify build-revision style so the public
Changelog tab is easier to scan and does not mix code changes, Store copy,
and old marketing notes in one release.
Removed stale pricing-description notes from the changelog body. Pricing is
maintained in the Apify Pricing tab, not in the README or changelog.
[0.4.2] - 2026-06-05
Store title encoding
Actor title now uses an ASCII hyphen instead of a long dash. This avoids
Windows CLI / API encoding issues that could render the title with a ?
on the Apify page.
[0.4.1] - 2026-06-05
Single-mode input guardrails
datasets remains a single-select input with three supported modes:
base_data, similar_sites, and aitdk. The Actor does not expose an
all mode, preserving one paid dataset mode per run.
JSON/API inputs that pass an old array-style datasets value now use only
the first supported dataset instead of enabling multiple streams.
Legacy boolean inputs are resolved to one dataset stream, so older tasks
cannot silently run several modes at once.
Input schema examples now show a valid one-domain base_data run instead
of relying only on UI prefill values.
Empty-result billing safety
Aggregated mode no longer pushes an empty dataset item when the selected
stream fails for a domain. A failed domain without data now produces no
default-dataset row.
Runtime defaults
Hidden default concurrency is now 10, matching the README and the intended
bulk-run behavior.
README no longer includes a pricing section. Users see pricing in Apify's
Pricing tab.
[0.3.15] - 2026-05-26
Supported dataset cleanup
Removed the public indepth dataset path to avoid expensive traffic through
captcha-gated similarweb.com/website/{domain}/ pages.
Removed AWS WAF solver code and related dependencies because the supported
modes no longer need captcha-gated website pages.
Dataset selection is limited to base_data, similar_sites, and aitdk.
The input form uses a single starter domain so new users can test with one
click and no minimum batch size.
Aggregated output keeps similar_sites_data / aitdk_data and mirrors
selected-mode fields at the top level so Apify dataset views show useful
columns for every mode.
TLS impersonation remains pinned to the Chrome 120 profile for supported
HTTP calls.
[0.3.0] - 2026-05-17
Apify Store presentation
Polished the README, Actor title, input labels, dataset option titles, and
Apify dataset views for the public Store page.
Removed memory limits from actor.json; memory is configured per run in
Apify Console.
Hid advanced retry, timeout, proxy, country, and concurrency inputs from
the default UI. Power users can still override these values through JSON
input.
[0.2.0] - 2026-05-17
AITDK mode
Added aitdk, combining RDAP WHOIS lookup with homepage keyword-density
analysis.
RDAP output includes registrar, IANA ID, registration / expiration /
last-changed dates, name servers, EPP status codes, DNSSEC flag, and domain
handle.
Keyword-density output returns the top non-stopword homepage tokens with
count and density.
A partial AITDK result is accepted when either RDAP or keyword density
succeeds. Both sub-fetches must fail before the stream is retried.
Added an AITDK dataset view with registrar, registration date, expiry date,
name servers, EPP status, total word count, and top keywords.
Input naming
Renamed urls to domains, while keeping legacy urls parsing for older
tasks.
Added aitdk_data to the default output and added "aitdk" to
types_included.
[0.1.0] - 2026-05-17
Initial public release
Added base_data fetching from Similarweb's open SPA data endpoint with
ranks, engagement, monthly visits, traffic sources, top keywords, AI traffic
share, country breakdown, screenshots, and favicon.
Added similar_sites fetching from SimilarSites embedded page data.
Added initial indepth fetching for Similarweb website pages.
Added concurrent domain processing with configurable max_concurrency.
Added independent retry budgets with exponential backoff and jitter.
Added per-request Apify Proxy session rotation.
Added Chrome TLS / JA3 impersonation through curl_cffi.
Added optional proxy country pinning.
Added the initial structured output layout.
Added run summary logging.
Added input, output, and dataset schema files.
Added local parser tests and dataset validation tooling.
![Similarweb [$0.99💰] AI Traffic·Rank·Visits·Keywords avatar](https://images.apifyusercontent.com/lCPT9t5f7ENS4nz43dVak8U0-jaH6kQEvgRNiS2zYh0/rs:fill:76:76/cb:1/aHR0cHM6Ly9hcGlmeS1pbWFnZS11cGxvYWRzLXByb2QuczMudXMtZWFzdC0xLmFtYXpvbmF3cy5jb20vbmduOHZSRVNUa3NleVdFNnAtYWN0b3ItSDFEd2wzZHNreFBRTUk0b2wtMlJOOGhvN000MS1zaW1pbGFyd2ViLWx0ZC1sb2dvLXBuZ19zZWVrbG9nby0zOTA4NDcucG5n.webp){kind=avatar}
