# Changelog of Similarweb Scraper - Traffic, AI Traffic & WHOIS (`vortex_data/similarweb-scraper`) Actor - **URL**: https://apify.com/vortex\_data/similarweb-scraper/changelog.md - **Full Actor documentation**: https://apify.com/vortex\_data/similarweb-scraper.md ## Changelog All notable changes to this Actor are documented here. Public builds use Apify build revisions such as `0.4.19` and `0.3.15`. ### \[Unreleased] #### Removed - Keyword density is no longer taken from the public web archive. The column says "the keywords of this homepage", and a capture is the homepage as it was on some past day: measured over a 290-domain batch, the archive supplied 16 rows of 285 with a median age of 16 days, four of them over a year old and one from 2022. It also cost 78% of the run's wall clock, because it is a single shared host and every domain that reached it queued behind the others. Keyword density now comes from the site itself or is reported as unavailable, which is the true answer for a site that refuses everyone. Rows that used to be filled this way keep their WHOIS record and simply carry no keyword density. #### Added - A run now saves a `TIMELINE` record beside `OUTPUT`: every request, domain, retry and resource sample with the second it happened on. It answers what a run total cannot - when a batch slowed down or started being refused, which host or proxy exit was refusing, and how long each domain took. On a very large batch the requests that answered are thinned evenly so the record stays storable; every request that failed is kept, and its `counts` says how many were left out. #### Changed - Reading a page costs less. Two things it did on every document were paid for whether or not there was anything to find: the walk over every script tag looking for structured data, which a single scan for the media type settles when there is none, and folding each word to lower case one at a time where folding the text once does the same. Measured on the shapes a run actually meets - median page 203 kB, most of it script - a page evaluation falls from 14 ms to 9.8. This is the only thing that shortens a run at a given memory: the wall clock is the processor work a batch needs divided by the share the platform gives it, and at 512 MB a batch needing 27.6 processor seconds cannot finish in less than 293 however it is scheduled. - The pool now widens by one worker at a time while it is filling, instead of doubling four times a second. Doubling reached 42 domains in flight within twenty seconds, before twenty of them had finished and while the model that sizes the pool still had nothing to measure; the queueing that created inflated the very figure the model reads, and the model settled at 59 for the rest of the run. Pinned by hand, that width is worth nothing - the same batch takes the same time at 5 as at 50 - and it costs a round trip of 4.8 seconds against 0.5, 68 timed-out requests against 43, and 315 MB against 204. The ramp does not have to arrive anywhere: the model sets the width outright once it can speak, so a machine that wants 180 workers still gets them. The two light dataset modes pay a little for this, because their runs are over in seconds and the ramp is a larger part of them: `base_data` is unchanged and `similar_sites` takes 12.0 seconds where it took 10.6. Growing the ramp at the rate domains are finishing, so that it scales itself to the mode, was tried and was worse for both. - A proxy exit is retired when a source refuses it, which needs the source to be capable of accepting some other one. A site that refuses every exit is refusing the Actor, not the address, and retiring an exit for it buys a fresh TLS handshake and the same refusal. Measured on a 290-domain run: 101 proxied requests would have burned their exit and 54 of them came from 27 hosts that refused every exit they ever saw - booking.com, expedia, globo and hyatt among them, which the homepage ladder already records as refusing everything tried. A refusal now retires an exit only from a host that has answered one, and the run is bound by processor time, where a handshake is the most expensive thing a request does. - The blind guess at an app manifest is asked last instead of fifth. It is two conventional paths that most sites serve on neither: over two 290-domain batches it was reached by 109 and 119 domains and answered none of them, and one row in the whole corpus has come from it. It sat ahead of the locale variants, which are what answer for amazon.com, so a candidate that has answered waited behind one that had not. Nothing waits behind it now. A page that names its own manifest is a different matter and is still read. - A redirect to an anti-bot challenge is recognised by the vendor's own paths as well as by four English words. `captcha`, `challenge`, `are-you-a-robot` and `robot-check` only catch a refusal written in English; a site that sends a browser to DataDome's or PerimeterX's interstitial is now seen whatever language it speaks. It is the same list the page body is read against, because a redirect to a challenge and a challenge served in place are the same refusal. - The pause before a repeat is as long as the attempt that earned it. It was an exponential sequence from half a second, capped at four, jittered by a quarter - four magnitudes, none derived from anything, and two more of them again for the registry ladder. A pause is owed for two different reasons that were being served by one mechanism. The first is that the source said so, and `Retry-After` is not a guess: where a server sends it, it wins outright. The second is not returning to a struggling source before it has recovered, and there the only scale the source offers is what it just spent - a server that took eight seconds to fail is under strain, one that refused in twenty milliseconds is not. It no longer grows from repeat to repeat, because growth is congestion control and this Actor already has one acting on the same signal: the per-server window halves the moment a server says it is being asked too often. The same reasoning removes the second that a rate-limited registry used to earn, doubling from refusal to refusal - it gets the room it just spent refusing, and follows the server back down when the refusals get cheap. The pause is spread evenly between nothing and its full length, which is the whole of what jitter is for and needs no fraction of its own. A ladder whose next attempt goes somewhere else owes no pause at all: every attempt of the homepage ladder is a different address of the same site, and waiting there rests a server that is not being asked again. That is a fact about the ladder, so it is a yes or no rather than a delay set to zero. A domain's budget follows from this without a ceiling on a pause: a branch that pauses may spend what its requests cost twice over, and one that does not spends it once. - How many times a source is asked is now learned on the run instead of being fixed at four. Two of those four were justifiable - a source that fails once may have failed for the moment or for good, and asking again is the only thing that tells them apart - and the third and fourth asked a question that had already been answered. Whether a third attempt pays is not a fact about domains in general but about the run in progress, so the ladder starts at two and opens one step at a time: a depth is reached only once the depth before it has actually produced an answer on this run. A clean run stays at two and costs half of what four cost; a run against a source that is refusing opens as far as retrying keeps working, which is further than four if that is what the answers say. `max_attempts_base` / `_similar` / `_aitdk` become a ceiling for a caller who wants one, rather than a target. - The WHOIS deadline of five seconds is gone; a lookup gets the deadline the caller set, like every other request. Replayed against a 290-domain run, the five seconds bought nothing: they were applied to each socket operation rather than to the lookup, so the lookup was never bounded by them at all, and those 201 lookups spent 1022 seconds under them. The caller's own deadline, applied once to the whole lookup, spends the same 1022 seconds and refuses one answer of 148 - the one that took 10.09s against a declared 10 - while actually bounding the lookup, which a server dribbling one byte at a time could previously stretch without limit. - A domain's time budget is now the work it is allowed to do rather than a request timeout multiplied by a chosen figure. Five numbers decided it before - a 4x for the simple modes, an 8x for aitdk, a 60s floor, a 180s floor for aitdk and a 900s ceiling - and none was derived from anything; they were sized until no domain in one 290-domain batch hit them. Every ladder a domain runs is bounded, so the budget is the count of those bounds times what one request costs. What one request costs is two things the run knows about itself: the deadline the caller set, and how long a request is currently waiting for a place before it is even sent. That second term is the difference the multipliers were standing in for - the same batch through a narrow pool queues for minutes and through a wide one for nothing - and it is measured rather than assumed. The budget is worked out per domain, at the moment the domain starts, because a domain with more copies of its homepage to try has more work it is allowed to do. For `aitdk` at the default settings this comes to 220 seconds on an unqueued run against the 320 it used to be given, and the slowest domain of the measured batch took 216.6. It rises with queueing, which is when domains actually need it. - How much one server is asked at once is now settled by that server. It was three, applied to every registry there is - a guess about a stranger, and no batch of domains settles it for the next zone. Measured on one 290-domain run, rdap.verisign.com answered 118 of 135 requests while rdap.nic.ch refused 54%: the same cap was far too small for one and too large for the other. The window now opens at one, which is the absence of concurrency rather than a small guess, widens by one for each window of answers, and halves whenever a server says it is being asked too often. Its ceiling is what the run is allowed to have in flight altogether, so it is a fact of the run rather than a number of its own; a caller who owes a particular source less still says so with `rdap_concurrency`. - A proxy exit is no longer retired after a fixed number of requests. The default was fifty, on the reasoning that an address which has answered fifty times looks less like a browser. The run's own record refutes it: across all 398 proxied requests, no exit was used more than ten times, so the fifty never decided anything, and the refusal rate did not rise with reuse - 6% on a fresh exit, 28% over uses one to five, 12% over six to ten. An exit is retired when a source refuses it, which is the same policy driven by evidence rather than by a guess. A caller who meets a source that does punish reuse still says so with `proxy_slot_max_uses`. - An anti-bot page is now recognised by the machinery it ships and the name it gives itself, not by a character budget and a word count. Two numbers decided it before: only the first 5000 characters of a page were read, and a page of 120 usable words or more was assumed to be content. Both were fitted to pages someone had looked at, and a refusal longer than either slipped through while a real page carrying the wrong sentence was thrown away. What replaces them is exact. A page that loads Cloudflare's challenge platform, Imperva's resource, DataDome's captcha or PerimeterX's is a refusal, whatever language it is written in - the earlier check was thirty-two English phrases and could not see a site refusing in Turkish or Japanese at all. Failing that, the page's own title settles it: a challenge titles itself one, or has no title, while a real page carries its own name however long it is and wherever the sentence sits in it. Widgets an ordinary site puts in its own login form - reCAPTCHA, hCaptcha, Turnstile - are deliberately not treated as refusals. - Two request deadlines fitted to a sample are gone: four seconds for a follow-up homepage attempt and two for an app manifest. A caller already declares how long it will wait, in `request_timeout_seconds`, and these sat underneath it for particular paths on the strength of one 50-domain batch. The manifest's own justification - "a guess at conventional paths most sites do not serve" - stopped being true when the manifest became something read because the page named it. What remains is the caller's deadline, and above it the measurement the run makes of itself: the ninety-ninth percentile of what requests are taking now, counting the ones still running, which widens when the pool is busy instead of cutting off answers the pool itself delayed. That is a reading of the present rather than a fit to a past batch. - Keyword density no longer depends on three thresholds fitted to one corpus. Whether a domain got a keyword density at all was decided by 30 usable words for a page body, 10 for structured data and 8 for a title and description - each drawn on 43 homepages, and a line drawn on 43 pages is a fact about 43 pages. A page with 29 usable words was published as having none, and the caller was never told it had 29. A page is now read every way it can be read - the text it serves, what it says about itself, its structured data, and the manifest when it names one - and the reading that yielded the most usable words is published, with that count in `keyword_density_total_words`. That is a comparison between readings of one page, so it needs no constant and means the same for any domain. A caller who wants the old behaviour writes `total_words >= 30`, on their own domains, where such a line can be drawn. Three consequences worth knowing. More rows carry a keyword density, some of them small - the count says which. Fewer requests are made: a page that says something is not chased through ten more copies of itself hoping for one that clears a threshold. And the conventional `/site.webmanifest` guess is made only for a page that named no manifest and said nothing at all, where before it was tried for 119 domains in one batch and answered none of them. #### Fixed - A run used to end with a long stretch in which almost nothing was running. On a 290-domain batch it was 48 seconds of 219, and it was the same 48 to 70 seconds however wide the pool was and however much memory the run was given. It was one domain. Read request by request: everything about the site was finished in 1.8 seconds, and the remaining 46 went on four attempts at a registry that never answers, ten seconds apiece, every one of them after the last domain had been started and the pool stood empty. Four attempts because the registry budget counted the routes that can exist - the registry's own RDAP twice, then WHOIS, then the public mirror - while a zone that publishes no RDAP and a mirror with nothing to add leave one route, and the route table hands that same route back on all four attempts. A domain is now asked as often as it has routes: the first one twice, because a repeat is what tells a failure of the moment from a failure for good, and each further route once, because it is a different source and not a repeat. The same batch now takes 157 seconds with a 7-second tail and returns the same registration records, one more row and 3 fewer processor seconds. Cutting the depth instead was tried first and was wrong: depth there is not repetition - the third attempt at a domain that has RDAP is the WHOIS route and the fourth is the mirror - and cutting it cost three domains their whole registration record. - A registry is now waited for as long as registries take. The wait was the caller's request deadline, which is meant for a page fetched through a residential proxy and was being applied to a raw socket to a registry. Over eighteen runs and 2,253 answers a registry answers in 0.47 seconds at the median and 4.6 at the ninety-ninth percentile, while the wait was being stopped at 11.2 - so a registry that was never going to answer cost more than twice the slowest one that did. It now sets its own deadline from its own answers, which is what the page fetches have always done. - A deadline measured from what answered was measuring the wrong thing. Only the requests that beat the deadline were being recorded, so the deadline descended: each step down cut off more, which removed more of the slow ones from the record, which let it descend again. A request stopped before it answered is now kept as a lower bound - it took at least that long - which closes the loop, because the share of the record sitting at the deadline is the share of requests that exceed it. - A quantile needs as many observations as the event it describes. Twenty answers were enough for the hedge threshold, which is a ninety-fifth percentile and asks about one in twenty, and were being used for the deadline as well, which is a ninety-ninth and asks about one in a hundred - over twenty answers that is the largest of the twenty. It cost a domain its whole registration record: it was processed early in a run, when a couple of dozen registries had answered and all of them quickly, and its own registry, which answers in 0.3 seconds, was cut off four times at 0.6 to 0.8. - Every proxy exit was being retired after a single request, so each request paid for a fresh TLS handshake. Retiring exits early was removed a few changes ago in favour of retiring one when a source refuses it, and the default was set to zero to say "no early retirement" - but the coercion that reads it floors at one, so zero became one and the pool churned an exit per request. Found by writing the first tests for the code that decides what a run will do, which had none. - A registrant's country was read from one of the two places RDAP keeps it. A vCard address has seven components and the last is the country's name; RDAP also carries it as an ISO code in the address's own `CC` parameter, which RFC 8605 defines for exactly this, and that is where registries put it when they leave the component empty. Two of the 33 recorded responses - flipkart.co.in and lemonde.fr - name their registrant's country there and nowhere else, and both published none. The written name still wins where there is one, and a country on the registrar's address is still the registrar's. - `aiTopPromptsUnavailableReason` was empty for most of the domains it exists for. It exists to tell "this domain has no AI prompts" from "this domain is excluded from the feature", and it read only the source's written message. Measured over 54 payloads, 18 came back with no prompts and 4 of those carried a message: for the other 14 the source had answered in a status rather than in words, and the column said nothing. The status is now repeated as the source gave it - what the number means is Similarweb's to say and it does not, so it is not interpreted - and the sixteen payloads that carry no such block at all are still distinguishable, because there is nothing to repeat and the column stays empty. - Two shapes registries write dates in were dropped whole. Audited over the fifty recorded responses, 153 date values parsed and five looked like dates and did not - and those five were every date published by two registries. `2024-Aug-26` leads with the year and spells the month, which nothing here covered; `1.1.1991` and `31.8.2027` write a single-digit day or month, where the dotted shape required two. Both are read now, the dotted one still day first, and a day that does not exist is still refused. - A registry that lays its answer out with dot leaders - `domain name................: yle.fi` - was being read as though the dots were part of the field's name, so every one of its keys missed every list of spellings this Actor knows. Measured over the recorded responses: yle.fi published no columns at all, and now publishes eight - its name, its registrar, its four name servers, its status, its address and its country. The dots are typography and are removed from a key, in runs only: no key in those responses ends in a single dot, and two comment lines contain abbreviations that do. - A page written without spaces between its words was answered with fragments of them. Every adjacent pair of characters was published as a keyword, so `ニュース` - news - came back as `ニュ`, `ュー` and `ース`, three overlapping halves of one word, beside pairs straddling two words that mean nothing at all. Japanese, Chinese and every other language written this way got that. A dictionary would settle where a word ends and none is affordable here, but where a pair always sits can be known without one: if every occurrence of a pair is preceded and followed by the same characters, the longer sequence occurs exactly as often and says the same thing for more of the word. Run out in both directions, a pair grows into the word it was cut from and stops where what surrounds it starts to vary. A pair seen only once is left alone - nothing disagrees with its single occurrence, so growing it runs on to whatever is beside it, and one Chinese page answered with two whole sentences. A term wholly inside a longer one seen just as often is dropped, because it is the same count said twice. Measured on a Japanese news page: `ニュース`, `天気`, `天気予報`, `政治`, `スポーツ` where it used to be `ニュ`, `ュー`, `ース`, `気予`, `スポ`. A 30,000-character page takes 12 milliseconds. Korean is not helped by this - it is written with spaces and glues its particles to its words, which needs morphology rather than counting. - Keyword density was computed against 151 English words, whatever language the page was written in. A German homepage's six leading "keywords" came back `der`, `und`, `die`, `mit`, `dem` and one real word; a Russian or Turkish page fared the same. Most of the web is not in English, so most of what this column published was that language's articles and prepositions. A page says which language it is in, the same way it names its manifest, and that is now read: ``, `Content-Language`, `og:locale`, and a web app manifest's own `lang`. Where a page says nothing, the language whose stop words account for more of it than any other's do is the one used - a comparison between 58 lists rather than a share anyone has to choose - and if not one of them accounts for a single word, nothing is dropped. The lists come from `stopwordsiso`, which is reference data and nothing else: no models, and nothing fetched at runtime. Every stop-word list eats some common nouns, because common nouns are common. Measured over 55 plainly content words, the English one takes `research`, `world`, `work` and `home`. That is the price of the rest. #### Changed - The furniture of a website is no longer counted as what the website is about. `