π Spy on any website in seconds: traffic, rankings, top keywords, AI traffic share (ChatGPT/Claude/Gemini), competitors, similar sites & WHOIS β all from Similarweb. No login or API key. Bulk parallel scrape, captcha-resilient. Export to JSON/CSV/Excel. SEO, lead gen, research.
All notable changes to this Actor are documented here. Public builds use
Apify build revisions such as 0.4.19 and 0.3.15.
[Unreleased]
Removed
Keyword density is no longer taken from the public web archive. The column
says "the keywords of this homepage", and a capture is the homepage as it was
on some past day: measured over a 290-domain batch, the archive supplied 16
rows of 285 with a median age of 16 days, four of them over a year old and one
from 2022. It also cost 78% of the run's wall clock, because it is a single
shared host and every domain that reached it queued behind the others.
Keyword density now comes from the site itself or is reported as unavailable,
which is the true answer for a site that refuses everyone. Rows that used to
be filled this way keep their WHOIS record and simply carry no keyword
density.
Added
A run now saves a TIMELINE record beside OUTPUT: every request, domain,
retry and resource sample with the second it happened on. It answers what a
run total cannot - when a batch slowed down or started being refused, which
host or proxy exit was refusing, and how long each domain took. On a very
large batch the requests that answered are thinned evenly so the record stays
storable; every request that failed is kept, and its counts says how many
were left out.
Changed
Reading a page costs less. Two things it did on every document were paid for
whether or not there was anything to find: the walk over every script tag
looking for structured data, which a single scan for the media type settles
when there is none, and folding each word to lower case one at a time where
folding the text once does the same. Measured on the shapes a run actually
meets - median page 203 kB, most of it script - a page evaluation falls from
14 ms to 9.8. This is the only thing that shortens a run at a given memory:
the wall clock is the processor work a batch needs divided by the share the
platform gives it, and at 512 MB a batch needing 27.6 processor seconds
cannot finish in less than 293 however it is scheduled.
The pool now widens by one worker at a time while it is filling, instead of
doubling four times a second. Doubling reached 42 domains in flight within
twenty seconds, before twenty of them had finished and while the model that
sizes the pool still had nothing to measure; the queueing that created
inflated the very figure the model reads, and the model settled at 59 for the
rest of the run. Pinned by hand, that width is worth nothing - the same batch
takes the same time at 5 as at 50 - and it costs a round trip of 4.8 seconds
against 0.5, 68 timed-out requests against 43, and 315 MB against 204. The
ramp does not have to arrive anywhere: the model sets the width outright once
it can speak, so a machine that wants 180 workers still gets them.
The two light dataset modes pay a little for this, because their runs are
over in seconds and the ramp is a larger part of them: base_data is
unchanged and similar_sites takes 12.0 seconds where it took 10.6. Growing
the ramp at the rate domains are finishing, so that it scales itself to the
mode, was tried and was worse for both.
A proxy exit is retired when a source refuses it, which needs the source to be
capable of accepting some other one. A site that refuses every exit is
refusing the Actor, not the address, and retiring an exit for it buys a fresh
TLS handshake and the same refusal. Measured on a 290-domain run: 101 proxied
requests would have burned their exit and 54 of them came from 27 hosts that
refused every exit they ever saw - booking.com, expedia, globo and hyatt among
them, which the homepage ladder already records as refusing everything tried.
A refusal now retires an exit only from a host that has answered one, and the
run is bound by processor time, where a handshake is the most expensive thing
a request does.
The blind guess at an app manifest is asked last instead of fifth. It is two
conventional paths that most sites serve on neither: over two 290-domain
batches it was reached by 109 and 119 domains and answered none of them, and
one row in the whole corpus has come from it. It sat ahead of the locale
variants, which are what answer for amazon.com, so a candidate that has
answered waited behind one that had not. Nothing waits behind it now. A page
that names its own manifest is a different matter and is still read.
A redirect to an anti-bot challenge is recognised by the vendor's own paths
as well as by four English words. captcha, challenge, are-you-a-robot
and robot-check only catch a refusal written in English; a site that sends
a browser to DataDome's or PerimeterX's interstitial is now seen whatever
language it speaks. It is the same list the page body is read against,
because a redirect to a challenge and a challenge served in place are the
same refusal.
The pause before a repeat is as long as the attempt that earned it. It was an
exponential sequence from half a second, capped at four, jittered by a
quarter - four magnitudes, none derived from anything, and two more of them
again for the registry ladder.
A pause is owed for two different reasons that were being served by one
mechanism. The first is that the source said so, and Retry-After is not a
guess: where a server sends it, it wins outright. The second is not returning
to a struggling source before it has recovered, and there the only scale the
source offers is what it just spent - a server that took eight seconds to
fail is under strain, one that refused in twenty milliseconds is not.
It no longer grows from repeat to repeat, because growth is congestion
control and this Actor already has one acting on the same signal: the
per-server window halves the moment a server says it is being asked too
often. The same reasoning removes the second that a rate-limited registry
used to earn, doubling from refusal to refusal - it gets the room it just
spent refusing, and follows the server back down when the refusals get
cheap. The pause is spread evenly between nothing and its full length, which
is the whole of what jitter is for and needs no fraction of its own.
A ladder whose next attempt goes somewhere else owes no pause at all: every
attempt of the homepage ladder is a different address of the same site, and
waiting there rests a server that is not being asked again. That is a fact
about the ladder, so it is a yes or no rather than a delay set to zero.
A domain's budget follows from this without a ceiling on a pause: a branch
that pauses may spend what its requests cost twice over, and one that does
not spends it once.
How many times a source is asked is now learned on the run instead of being
fixed at four. Two of those four were justifiable - a source that fails once
may have failed for the moment or for good, and asking again is the only
thing that tells them apart - and the third and fourth asked a question that
had already been answered. Whether a third attempt pays is not a fact about
domains in general but about the run in progress, so the ladder starts at two
and opens one step at a time: a depth is reached only once the depth before
it has actually produced an answer on this run. A clean run stays at two and
costs half of what four cost; a run against a source that is refusing opens
as far as retrying keeps working, which is further than four if that is what
the answers say. max_attempts_base / _similar / _aitdk become a
ceiling for a caller who wants one, rather than a target.
The WHOIS deadline of five seconds is gone; a lookup gets the deadline the
caller set, like every other request. Replayed against a 290-domain run, the
five seconds bought nothing: they were applied to each socket operation
rather than to the lookup, so the lookup was never bounded by them at all,
and those 201 lookups spent 1022 seconds under them. The caller's own
deadline, applied once to the whole lookup, spends the same 1022 seconds and
refuses one answer of 148 - the one that took 10.09s against a declared 10 -
while actually bounding the lookup, which a server dribbling one byte at a
time could previously stretch without limit.
A domain's time budget is now the work it is allowed to do rather than a
request timeout multiplied by a chosen figure. Five numbers decided it
before - a 4x for the simple modes, an 8x for aitdk, a 60s floor, a 180s
floor for aitdk and a 900s ceiling - and none was derived from anything;
they were sized until no domain in one 290-domain batch hit them.
Every ladder a domain runs is bounded, so the budget is the count of those
bounds times what one request costs. What one request costs is two things the
run knows about itself: the deadline the caller set, and how long a request
is currently waiting for a place before it is even sent. That second term is
the difference the multipliers were standing in for - the same batch through
a narrow pool queues for minutes and through a wide one for nothing - and it
is measured rather than assumed. The budget is worked out per domain, at the
moment the domain starts, because a domain with more copies of its homepage
to try has more work it is allowed to do.
For aitdk at the default settings this comes to 220 seconds on an unqueued
run against the 320 it used to be given, and the slowest domain of the
measured batch took 216.6. It rises with queueing, which is when domains
actually need it.
How much one server is asked at once is now settled by that server. It was
three, applied to every registry there is - a guess about a stranger, and no
batch of domains settles it for the next zone. Measured on one 290-domain
run, rdap.verisign.com answered 118 of 135 requests while rdap.nic.ch refused
54%: the same cap was far too small for one and too large for the other. The
window now opens at one, which is the absence of concurrency rather than a
small guess, widens by one for each window of answers, and halves whenever a
server says it is being asked too often. Its ceiling is what the run is
allowed to have in flight altogether, so it is a fact of the run rather than
a number of its own; a caller who owes a particular source less still says so
with rdap_concurrency.
A proxy exit is no longer retired after a fixed number of requests. The
default was fifty, on the reasoning that an address which has answered fifty
times looks less like a browser. The run's own record refutes it: across all
398 proxied requests, no exit was used more than ten times, so the fifty never
decided anything, and the refusal rate did not rise with reuse - 6% on a fresh
exit, 28% over uses one to five, 12% over six to ten. An exit is retired when
a source refuses it, which is the same policy driven by evidence rather than
by a guess. A caller who meets a source that does punish reuse still says so
with proxy_slot_max_uses.
An anti-bot page is now recognised by the machinery it ships and the name it
gives itself, not by a character budget and a word count. Two numbers decided
it before: only the first 5000 characters of a page were read, and a page of
120 usable words or more was assumed to be content. Both were fitted to pages
someone had looked at, and a refusal longer than either slipped through while
a real page carrying the wrong sentence was thrown away.
What replaces them is exact. A page that loads Cloudflare's challenge
platform, Imperva's resource, DataDome's captcha or PerimeterX's is a
refusal, whatever language it is written in - the earlier check was thirty-two
English phrases and could not see a site refusing in Turkish or Japanese at
all. Failing that, the page's own title settles it: a challenge titles itself
one, or has no title, while a real page carries its own name however long it
is and wherever the sentence sits in it. Widgets an ordinary site puts in its
own login form - reCAPTCHA, hCaptcha, Turnstile - are deliberately not
treated as refusals.
Two request deadlines fitted to a sample are gone: four seconds for a
follow-up homepage attempt and two for an app manifest. A caller already
declares how long it will wait, in request_timeout_seconds, and these sat
underneath it for particular paths on the strength of one 50-domain batch.
The manifest's own justification - "a guess at conventional paths most sites
do not serve" - stopped being true when the manifest became something read
because the page named it. What remains is the caller's deadline, and above
it the measurement the run makes of itself: the ninety-ninth percentile of
what requests are taking now, counting the ones still running, which widens
when the pool is busy instead of cutting off answers the pool itself delayed.
That is a reading of the present rather than a fit to a past batch.
Keyword density no longer depends on three thresholds fitted to one corpus.
Whether a domain got a keyword density at all was decided by 30 usable words
for a page body, 10 for structured data and 8 for a title and description -
each drawn on 43 homepages, and a line drawn on 43 pages is a fact about 43
pages. A page with 29 usable words was published as having none, and the
caller was never told it had 29.
A page is now read every way it can be read - the text it serves, what it
says about itself, its structured data, and the manifest when it names one -
and the reading that yielded the most usable words is published, with that
count in keyword_density_total_words. That is a comparison between readings
of one page, so it needs no constant and means the same for any domain. A
caller who wants the old behaviour writes total_words >= 30, on their own
domains, where such a line can be drawn.
Three consequences worth knowing. More rows carry a keyword density, some of
them small - the count says which. Fewer requests are made: a page that says
something is not chased through ten more copies of itself hoping for one that
clears a threshold. And the conventional /site.webmanifest guess is made
only for a page that named no manifest and said nothing at all, where before
it was tried for 119 domains in one batch and answered none of them.
Fixed
A run used to end with a long stretch in which almost nothing was running.
On a 290-domain batch it was 48 seconds of 219, and it was the same 48 to 70
seconds however wide the pool was and however much memory the run was given.
It was one domain. Read request by request: everything about the site was
finished in 1.8 seconds, and the remaining 46 went on four attempts at a
registry that never answers, ten seconds apiece, every one of them after the
last domain had been started and the pool stood empty.
Four attempts because the registry budget counted the routes that can exist -
the registry's own RDAP twice, then WHOIS, then the public mirror - while a
zone that publishes no RDAP and a mirror with nothing to add leave one route,
and the route table hands that same route back on all four attempts. A domain
is now asked as often as it has routes: the first one twice, because a repeat
is what tells a failure of the moment from a failure for good, and each
further route once, because it is a different source and not a repeat. The
same batch now takes 157 seconds with a 7-second tail and returns the same
registration records, one more row and 3 fewer processor seconds.
Cutting the depth instead was tried first and was wrong: depth there is not
repetition - the third attempt at a domain that has RDAP is the WHOIS route
and the fourth is the mirror - and cutting it cost three domains their whole
registration record.
A registry is now waited for as long as registries take. The wait was the
caller's request deadline, which is meant for a page fetched through a
residential proxy and was being applied to a raw socket to a registry. Over
eighteen runs and 2,253 answers a registry answers in 0.47 seconds at the
median and 4.6 at the ninety-ninth percentile, while the wait was being
stopped at 11.2 - so a registry that was never going to answer cost more than
twice the slowest one that did. It now sets its own deadline from its own
answers, which is what the page fetches have always done.
A deadline measured from what answered was measuring the wrong thing. Only
the requests that beat the deadline were being recorded, so the deadline
descended: each step down cut off more, which removed more of the slow ones
from the record, which let it descend again. A request stopped before it
answered is now kept as a lower bound - it took at least that long - which
closes the loop, because the share of the record sitting at the deadline is
the share of requests that exceed it.
A quantile needs as many observations as the event it describes. Twenty
answers were enough for the hedge threshold, which is a ninety-fifth
percentile and asks about one in twenty, and were being used for the deadline
as well, which is a ninety-ninth and asks about one in a hundred - over
twenty answers that is the largest of the twenty. It cost a domain its whole
registration record: it was processed early in a run, when a couple of dozen
registries had answered and all of them quickly, and its own registry, which
answers in 0.3 seconds, was cut off four times at 0.6 to 0.8.
Every proxy exit was being retired after a single request, so each request
paid for a fresh TLS handshake. Retiring exits early was removed a few
changes ago in favour of retiring one when a source refuses it, and the
default was set to zero to say "no early retirement" - but the coercion that
reads it floors at one, so zero became one and the pool churned an exit per
request. Found by writing the first tests for the code that decides what a
run will do, which had none.
A registrant's country was read from one of the two places RDAP keeps it. A
vCard address has seven components and the last is the country's name; RDAP
also carries it as an ISO code in the address's own CC parameter, which RFC
8605 defines for exactly this, and that is where registries put it when they
leave the component empty. Two of the 33 recorded responses - flipkart.co.in
and lemonde.fr - name their registrant's country there and nowhere else, and
both published none. The written name still wins where there is one, and a
country on the registrar's address is still the registrar's.
aiTopPromptsUnavailableReason was empty for most of the domains it exists
for. It exists to tell "this domain has no AI prompts" from "this domain is
excluded from the feature", and it read only the source's written message.
Measured over 54 payloads, 18 came back with no prompts and 4 of those
carried a message: for the other 14 the source had answered in a status
rather than in words, and the column said nothing. The status is now repeated
as the source gave it - what the number means is Similarweb's to say and it
does not, so it is not interpreted - and the sixteen payloads that carry no
such block at all are still distinguishable, because there is nothing to
repeat and the column stays empty.
Two shapes registries write dates in were dropped whole. Audited over the
fifty recorded responses, 153 date values parsed and five looked like dates
and did not - and those five were every date published by two registries.
2024-Aug-26 leads with the year and spells the month, which nothing here
covered; 1.1.1991 and 31.8.2027 write a single-digit day or month, where
the dotted shape required two. Both are read now, the dotted one still day
first, and a day that does not exist is still refused.
A registry that lays its answer out with dot leaders -
domain name................: yle.fi
- was being read as though the dots were part
of the field's name, so every one of its keys missed every list of spellings
this Actor knows. Measured over the recorded responses: yle.fi published no
columns at all, and now publishes eight - its name, its registrar, its four
name servers, its status, its address and its country. The dots are
typography and are removed from a key, in runs only: no key in those
responses ends in a single dot, and two comment lines contain abbreviations
that do.
A page written without spaces between its words was answered with fragments
of them. Every adjacent pair of characters was published as a keyword, so
γγ₯γΌγΉ - news - came back as γγ₯, γ₯γΌ and γΌγΉ, three overlapping
halves of one word, beside pairs straddling two words that mean nothing at
all. Japanese, Chinese and every other language written this way got that.
A dictionary would settle where a word ends and none is affordable here, but
where a pair always sits can be known without one: if every occurrence of a
pair is preceded and followed by the same characters, the longer sequence
occurs exactly as often and says the same thing for more of the word. Run out
in both directions, a pair grows into the word it was cut from and stops
where what surrounds it starts to vary. A pair seen only once is left alone -
nothing disagrees with its single occurrence, so growing it runs on to
whatever is beside it, and one Chinese page answered with two whole
sentences. A term wholly inside a longer one seen just as often is dropped,
because it is the same count said twice.
Keyword density was computed against 151 English words, whatever language the
page was written in. A German homepage's six leading "keywords" came back
der, und, die, mit, dem and one real word; a Russian or Turkish page
fared the same. Most of the web is not in English, so most of what this column
published was that language's articles and prepositions.
A page says which language it is in, the same way it names its manifest, and
that is now read: <html lang>, Content-Language, og:locale, and a web
app manifest's own lang. Where a page says nothing, the language whose stop
words account for more of it than any other's do is the one used - a
comparison between 58 lists rather than a share anyone has to choose - and if
not one of them accounts for a single word, nothing is dropped. The lists come
from stopwordsiso, which is reference data and nothing else: no models, and
nothing fetched at runtime.
Every stop-word list eats some common nouns, because common nouns are common.
Measured over 55 plainly content words, the English one takes research,
world, work and home. That is the price of the rest.
Changed
The furniture of a website is no longer counted as what the website is about.
<nav>, <header>, <footer> and <aside> are what HTML names for that
content, so a page is read without them - which needs no list of words and
works in every language. Twenty-two words were doing this job inside the
English stop-word list - cookie, privacy, terms, login, menu, search - where
they had nothing to do with grammar and reached only English sites. They are
kept, named for the job they do, for the sites that mark up neither.
A page's title was counted twice in its own keyword density: once as the
metadata it is, and again as visible text, because the document head was not
stripped before the page's text was read. Every title word carried double the
weight of a word from the page. The head is metadata and is now read only as
metadata - which also makes "did the visible text say anything" an exact
question, and that is what now decides whether a reading is labelled
homepage or homepage_metadata instead of which side of 30 words it fell.
Requests to a registry are no longer spaced out by a fixed pause. On a
290-domain AITDK batch the 0.35-second default spread 123 requests to
Verisign's RDAP over 42.7 seconds and set the length of the whole run, while
Verisign refused none of them for rate - the pause was protecting a server
from something it was not objecting to. The standing politeness is now the
concurrency cap alone, a handful of requests in flight to one server, and any
pause on top of that is what a server asks for: it appears the moment one
answers that it is being asked too often, follows its Retry-After where it
sends one, and doubles on each further refusal. rdap_min_interval_seconds
still sets a floor for a caller who wants one.
Every field in the dataset schema now says what it is. Fifty-six of the
ninety-one carried a title and nothing else - including every traffic
channel, the ranks, the engagement figures and the WHOIS dates - so a reader
of the results had no way to tell a share from 0 to 1 apart from a
percentage, or whois_last_changed_date from the date a contact record
changed. The descriptions state the unit and, where it matters, what the
value is not.
Batches are no longer processed ten at a time. The Actor now runs as many
domains at once as the run's memory will hold, widening while it works and
measuring what the container is actually charging rather than following a
figure written into the source. A 290-domain AITDK batch that took about
three minutes now takes between eighty seconds and two minutes at 4096 MB -
the spread is the network, not the Actor - and returns the same rows. More
memory buys a larger share of CPU: the same batch takes about two minutes at
2048 MB, where the container spends thirty seconds waiting for processor
time, and four seconds at 4096 MB. Above that the wait is already gone and
the extra CPU sits idle, so a larger size buys time only for batches much
bigger than this one.
Changed
Asking for all, multi or multi-dataset is now reported as the
unsupported request it is. This Actor runs one dataset per run, and those
names had been aliases for base_data - so a caller asking for all three
was given one of them, and the alias made the request look recognised, so
the warning an unrecognised mode gets was suppressed too.
When a slow request gets a second copy on another proxy exit is now measured
rather than fixed at two seconds. On a 290-domain batch the requests that
answered had a median of 1.43s, so two seconds duplicated a third of the
requests that were going to answer anyway - each duplicate a wasted request
and a retired exit. The delay is the ninety-fifth percentile of what has
answered on the run so far, and nothing is hedged until enough requests have
answered for that to mean anything.
Fixed
A registration attempt that never asked the registry anything still spent one
of the domain's four. Whether a zone has a WHOIS server is only known after
asking IANA, and IANA times out - 17 of 56 lookups in one 290-domain run. The
answer came back as "no server" either way, so the ladder reported the domain
as having no registration route and moved on, while the route existed and was
reached two attempts later. Measured over that run, 25 attempts across 14
domains were spent without a single query going out, and sabah.tr, publico.pt
and tokopedia.my lost half their tries at the registry that way. IANA's answer
and the absence of one are now separate facts: a zone that publishes no WHOIS
server still moves the ladder along, while a lookup that did not happen gives
the attempt back - up to as many times as the budget has attempts, so a source
that can never be addressed still ends, and the domain's own deadline remains
the ceiling on how long it may take.
The TIMELINE record could still grow past what the platform will store. Its
thinning bounded the requests and nothing else, on the reasoning that they
are the most numerous; measured on a 290-domain run they are 370 kB of 697,
so that bounded the record by about half. A batch of five thousand domains
would have written about 12 MB and been refused whole - losing the record on
exactly the runs that raise questions, which is what the thinning existed to
prevent. The budget is now the platform's own limit, taken from the Apify
client's _MAX_PAYLOAD_SIZE_BYTES rather than chosen here, and it covers
requests, domains, retry events and ticks together. Everything that answers
a question no other row can - a request that failed, a domain that came back
with nothing, a retry that gave up - is kept before anything else, and the
record says how many rows of each kind there really were. Replayed against a
real record scaled to 58,000 domains: 6.5 MB, inside the limit.
Four columns published whatever type the source sent, against a schema that
promises one. category, description and snapshotDate passed the
extension payload's value through unread, and whois_status - declared a
list of strings - carried a bare string, a number, or a list with an object
in it, whichever the registry's JSON held. All four are read for their type
now. The dataset schema is the only thing telling a reader what a column
holds, and nothing had ever checked that the Actor agreed with it; a test
now parses hostile payloads through every key and checks every column
against the type its schema declares.
What a registry writes where it has nothing was being published as though it
were something. .tr answers -, .it writes a bare *; read into a
column those became a fax number of "-" and a status of "-". A line whose
value has no letter and no digit in it is now read as the empty box it is -
a shape rather than a list of the words registries use for nothing, so it
cannot be wrong about a word nobody has written yet.
whois_handle published the word REDACTED. .org answers
Registry Domain ID:REDACTED
, which two of the fifty recorded responses do
(sqlite.org and wikipedia.org), and a column of identifiers held a word that
identifies nothing. It goes through the same rule that already recognised a
refusal in the owner's fields. The registrar's name deliberately does not:
the Korean registry writes Gabia, Inc.(http://www.gabia.co.kr), which is a
registrar naming itself, not a refusal pointing elsewhere.
whois_name_servers accepted anything in the first column of the line,
including - and a web address. A name server is a host name, and the
column is now checked for one on both the WHOIS and the RDAP path - the same
reason the glue records beside it are asked of the address parser. Checked
against all fifty recorded responses: no real name server is refused.
Contact columns were publishing whatever the registry wrote there, including
prose. whois_abuse_email accepted n/a, REDACTED, - and the sentence
Please query the RDDS service; whois_abuse_phone accepted text with no
digit in it; whois_registrar_url accepted javascript:alert(1); and
whois_port43 accepted a web address or anything else. Read from the
dataset, each of those looks like a contact that simply does not work. The
four columns are now checked for the shape their name promises - the same
check the registrant's own email has always had, which is now the single
rule both use - and hold either a contact or nothing. Written-out at-signs
(domainreg AT walla.net.il, the .il convention) still count as
addresses.
Glue records that are not addresses were being published. The check was a
pattern - four groups of up to three digits - which also accepts
999.999.999.999 and 256.1.1.1, and an RDAP registry's address list was
passed through without any check at all. Both are asked of the address parser
now, the way a date is asked of the calendar.
One domain written two ways was looked up twice and saved twice. Duplicates
were removed by comparing the text, so example.com and www.example.com
passed as two, and so did https://example.com/ beside example.com, a
trailing dot, and two paths on the same site - each pair costing a second
lookup and putting a second row in the dataset for one site. They are
compared by the domain they name now.
Dates that do not exist were being published. The day was checked against 31,
which let through 2020-02-31, 2020-04-31 and 2021-02-29 - a day in a month
that has no such day, and a leap day in a year that has none - and they were
saved as registration and expiry dates where a reader could not tell them
from real ones. The calendar decides now, so a leap day is kept in a leap
year and refused in every other, and every shape the registries actually
write still reads.
IANA's bootstrap file, which decides where every RDAP lookup goes, is now
read as a document rather than assumed to be well formed. A page served in
its place, or a copy with a service written as a number, raised an error
where the honest reading is a file this Actor does not understand; the run
survived because the fetch is wrapped, but it recorded a crash.
An IP address passed as a domain is now rejected with the other invalid
entries instead of being looked up. 127.0.0.1 has dots and legal labels, so
it reached the sources and spent a whole domain's budget - minutes of it in
the aitdk mode - discovering that no registry has a record of it. It is
reported in invalidEntries, where a caller is told not to re-run something,
rather than in FAILED_DOMAINS, where they are told to.
A source that answered with the wrong shape lost the domain outright.
Twenty-three places read a block of fields, a list or a string without
checking it was one, so a payload carrying a number where a block belongs
raised an error the callers do not catch: the domain was dropped with no
retry and "non-retryable" in the log, rather than being reported as the parse
failure it is. The extension source's own parse failures escaped the same way
and skipped the website source that would have been tried next. A further
thirty were found inside nested structures, all of them in the RDAP reader,
which takes its payloads from several hundred independent registry
implementations: a registry that writes a single role where a list belongs,
or a block where a field name belongs, no longer takes the domain with it.
Found by putting every shape a source could send into every value at every
depth of all four parsers - sixty-nine unguarded reads in all - which is now
a test.
snapshotDate was an empty string where Similarweb reported no month, while
the item's own contract says null. An empty string is not a month, and a
consumer parsing dates breaks on it where a null it can test for.
redirect repeated the row's own domain instead of saying where a site went.
Measured across a 290-domain batch it did so for all 290 in base_data and
276 of 290 in similar_sites, because both sources name the site they
measured - which is usually the one that was asked about. A reader could not
tell a site that redirects from one that does not. The column now carries a
domain only when the source names a different one, in every mode, and is null
otherwise. The page title no longer falls back to the domain either, for the
same reason.
Widening the pool was throwing away answers that its own width had delayed.
A homepage attempt after the first was given four seconds, a number written
into the source; measured on a 290-domain AITDK batch, the median request
took 1.42 seconds with fewer than twenty domains in flight and 7.32 with
sixty to a hundred and fifty, with a third of them timing out. That deadline
is generous at the first and cuts off more than half the answers at the
second, so a busier run reported refusals that were really its own queue.
The deadline is now the ninety-ninth percentile of what requests are actually
taking on the run, which rises with the pool instead of being outrun by it.
The CPU share in the run summary was the platform's claim rather than a
measurement, and the two differ threefold: APIFY_DEDICATED_CPUS reports
0.094 cores at 512 MB while the container's own counters, and the platform's
own CPU-usage figure, both show a run there reaching 0.29. What the container
is seen to use is now measured and reported as cpuCoresSeen beside what it
was promised.
A similar_sites batch was killed for running out of memory at 512 MB, the
platform's default, which none of this had been measured at. The pool widened
to two hundred and seventy domains while almost nothing had come back, judged
them nearly free because a domain still in flight had not yet paid for its
answer, and the container was gone at 511.9 MB eight seconds in. Memory is now
charged against the domains that have finished, which makes the estimate wrong
on the safe side exactly while a batch is arriving.
The pool no longer widens until memory stops it. Width is bought with proxy
exits - each a TLS handshake through a residential address - and with the
duplicate requests a wider pool makes room for, and past the point where the
network is busy it buys nothing back. Measured on 290 domains of base_data
at 512 MB with the width fixed by hand: 25 finished in 5.6 seconds, 200 in
11.4, and both returned the same 290 rows. The width now follows domains
finished per second, which is the thing worth maximising.
A registry that never answers no longer costs ten seconds a domain. Measured
over 376 answered WHOIS lookups: median 0.29 seconds, ninety-ninth percentile
3.30, slowest 4.35 - not one arrived later than that. Against it, 38 lookups
that never answered spent 214 seconds between them, because they were being
given the general request timeout. The wait is now bounded by what the
protocol has been seen to need, and request_timeout_seconds still shortens
it further for a caller who wants that.
A domain whose server never answers no longer spends the whole homepage
ladder finding that out. Measured on excite.com: eleven candidates, ten of
them timing out at four seconds each against a host that said nothing over
HTTPS, and the eleventh - plain HTTP, a different scheme - answering in 0.27
seconds. Fifty-eight seconds for one domain, fifty-two of them waiting. A
different query string, path or certificate policy leaves the address the
same, so once an address has been asked and said nothing, the candidates that
would ask it again go to the back of the list and the ones that change the
address or the scheme are reached straight away. They go to the back rather
than away: amazon.com times out on its bare HTTPS address and then answers
that same address with a language parameter on it, so the poor bet is still
taken once the better ones are exhausted. A refusal is an answer, so a server
that refuses is still tried every way from the start.
A batch spent a third of its time closing proxy connections after the work
was already done. On a 290-domain base_data run every domain finished at
6.6 seconds and the run took 24, because the several hundred exits a batch
leaves open were being closed one after another. They are closed together
now.
The run summary counted a hedge as won whenever one was in flight and any
attempt answered, not when the hedge was the one that answered. A
290-domain similar_sites batch reported 197 wins out of 198 hedges while the
run's own record shows the hedge answered 42 times and the first attempt 173,
which made a second copy of every slow request look free.
The public RDAP mirror was asked about zones it cannot know. It redirects to
whichever service IANA's bootstrap file names, and that is the same file this
Actor reads for itself, so asking it about a zone the file does not name is
refused by construction: measured across four 290-domain batches, 224
requests, one answer, 235 seconds. It is now asked only when this Actor could
not fetch that file at all, which is the one case where it knows more.
Japanese domains returned no registration record at all, for two reasons.
JPRS answers in Japanese unless asked otherwise - it says so in the banner
above every response - so the keys came back as [η»ι²εΉ΄ζζ₯] and only the
name servers were readable; the Actor now asks in the way that registry
documents. And JPRS writes its fields as [Key] Value, sometimes behind
the item letter it numbers them with, rather than as Key: Value. A .jp
domain now returns its organisation, dates, name servers and status.
Dates were matched against four lists of exact key spellings, so a registry
that words one differently lost it. The Hungarian registry writes
record created
, which is plainly a creation date and matched none of the eight
spellings collected, and index.hu came back with no registration record at
all. A key is now recognised by the words registries actually use, so a
spelling nobody has met yet is read correctly the first time.
Swiss domains came back with no registration data at all. .ch publishes
RDAP, but IANA's bootstrap file does not list it, and SWITCH answers port-43
WHOIS with "Requests of this client are not permitted" - so every .ch domain
in a batch lost its WHOIS section, and those with no other data were dropped
entirely. A registry that is missing from the bootstrap is now asked at the
address RDAP conventionally answers on, and believed only if it resolves. That
covers .ch and .li, and it replaces two of the three TLDs that had been
written into an override table by hand.
A domain that returned nothing was reported as "No usable source data
returned", which is not something anyone can act on. Each source already said
why it gave up and all of it was being discarded. FAILED_DOMAINS now
carries what each source actually said - the status a site refused with, a
name that does not resolve, a registry with no record.
A domain that is not registered came back as though it were. When a registry
answers "no match", the WHOIS parser filled the record's name from the query
it had just sent, and the Actor treats that field as proof the lookup
succeeded - so an unregistered domain produced a WHOIS section holding its
own name and nothing else, counted as a complete item. A response is now
taken as a record only if it carries at least one fact about a registration,
which every real one does and no rejection does. This also covers the shape
that has no "no match" wording at all: DENIC answers an available .de with
Status: free, which used to read as a sparse record. Such domains are now
reported as having no record, and land in FAILED_DOMAINS for the caller to
see rather than in the dataset as a row of nothing.
One failed lookup to IANA cost every remaining domain in that TLD its WHOIS
section. The Actor asks IANA once per TLD which server to query and caches
the answer; a timeout or a rate-limit refusal was cached as "this TLD has no
WHOIS server", so a single hiccup at the start of a bulk run silently removed
the WHOIS section from every .de domain after it. Only IANA's actual answer
is cached now - including a genuine "no server", which .gr, .za and .ph
really do publish - and a failure to reach IANA is retried.
Dates from ccTLD registries were exported as the registry wrote them, so one
column held 2016-07-18T16:06:12.0Z, 17-01-2019, 10-Dec-2027,
1997/02/07, 2007. 02. 28. and Fri Feb 24 1995, and could not be sorted
or compared. They are now ISO 8601 throughout: a timestamp that was already
ISO is kept whole, so the WHOIS and RDAP routes fill the field identically,
and the rest become YYYY-MM-DD. A value that is not a date is no longer
passed off as one - .il publishes
changed: domain-registrar AT isoc.org.il 20190117(Assigned)
, a log line, which was exported verbatim as
the date the domain last changed. Registries that write one such line per
change now report the most recent, not the oldest.
A status made of more than one word was cut in half - .il's
Transfer Locked became Transfer - while a gTLD status carried the ICANN
page explaining it as though that were a second status. Statuses that name
several at once, such as .ru's REGISTERED, DELEGATED, VERIFIED, were
reported as one.
A domain Similarweb has not measured was reported as one measured at zero.
The source fills a block it has no data for with zeros rather than omitting
it, and those were exported as written - so a site came back stating that 0%
of its traffic arrives from search, 0% from direct and 0% from social, with a
bounce rate of 0% and sessions lasting 0 seconds. Those are claims about the
site; the truth is that the source does not know. Traffic shares are a
partition of a site's traffic, so all of them at zero is not a measurement a
site can have: naver.kr is measured at 5,273 visits with every channel at
zero. A block that is entirely zero is now reported as unknown, and the
metrics the source did measure are untouched. A single zero among real values
is still zero - a site can genuinely have no paid search.
Name servers from some registries carried a status on the end of the host.
The Finnish registry publishes ns-997.awsdns-60.net [OK] as the name, and
that went into whois_name_servers whole, where no resolver would accept it.
A single-country site could report a traffic share of 1.0000000000000002, so
a consumer filtering on share <= 1 dropped the row. Shares are rounded to a
precision finer than the source's own.
categoryRank was null for a whole class of domains. Similarweb returns the
same rank under either CategoryRank or GlobalCategoryRank and never under
both - over 54 domains, 42 carried the first, 3 carried the second and none
carried the two together - and only the first was read. baidu.cn, sina.com.cn
and visir.is are ranked 5062, 152 and 944 in their categories, and the Actor
reported no rank at all for each.
A postal address could be built out of three different people's. Registries
that write a flat list of contacts repeat address: for the registrant, then
the technical contact, then the registry's own - seznam.cz reported a
Radlicka/Praha/Milesovska hybrid. Only the run of lines belonging to the owner
is used now. The same fault put a street number where the owner's name goes:
repubblica.it reported its registrant as "15".
WHOIS responses from registries that answer in Latin-1 were decoded as UTF-8,
so every accented character became a replacement mark and globo.com.br
reported its owner as "Globo ComunicaοΏ½οΏ½o e ParticipaοΏ½οΏ½es". Port 43 carries no
charset, so UTF-8 is now tried strictly and Latin-1 used when it fails.
Failure reasons in the run summary were decided by searching the error
message for words, which made some of them wrong. Any domain whose name
contains "thin" - thinkpad.com, thinkgeek.com - had an empty payload, an
unreadable one, a parse failure and the source's own 500 all reported alike
as "Source returned too little usable text"; a proxy that could not be
reached at all was reported as "Other source issue". Three parts of the Actor
each kept their own list of words and their own order, so one failure could
be a timeout in the summary and a TLS issue to the proxy pool, and rewording
any message quietly changed how requests were retried and which proxy exits
were rotated away. Each failure
now carries its reason and HTTP status as data, decided where the failure
happens. The run summary also separates reasons that used to be lumped into
"Other source issue": a source's own server error, an unexpected status, and
an anti-bot page served with HTTP 200.
aiTopPrompts was empty on every domain, including the ones Similarweb does
answer for. The parser looked for the prompts block one level too deep -
under AiTrafficDetails.Traffic - while the source hangs it off
AiTrafficDetails itself. Across a 22-domain check the block was in the
outer position every time and the nested one never existed. Domains that get
prompts now carry them; domains Similarweb excludes from the feature keep
returning an empty list, as they always did.
WHOIS in aitdk mode reported the public suffix instead of the domain for
every multi-label suffix that was not in a hand-kept list of sixteen.
walla.co.il was looked up as co.il, rspb.org.uk as org.uk,
flipkart.co.in as co.in and clarin.com.ar as com.ar, so the
registry's own record came back looking like the domain's - registrar,
status and a plausible registration date included - and the result was still
labelled complete. The registrable domain is now read from the Public Suffix
List, which covers every suffix rather than the ones someone remembered.
flipkart.co.in now reports its real 2011 registration through GoDaddy in
place of the .co.in registry entry. Hosts with no registrable domain
beneath a public suffix - a bare co.uk, an IP address, an intranet name -
are looked up as given, so the query fails on its own terms instead of
quietly describing something else. The list is read from a snapshot shipped
inside the image, so nothing is fetched at run time.
A domain whose name does not resolve threw away a working proxy exit. Every
transport-level failure was treated as evidence that the exit was bad, and a
DNS failure reads as one - so the exit was retired and replaced, at the cost
of a fresh TLS handshake, for a name that would not have resolved from the
replacement either. A list with dead domains in it could work its way through
the pool this way. Those failures are now identified as DNS failures, keep
the exit, and are reported as "Domain name could not be resolved" instead of
a generic network issue.
A blocked proxy exit was kept and reused instead of being rotated away,
whenever the caller had asked to receive the block status rather than catch
it. The pool retires an exit that produces a 202 / 403 / 429 so the retry
lands on a fresh IP, but that only happened on the raised path: base_data
asks for 403 and 429 so it can word its own error, and the homepage keyword
ladder asks for 202 and 403 so it can move to the next candidate, and both
handed the refused exit straight back to the pool. A domain could then spend
its whole retry budget on the one IP that had already been turned away, and
be reported as having no data while other exits were available. Rotating a
refused exit is now the client's decision and retrying is the caller's, which
is how the pool was documented to work. A 5xx still does not retire an exit:
that is the origin failing, not the IP being refused.
Homepages were being refused because the Actor introduced itself as Chrome
120, a browser from December 2023. Holding the proxy exit and the headers
fixed and changing only that, openai.com served Chrome 142 on 6 of 6 exits
and Chrome 120 on 2 of 8 - and where Chrome 120 was refused, Chrome 142 was
served over the same address, so the address was never the problem. Over six
consecutive aitdk runs on that domain the old fingerprint lost it once and
hit 22 refusals along the way; the new one lost it never and hit none. The
Actor now presents Chrome 142, checked against every source it reads first.
Eleven columns across four of the six Console views were rendering empty no
matter what the run collected. A view lists the columns to show and,
separately, the fields to fetch, and the two had drifted apart: fetching
?view=aitdk returned eight fields for eleven declared columns. dataCoverage
on the overview, the organic/paid split on traffic sources, and
aiTrafficVisits, aiTrafficSources and aiReferralTraffic on AI traffic
were all affected - present in every export, invisible on the tab meant to
show them. Every view now fetches exactly what it displays. The AI traffic tab
drops its Copilot share column instead: the source caps the share chart at
three assistants and ranks Copilot fourth at best, so that column can never
fill. Copilot is still reported, with its rank, in aiTrafficSources, and the
aiTrafficShare* fields now say in the schema why a ranked assistant can
still have a null share.
The WHOIS tab led with whois_unicode_name, which was one of those dead
columns and would have been near-empty anyway: across 94 items it is set on 12
and repeats whois_ldh_name on all 12. The tab now leads with
whois_ldh_name, which is set on every item, and shows the registrar abuse
contact that was declared but never fetched. whois_unicode_name stays in the
exports for genuinely internationalised domains.
Entries that cannot be a domain no longer reach the sources. hello world,
notadomain, localhost and a..b used to be counted into the run total and
turned into real lookups; they now land in invalidEntries with a reason. The
check is deliberately shallow - two or more labels, each a legal hostname
label - and was verified against every domain string the Actor has been sent
in production, none of which it refuses. Internationalised domains still pass
in either form, and a trailing DNS-root dot is now trimmed instead of being
forwarded to sources that reject it.
A dataset mode the Actor does not support was silently swapped for
base_data, so a caller who wrote similar-sites with a hyphen paid for a
mode they had not asked for with nothing anywhere saying so. The fallback
stays, because an old task should still return something, but the run now
warns in the log and records it in the summary notes.
A proxy exit could go missing from the pool. Two requests arriving on one
pinned key before either was registered both opened an exit; the one that
lost the race still held the key, took the "stays pinned" path on release and
ended up in neither the pool nor the pin table while still counting against
the live total. The pool shrank by one exit each time it happened.
Added
dataSource on every base_data item, naming which Similarweb route
answered: extension for the data API, website for the public page used
when the API does not. It matters because the website page carries no AI
traffic breakdown at all - an empty AI block on an item from that route means
the route cannot see it, not that Similarweb reports none, and until now
nothing in the item said which.
Who a domain belongs to. Every route already received it and none of it was
read: whois_registrant_name, whois_registrant_organization,
whois_registrant_email, whois_registrant_phone,
whois_registrant_country and whois_registrant_address are now filled from
whichever of RDAP or WHOIS answered. Measured over the corpus, 16 of 50 WHOIS
responses and 10 of 33 RDAP responses carry owner information -
repubblica.it names Gedi Digital S.r.l, walla.co.il names Walla
Communications Ltd., naver.kr names NAVER Corp.
whois_redacted_fields says which owner fields the registry holds but will
not publish, so an empty owner column can be told from an unknown one. gTLD
registries redact these under ICANN's temporary specification and declare it
in RDAP's redaction block; others write the refusal into the field itself.
Those refusals are no longer exported as data: a domain used to be reported
as belonging to a company called "Not shown, please visit
www.dnsbelgium.beΒ ", with "Please query the RDDS service of the Registrar of
Record" as its email address.
whois_registrant_handle for registries that name the owner by an identifier
rather than a name - .se answers CSC09-56902750, .cz answers
SEZNAM-CZ-AS. Those are not names and no longer sit in the name column.
countryName beside every country code in countryShare and
website_traffic_by_country. Similarweb ships the country table in the same
response and it was being discarded, leaving every consumer to do its own ISO
lookup.
icon on each entry of aiTrafficSources, which the source publishes for
every assistant it names.
aiTopPromptsUnavailableReason. When aiTopPrompts came back empty there was
no way to tell a domain with no prompts from one Similarweb excludes from the
feature - the source says which, in a message that was being dropped. Four of
the 54 domains measured are excluded and now say so.
aiTrafficDistributionBoundary, the source's own label for the band it places
a site's AI traffic in ("<20K", "<1M", "<100M"). Passed through as written.
WHOIS in aitdk mode now reads the fields ccTLD registries publish, not only
the ones gTLD registries do. Measured across the WHOIS servers of 17
registries, the number of populated fields went from 74 to 103. .il and
.cl report a registrar the Actor used to leave empty; .nl, .be and .dk
report their name servers; .ru reports its status and expiry; .se reports
when it was last modified and when it was transferred; .cn reports its
registry id and both its dates. This matters more than it sounds: 42 of the
65 most common TLDs - .de, .ru, .jp, .it, .es, .ch, .cn, .se,
.kr and .co among them - have no RDAP service at all, so plain WHOIS is
the only route to their registration data.
Registries that answer in an indented block - a heading line, then the values
under it - are now read. Nominet publishes every .uk field that way, SIDN
publishes the registrar and name servers of every .nl that way, and the
Italian registry writes its headings as bare words with no colon at all; all
of it was previously skipped. .it now reports its registrar, the registrar's
website and its name servers, which were empty on every Italian domain.
Fields inside a block that names a party rather than the domain - Registrant,
Registrar, the admin and technical contacts - stay attached to that party, so
a contact record's own change date is no longer reported as the date the
domain changed.
Glue records from the WHOIS route. whois_name_server_details was documented
as always empty of addresses because "plain WHOIS lists nameservers without
glue records", which is not true of .ru: it writes them beside each name
server, and they were being discarded.
Keyword density now falls back to the public archive of the same homepage
when a site refuses every live route. Some homepages are served only to
addresses the site approves of: the identical request that is refused here
returns the ordinary page through a different address range, so no choice of
URL, header or browser fingerprint changes the outcome. The Internet Archive's
copy of that homepage is one URL of the same shape for every domain, which is
what makes it a rule rather than another special case. Measured over the
previously unreachable sites, it recovers amazon.com - which no per-domain
case ever did - and domains whose site has since gone offline entirely.
It is tried only after every live candidate has failed, so an ordinary run
never touches it.
keyword_density_source names the archive when it is used, and
keyword_density_source_urls carries the exact capture that was counted, its
date included. An archived copy is dated, sometimes by years, and an item now
says so rather than presenting it as today's page. Archived anti-bot pages are
rejected by the same checks as live ones, so a captured "checking your
browser" screen never becomes keyword data.
Changed
All six per-domain special cases in the homepage keyword ladder are gone,
replaced by one general rule that applies to every domain. Measured on the
platform: amazon, paypal, espn and x.com were served by the ordinary first
candidate and their special cases were never reached at all, while adidas
answers 403 with a block page to every URL and profile tried, so its special
case never worked either. booking.com's did work - it read two
component-metadata URLs on a host outside the site's firewall - but their
addresses are hashes of one front-end build and stop resolving whenever that
site next deploys, which would have cost a domain its keywords with nothing
in the log to say why. What replaces all six is described under Added.
booking.com now reports its keyword density as unavailable in aitdk mode,
and the item is marked partial. Its WHOIS section is unaffected, and so are
base_data and similar_sites. There is no public route to that homepage's
text: the live page answers 202 to every impersonation profile tried, the
Internet Archive excludes the domain, and Common Crawl's own crawler recorded
202 as well. Reporting that as unavailable is the accurate answer.
similar_sites and aitdk items no longer carry their payload twice. Each
was written once nested under similar_sites_data / aitdk_data and once
flattened alongside it - a container left from when one run could return
several datasets at a time. A run carries one mode now, so the item simply is
that payload: a 20-competitor similar_sites record halves in size, and CSV
exports stop offering two columns for every value. base_data items are
byte-for-byte unchanged, verified field by field over 22 live payloads.
Added whois_registrar_url, the registrar's own web page. RDAP publishes it
as the registrar entity's about link and gTLD WHOIS as Registrar URL; six
of the ten registries sampled provide it, and ccTLD registries mostly do not,
so it is null for those. All three WHOIS routes return it, so an item's shape
does not depend on which one answered.
similar_sites: each competitor now carries its own category, which the
source names for every entry and the Actor was dropping.
similar_sites: removed four per-competitor fields and one item field that
the source has stopped returning - title, globalRank, categoryRank and
tags on each competitor, and top_country on the item. A 300-entry sample
across 15 pages contains none of them, so they were exported as columns that
are null in every row while the documentation described them as data. The
requested site's own title, category, categoryRank and tags are
unaffected and still populated.
[0.4.81] - 2026-07-25
Fields the sources were returning but the Actor discarded
Added aiTrafficSources: every AI assistant Similarweb names for the domain,
with its rank and share. The five aiTrafficShare* columns are a fixed shape
over a variable ranked list, so assistants such as Grok and DeepSeek had
nowhere to go, and the share chart only carries the top few, which is why a
named column could read as null while the assistant was clearly ranked. The
old columns are unchanged for anyone already reading them.
Added aiTrafficVisits, the absolute number of AI-referred visits, and
aiReferralTraffic, which the source reports separately from genAiTraffic.
Added similarityRank and similarityScore to each entry in
similar_sites. The list was flat, giving no sign that the first competitor
is often far closer than the twentieth.
Added related_apps: mobile apps associated with the site, with platform,
store ranking and link.
Added searchOrganicTraffic, searchPaidTraffic, socialOrganicTraffic and
socialPaidTraffic. Both sources report search and social split by organic
versus paid, and the Actor was summing them away, hiding how much of a site's
search or social presence is bought. searchTraffic and socialTraffic keep
their combined meaning.
Made both base_data sources emit the same field set. The website fallback
was missing the new AI keys entirely rather than returning them empty, so a
record could change shape depending on which source answered.
Added whois_name_server_details with the IPv4 and IPv6 glue records the
registry publishes for each name server, and whois_unicode_name for the
readable form of an internationalised domain. All three WHOIS paths - RDAP,
the plain WHOIS fallback and the no-result case - now return one field set.
Added whois_abuse_email and whois_abuse_phone. Registrant, administrative
and technical contacts are redacted by the registries, but ICANN requires the
registrar abuse contact to stay public and it was present on every domain
checked. Registries nest it inside the registrar entity, which is why a
glance at the top-level roles suggests no contact data survives.
Added whois_transfer_date, whois_port43 and whois_dnssec_ds_records.
DNSSEC was reduced to a single boolean; the delegation signer records carry
the key tag, algorithm and digest.
These came from checking the parser against the RDAP object model in RFC 9083
rather than against sampled responses. Sampling three registries had suggested
the source was fully exploited; the specification named four members, five
secureDNS members and eleven entity roles that sampling had not surfaced.
Console views
Added a πͺ Similar sites view. The mode had no view of its own, so its
results were rendered through Overview, which shows five mostly empty columns
and none of the competitor list, similarity scores or related apps.
Removed aiTrafficShareCopilot from the AI view, where it was empty on every
row of a 500-domain sample: the source publishes shares only for its top few
assistants. The field itself is unchanged, and aiTrafficSources now covers
every assistant including the ones without a column.
Added aiReferralTraffic to the AI view, and the readable domain plus name
server addresses to the AITDK view.
Every column in every view is now a declared field that carries data on real
runs. One field stays permanently empty and is deliberately left out of the
views: top_country in similar_sites, because the upstream payload no longer
contains the key it was read from.
Checked and deliberately left alone: Competitors.TopSimilarityCompetitors is
empty on every domain sampled, GlobalCategoryRank is always null, Countries
is a static 252-entry country lookup identical for every domain, and the unused
RDAP keys are protocol metadata. Registrant, admin and technical contacts are
absent from RDAP responses themselves.
[0.4.80] - 2026-07-25
Faster aitdk runs
Capped the request deadline for follow-up homepage candidates at 4 seconds
while the first candidate keeps the full deadline. Tracing a 50-domain sample
showed 76% of the time went into attempts that ran to the full deadline, and
35 of 39 successes came from the very first candidate, so the later ones were
long shots against hosts that had already stopped answering.
Made RDAP rate limiting per registry instead of global. Politeness is owed to
each RDAP server, and a single shared limit made unrelated registries queue
behind each other. The per-server limit keeps the same conservative defaults
(3 concurrent requests, 0.35 s apart) while removing the queueing.
Measured over four runs of each build against the same 50-domain sample, since
a single run of this mode varies by up to a fifth depending on how the sampled
sites happen to respond: median wall time fell from 105.6 s to 68.9 s, a 35%
reduction, and the two ranges do not overlap (102.3-107.2 s before, 63.0-89.4 s
after). Keyword coverage was unchanged at 11-13 of 50 domains missing, and
WHOIS stayed at 50/50.
Raising worker concurrency was measured as well and does not help this mode:
10, 20 and 25 workers all landed within the same band, because wall time is set
by a dozen domains grinding the full candidate ladder against hosts that refuse
or never answer. Lifting the RDAP limits outright was also tried and produced
RDAP timeouts, which the per-server limit avoids.
Added character-bigram counting for Han and Katakana, which are written
without spaces and therefore cannot be split by a word pattern. Hiragana is
treated as a separator, since bigramming grammatical particles buries the
real terms.
This also fixes silent quality loss rather than only outright failures: a
Japanese page previously scraped past the 30-word threshold on stray English
boilerplate, producing keyword density made of words like "home" and
"copyright". The same page now yields 838 real terms.
Added two last-resort homepage candidates for aitdk, tried only after every
other option failed: one that ignores an invalid TLS certificate, and one
over plain HTTP for hosts that never answer on 443. Certificate validation is
unchanged everywhere else; these requests only read public page text.
On a fixed 50-domain sample, domains ending without keyword density dropped
from 19 to 12. The remainder are refusals and dead hosts rather than parsing
gaps: 3 x HTTP 403, 3 x too little text on the page, 2 x transport failure, and
one each of empty response, HTTP 429, timeout and an unclassified error. The
extra candidates make an aitdk run about 18% longer, since they are attempted
for exactly the domains that would otherwise return nothing.
[0.4.78] - 2026-07-25
Honest reporting for domains Similarweb barely covers
Added a dataCoverage field to every base_data item: full when the
domain is ranked, small_site when it sits below Similarweb's ranking
threshold but descriptive data or visit estimates are still returned, and
no_data when the source answers with every metric null or zero.
Reported those rows in a new sourceCoverage block (ranked, smallSites,
noData) kept deliberately separate from dataQuality. Thin coverage is a
property of Similarweb's data, not a scraping shortfall, so these rows stay
counted as complete results: they already carry every value the source
provides. dataQuality.partialItems continues to mean only what the Actor
itself could not retrieve.
Added the coverage column to the Console overview so a thin row is visible
without opening the record.
On a 500-domain sample spread across the whole Majestic rank range, the split
was 301 full, 120 small_site and 79 no_data. The small_site rows are
not empty: they keep title, description, category, monthly visit estimates and
screenshots, and the existing isSmall flag already matched every unranked row
exactly. The no_data rows cannot be filled from the second source either -
Similarweb returns the same empty record for them.
[0.4.76] - 2026-07-25
Sticky proxy exits instead of a new session per request
Replaced the "fresh Apify Proxy session on every HTTP call" behaviour with a
pool of sticky exits. Each slot owns one AsyncSession pinned to a single
proxy session id, so consecutive requests reuse the established tunnel.
Profiling put the Chrome-fingerprint TLS handshake at ~96% of per-domain CPU,
which was the binding constraint on the 512 MB default run.
Kept IP rotation, but driven by evidence rather than by every request: an
exit is burned and replaced on a block signal (202 / 403 / 429 / proxy
connect / timeout) or after proxy_slot_max_uses requests (default 50).
Sized the pool at twice the worker count so a hedged request always lands on
a different exit instead of waiting for the worker holding the primary one.
Released the domain-pinned exit when aitdk finishes a domain; without it
the pool drained after one pinned slot per concurrent worker.
Measured on the same 500-domain list, base_data, concurrency 10: a run that
took 83.4 s now takes 21.4 s (5.99 -> 23.35 domains/s), with retry attempts
down from 18 to 6 and the proxy-layer error classes gone entirely. Saved rows
and failure count were unchanged at 500/500 and 0. similar_sites over 50
domains went from 7.1 s to 3.1 s with an identical 14 complete / 36 partial
split. aitdk was unchanged in both speed and quality, since that mode fetches
RDAP and the first homepage candidates without a proxy.
Re-runnable list of missing domains
Added a FAILED_DOMAINS key-value store record with the complete list of
domains that produced no dataset item, plus a domainsText form that can be
pasted straight back into the Domains field. OUTPUT.dataQualityfailedResultsSample stays capped at 50 entries, so on bulk runs the full
list previously existed only in the run log and users had to repeat the whole
batch to recover a handful of domains.
Reported input lines rejected during validation as invalidEntries in the
same record and as dataQuality.invalidEntries in OUTPUT, kept out of the
re-run list because they would fail again.
Wrote the record on every terminating path, including empty / invalid input
runs, so count: 0 is a reliable "nothing is missing" signal.
Named the missing-domain count in the terminal run status message and the run
summary log instead of leaving a partially complete batch looking identical
to a complete one.
[0.4.75] - 2026-07-14
Input compatibility
Added support for copied/API inputs that pass domains through websites,
websiteUrls, or website_urls. These aliases now resolve the same way as
domains and urls instead of finishing with zero processed domains.
[0.4.74] - 2026-06-29
Expanded traffic-source output
Added displayAdsTraffic and genAiTraffic to base_data output.
Added both fields to the Apify Dataset schema and the Traffic sources table
view so users can see the full traffic-source breakdown returned by the
current Similarweb payload.
[0.4.73] - 2026-06-29
Traffic-source parsing
Fixed base_data traffic-source fields for the current Similarweb extension
payload. SearchOrganic + SearchPaid now populate searchTraffic,
SocialOrganic + SocialPaid populate socialTraffic, and Affiliate
populates paidReferralsTraffic.
[0.4.72] - 2026-06-26
Input prefill visibility
Added an explicit datasetMode prefill value so Apify Console's JSON input
view shows "datasetMode": "base_data" together with the 20 prefilled
domains.
[0.4.71] - 2026-06-26
Dataset selector conflict handling
Fixed mixed old/new inputs where the visible datasetMode is
similar_sites but a stale legacy datasets=base_data value is also
present. The explicit public selector now wins in that case.
Kept the intended API compatibility path where a legacy datasets value can
still override Apify's injected public default datasetMode=base_data.
[0.4.70] - 2026-06-25
Build archive cleanup
Kept local tmp_*.json test inputs out of future Apify upload archives.
No runtime behavior changed.
[0.4.69] - 2026-06-25
Base-data default restored
Restored base_data as the public input default now that the clean HTTP
Similarweb data endpoint is stable again.
Expanded the public input prefill to 20 common domains so first-time users
can run a meaningful bulk base_data test immediately.
Updated README examples to match the public input default.
[0.4.68] - 2026-06-25
Base-data clean HTTP source
Switched base_data back to the Similarweb extension JSON endpoint with
extension-style HTTP headers instead of relying on the captcha-gated website
page.
Treats temporary 403 / 429 responses from the data endpoint as retryable
source issues so proxy session rotation can recover before the run finishes.
Added a generic AITDK app-manifest metadata candidate as a clean HTTP
fallback for blocked homepages.
[0.4.64] - 2026-06-25
Base-data source refresh
Added a parser for Similarweb's current website SSR payload
(window.__APP_DATA__) and made base_data try
https://www.similarweb.com/website/{domain}/ before the legacy extension
endpoint.
Kept the base-data contract strict: if Similarweb returns an anti-bot
challenge or a payload for a different domain, no guessed/fallback item is
saved as base_data.
Verified in Apify cloud that the new parser is ready, but the public
no-browser HTTP path can still return challenge_202; similar_sites
remains unaffected.
[0.4.60] - 2026-06-25
Base-data zero-result cost reduction
Stopped retrying the public Similarweb base-data endpoint after HTTP 403.
This endpoint currently returns a CloudFront refusal for tested domains, so
repeated proxy attempts only make zero-result runs slower and more expensive.
Kept the result contract unchanged: no fallback data is saved as base_data
unless the base-data source itself returns a usable payload.
Fixed copied/API inputs where Apify injects the public datasetMode default
(base_data) alongside a legacy dataset or datasets value. A supported
legacy value such as similar_sites or aitdk now wins over the injected
default instead of accidentally running base_data.
Changed the public input default to similar_sites so first-time UI runs use
a currently available source instead of the public base-data endpoint that is
returning HTTP 403.
[0.4.59] - 2026-06-25
Zero-result run cost guardrail
Added an optional actor-start pay-per-event charge hook. When this event is
configured in the Actor pricing tab, empty or invalid-input runs can cover
their startup/platform cost instead of relying only on per-result charges.
Stopped before source requests when the user's max charge limit does not
allow the start event to be charged.
Kept the hook backward-compatible: if the actor-start event is not yet
configured, the Actor continues normally without calling the charge API.
[0.4.58] - 2026-06-21
Documentation alignment
Updated README input examples to use the current public datasetMode field;
the older datasets key remains accepted as a legacy/API alias.
[0.4.57] - 2026-06-21
Safer copied-input handling
Moved the public dataset selector to datasetMode while keeping datasets
as a flexible legacy/API input, so copied JSON with a foreign datasets
array no longer fails schema validation before the Actor starts.
Stopped declaring flexible copied-input aliases as hidden schema fields; the
Actor still parses them, but Apify no longer rejects valid copied shapes such
as startUrl: { "url": "https://example.com" }.
Accepted top-level requestsFromUrl values in addition to request-list
objects inside startUrls, requests, or requestListSources.
[0.4.56] - 2026-06-21
Broader copied-input compatibility
Accepted more common copied Actor input aliases: single url, domain,
website, websiteUrl, startUrl, requests, and requestListSources.
Added support for standard Apify request-list sources with requestsFromUrl,
so inputs copied from Web Scraper or Website Content Crawler can resolve
remote URL lists instead of finishing with zero usable domains.
Accepted the common camelCase proxyConfiguration field and hidden dataset
aliases such as dataset / selectedDataset.
[0.4.55] - 2026-06-21
Tightened async cancellation handling
Kept the per-source stream guardrail from 0.4.54, but let forced task
cancellation and other system-level interruptions propagate normally.
[0.4.54] - 2026-06-21
Hardened run completion paths
Guarded OUTPUT and status-message writes so temporary Apify metadata API
issues do not turn an otherwise completed run into a failed run.
Isolated per-domain source stream exceptions so one unexpected source error is
recorded as unavailable data for that domain instead of bubbling out of the
worker pool.
Converted proxy URL generation failures into retryable source errors and made
HTTP client shutdown tolerant of close-time transport issues.
[0.4.53] - 2026-06-21
Hardened API input guardrails
Sanitized hidden proxy_configuration values before creating an Apify proxy
configuration, so malformed API inputs such as numeric proxy groups no longer
fail the whole run before a summary can be written.
Added a safe proxy fallback path: if a custom proxy configuration is rejected,
the Actor falls back to the default Residential proxy and, if that is
unavailable, finishes the run with normal per-domain source handling.
Clamped hidden retry, timeout, RDAP and concurrency settings to bounded values
so malformed API inputs cannot accidentally stretch a run to the platform
timeout.
Switched domain scheduling to a bounded worker pool, keeping large input lists
from creating every domain task at once.
[0.4.52] - 2026-06-17
Clearer malformed input summary
Inputs that include a domain field but no usable domain values now report
that no usable domains were found, instead of saying the domain input was
entirely missing.
[0.4.51] - 2026-06-17
Non-failing empty input handling
Empty domain input and inputs where every provided domain is invalid now
finish with a structured OUTPUT summary and zero dataset items instead of
marking the whole Apify run as failed.
Legacy API inputs that disable every old include flag now fall back to the
single default base_data mode instead of failing dataset selection.
[0.4.50] - 2026-06-16
Explicit input aliases
Updated the public input schema to accept the domain formats the Actor code
supports: the UI domains list, domainsText for newline/comma-separated
API input, legacy urls, and Apify-style startUrls request objects.
Kept dataset selection strict in the public schema: only base_data,
similar_sites, and aitdk are accepted, so old multi-dataset values do
not silently enable extra modes.
[0.4.49] - 2026-06-16
Corrected input guardrails
Invalid custom domain_timeout_seconds values are now ignored so the normal
per-mode timeout is used instead of accidentally shortening a domain to a
one-second budget.
Verified the public schema still rejects unsupported dataset names before a
container starts; the Actor continues to expose one selected dataset per run.
[0.4.48] - 2026-06-16
Safer public API inputs
Accepted common API shapes inside domains / legacy urls, including
newline/comma-separated strings and objects with url, domain, or
website fields.
Guarded hidden numeric and proxy input fields so malformed API values are
handled defensively instead of crashing the run.
[0.4.47] - 2026-06-16
Compact homepage metadata
Added a compact homepage metadata threshold for title/meta descriptions with
at least 8 usable words. This keeps concise homepage descriptions usable
without lowering the threshold for manifests or JSON bundle sources.
Improved anti-bot diagnostics for final URLs that redirect to captcha or
challenge pages, and for pages titled like Are you a robot?.
[0.4.46] - 2026-06-16
Direct-first AITDK homepage checks
Reordered AITDK homepage checks to try direct root and www homepage
requests before residential proxy requests. Many public homepages are less
likely to return anti-bot pages when they are fetched directly.
Kept residential proxy root and www checks as the immediate fallback, using
one sticky proxy session across both proxy homepage candidates.
[0.4.45] - 2026-06-16
Leaner AITDK homepage variants
Narrowed generic homepage variants to language query forms that keep the
final path on /, such as /?lang=en, /?hl=en, and /?locale=en_US.
Removed generic path variants such as /home and /en-US after volume
testing showed they often redirect into site sections or error pages instead
of the requested homepage.
[0.4.44] - 2026-06-16
Safer AITDK homepage variants
Added direct same-site homepage variant candidates for language and canonical
homepage paths such as /?lang=en, /?hl=en, /home, and /en-US.
Variants are accepted only when the final URL still points to the requested
domain homepage shape. Redirects into unrelated sections such as questions,
flights, premium pages, app configs, or sitemaps are rejected.
Removed keyword-candidate backoff delays. AITDK now moves between homepage
candidates immediately instead of waiting between different source checks.
Improved anti-bot detection for Just a moment, Bot or Not, and network
security block pages while keeping content-rich homepage pages usable.
[0.4.43] - 2026-06-16
Actor memory default removed
Removed the Actor-level defaultMemoryMbytes setting. Runs now use Apify's
platform default unless the user explicitly chooses another memory value.
Validation runs are performed with an explicit 512 MB memory setting so
network/source behavior is tested under the lowest practical resource
profile instead of being hidden by higher CPU allocation.
[0.4.42] - 2026-06-16
Explicit AITDK run memory default
Set the Actor's default memory to a fixed 2048 MB. This keeps the public
default explicit and predictable, avoids the previous input-size memory
formula, and performed better than 1024 MB in the 80-domain AITDK benchmark
while staying cheaper than 4096 MB.
[0.4.41] - 2026-06-16
Safer AITDK homepage fallbacks
Removed the generic app-state JSON fallback from AITDK keyword extraction.
Hydration state such as __NEXT_DATA__ can contain share URLs, tracking
parameters, build metadata, and client state that are not reliable homepage
content. AITDK now keeps only visible homepage text, title/meta fields,
JSON-LD structured data, and same-site Web App Manifest sources.
Replaced the dynamic memory expression with a fixed 1024 MB default run
memory. Bulk users can still raise run memory manually, but the Actor no
longer hides a 4096 MB allocation behind an input-size formula.
[0.4.39] - 2026-06-16
Generic AITDK homepage app-data fallbacks
Added dynamic default memory for Apify runs: small inputs start at 1024 MB,
larger domain batches scale up automatically, and 4096 MB is used for large
AITDK batches. This avoids the slow 256 MB default that can make network-heavy
runs hold proxy sessions much longer.
Added a generic app-state JSON fallback for thin homepages. When the visible
homepage body is only an app shell, AITDK can now use official same-page
embedded JSON such as __NEXT_DATA__, Nuxt data, Remix context, Apollo state,
or initial/preloaded state before marking keyword density unavailable.
App-state parsing is intentionally limited to semantic SEO/content fields
such as title, description, headline, summary, keywords, and tags. Generic
client state, share URLs, tracking parameters, build ids, and chunks are not
counted as keywords.
Added linked Web App Manifest discovery from the homepage HTML. AITDK now
tries same-site <link rel="manifest"> URLs before the standard manifest
paths, so hashed or custom manifest routes can be used without site-specific
rules.
Manifest parsing continues to ignore routes, chunks, URLs, icons, colors,
and other technical fields so keyword density is based on human-readable
homepage metadata only.
[0.4.38] - 2026-06-16
Internal AITDK validation build
Tested broad app-state extraction on the 80-domain benchmark and rejected the
broad variant because it could count client/share tracking state on some
sites. The public follow-up build keeps only the stricter semantic extractor.
[0.4.35] - 2026-06-16
Generic AITDK homepage metadata fallbacks
Added a generic same-page JSON-LD fallback for thin homepages. When the
visible homepage body is only an app shell, AITDK can now use official
application/ld+json WebSite/Organization metadata before marking keyword
density unavailable.
Added a fast generic same-domain Web App Manifest fallback for thin app-shell
homepages: /manifest.json and /site.webmanifest on the requested host and
www host. It is used only after an accessible homepage returns too little
text, and accepted only when it contains enough human-readable words.
Generic manifest checks use a short per-URL deadline so unavailable manifest
routes cannot slow a domain down for long.
Manifest JSON payloads are deduplicated across bare and www hosts before
keyword density is calculated.
AITDK homepage HTML checks can inspect the response body on HTTP 202/403
before discarding it. Real WAF/interstitial bodies are still rejected by the
visible-text block detector, but valid metadata returned with a non-200 edge
status can now be used.
[0.4.32] - 2026-06-16
Cleaner AITDK homepage sources
AITDK block-page detection now checks visible page text instead of raw HTML.
Valid homepages that load captcha-related helper scripts are no longer
discarded as captcha pages.
AITDK homepage fetches now keep one sticky proxy session per domain and then
try the same homepage directly when the proxy route is blocked. This reduces
missing keyword density caused by proxy-only WAF responses.
Thin SPA homepages can now use their official title/meta description as the
keyword source instead of being marked missing when the visible body is only
an app shell.
Added verified same-site metadata candidates for proxy-sensitive public
pages: PayPal's canonical US homepage, Booking's homepage component
metadata, ESPN's public active-sports homepage data endpoint, and X's app
manifest.
Captcha detection no longer treats normal Google reCAPTCHA form disclaimers
as an anti-bot page. Valid landing pages such as Netflix are kept when the
actual homepage content is visible.
Added Adidas' canonical US homepage as a clean same-site fallback, with both
direct and residential-proxy attempts when the root homepage is blocked by
the upstream edge.
Added stricter detection for real visible WAF/ad-block/interstitial pages so
blocked pages stay partial instead of becoming misleading keyword density.
[0.4.31] - 2026-06-14
Timeout cleanup logs
Hedged HTTP child tasks are now always cancelled and drained when a
per-domain timeout cancels the parent request. This removes scary
Task exception was never retrieved messages after an intentional timeout.
[0.4.30] - 2026-06-14
Timeout guardrails
Added a hard deadline around each low-level HTTP request so a stuck proxy,
TLS handshake, or HTTP client call is converted into a retryable source
timeout instead of holding the Actor run open.
Added a total deadline for hedged proxy requests. The backup proxy attempt
can still recover slow sessions, but the hedge loop can no longer wait
indefinitely.
Added a per-domain processing timeout so one stuck domain is saved as a
failed domain and the rest of the run can finish normally.
[0.4.29] - 2026-06-14
AITDK source quality checks
Added an official WHOIS fallback discovered through IANA for TLDs where a
registry RDAP route is missing or unavailable. The fallback maps into the
existing WHOIS/RDAP fields, so dataset shape stays unchanged.
AITDK keyword-density now requires enough usable homepage text before it is
saved as complete data. Very small source payloads and error-shell pages are
returned as partial keyword data instead of misleading top keywords.
Retry summaries now label these cases as thin_payload, making run output
clearer without treating low-content source responses as network failures.
[0.4.28] - 2026-06-14
Emoji schema labels restored
Restored UTF-8 emoji labels in the public input form and dataset views after
verifying the local README and schema files are stored as UTF-8.
Dataset tabs now use the visual labels shown in the README:
π Overview, π¦ Traffic sources, π« Engagement,
π€ AI traffic share, and π AITDK (WHOIS + keywords).
[0.4.27] - 2026-06-14
Booking metadata cloud route
Switched the Booking.com homepage component metadata candidate from the
www.booking.com/page/v0/chunk-metadata route to Booking's
accommodations.booking.com/orca/chunk-metadata route. It returns the same
homepage component metadata locally and avoids the www.booking.com WAF
challenge seen in Apify cloud.
[0.4.26] - 2026-06-14
Verified homepage component endpoints
Replaced the Booking.com weekend-deals GraphQL candidate with verified
CAPLA homepage component metadata endpoints loaded by Booking.com's own
homepage. The Actor now uses hero/search and trust/USP homepage copy instead
of unrelated deal-card data.
Booking.com homepage metadata is fetched with normal JSON GET requests and
no Residential proxy session, reducing retries for this hard-blocked
homepage pattern.
JSON keyword extraction now strips template placeholders such as
{num_reviews} before counting tokens.
[0.4.25] - 2026-06-14
Booking homepage endpoint routing
Booking.com's homepage data endpoint is now fetched directly instead of
through Residential proxy sessions. The endpoint is not a Similarweb source,
and direct routing avoids proxy-side empty responses observed in Apify runs.
[0.4.24] - 2026-06-14
Homepage endpoint candidates
Added a verified homepage-candidate layer for AITDK keyword extraction.
Standard domains still use only the bare-domain and www. homepages.
Added an Amazon mobile homepage candidate (/gp/aw/h.html) for Amazon
domains when the root homepage is blocked by an upstream challenge.
Added a Booking.com homepage data endpoint candidate for the public landing
page's weekend-deals component when the root homepage is blocked.
JSON homepage data is parsed from human-readable values only; technical
keys, IDs, image URLs, prices, and GraphQL error payloads are ignored.
[0.4.23] - 2026-06-14
Source summary labels
Kept the original homepage anti-bot reason when switching from bare-domain
homepage to the www. homepage variant, so run summaries group those source
checks as anti-bot challenges instead of generic source issues.
[0.4.22] - 2026-06-14
Homepage variant retry
AITDK homepage keyword extraction now tries both the bare domain homepage
and the www. homepage variant before marking hard-blocked homepage
keywords as unavailable. This keeps the fix within the homepage contract
without falling back to other same-domain pages.
[0.4.21] - 2026-06-14
AITDK partial log cleanup
Cleaned the AITDK progress and summary log wording for partial homepage
keyword results.
[0.4.20] - 2026-06-14
Homepage-only AITDK keywords
Removed the official sitemap keyword fallback from the default AITDK result.
keyword_density now contains homepage text only, so blocked homepages are
returned as partial AITDK items instead of mixing in keywords from other
same-domain pages.
Kept fast proxy recovery for transient homepage network issues, but stop
early on hard homepage anti-bot responses after a fresh proxy session has
already been tried.
Removed the redundant keyword-source column from the AITDK table view.
[0.4.19] - 2026-06-14
Clean schema labels
Removed emoji and corrupted Unicode symbols from Apify input and dataset
schema labels. Console tabs and field titles now use plain ASCII labels such
as Overview, Traffic sources, Engagement, AI traffic share, and
AITDK (WHOIS + keywords).
[0.4.18] - 2026-06-14
Faster fallback decisions
Stopped guessing default sitemap paths when a domain's robots.txt is
available but does not declare any Sitemap: directives. This avoids slow
timeouts on hard-block domains that publish no official sitemap entry.
Added early retry stop for blocked AITDK homepage keyword checks when the
official sitemap fallback is also unavailable.
[0.4.17] - 2026-06-14
Official keyword fallback
Added a generic AITDK keyword fallback for blocked homepages. When the
homepage is unavailable because of an upstream challenge, block, rate limit,
or empty text, the Actor now checks official domain-published sources:
robots.txt sitemap directives, sitemap indexes, gzip sitemaps, and a small
bounded set of same-domain crawlable pages.
Added keyword_density_source and keyword_density_source_urls to AITDK
output so users can see whether keyword density came from the homepage or
from official sitemap pages.
Added English/no-language sitemap preference and common web UI stopwords to
keep fallback keyword density cleaner and avoid navigation/cookie/footer
terms dominating the result.
[0.4.16] - 2026-06-14
Output tab default
Reordered the Actor output schema so Dataset items opens first in Apify
Console. The structured Run summary remains available as the second output
option.
[0.4.15] - 2026-06-14
Cleaner run output
Added a structured OUTPUT record in the default key-value store. Each run
now saves a machine-readable summary with processed domains, saved items,
data-quality counts, retry/source-availability counters, and samples of
partial or unsaved domains.
Added an Apify key-value store schema and linked the OUTPUT summary from
the Actor output schema, so the ru
