-
Faster and steadier when a connection misbehaves. Each page is now asked
for directly first, and a request that gets no answer is cut after 20 seconds
(it was 60) and retried straight away by another route. A page that would not
load can hold a run for about 100 seconds at most, down from 240. Nothing changes
in what is delivered or what it costs you.
-
When a page cannot be read, the run status now names the routes that were
actually tried instead of a general "after proxy escalation".
-
The run log ends with a one-line route summary (how many requests each route
carried), so a slow or partial run can be explained from its log.
-
Fixed: isSponsored was false on every ad. Promoted products are now
flagged (about 8 of 44 on a typical search page).
-
Fixed: marketplace products were being merged. Item numbers that contain
hyphens (for example 1FT-00EY-00036) were cut at the first hyphen, so different
products looked like duplicates and were dropped — a search page showing 44
products delivered 26. Each product is now delivered, with its full item number.
-
"No products" is now said only when Newegg itself says so: the run reads
Newegg's own item count ("We have found 0 items that match …") and reports an
empty search only when that count is 0. A listing page with no products and
no count is reported as a possible layout change on Newegg's side, and a run
where some searches came back that way is marked incomplete instead of clean.
-
New: products from a widened search are flagged. A keyword Newegg does
not recognise usually does not come back empty — Newegg answers it with a
broad "closest match" list across unrelated categories. Those products are
still delivered, but each one now carries searchWidened: true, and the run
status names the search: "Newegg did not find an exact match for '…' and
widened the search; the N products delivered from it may be unrelated".
Filter on searchWidened if you only want exact matches, or use a more
specific keyword or a category URL.
-
Fixed: a search with no matching product could be reported as blocked.
The check that tells a bot-wall page from an empty listing matched a script
Cloudflare adds to every Newegg page, healthy ones included, so any page
without product tiles — a genuine "0 items" search first of all — was
reported as "bot-wall page (HTTP 200)" and the search was dropped. The page
is now read payload first: a page carrying products or Newegg's own item
count is a listing page and is never called a wall; only a page with neither
is tested against the wall signatures.
-
A real bot-wall page served with HTTP 200 is now treated like an HTTP 403:
the same page is retried on the next proxy rung before the search is given
up, and the run log names the rung, the signature that matched, the page
size and its title. A wall that survives the last rung is still reported as
a refusal — never as an empty search — and is no longer counted among the
listing pages read.
-
New diagnostic: the first page of a run that is classified as a wall, or
that comes back with no product and no item count, is saved to the run's
default key-value store as BLOCKED_PAGE_SNAPSHOT (HTML, at most one per
run, capped at 2 MB), so a blocked or unexplained result can be inspected
after the run.
-
Fixed: searches served in Newegg's redesigned listing page returned no
products. Newegg now serves two page layouts for the same search — the
long-standing one and a redesign with different product cards — and a run
that received the redesign read a healthy page as empty and reported the
search as incomplete. Both layouts are now read, into the same output
fields: a product has the same itemNumber, price, rating and the rest
whichever layout Newegg served, so a result set that spans both is not
duplicated. The run log names the layout of every page.
-
Status wording: a page with no readable product is now described as
"Newegg's own item count did not say 0" rather than "no item count from
Newegg" — on the redesigned page Newegg did state a count (1,347 items), so
the old sentence was untrue for exactly this case.
