Dhgate Scraper
Pricing
from $6.80 / 1,000 results
Source wholesale products from DHgate with real tiered pricing, MOQ, supplier feedback score, and active promo discounts - everything an importer needs to compare suppliers fast, in one search. 26 real fields per product, filters that actually work.
Real limitation found by testing at real scale (1,000 items), fixed properly - not worked
around: DHgate's own per-keyword pagination has a hard ceiling, confirmed live - a keyword
claiming 70,566 matches only ever yields ~550-600 real, reachable unique products from a flat
search before every further page comes back degenerate (checked live to page 25; nothing new past
page ~17). Investigated DHgate's own real UI for its own real answer before inventing one: its
search response carries its own real category facets (cateTreeList) - genuinely narrower,
largely-distinct slices of the same keyword's results, independently reachable via catalog={id}
on the same open (non-Cloudflare-protected) search endpoint. Confirmed live combining several
roughly TRIPLED the real reachable total for one keyword (578 flat -> 1,561 across 6 of its 20 real
categories). Now walks a keyword's own real categories, largest first, once its flat search hits
the real ceiling - a genuinely real 1,000/1,000 delivered from a single keyword in testing.
Fixed the "last page" detection to match: a page returning far fewer than the real ~40-50/page (checked live: DHgate's own real page size drops from ~50 to a stable 40 after page 3, not constant as first assumed) is the real ceiling, not a normal short last page - the earlier size-based check was stopping 5-6 real pages early on every run before this fix.
22 unit tests, 94 expect() calls (up from 19/53), including coverage for the real category
extraction and sort order.
Real bug found by a 700-row freeShippingOnly run and fixed: DHgate's own freeshipping=1
server filter has the same real gap as minprice/maxprice - 3 of 700 real rows came back with
free_shipping: false despite the filter. Re-checked client-side like the price filters already
were; re-verified at the same 700-row scale afterward: 0 mismatches.
Category walking batched (3 concurrent, each targeting its own share of what's still needed) after measuring the fully-sequential version at 130-143s for a real 700-row, single-keyword, heavily filtered run - the same real scenario measured at 91s after batching (~35% faster), re-verified at 0 duplicates and 0 filter mismatches, and multi-keyword fairness re-checked unaffected.
Initial build: multi-keyword search with price-range, free-shipping, and sort filters, reading
DHgate's own real search-results JSON (__NEXT_DATA__, totalProducts) directly.
Confirmed live that DHgate's individual product pages and store/supplier pages are Cloudflare-protected (a real "Just a moment..." challenge), while the search results page itself is not - scope kept to search only, avoiding the protected pages entirely rather than working around them.
Real bug found and fixed before shipping: DHgate's own minprice/maxprice search params
narrow the real result count server-side but do not reliably bound every returned product's real
price (verified live: a minPrice=5, maxPrice=10 search returned real products priced at $0.98 and
$22.22). Fixed by re-checking every row against its own parsed price client-side.
Real bug found and fixed: the sort control was implemented as a single combined value
(price_up/price_down), which silently fell back to DHgate's own default "best match" ordering -
5 real "ascending" results came back in no real price order. Root cause: DHgate's real sort is two
separate parameters (sinfo=price + stype=up/down), found by reading the page's own default
params echoed back in its data. Fixed and verified against a real 5-result ascending sequence.
Real bug found and fixed: running several keywords in one call let whichever one's first page
happened to respond fastest consume the entire maxItems budget - 3 keywords requesting 12 rows
total came back with all 12 from a single keyword, 0 from the other two. Fixed with a fair
per-keyword cap (maxItems divided evenly across queries) instead of a shared free-for-all pool.
Real bug found and fixed: the price string's real "US $..." prefix was read as the literal
currency code ("US") instead of resolved to the real ISO-4217 code ("USD").
19 unit tests, 53 expect() calls, against a real saved search-results fixture.
Full live field audit: 26 fields diffed to zero discrepancy against dataset_schema.json.
