# Changelog of Dhgate Scraper (`normdata/dhgate-scraper`) Actor

- **URL**: https://apify.com/normdata/dhgate-scraper/changelog.md
- **Full Actor documentation**: https://apify.com/normdata/dhgate-scraper.md

## Changelog

### 2026-09-19

- **Real limitation found by testing at real scale (1,000 items), fixed properly - not worked
  around:** DHgate's own per-keyword pagination has a hard ceiling, confirmed live - a keyword
  claiming 70,566 matches only ever yields ~550-600 real, reachable unique products from a flat
  search before every further page comes back degenerate (checked live to page 25; nothing new past
  page ~17). Investigated DHgate's own real UI for its own real answer before inventing one: its
  search response carries its own real category facets (`cateTreeList`) - genuinely narrower,
  largely-distinct slices of the same keyword's results, independently reachable via `catalog={id}`
  on the same open (non-Cloudflare-protected) search endpoint. Confirmed live combining several
  roughly TRIPLED the real reachable total for one keyword (578 flat -> 1,561 across 6 of its 20 real
  categories). Now walks a keyword's own real categories, largest first, once its flat search hits
  the real ceiling - a genuinely real 1,000/1,000 delivered from a single keyword in testing.

- Fixed the "last page" detection to match: a page returning far fewer than the real ~40-50/page
  (checked live: DHgate's own real page size drops from ~50 to a stable 40 after page 3, not
  constant as first assumed) is the real ceiling, not a normal short last page - the earlier
  size-based check was stopping 5-6 real pages early on every run before this fix.

- 22 unit tests, 94 `expect()` calls (up from 19/53), including coverage for the real category
  extraction and sort order.

- **Real bug found by a 700-row `freeShippingOnly` run and fixed:** DHgate's own `freeshipping=1`
  server filter has the same real gap as `minprice`/`maxprice` - 3 of 700 real rows came back with
  `free_shipping: false` despite the filter. Re-checked client-side like the price filters already
  were; re-verified at the same 700-row scale afterward: 0 mismatches.

- Category walking batched (3 concurrent, each targeting its own share of what's still needed) after
  measuring the fully-sequential version at 130-143s for a real 700-row, single-keyword, heavily
  filtered run - the same real scenario measured at 91s after batching (~35% faster), re-verified at
  0 duplicates and 0 filter mismatches, and multi-keyword fairness re-checked unaffected.

- Initial build: multi-keyword search with price-range, free-shipping, and sort filters, reading
  DHgate's own real search-results JSON (`__NEXT_DATA__`, `totalProducts`) directly.

- Confirmed live that DHgate's individual product pages and store/supplier pages are
  Cloudflare-protected (a real "Just a moment..." challenge), while the search results page itself is
  not - scope kept to search only, avoiding the protected pages entirely rather than working around
  them.

- **Real bug found and fixed before shipping:** DHgate's own `minprice`/`maxprice` search params
  narrow the real result count server-side but do not reliably bound every returned product's real
  price (verified live: a `minPrice=5, maxPrice=10` search returned real products priced at $0.98 and
  $22.22). Fixed by re-checking every row against its own parsed price client-side.

- **Real bug found and fixed:** the sort control was implemented as a single combined value
  (`price_up`/`price_down`), which silently fell back to DHgate's own default "best match" ordering -
  5 real "ascending" results came back in no real price order. Root cause: DHgate's real sort is two
  separate parameters (`sinfo=price` + `stype=up`/`down`), found by reading the page's own default
  params echoed back in its data. Fixed and verified against a real 5-result ascending sequence.

- **Real bug found and fixed:** running several keywords in one call let whichever one's first page
  happened to respond fastest consume the entire `maxItems` budget - 3 keywords requesting 12 rows
  total came back with all 12 from a single keyword, 0 from the other two. Fixed with a fair
  per-keyword cap (`maxItems` divided evenly across `queries`) instead of a shared free-for-all pool.

- **Real bug found and fixed:** the price string's real `"US $..."` prefix was read as the literal
  currency code (`"US"`) instead of resolved to the real ISO-4217 code (`"USD"`).

- 19 unit tests, 53 `expect()` calls, against a real saved search-results fixture.

- Full live field audit: 26 fields diffed to zero discrepancy against `dataset_schema.json`.