Root cause 1 — every HTTP request was cookie-less, so eBay's edge bot-defense
blocked all of them.http_get() called curl_cffi.requests.get(...) fresh
for every single request (item page, feedback pages), never reusing a session or
carrying cookies between calls. Live testing (curl_cffi impersonating Chrome,
against the actor's own prefill URLs and several currently-live eBay listings)
showed eBay's edge defense (Akamai-style; sets bm_s / bm_so cookies) blocks a
session's very first request with HTTP 403 and a "Pardon our interruption"
style bot-block page — but that same blocked response sets the tracking
cookies needed to pass. A brand-new stateless request never picks those cookies
up, so it was blocked again on the very next call, every time, on every run.
Evidence: a fresh requests.get() per call reproduced 403 with a ~1.9 KB
bot-block body on 100% of attempts across 6 different impersonate profiles and
6 different item ids; reusing one curl_cffi.requests.Session across calls (so
the cookies set by the first blocked response are sent on the next request)
consistently returned 200 with real HTML from the second request onward.
Fix: src/main.py — replaced the stateless http_get() with a persistent
curl_cffi.requests.Session created once per item (_new_session), a
throwaway warm-up hit that seeds the session's cookies before any request that
matters (_warm_up_session), and a session_get() helper that retries a
request once on the same session if the response looks like a bot
block/challenge (_looks_blocked) before giving up. _fetch_item_page_once
and fetch_feedback_page now take that session instead of a bare proxy_url.
Root cause 2 — the feedback-pagination AJAX endpoint now returns an
always-empty body. Even inside a properly warmed session with the correct
referer, GET /fdbk/mweb_profile?...&_ajax=pagination answers HTTP 200 with
{"modules": {}} — a soft failure with no error status, so parse_feedback_cards
always had zero cards to parse and only the handful of reviews embedded on the
item page itself ever made it out (and none at all for a seller whose feedback
wasn't shown on the item page). The identical FEEDBACK_CARD_MODULE_<id> JSON
records are still embedded verbatim in the full (non-ajax) HTML of the same
mweb_profile URL, inside its inline hydration script, and page_id genuinely
pages through them (verified against a high-volume seller: 25 distinct cards on
page 1, 25 more distinct cards on page 2, zero overlap).
Fix: src/main.py — build_feedback_page_url no longer appends
&_ajax=pagination (fetches the real HTML page instead of the dead ajax
fragment). Added extract_feedback_card_modules(), which brace-matches each
FEEDBACK_CARD_MODULE_<id> object out of the HTML the same way
extract_product_json_ld() already does for the item page's Product JSON-LD,
and wraps them as {"modules": {...}} so the existing parse_feedback_cards()
needs no changes. Removed get_total_pages() (read a PAGINATION_MODULE that
no longer exists in this response) and replaced the
for page inrange(1, total_pages +1)
loop in scrape_single_item with a while loop
that walks page_id until a page contributes zero new card keys (capped at
_MAX_FEEDBACK_PAGES = 400 as a safety backstop).
Verified
Ran the fixed scrape_single_item() directly (no mocks) against a currently-live
eBay listing (https://www.ebay.com/itm/128039732630): 22 structured review rows
returned, 22/22 with a non-empty ReviewComment, product details populated
(title/price/seller/images). Re-ran with maxReviewsPerUrl=5 (capped correctly
at 5), ratingType=POSITIVE (0 rows violated the filter), sortReviewsBy=TIME,
and withPhotosOnly=true (correctly fell back to the pre-existing empty-result
placeholder row, since this seller has no photo reviews in the collected window).
The two original prefill URLs (ebay.com/itm/364771787371,
ebay.co.uk/itm/194488920284) were confirmed to be genuinely dead/ended listings
(HTTP 404 on every regional mirror, independent of this fix) rather than a
symptom of the bug — a live substitute item was used to verify the actual fix.
