Cost optimization pass. All investigation done against the live site in Chrome, so the
byte measurements below cost nothing to obtain.
- Request blocking in the browser tier. Measured per-render wire bytes and dropped
everything that isn't the server-rendered document or Akamai's sensor script:
~926 KB → ~334 KB first page, ~143 KB steady-state. Confirmed live that the trader
block (module, email, VAT) is present in a plain fetch of the raw HTML, that search
pages are server-rendered too, and that
/itemmodules/{id} does not carry it.
Written as a block-list, not an allow-list, because Akamai's sensor sits on a
randomised path an allow-list would silently drop.
- Removed the search-page hydration wait — search results are server-rendered; only
a short guarded fallback remains for a challenge shell.
- Custom residential proxy via the
CUSTOM_RESIDENTIAL_PROXY_URLS secret env var.
Apify residential is $8/GB against Decodo's $4/GB pay-as-you-go.
- Warm HTTP tier (
warmHttpTier flag, off by default): the browser renders search
pages and harvests the Akamai cookie set (_abck is HttpOnly, so it comes from
Playwright's context), then replays it over impit on one sticky residential session
for every item page. Falls back to the browser on failure and logs a hit rate.
- Item→seller cache (90-day TTL) in the existing per-task store. Caching by seller
saves nothing — you can't consult it before the render that identifies the seller —
so the cached edge is item→seller, which skips listings whose seller we already hold
and collapses several listings from one wanted seller into a single render. Private
sellers are marked permanently skippable; email-less sellers are not, since a later
requireEmail=false run still wants them. No effect on a cold run; large effect on
scheduled monitor tasks.
Input model reworked around keyword-first targeting, after a survey of every comparable
eBay actor on the Apify store (none of which return the trader block at all).
- Keywords are now the primary input and the first field in the form.
startUrls is
retained as the advanced path — paste a pre-filtered eBay search/category URL and the
actor inherits every filter, so the form never has to mirror eBay's own filter UI.
- Untargeted sitemap sweep removed from the product. A run with no keywords and no
start URLs now fails fast instead of returning a blind list of whatever sellers a market
happened to surface.
candidatesFromSitemaps() stays in src/discovery.ts, unwired, as
the reference implementation of eBay's only Akamai-free discovery surface.
- New
requireEmail input (default on) — sellers whose trader block publishes no email
are dropped and never billed, so the customer only pays for contactable leads. The dropped
count is logged rather than silently swallowed.
- Two-tier pay-per-event billing:
unique-business-seller-with-email (premium) for a
contactable lead, unique-business-seller (base) for a verified record without one. Both
events must exist in Console — see docs/BUILD_NOTES.md §1.
businessSellersOnly input removed; skipping private sellers is now unconditional, since
they carry no legally-mandated trader block and so no contact data by definition.ebay.com dropped from the domain list (no DSA trader block — it could only ever return
empty rows). Default domain is now a single ebay.co.uk; maxSellers default 1000 → 100.
Initial build (greenfield). TypeScript + Crawlee Apify actor.
- Two-stage, dedup-before-fetch crawl: sitemap/startUrl/keyword discovery → seller-first dedup → trader-block extraction.
- Unblocked Stage-0 discovery from eBay's public GTC (fixed-price) sitemaps.
- Tiered transport: CheerioCrawler (datacenter) with fallback to PlaywrightCrawler (residential) for Akamai-blocked listings.
full and monitor run modes; cross-run seen-set in a named key-value store (firstSeenAt).- Pay-per-event charging on
unique-business-seller, guarded by the per-run charge limit.
- Input schema, dataset schema + Console table view, SEO README.
Not yet validated against live business-seller pages — trader-block selectors and block
signatures are marked LIVE-TUNE and must be confirmed during the Phase-1 yield gate. See
docs/BUILD_NOTES.md.
![Ebay Search Country Filter Spider[Client] avatar](https://images.apifyusercontent.com/JgV9UIBei4ZmI07eL4Iias4UFxC5Pkw4hGe2PRYMzSU/rs:fill:76:76/cb:1/aHR0cHM6Ly9hcGlmeS1pbWFnZS11cGxvYWRzLXByb2QuczMudXMtZWFzdC0xLmFtYXpvbmF3cy5jb20vVG1nM0VoZEppNmgzQlN1OFAtYWN0b3ItRW9UZ0tmUXFXczRKa1Uyb0ItcGx1UHFjdmRLaC1HZW1pbmlfR2VuZXJhdGVkX0ltYWdlXzdueXk1OTdueXk1OTdueXkucG5n.webp){kind=avatar}
