Yahoo! Auctions Japan Scraper

Listings from Yahoo! Auctions Japan (ヤフオク!), Japan's largest auction marketplace: current price and buy-now price in yen, start price, bids, time left and end time, condition, postage and free-shipping flag, seller id, category and category path, image, and the listing URL — with Yahoo! Flea Market (フリマ) fixed-price listings included or excluded on request.

HTTP only, no login, no key, no browser. 100 listings per page.

What it is for

• Price research — sold-out Switch consoles, film cameras, trading cards: what is asked and what is bid, right now.
• Sniping lists — sortBy: endingSoon with a price band.
• Reseller sourcing — condition: unused, buyNowOnly, freeShippingOnly.

Input

field	what it does
searchTerms	Keywords (Japanese or English).
categoryIds	Yahoo category ids (auccat), alone or with terms.
priceMin/Max, condition, newListingsOnly, sortBy	Filters the site applies.
buyNowOnly, freeShippingOnly	Filters applied locally (see below).
includeFleaMarket	Which of the site's two catalogues to read (see below).
maxItems, maxConcurrency, minRequestInterval, proxyConfiguration	Limits.

Targets are searchTerms × categoryIds; each gets its own SEARCH_SUMMARY.

Five things about this site worth knowing before you trust a run

1. The first request of a session sees half the catalogue

A cookie-less request for "nintendo switch" answers 36,791 results. The same request again, once the site's browser-id cookie is set, answers 77,080 — the extra 40,000 are Yahoo! Flea Market fixed-price listings. A crawler that paginates from its first response gets page 1 from one catalogue and page 2 from the other. This Actor warms every session with one discarded request before the first real page (includeFleaMarket: true, the default), or clears cookies before every request to get the auctions-only catalogue deterministically (false). The summary reports sessionWarmups, the site's three totals (all / auctions / fixed price), and fleaMarketRows.

2. The site stops at 15,000 rows and then says "0件"

Offset 14,901 serves 100 cards; offset 15,001 serves none and every count on the page reads 0 — for a search that had 729,000 results a page earlier. The Actor never asks past the wall, reports wallReached: true, and keeps the real total. Narrow by category, price or condition to see more.

3. Some URL filters are honoured, some are silently ignored

Honoured (each answered an impossible value with nothing): price band, condition (istatus), category, recent-only, sort. Ignored (the total never moved): the buy-now-only, free-shipping-only and sold-items parameters the site's own UI generates. buyNowOnly and freeShippingOnly are therefore applied locally from the card flags and counted in filteredOut; sold items are not offered. An unknown category id is answered with an HTTP 500 (reported as upstream_error), an unknown keyword with a clean 404 (no_matches), an empty keyword with a landing page (refused here).

4. Condition ids were verified, not assumed

istatus=1 returns 28,262 rows, every one labelled 未使用; istatus=2 returns 48,825 — exactly the remainder. So unused is 1 and used is 2. The finer used grades (3/4/5) are not exposed. istatus=99 is silently ignored by the site, which is why the input is an enum.

5. Featured cards repeat, and the listing's fields are scattered

Page 1 carries three featured (paid) cards that appear again on page 2; flagged isFeatured and de-duplicated. A card's data-auction-* attributes are split across its image link, title link and watch button — end time, start price, buy-now price and seller live on the button. All are merged.

Output

• LISTING — auctionId, url, title, currentPrice, buyNowPrice, startPrice, currency, bids, timeLeft, endsAt (UTC), condition, postage, freeShipping, isFleaMarket, isShoppingItem, easyPayment, categoryId, categoryPath, sellerId, imageUrl, isFeatured, query, resultPosition, pageFound.
• SEARCH_SUMMARY — totalResults, totalAuctions, totalFixedPrice, listingsReturned, pagesFetched, sessionWarmups, stoppedReason, wallReached, duplicateRowsDropped, featuredRows, filteredOut, fleaMarketRows, buyNowRows, freeShippingRows, minPrice, maxPrice.
• ERROR — invalid_input, upstream_error, page_shape_changed, fetch_failed, with detail.

Known limits

• Flea-market rows have no bids, time left, end time or seller id — they are fixed-price listings, and the search card does not carry those for them.
• condition is shown on the card for only some listings; the filter is the site's, the label is emitted when present.
• Sold / closed auctions are a different search on the site and are not fetched.
• Connections from outside Japan occasionally stall at the TCP handshake; the Actor retries on a fresh session.