# Changelog of WooCommerce Scraper (`autofacts/woocommerce-scraper`) Actor - **URL**: https://apify.com/autofacts/woocommerce-scraper/changelog.md - **Full Actor documentation**: https://apify.com/autofacts/woocommerce-scraper.md ## Changelog ### Unreleased #### Fixed - The Apify Store Output tab lists the real fields. `fields` was a bare `oneOf`/`definitions` pair, which rendered two rows named `oneOf` and `definitions`; it is now a flat property map over the same definitions. - Eleven output fields that carried no description now carry one. - The prefilled start URL points at a store that answers. The previous one returns 403 to datacenter IPs on every path, so the default run failed. - A run that stores nothing now exits FAILED, naming the start URLs it gave up on. A run that stored anything at all still succeeds. - Runs are reported to Scraper Insight under this actor's own app id and secret, not shopify-scraper's. ### 0.2.0 — 2026-09-05 Closes the two gaps the competitive review found (reviews, usability of the caps) and restructures the README as a store listing. #### Added - **Customer reviews** as a fourth record kind, `Review`, billed by the new `review` event. Opt-in via `includeReviews`; `maxReviewsPerProduct` (default 20, `0` = all). Records carry rating, body, reviewer, verified-buyer flag, date, product id and URL. Stores on the HTML fallback yield none. - Reviews are fetched by whichever of two routes is cheaper for that store: per-product requests, or a single store-wide walk ordered by product. Both keep the same newest-first subset. - `maxProductsPerStore`: a per-store share of the run, so one large catalog can no longer consume the whole budget before the next start URL is reached. Two start URLs on the same store share one cap. - `domains`: bare store domains, one per line, crawled as store roots. The platform's URL-list editor rejects anything without a scheme, so domains get their own field. Start URLs given without a scheme are also accepted when they reach the crawler. #### Changed - `maxRequestsPerCrawl` is renamed `maxProducts` and counts product records only; categories and reviews have their own budgets. The old name remains as a hidden alias, so saved tasks keep working. - README rewritten for people using the actor: quick start with API examples, the URL types accepted, the input reference, the four record kinds with real examples, what is billed, and the limitations to plan around. - Run summary now reports products, categories and reviews separately. #### Fixed during live verification - The `query` input was accepted and passed through but never read: only a `?s=` start URL actually searched. It now searches every store root or domain in the run and narrows category and tag URLs to matching products. - Every name field is decoded from the HTML entities WordPress stores it with (`Monk’s Robe` becomes `Monk's Robe`): titles, category and tag names, brand, attribute labels and values, option values, variation titles and reviewer names. Descriptions are HTML by design and are unchanged. - With reviews requested on a store read through the HTML fallback, the run produced no reviews and said nothing. The per-store warning now fires for every such store, not only for the Store API code paths. - Once the HTML-fallback cap was reached, a `wp_rest` store kept fetching the remaining id pages (ten wasted requests on a 1,025-product store) and repeated the cap warning for each. Enumeration now stops with the budget, and the warning is logged once per store. #### Deploying this version - Create the `review` event in the Console pricing config before publishing. Pushing against an event the Console does not know stores the record and charges nothing. - The synthetic `apify-default-dataset-item` event must stay removed, as before. ### 0.1.0 — 2026-09-03 First implementation. Extracts products, variations and categories from any WooCommerce store, adapting to whichever data surfaces that store exposes. #### Added - **Two-axis surface model.** A store's `enumerate` surface (`store_api`, `wp_rest`, `sitemap`) and its `detail` surface (`inline`, `store_api`, `html`, `none`) are probed and recorded independently, because they fail independently. - **Capability probing** (`probe.ts`), ≤8 unbilled requests per host, API before homepage. Stores whose firewall guards HTML but not `/wp-json` are classified correctly instead of being written off. - **Store profile cache** in the `WOO_STORE_PROFILES` Key-Value Store, with an asymmetric TTL: 30 days for a healthy profile, 24 hours for a degraded one, so a store that repairs its REST surface is promoted back within a day. A blocked probe is never cached. - **Failure-signature handling.** Transport failures are classified (`waf`, `challenge`, `captcha`, `rest_no_route`, `plain_permalink`, `not_found`, `server_error`, `param_rejected`, `rate_limited`, `transport`) and the crawler acts on the signature, not the status code. - **Data-source routing** in place of permalink pattern matching, since WooCommerce permalinks are store-configurable. A URL is classified by asking the Store API for the slug, then by reading the WordPress ``, and only then by pulling the taxonomy list. - **Pay-per-event billing** with four events: `product`, `category`, an `html-detail` surcharge for products read off a product page, and `real-inventory` for products whose store published an exact stock count. `StoreSummary` is stored unbilled. - **HTML fallback** for stores with no open API: JSON-LD plus the inline `data-product_variations` matrix, capped separately by `maxHtmlDetailRecords`. - `StoreSummary` record carrying catalog-level price range, stock split and product-type mix — one request, not billed. - `extraInfo.stockCountSource` distinguishes an exact published stock count from one inferred from the purchase maximum, and both from an unknown zero. #### Fixed during live verification - Categories have their own ceiling and may take at most a fifth of a capped run, so a large taxonomy can no longer consume the whole budget before a product is reached. - `Category.canonicalUrl` was read from `link`, which is the key the abbreviated category copies embedded in a product use. The standalone categories endpoint calls it `permalink`, so every category record crawled from the taxonomy had an empty URL. - The category deduplication key fell back to a bare term id when no URL was present. Term ids are per-site, so one store's category could suppress another's. The host is now always part of the key, as it already was for products. - `Actor.getInput()` returning null (an actor started with no input) threw a TypeError on the first field read instead of a validation error. #### Fixed verifying the sitemap enumeration path - `enumerate=sitemap` produces records again. The probe walks robots.txt down to a sitemap that contains product URLs and keeps a sample, so the HTML detail probe runs. The `wp_rest` branch had the same defect and is fixed with it. - The resolved product sitemap is cached on the profile, so a crawl no longer repeats the robots.txt/index walk, and numbered continuations (`product-sitemap2.xml`, …) are followed to the end of the catalog. - The probe requires the crawl's own parser to return a product before calling a surface usable, and tries up to three sitemap entries. - The probe had a second, private currency regex separate from the crawl's parser. It is gone; the probe runs the real parser, so the two cannot disagree. - Removed a redundant Store API request from the probe. It re-requested the exact route step 1 had already tried, wasting one of eight probe requests on every store that is not on the Store API — the half of the population that can least afford it. - `StoreSummary.productCount` stayed -1 on sitemap stores. A sitemap reports its total by having that many entries, which is the same fact the Store API reports in a header. - Non-product URLs skipped during an HTML crawl are now counted and reported, so a run that picked the wrong sitemap is visible rather than silently thin. #### Fixed auditing the output - The HTML path emits categories, read from the product page's `posted_in` element and falling back to the breadcrumb trail. Themes that render neither still yield none. #### Notes - Product ids are per-site WordPress post ids, so the run's deduplication key is `host:id`. Two stores each holding a `postid-99119` is normal. - `extraInfo.productType` is an open string. Plugin-registered types (`subscription`, `bundle`, `variable-subscription`, `gift-card`) occur in the wild and a closed enumeration would reject those records. - Variation records return an empty `attributes` array, so a variant's option values are recovered from the `variation` display string, anchored on the parent's axis names. Verified against 427 variation records from 20 stores. - The `slimImages` input option was removed before release: the shared retail schema only carries an image's full-size URL, so the option had nothing to drop.