Pottery Barn Product Scraper

Scrape PotteryBarn.com — Williams-Sonoma's US furniture & home-decor e-commerce. Product discovery is sitemap-driven: live product URLs come from the public product sitemap (16k+ URLs), so the actor never relies on JS-rendered category grids or site search. Product details (prices, images, availability, SKUs, category breadcrumbs) are parsed from the server-rendered window.__INITIAL_STATE__ payload of each product page. Direct product URLs are also accepted.

Why sitemap-driven?

The category/search result grids are JS-rendered (absent from the plain HTML). The product sitemap — netstorage/sitemaps/product-sitemap-1.xml.gz — is publicly served and lists ~16,706 live product URLs, sidestepping that problem entirely. Stale/dead deep links that redirect to the homepage are never used.

What this actor does

• Three modes: byCategory (default), byKeyword, byProductUrls
• 13 shop categories as a dropdown (furniture, bedding, rugs, lighting, outdoor, …)
• curl_cffi (chrome131 impersonation) as the primary fetcher — fast and cheap when it gets through
• Automatic anti-bot escalation: lazy-engages Apify AUTO datacenter proxy on block, rotates to a fresh exit IP after repeated blocks, and escalates to Apify RESIDENTIAL proxy if AUTO stays blocked. If curl_cffi (even with residential) can't get real data through in a fair trial, the actor automatically falls back to a real anti-detect browser (Camoufox + residential proxy) for the rest of the run — no user configuration needed either way
• Filters: price range (selling price), title/category keyword, in-stock-only
• Bounded discovery: byCategory / byKeyword walk the sitemap in order (fetch budget = max(1500, maxItems×50) pages, walked 8 URLs concurrently) and keep records matching the category path / keyword, stopping at maxItems
• Empty fields are omitted; every record carries sourceUrl, scrapedAt, recordType

Output per product

• productId (group id), title, sku (leader SKU)
• description (SEO materials copy from <meta name="description">)
• price (low selling), priceMax (high selling), regularPrice, retailPrice, wasPrice (when retail > selling)
• brand (Pottery Barn), categoryPath[], categoryUrl, superCategoryId (canonical shop-category slug)
• availability, skuCount
• images[] (pbimgs CDN URLs), imageUrl
• productUrl, recordType, scrapedAt

Input

Field	Type	Default	Description
mode	select	byCategory	byCategory / byKeyword / byProductUrls
category	select	furniture	Shop category (13 options)
searchQuery	string	vanity	Keyword (mode=byKeyword)
productUrls	array	–	Product URLs (mode=byProductUrls)
priceMin / priceMax	int	–	Selling-price range (USD)
containsKeyword	string	–	Title/category substring filter
inStockOnly	bool	false	Only available products
maxItems	int	20	Hard cap (1–500)

Example: furniture under $500, in stock

{
  "mode": "byCategory",
  "category": "furniture",
  "priceMax": 500,
  "inStockOnly": true,
  "maxItems": 10
}

Example: keyword match

{
  "mode": "byKeyword",
  "searchQuery": "vanity",
  "maxItems": 10
}

Example: direct URLs

{
  "mode": "byProductUrls",
  "productUrls": ["https://www.potterybarn.com/products/sausalito-dressing-vanity/"]
}

Image URLs

Product images are served from assets.pbimgs.com — verified HTTP 200 image/jpeg from a clean shell (no Referer/cookies), so they are shipped as-is.

Anti-bot notes

• Pottery Barn sits behind Akamai bot management, which can escalate from simple IP-reputation blocking (fixed by proxy rotation) up to a JS-sensor challenge (_abck) that no static HTTP client can satisfy regardless of IP quality or TLS fingerprint — only a real browser executing Akamai's JS clears that. The actor handles both: curl_cffi + proxy rotation for the common case, Camoufox (anti-detect Firefox) + residential proxy as an automatic fallback for the JS-sensor case.
• The fallback triggers on a success-rate probe (first 3 batches of the sitemap walk, or the first few byProductUrls requests) rather than a strict "zero successes" check, so an occasional lucky curl_cffi request doesn't mask a mostly-blocked run and disable the fallback.
• Ratings require the JS review widget (Bazaarvoice), which is not rendered server-side and is disabled for Pottery Barn desktop in some regions — rating/reviewCount are not shipped in v1 (documented, not sentinel).

Data source

potterybarn.com product pages (window.__INITIAL_STATE__.product.productDetails) + public product sitemap. No login required.

Limitations

• byCategory / byKeyword walk the sitemap with a bounded fetch budget (budget = max(1500, maxItems×50)); all 13 dropdown categories are reachable at the default maxItems=20 — for keywords that match extremely deep in the sitemap (past ~1500 URLs), raise maxItems to widen the budget. Category membership is matched on the canonical superCategoryId slug (with breadcrumb-label aliases as fallback).
• The category enum contains only the 13 categories that actually appear as superCategoryId on product pages. Pottery Barn's "New", "Sale", "Registry", "PB Apartment" and "Small-Space Furniture" sections are curated landing pages, not product categories — products there carry their real category ID (e.g. furniture), so they are not offered as byCategory options; use byKeyword (e.g. sale, outlet) or direct product URLs to reach those products.
• If every candidate page within the budget is filtered out, the actor soft-fails with a status message instead of silently dropping. Invalid product URLs in byProductUrls produce typed recordType: "error" records; duplicate URLs in the input are deduplicated to single records.