Pottery Barn Product Scraper

Scrape PotteryBarn.com — Williams-Sonoma's US furniture & home-decor e-commerce. Product discovery is sitemap-driven: live product URLs come from the public product sitemap (16k+ URLs), so the actor never relies on JS-rendered category grids or site search. Product details (prices, images, availability, SKUs, category breadcrumbs) are parsed from the server-rendered window.__INITIAL_STATE__ payload of each product page. Direct product URLs are also accepted.

Why sitemap-driven?

Akamai returns 403 "Restricted Access" to headless browsers from datacenter IPs (verified on cloud), and the category/search result grids are JS-rendered (absent from the plain HTML). The product sitemap — netstorage/sitemaps/product-sitemap-1.xml.gz — is publicly served and lists ~16,487 live product URLs, sidestepping both problems. Stale/dead deep links that redirect to the homepage are never used.

What this actor does

• Three modes: byCategory (default), byKeyword, byProductUrls
• 13 shop categories as a dropdown (furniture, bedding, rugs, lighting, outdoor, …)
• curl_cffi (chrome131 impersonation) everywhere — product pages return 200 with the full embedded product state
• Filters: price range (selling price), title/category keyword, in-stock-only
• Auto-escalation: lazy-engages Apify AUTO datacenter proxy + backoff on 403/429/5xx
• Bounded discovery: byCategory / byKeyword walk the sitemap in order (fetch budget = max(1500, maxItems×50) pages, walked 8 URLs concurrently) and keep records matching the category path / keyword, stopping at maxItems
• Empty fields are omitted; every record carries sourceUrl, scrapedAt, recordType

Output per product

• productId (group id), title, sku (leader SKU)
• description (SEO materials copy from <meta name="description">)
• price (low selling), priceMax (high selling), regularPrice, retailPrice, wasPrice (when retail > selling)
• brand (Pottery Barn), categoryPath[], categoryUrl, superCategoryId (canonical shop-category slug)
• availability, skuCount
• images[] (pbimgs CDN URLs), imageUrl
• productUrl, recordType, scrapedAt

Input

Field	Type	Default	Description
mode	select	byCategory	byCategory / byKeyword / byProductUrls
category	select	furniture	Shop category (13 options)
searchQuery	string	vanity	Keyword (mode=byKeyword)
productUrls	array	–	Product URLs (mode=byProductUrls)
priceMin / priceMax	int	–	Selling-price range (USD)
containsKeyword	string	–	Title/category substring filter
inStockOnly	bool	false	Only available products
maxItems	int	20	Hard cap (1–500)

Example: furniture under $500, in stock

{
  "mode": "byCategory",
  "category": "furniture",
  "priceMax": 500,
  "inStockOnly": true,
  "maxItems": 10
}

Example: keyword match

{
  "mode": "byKeyword",
  "searchQuery": "vanity",
  "maxItems": 10
}

Example: direct URLs

{
  "mode": "byProductUrls",
  "productUrls": ["https://www.potterybarn.com/products/sausalito-dressing-vanity/"]
}

Image URLs

Product images are served from assets.pbimgs.com — verified HTTP 200 image/jpeg from a clean shell (no Referer/cookies), so they are shipped as-is.

Anti-bot notes

• Plain curl gets 403; curl_cffi (chrome131) + parsing window.__INITIAL_STATE__ works (verified locally, 200).
• Headless browsers are hard-blocked by Akamai from datacenter IPs (403 Restricted Access, verified on cloud) → discovery is sitemap-driven instead of browser-driven.
• Ratings require the JS review widget (Bazaarvoice), which is not rendered server-side and is disabled for Pottery Barn desktop in some regions — rating/reviewCount are not shipped in v1 (documented, not sentinel).

Data source

potterybarn.com product pages (window.__INITIAL_STATE__.product.productDetails) + public product sitemap. No login required.

Limitations

• byCategory / byKeyword walk the sitemap with a bounded fetch budget (budget = max(1500, maxItems×50)); all 13 dropdown categories are reachable at the default maxItems=20 — for keywords that match extremely deep in the sitemap (past ~1500 URLs), raise maxItems to widen the budget. Category membership is matched on the canonical superCategoryId slug (with breadcrumb-label aliases as fallback).
• The category enum contains only the 13 categories that actually appear as superCategoryId on product pages. Pottery Barn's "New", "Sale", "Registry", "PB Apartment" and "Small-Space Furniture" sections are curated landing pages, not product categories — products there carry their real category ID (e.g. furniture), so they are not offered as byCategory options; use byKeyword (e.g. sale, outlet) or direct product URLs to reach those products.
• If every candidate page within the budget is filtered out, the actor soft-fails with a status message instead of silently dropping. Invalid product URLs in byProductUrls produce typed recordType: "error" records; duplicate URLs in the input are deduplicated to single records.