BigCommerce Scraper: Catalog, Variants & Pricing

Agent skill: SKILL.md

https://api.apify.com/v2/key-value-stores/t7YoTxpZEJOWvw4Ug/records/bigcommerce-store-scraper.md

What it does

Scrape BigCommerce storefronts for product catalogues: names, SKUs, prices, options, availability, categories and images. Covers a large mid-market merchant base. Give it stores and it returns one structured record per product.

Point it at a storefront domain and it walks the store's public product sitemap, then reads each product from two public storefront surfaces: the product page's structured-data island and BigCommerce's /remote/v1/product-attributes/{id} JSON route, the same endpoint the storefront's own JavaScript calls to price a product. That second call is what makes SKUs, live prices, sale prices, currency and stock reliable instead of scraped off rendered text.

Verified live against three unrelated merchants running three different themes: cornerstone-light-demo.mybigcommerce.com, www.bulkapothecary.com and www.berlinpackaging.com.

Which BigCommerce surface this uses

This matters, so it is stated plainly. This Actor targets the public storefront a shopper sees, not the authenticated BigCommerce management API, and not the Storefront GraphQL API. No token, no store hash, no API account.

Two routes that are often assumed to work do not work unauthenticated, and this Actor does not use them:

Route	What actually happens
public API	401, {"errors":[{"message":"GraphQL credentials were missing. No token was sent."}]}. The Storefront GraphQL API needs a token the merchant must mint; storefronts do not publish one in the page.
public API	404, no such route. /api/storefront/* covers carts, checkouts and orders only.

What this Actor uses instead, all anonymous, all confirmed 200 on live merchant storefronts:

Route	Purpose
GET /xmlsitemap.php → ?type=products&page=N	Product discovery. Served on every BigCommerce storefront regardless of theme.
GET /remote/v1/product-attributes/{productId}	Authoritative JSON: SKU, UPC/MPN/GTIN, price, MSRP/was-price, currency, stock, purchasability, weight.
GET /{product-url}	Product page, JSON-LD Product and BreadcrumbList, Open Graph tags, image gallery, option widgets.
GET /{category-path}?page=N&limit=50	Optional category-scoped discovery on Stencil-standard themes.

Input

Field	Type	Default	Notes
stores	array	-	Required. Storefront domains or URLs. Anything past the host is ignored, so https://store.com/some/page and store.com are the same input.
categoryIds	array	[]	Restrict the sweep to categories. Pass a category page path or full URL, fragrance-oils, /shop-all/kitchen/. Empty = walk the whole product sitemap.
maxProductsPerStore	integer	50	Cap per store. 0 = no limit. Each product costs two requests.
includeOptions	boolean	true	Include configurable options and modifiers. Set false to drop the field entirely.

{
 "stores": ["www.bulkapothecary.com", "https://www.berlinpackaging.com"],
 "maxProductsPerStore": 50,
 "includeOptions": true
}

Which combinations make sense.

• Whole-catalogue sweep. stores only, maxProductsPerStore: 0. Uses the sitemap route, which works on every theme. Budget two requests per product.
• Narrow, repeatable price watch. Add categoryIds with the category paths you care about and keep maxProductsPerStore small. Uses the category route, which is faster but theme-dependent (see below).
• Cheap change detection. includeOptions: false. Records shrink a lot on stores with large option sets, and you still get price, sale price and availability.

About categoryIds. The field keeps its name for compatibility, but numeric BigCommerce category IDs cannot be used. Public storefronts expose no id-addressable category route, ?category=23, /categories/23 and rss.php?categoryid=23 are all either ignored or 404, verified live. Numeric entries are skipped with a warning and listed in RUN_SUMMARY.filters.ignoredNumericCategoryIds. Pass the category page path instead; that is the selector the storefront actually honours.

Output

One dataset item per product. Annotated sample, real values from a live run against www.bulkapothecary.com:

{
 "store": "www.bulkapothecary.com",
 "productId": 479,
 "name": "Clear Stick Lip Balm (Filled and Unlabeled)",
 "sku": "B850-016-000",
 "price": 21.84,
 "salePrice": null,
 "currency": "USD",
 "categories": ["Make It", "Lip Balm", "Filled and Unlabeled Product"],
 "brand": null,
 "availability": "InStock",
 "inventoryLevel": null,
 "options": [
 {
 "name": "Flavor",
 "type": "set-radio",
 "required": true,
 "values": [
 { "id": 75486, "label": "Original" },
 { "id": 75484, "label": "Vanilla" },
 { "id": 75483, "label": "Cherry" }
 ]
 },
 {
 "name": "Quantity",
 "type": "set-radio",
 "required": true,
 "values": [
 { "id": 75478, "label": "Bag of 50 - ($0.44 / unit) - $21.84" },
 { "id": 75479, "label": "Case of 1000 - ($0.35 / unit after discount) - $347.20" }
 ]
 }
 ],
 "images": ["https://cdn11.bigcommerce.com/s-99si0d/images/stencil/1280x1280/products/479/55514/Clear_Stick_Cherry_1__17136.1684366114.jpg?c=2"],
 "description": "Bulk Apothecary offers the best prices in the USA on unlabeled clear stick lip balm…",
 "url": "https://www.bulkapothecary.com/clear-stick-lip-balm-filled-and-unlabeled/",
 "scrapedAt": "2026-08-06T11:38:57.406Z"
}

Field	Meaning
store	Storefront host the record came from
productId	BigCommerce product ID, read from the add-to-cart form, the id you pass to /remote/v1/product-attributes/
name	Product name (JSON-LD, falling back to the page <h1>, then og:title)
sku	Merchant SKU from the storefront JSON route. null when the merchant publishes none
price	List price. The MSRP or was-price when the store publishes one, otherwise the current price
salePrice	Discounted price currently charged, or null when not on sale. What a shopper pays today is salePrice ?? price
currency	ISO currency of the price, per the storefront's active currency
categories	Breadcrumb trail with Home and the product itself removed
brand	Brand name where the theme publishes one, else null
availability	InStock, OutOfStock or PreOrder, from the JSON route's instock / purchasable, falling back to schema.org
inventoryLevel	Units in stock, only when the merchant enables public inventory display, otherwise null
options	Configurable options: name, widget type, whether required, and every value with its BigCommerce value ID. Omitted when includeOptions is false
images	Full-resolution gallery URLs (1280px zoom variants where the theme provides them)
description	Product description as plain text, HTML stripped, entities decoded
url	Canonical product URL
scrapedAt	Run timestamp

Sale detection works: in the same run, Openhouse No. 3 came back with "price": 30, "salePrice": 27.95, "currency": "CAD".

RUN_SUMMARY

A RUN_SUMMARY record in the key-value store carries per-run counts, per-source failures and the applied filters:

{
 "storesRequested": 3,
 "storesFailed": 0,
 "perStore": [
 { "store": "cornerstone-light-demo.mybigcommerce.com", "route": "sitemap", "discovered": 6, "saved": 6, "bigcommerceConfirmed": true },
 { "store": "www.bulkapothecary.com", "route": "sitemap", "discovered": 6, "saved": 6, "bigcommerceConfirmed": true },
 { "store": "www.berlinpackaging.com", "route": "sitemap", "discovered": 6, "saved": 6, "bigcommerceConfirmed": true }
 ],
 "failures": [],
 "productsSaved": 18,
 "filters": {
 "categoryIds": [],
 "ignoredNumericCategoryIds": [],
 "maxProductsPerStore": 6,
 "includeOptions": true
 },
 "finishedAt": "2026-08-06T11:39:07.481Z"
}

perStore tells you which discovery route each store resolved to, how many product URLs it found, how many became records, and whether a BigCommerce fingerprint was seen on the page, the fastest way to tell "wrong platform" from "cap reached" from "theme renders client-side".

Use cases

1. Mid-market retail price monitoring. Watch a competitor's catalogue on a schedule and diff price, salePrice and availability on productId. Drop options to keep records small and the diff clean.

{
 "stores": ["www.bulkapothecary.com"],
 "maxProductsPerStore": 0,
 "includeOptions": false
}

2. Competitive assortment analysis. Pull several stores in one run and compare on categories and brand, who carries what, at what price, and where the gaps in your own range are.

{
 "stores": [
 "www.bulkapothecary.com",
 "https://www.berlinpackaging.com",
 "cornerstone-light-demo.mybigcommerce.com"
 ],
 "maxProductsPerStore": 500
}

3. Supplier catalogue extraction. Take a supplier's storefront and lift the section you resell into your own system, SKU, options with their value IDs, tiered pricing text, images and description, ready to map.

{
 "stores": ["www.bulkapothecary.com"],
 "categoryIds": ["fragrance-oils", "essential-oils"],
 "maxProductsPerStore": 0,
 "includeOptions": true
}

Limits and behaviour

• Storefront access varies by merchant configuration. The Actor detects the available route per store instead of assuming one shape, and reports the route it used in RUN_SUMMARY.perStore.
• The sitemap route is the reliable one. /xmlsitemap.php is served by every BigCommerce storefront and does not depend on the theme. Use it (leave categoryIds empty) whenever you want a complete catalogue.
• The category route is theme-dependent. It reads Stencil's standard product-card markup. Themes that render their product grid client-side, www.berlinpackaging.com is a live example, return a page with no server-rendered cards, so that category yields nothing. This is logged, recorded in RUN_SUMMARY.failures, and the run continues.
• Numeric category IDs do not work on public storefronts. See the input section.
• A failing store never aborts the run. Discovery failures, non-BigCommerce hosts and individual product errors are all recorded in RUN_SUMMARY.failures. The Actor only throws when every store produced zero products.
• Transient errors are retried. 429 and 5xx get three attempts with linear backoff. 404 and 403 fail fast.
• Sparse fields are honest, not broken. sku, brand and inventoryLevel are null when the merchant does not publish them. inventoryLevel in particular is only populated when the store enables public stock display; most do not.
• price: 0 is real. Sample and quote-only products price at zero on some merchants, www.berlinpackaging.com sample SKUs return $0.00 from the storefront's own pricing endpoint.
• JSON-LD is parsed defensively. Some themes inject raw newlines into the structured-data island, which is invalid JSON; the Actor retries the parse with control characters flattened rather than dropping the product.
• Public data only. No authentication, no personal data, no access-control bypass, every route used is one an anonymous shopper's browser calls.

Defaults: 2 GB memory · 30 min timeout.

FAQ

Do I need a proxy? No. Proxy configuration is not required to run this Actor.

Do I need an account on the source? No. You supply no credentials.

Does this use the BigCommerce API? Not the management API and not the Storefront GraphQL API, both need credentials the merchant controls. This reads the public storefront, including the /remote/v1/product-attributes/ JSON route that the storefront's own JavaScript uses.

What happens if a source is unavailable? It is reported in RUN_SUMMARY.failures and the run continues with the remaining stores.

Can I schedule it? Yes, it is designed for scheduled runs. Diff on productId plus price / salePrice / availability.

How do I know a store is BigCommerce? Run it and check RUN_SUMMARY.perStore[].bigcommerceConfirmed. A store on another platform returns 404 on /xmlsitemap.php and is reported as failed.

Why did I get fewer products than the store sells? Either maxProductsPerStore capped the run, or you used categoryIds on a theme that renders its grid client-side. perStore.discovered versus saved distinguishes the two immediately.

Does it return variant-level prices? It returns the base product price plus the full option matrix with each value's ID. Per-variant pricing usually appears in the option labels themselves ("Bag of 50 - ($0.44 / unit) - $21.84"), because that is how BigCommerce presents it.

Can I integrate it with something else? Yes, Apify API, client libraries, webhooks, scheduled runs, dataset exports (JSON/CSV/Excel) or MCP. Output is structured JSON.

API example

curl -X POST "https://api.apify.com/v2/acts/arman-bd~bigcommerce-store-scraper/run-sync-get-dataset-items?token=YOUR_TOKEN" \
 -H "Content-Type: application/json" \
 -d '{
 "stores": ["www.bulkapothecary.com"],
 "maxProductsPerStore": 25
 }'

JavaScript example

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: 'YOUR_TOKEN' });
const run = await client.actor('arman-bd/bigcommerce-store-scraper').call({
 stores: ['www.bulkapothecary.com'],
 maxProductsPerStore: 100,
 includeOptions: false,
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
for (const p of items) {
 const paid = p.salePrice ?? p.price;
 console.log(`${p.sku ?? p.productId}, ${p.name}, ${paid} ${p.currency} (${p.availability})`);
}