# BigCommerce Scraper: Catalog, Variants & Pricing (`arman-bd/bigcommerce-store-scraper`) Actor

Scrape BigCommerce storefronts for product catalogues: names, SKUs, prices, options, availability, categories and images. Covers a large mid-market merchant base.

- **URL**: https://apify.com/arman-bd/bigcommerce-store-scraper.md
- **Developed by:** [Arman Hossain](https://apify.com/arman-bd) (community)
- **Categories:** E-commerce, Lead generation, MCP servers
- **Stats:** 1 total users, 1 monthly users, 0.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.74 / 1,000 product scrapeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## BigCommerce Scraper: Catalog, Variants & Pricing

![BigCommerce Scraper: Product catalogues from any BigCommerce storefront, SKUs, prices, sale prices, options and stock](https://api.apify.com/v2/key-value-stores/ZQOcNAOHrIgTacAmy/records/bigcommerce-store-scraper.jpg)

**Agent skill: [SKILL.md](https://api.apify.com/v2/key-value-stores/t7YoTxpZEJOWvw4Ug/records/bigcommerce-store-scraper.md)**

```
https://api.apify.com/v2/key-value-stores/t7YoTxpZEJOWvw4Ug/records/bigcommerce-store-scraper.md
```

### What it does

Scrape BigCommerce storefronts for product catalogues: names, SKUs, prices, options, availability, categories and images. Covers a large mid-market merchant base. Give it stores and it returns one structured record per product.

Point it at a storefront domain and it walks the store's public product sitemap, then reads each product from two public storefront surfaces: the product page's structured-data island and BigCommerce's `/remote/v1/product-attributes/{id}` JSON route, the same endpoint the storefront's own JavaScript calls to price a product. That second call is what makes SKUs, live prices, sale prices, currency and stock reliable instead of scraped off rendered text.

Verified live against three unrelated merchants running three different themes: `cornerstone-light-demo.mybigcommerce.com`, `www.bulkapothecary.com` and `www.berlinpackaging.com`.

### Which BigCommerce surface this uses

This matters, so it is stated plainly. **This Actor targets the public storefront a shopper sees, not the authenticated BigCommerce management API, and not the Storefront GraphQL API.** No token, no store hash, no API account.

Two routes that are often assumed to work do **not** work unauthenticated, and this Actor does not use them:

| Route | What actually happens |
|---|---|
| `public API` | `401`, `{"errors":[{"message":"GraphQL credentials were missing. No token was sent."}]}`. The Storefront GraphQL API needs a token the merchant must mint; storefronts do not publish one in the page. |
| `public API` | `404`, no such route. `/api/storefront/*` covers carts, checkouts and orders only. |

What this Actor uses instead, all anonymous, all confirmed `200` on live merchant storefronts:

| Route | Purpose |
|---|---|
| `GET /xmlsitemap.php` → `?type=products&page=N` | Product discovery. Served on every BigCommerce storefront regardless of theme. |
| `GET /remote/v1/product-attributes/{productId}` | Authoritative JSON: SKU, UPC/MPN/GTIN, price, MSRP/was-price, currency, stock, purchasability, weight. |
| `GET /{product-url}` | Product page, JSON-LD `Product` and `BreadcrumbList`, Open Graph tags, image gallery, option widgets. |
| `GET /{category-path}?page=N&limit=50` | Optional category-scoped discovery on Stencil-standard themes. |

### Input

| Field | Type | Default | Notes |
|---|---|---|---|
| `stores` | array | - | **Required.** Storefront domains or URLs. Anything past the host is ignored, so `https://store.com/some/page` and `store.com` are the same input. |
| `categoryIds` | array | `[]` | Restrict the sweep to categories. Pass a **category page path or full URL**, `fragrance-oils`, `/shop-all/kitchen/`. Empty = walk the whole product sitemap. |
| `maxProductsPerStore` | integer | `50` | Cap per store. `0` = no limit. Each product costs two requests. |
| `includeOptions` | boolean | `true` | Include configurable options and modifiers. Set `false` to drop the field entirely. |

```json
{
 "stores": ["www.bulkapothecary.com", "https://www.berlinpackaging.com"],
 "maxProductsPerStore": 50,
 "includeOptions": true
}
```

**Which combinations make sense.**

- **Whole-catalogue sweep.** `stores` only, `maxProductsPerStore: 0`. Uses the sitemap route, which works on every theme. Budget two requests per product.
- **Narrow, repeatable price watch.** Add `categoryIds` with the category paths you care about and keep `maxProductsPerStore` small. Uses the category route, which is faster but theme-dependent (see below).
- **Cheap change detection.** `includeOptions: false`. Records shrink a lot on stores with large option sets, and you still get price, sale price and availability.

**About `categoryIds`.** The field keeps its name for compatibility, but **numeric BigCommerce category IDs cannot be used.** Public storefronts expose no id-addressable category route, `?category=23`, `/categories/23` and `rss.php?categoryid=23` are all either ignored or `404`, verified live. Numeric entries are skipped with a warning and listed in `RUN_SUMMARY.filters.ignoredNumericCategoryIds`. Pass the category page path instead; that is the selector the storefront actually honours.

### Output

One dataset item per product. Annotated sample, real values from a live run against `www.bulkapothecary.com`:

```json
{
 "store": "www.bulkapothecary.com",
 "productId": 479,
 "name": "Clear Stick Lip Balm (Filled and Unlabeled)",
 "sku": "B850-016-000",
 "price": 21.84,
 "salePrice": null,
 "currency": "USD",
 "categories": ["Make It", "Lip Balm", "Filled and Unlabeled Product"],
 "brand": null,
 "availability": "InStock",
 "inventoryLevel": null,
 "options": [
 {
 "name": "Flavor",
 "type": "set-radio",
 "required": true,
 "values": [
 { "id": 75486, "label": "Original" },
 { "id": 75484, "label": "Vanilla" },
 { "id": 75483, "label": "Cherry" }
 ]
 },
 {
 "name": "Quantity",
 "type": "set-radio",
 "required": true,
 "values": [
 { "id": 75478, "label": "Bag of 50 - ($0.44 / unit) - $21.84" },
 { "id": 75479, "label": "Case of 1000 - ($0.35 / unit after discount) - $347.20" }
 ]
 }
 ],
 "images": ["https://cdn11.bigcommerce.com/s-99si0d/images/stencil/1280x1280/products/479/55514/Clear_Stick_Cherry_1__17136.1684366114.jpg?c=2"],
 "description": "Bulk Apothecary offers the best prices in the USA on unlabeled clear stick lip balm…",
 "url": "https://www.bulkapothecary.com/clear-stick-lip-balm-filled-and-unlabeled/",
 "scrapedAt": "2026-08-06T11:38:57.406Z"
}
```

| Field | Meaning |
|---|---|
| `store` | Storefront host the record came from |
| `productId` | BigCommerce product ID, read from the add-to-cart form, the id you pass to `/remote/v1/product-attributes/` |
| `name` | Product name (JSON-LD, falling back to the page `<h1>`, then `og:title`) |
| `sku` | Merchant SKU from the storefront JSON route. `null` when the merchant publishes none |
| `price` | **List price.** The MSRP or was-price when the store publishes one, otherwise the current price |
| `salePrice` | **Discounted price currently charged**, or `null` when not on sale. What a shopper pays today is `salePrice ?? price` |
| `currency` | ISO currency of the price, per the storefront's active currency |
| `categories` | Breadcrumb trail with `Home` and the product itself removed |
| `brand` | Brand name where the theme publishes one, else `null` |
| `availability` | `InStock`, `OutOfStock` or `PreOrder`, from the JSON route's `instock` / `purchasable`, falling back to schema.org |
| `inventoryLevel` | Units in stock, only when the merchant enables public inventory display, otherwise `null` |
| `options` | Configurable options: name, widget type, whether required, and every value with its BigCommerce value ID. Omitted when `includeOptions` is `false` |
| `images` | Full-resolution gallery URLs (1280px zoom variants where the theme provides them) |
| `description` | Product description as plain text, HTML stripped, entities decoded |
| `url` | Canonical product URL |
| `scrapedAt` | Run timestamp |

Sale detection works: in the same run, `Openhouse No. 3` came back with `"price": 30, "salePrice": 27.95, "currency": "CAD"`.

#### `RUN_SUMMARY`

A `RUN_SUMMARY` record in the key-value store carries per-run counts, per-source failures and the applied filters:

```json
{
 "storesRequested": 3,
 "storesFailed": 0,
 "perStore": [
 { "store": "cornerstone-light-demo.mybigcommerce.com", "route": "sitemap", "discovered": 6, "saved": 6, "bigcommerceConfirmed": true },
 { "store": "www.bulkapothecary.com", "route": "sitemap", "discovered": 6, "saved": 6, "bigcommerceConfirmed": true },
 { "store": "www.berlinpackaging.com", "route": "sitemap", "discovered": 6, "saved": 6, "bigcommerceConfirmed": true }
 ],
 "failures": [],
 "productsSaved": 18,
 "filters": {
 "categoryIds": [],
 "ignoredNumericCategoryIds": [],
 "maxProductsPerStore": 6,
 "includeOptions": true
 },
 "finishedAt": "2026-08-06T11:39:07.481Z"
}
```

`perStore` tells you which discovery route each store resolved to, how many product URLs it found, how many became records, and whether a BigCommerce fingerprint was seen on the page, the fastest way to tell "wrong platform" from "cap reached" from "theme renders client-side".

### Use cases

**1. Mid-market retail price monitoring.** Watch a competitor's catalogue on a schedule and diff `price`, `salePrice` and `availability` on `productId`. Drop options to keep records small and the diff clean.

```json
{
 "stores": ["www.bulkapothecary.com"],
 "maxProductsPerStore": 0,
 "includeOptions": false
}
```

**2. Competitive assortment analysis.** Pull several stores in one run and compare on `categories` and `brand`, who carries what, at what price, and where the gaps in your own range are.

```json
{
 "stores": [
 "www.bulkapothecary.com",
 "https://www.berlinpackaging.com",
 "cornerstone-light-demo.mybigcommerce.com"
 ],
 "maxProductsPerStore": 500
}
```

**3. Supplier catalogue extraction.** Take a supplier's storefront and lift the section you resell into your own system, SKU, options with their value IDs, tiered pricing text, images and description, ready to map.

```json
{
 "stores": ["www.bulkapothecary.com"],
 "categoryIds": ["fragrance-oils", "essential-oils"],
 "maxProductsPerStore": 0,
 "includeOptions": true
}
```

### Limits and behaviour

- **Storefront access varies by merchant configuration.** The Actor detects the available route per store instead of assuming one shape, and reports the route it used in `RUN_SUMMARY.perStore`.
- **The sitemap route is the reliable one.** `/xmlsitemap.php` is served by every BigCommerce storefront and does not depend on the theme. Use it (leave `categoryIds` empty) whenever you want a complete catalogue.
- **The category route is theme-dependent.** It reads Stencil's standard product-card markup. Themes that render their product grid client-side, `www.berlinpackaging.com` is a live example, return a page with no server-rendered cards, so that category yields nothing. This is logged, recorded in `RUN_SUMMARY.failures`, and the run continues.
- **Numeric category IDs do not work** on public storefronts. See the input section.
- **A failing store never aborts the run.** Discovery failures, non-BigCommerce hosts and individual product errors are all recorded in `RUN_SUMMARY.failures`. The Actor only throws when *every* store produced zero products.
- **Transient errors are retried.** `429` and `5xx` get three attempts with linear backoff. `404` and `403` fail fast.
- **Sparse fields are honest, not broken.** `sku`, `brand` and `inventoryLevel` are `null` when the merchant does not publish them. `inventoryLevel` in particular is only populated when the store enables public stock display; most do not.
- **`price: 0` is real.** Sample and quote-only products price at zero on some merchants, `www.berlinpackaging.com` sample SKUs return `$0.00` from the storefront's own pricing endpoint.
- **JSON-LD is parsed defensively.** Some themes inject raw newlines into the structured-data island, which is invalid JSON; the Actor retries the parse with control characters flattened rather than dropping the product.
- **Public data only.** No authentication, no personal data, no access-control bypass, every route used is one an anonymous shopper's browser calls.

**Defaults:** 2 GB memory · 30 min timeout.

### FAQ

**Do I need a proxy?** No. Proxy configuration is not required to run this Actor.

**Do I need an account on the source?** No. You supply no credentials.

**Does this use the BigCommerce API?** Not the management API and not the Storefront GraphQL API, both need credentials the merchant controls. This reads the public storefront, including the `/remote/v1/product-attributes/` JSON route that the storefront's own JavaScript uses.

**What happens if a source is unavailable?** It is reported in `RUN_SUMMARY.failures` and the run continues with the remaining stores.

**Can I schedule it?** Yes, it is designed for scheduled runs. Diff on `productId` plus `price` / `salePrice` / `availability`.

**How do I know a store is BigCommerce?** Run it and check `RUN_SUMMARY.perStore[].bigcommerceConfirmed`. A store on another platform returns `404` on `/xmlsitemap.php` and is reported as failed.

**Why did I get fewer products than the store sells?** Either `maxProductsPerStore` capped the run, or you used `categoryIds` on a theme that renders its grid client-side. `perStore.discovered` versus `saved` distinguishes the two immediately.

**Does it return variant-level prices?** It returns the base product price plus the full option matrix with each value's ID. Per-variant pricing usually appears in the option labels themselves (`"Bag of 50 - ($0.44 / unit) - $21.84"`), because that is how BigCommerce presents it.

**Can I integrate it with something else?** Yes, Apify API, client libraries, webhooks, scheduled runs, dataset exports (JSON/CSV/Excel) or MCP. Output is structured JSON.

### API example

```bash
curl -X POST "https://api.apify.com/v2/acts/arman-bd~bigcommerce-store-scraper/run-sync-get-dataset-items?token=YOUR_TOKEN" \
 -H "Content-Type: application/json" \
 -d '{
 "stores": ["www.bulkapothecary.com"],
 "maxProductsPerStore": 25
 }'
```

### JavaScript example

```js
import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: 'YOUR_TOKEN' });
const run = await client.actor('arman-bd/bigcommerce-store-scraper').call({
 stores: ['www.bulkapothecary.com'],
 maxProductsPerStore: 100,
 includeOptions: false,
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
for (const p of items) {
 const paid = p.salePrice ?? p.price;
 console.log(`${p.sku ?? p.productId}, ${p.name}, ${paid} ${p.currency} (${p.availability})`);
}
```

# Actor input Schema

## `stores` (type: `array`):

Storefront domains or URLs. 'www.bulkapothecary.com' or 'https://www.bulkapothecary.com/'. Anything past the host is ignored. Must be a live BigCommerce storefront; a store on another platform will report no products.

## `categoryIds` (type: `array`):

Restrict the sweep to specific categories. Pass the category page path or full URL. 'fragrance-oils', '/shop-all/kitchen/' or 'https://store.com/honey-jars/'. Leave empty to walk the store's whole product sitemap. Numeric BigCommerce category IDs are NOT usable: public storefronts expose no id-addressable category route, so numeric entries are skipped and reported in RUN_SUMMARY.

## `maxProductsPerStore` (type: `integer`):

Cap the number of products scraped per store. Each product costs two requests, so a large cap on a big catalogue needs a longer run timeout. Set 0 for no limit.

## `includeOptions` (type: `boolean`):

Include the product's configurable options and modifiers. option name, widget type, whether it is required, and every selectable value with its BigCommerce value ID. Disable for a smaller, flatter dataset.

## Actor input object example

```json
{
  "stores": [
    "www.bulkapothecary.com",
    "https://www.berlinpackaging.com"
  ],
  "categoryIds": [
    "fragrance-oils",
    "/shop-all/kitchen/"
  ],
  "maxProductsPerStore": 50,
  "includeOptions": true
}
```

# Actor output Schema

## `items` (type: `string`):

Every record the run produced.

## `runsummary` (type: `string`):

The RUN_SUMMARY record from the run's key-value store.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "stores": [
        "cornerstone-light-demo.mybigcommerce.com"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("arman-bd/bigcommerce-store-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "stores": ["cornerstone-light-demo.mybigcommerce.com"] }

# Run the Actor and wait for it to finish
run = client.actor("arman-bd/bigcommerce-store-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "stores": [
    "cornerstone-light-demo.mybigcommerce.com"
  ]
}' |
apify call arman-bd/bigcommerce-store-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,arman-bd/bigcommerce-store-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/vQvzcLb9ed68RGdZV/builds/YXwjczaeIbB8eMIQ7/openapi.json