Shopify API Scraper

Shopify API Scraper

Most eCommerce stores based on the Shopify platform expose a RESTful API to get product data, e.g. the sneaker boutique Social Status:

• https://socialstatuspgh.com/products.json — the whole catalogue
• https://socialstatuspgh.com/collections/sneakers/products.json — one collection

Other sneaker and apparel stores that work the same way:

• https://solefly.com/products.json
• https://www.allbirds.com/products.json
• https://pelacase.myshopify.com/products.json

Note, however, that in some cases this trick won't work with primary store domain. In that case one should inspect web site source with View Source feature of Chrome browser and look for *.myshopify.com subdomain being referenced in JavaScript code, e.g.:

Shopify.shop = "pelacase.myshopify.com";

Proxy — required in practice

Shopify rate-limits products.json per client IP, across stores: once the limiter trips, unrelated storefronts start answering HTTP 429 {"errors":[{"message":"You have sent too many requests..."}]}. Measured on one host, 9 of 18 unrelated stores returned 429 after a short burst, and the same request was 429 through Apify's datacenter pool while succeeding through residential:

direct       HTTP 200  products=250
datacenter   HTTP 429
residential  HTTP 200  products=250

So the scraper takes a fresh exit IP for every request and defaults to Apify Proxy residential. A full 10-page catalogue crawl then completes with no 429s at all. You can switch pools or disable the proxy in the input, but expect throttling past a few pages without one.

Input example

{
  "start_urls": [
    {
      "url": "https://socialstatuspgh.com/products.json"
    }
  ],
  "proxyConfiguration": {
    "useApifyProxy": true,
    "apifyProxyGroups": ["RESIDENTIAL"]
  }
}

Output example

This scraper outputs one row per product variant, e.g.:

{
  "product_id": 7985713840170,
  "title": "THE WHITAKER GRP SUEDE CODA ECHO POINTS (CHAMBRAY BLUE/ALPINE SNOW/BUTTERCREAM)",
  "description": "The Whitaker Grp introduces PUMA CODA a fresh take on an everyday classic. Crafted from soft premium suede, this low-top sneaker pairs a clean neutral upper with textured pink terry laces for an unexpected pop of color. Designed with subtle contrast stitching and a durable rubber sole, it delivers effortless comfort while making a statement with every step.\n\nDetails\n\n\nPremium suede upper\nPlush terry laces\nLow-top silhouette\nContrast stitching throughout\nCushioned footbed for all-day comfort\nDurable rubber outsole\nTrue to size\n\n\nSKU: 40874401",
  "handle": "the-whitaker-grp-coda-echo-points-chambray-blue-alpine-snow-buttercream",
  "web_url": "https://socialstatuspgh.com/products/the-whitaker-grp-coda-echo-points-chambray-blue-alpine-snow-buttercream",
  "vendor": "PUMA CODA",
  "product_type": "SNEAKERS",
  "tags": [
    "81026",
    "CODA",
    "FinalSale",
    "fw26",
    "maincolor-blue/cream",
    "mens",
    "new arrivals",
    "nosale",
    "puma",
    "sneakers"
  ],
  "images": [
    "https://cdn.shopify.com/s/files/1/0262/2475/files/PUMACODABLU-1.jpg?v=1786378574",
    "https://cdn.shopify.com/s/files/1/0262/2475/files/PUMACODABLU-2.jpg?v=1786378574",
    "https://cdn.shopify.com/s/files/1/0262/2475/files/PUMACODABLU-3.jpg?v=1786378574",
    "..."
  ],
  "featured_image": null,
  "variant_id": 44682235478058,
  "variant_title": "4",
  "option1": "4",
  "option2": null,
  "option3": null,
  "variant_sku": "40874401",
  "variant_available": false,
  "price": "130.00",
  "compare_at_price": null
}

When something goes wrong

The run status tells you whose problem it is, and the status message says what happened. A run never finishes green and empty without an explanation:

What happened	Run status	What you do
No usable products.json URL in the input	Succeeded, "Nothing to scrape: ..."	fix the URL
Store is password-protected, or the endpoint is disabled (401/403/404, or an HTML page)	Succeeded, "the catalogue is not public"	nothing — the store does not serve it
Rate-limited after 4 attempts on rotating IPs	Succeeded, "still failing after 4 attempts"	enable/raise the residential proxy, retry later
Run timeout hit mid-catalogue	Succeeded, "the run timeout was reached"	raise the run timeout; rows already fetched are kept
The endpoint changed shape, or an unexpected bug	Failed	report it — this one is ours

Partial results are always delivered: rows are flushed as pages complete, so a run that is throttled or cut short still hands over everything it fetched, and a page that fails does not discard the pages that succeeded. A run that hits any error also writes an ERROR_SUMMARY record to the key-value store with the counts and a sample of each.

Notes

• Some of the heavily customised Shopify stores (e.g. Supreme) do not expose the API this scraper relies on.
• It is recommended to disable timeout for larger scraping jobs that span across multiple stores or cover large amount of products.

Global marketplace toolkit

Part of the Global marketplace toolkit — product and pricing data across international B2B/B2C marketplaces and classifieds sites:

• DHgate API Scraper — Scrapes DHgate product search results and detail pages via mobile API.
• Made-in-China.com API Scraper — Scrapes product search results and details from Made-in-China.com's mobile app API.
• Back Market API Scraper — Scrape refurbished-device listings and full product detail from Back Market's API.
• Senukai.lt API Scraper — Scrape Senukai.lt product listings and full detail — prices, discounts, EAN/GTIN, images.
• Kleinanzeigen API Scraper — Scrape Kleinanzeigen (Germany) classified-ad listings and full details via mobile API.
• Algolia API Scraper — Generalised scraper for any Algolia-powered search index — scrape hits or discover facets.

Did you find this useful?

⭐ Rate this actor on Apify! Your feedback helps other users find it and helps us keep improving it.