Shopify Products Scraper

Shopify Products Scraper

Shopify Products Scraper is a production-ready Shopify product scraper that discovers product pages on Shopify stores and fetches each product’s .json for clean, structured extraction. It solves the manual, error-prone effort to scrape products from Shopify store catalogs by pairing HTML discovery of “/products/” links with Shopify’s “/products.json” pagination when detected. Built for marketers, developers, data analysts, and researchers, this Shopify product data scraper and Shopify product data extractor enables bulk competitor analysis, catalog building, and price monitoring at scale — with live-saving to datasets for effortless export and automation. 🚀

What data / output can you get?

Below are the exact dataset fields this Shopify product scraping tool pushes during a run. Each row represents a single product fetched from a store.

Data type	Description	Example value
store_url	The Shopify store URL where the product was found	https://lootcrate.com
product_url	Direct product page URL	https://lootcrate.com/products/loot-crate
json_url	The corresponding Shopify Product JSON endpoint	https://lootcrate.com/products/loot-crate.json
product_id	Unique Shopify product ID from the JSON	5083963261059
title	Product title from the JSON payload	Loot Crate
vendor	Product vendor/brand	Loot Crate Core
product_type	Product type/category from the JSON	Subscription Box
price	Current price (from the first variant if present)	29.99
compare_at_price	Compare-at (original) price if available	24.99
tags	Comma-separated product tags	Subscription, Collectibles, Pop Culture
total_found	Count of products discovered for the store in this run	5
successful	Running count of successfully extracted products for the store	5
full_data	Complete product JSON object as returned by Shopify	{ "product": { ... } }

Notes:

• full_data preserves the complete Shopify response for each product (including variants, images, options, and other metadata) — ideal for Shopify product variant scraper, Shopify product image scraper, and Shopify inventory scraper use cases.
• You can export results as CSV or JSON directly from the Apify dataset.

Key features

• 🔍 Automatic product discovery
  Detects Shopify storefronts and paginates “/products.json” for efficient discovery. Falls back to scanning HTML for “/products/” links and deduplicates them when needed — perfect for a Shopify store product scraper operating across diverse sites.

• 🧠 Intelligent proxy fallback
  Starts with no proxy, then automatically escalates to datacenter and finally residential proxies on 403/429 blocks. Once residential is activated, it stays sticky for the remaining requests, with clear logs of all switches.

• ⚡ Concurrent, async extraction
  Fetches product .json endpoints concurrently for high throughput on large catalogs — an ideal Shopify product scraping API workflow building block.

• 💾 Live, resilient data saving
  Pushes each product to the dataset as it’s processed to prevent data loss and enable real-time monitoring and exports.

• 🧱 Robust error handling
  Built-in retries, graceful handling of malformed/missing data, and detailed logs keep runs reliable — a dependable Shopify product scraping tool for production.

• 🧰 Developer-friendly integrations
  Built on the Apify SDK (Python). Access datasets via the Apify API, automate with webhooks, or plug into Make and n8n. Great for building a Shopify product feed scraper or Shopify product catalog scraper pipeline.

• 🔐 Public data only, no login required
  Targets publicly available product pages and JSON endpoints — no credentials, sessions, or extensions required.

How to use Shopify Products Scraper - step by step

1. Sign in to your Apify account and open the Apify Console.
2. Search for the “shopify-products-scraper” actor by scrapeengine and open it.
3. In the Input tab, add one or more Shopify store URLs in startUrls. For example:
   • https://lootcrate.com
   • https://www.decathlon.com
4. (Optional) In proxyConfiguration, keep the default (no proxy) or enable Apify Proxy. The actor will automatically fall back to datacenter and then residential proxies if blocking occurs.
5. Click Start to run. The actor detects if the site is Shopify and either paginates “/products.json” or extracts product links from HTML.
6. Monitor logs to see product discovery progress, proxy fallback events, and success counts in real time.
7. Go to the Dataset tab to view results and export as CSV or JSON.

Pro Tip: Use the Apify API to pull datasets into BI tools or data pipelines, or trigger runs from Make, n8n, or your backend. This enables automated “download Shopify product data” workflows end-to-end.

Use cases

Use case name	Description
Competitor catalog monitoring	Track competitor assortments and pricing across Shopify stores to inform merchandising and promotions.
Price monitoring for variants	Capture variant-level prices via the product JSON to analyze discounting and compare-at strategies using a Shopify price scraper approach.
Catalog building for marketplaces	Aggregate product data from target Shopify stores to seed or enrich your own product catalog.
SEO research and content mapping	Collect titles, tags, and product metadata to understand keyword usage and content strategies across niches.
Bulk product data extraction (API pipeline)	Feed structured product rows from the dataset into warehouses, BI tools, or custom APIs for analytics.
Market research across niches	Discover and compare product types and vendors across many stores for trend and gap analysis.
Shopify variants & images analysis	Use full_data for detailed Shopify product variant scraper and Shopify product image scraper workflows.

Why choose Shopify Products Scraper?

Built for precision, automation, and reliability, this Shopify product scraper outperforms manual methods and brittle browser extensions.

• ✅ Accurate Shopify detection and data capture using “/products.json” when available, with HTML discovery fallback.
• 🚀 Scales to many stores with concurrent requests and live dataset saving for long runs.
• 🔄 Smart proxy management with automatic escalation and sticky residential mode for tough stores.
• 🧩 Developer access via the Apify SDK and Apify API — ideal for integrating a Shopify product scraping API into ETL pipelines.
• 🔐 Ethical by design: only public endpoints and pages, no login or cookies required.
• 📤 Export-ready outputs (CSV/JSON) for analytics, catalog ops, and downstream automations.
• 🧱 Production-grade retries and error handling for consistent results versus unstable alternatives.

In short, it’s a dependable Shopify product data scraper that combines discovery, speed, and resilience for real-world workloads.

Is it legal / ethical to use Shopify Products Scraper?

Yes — when used responsibly. This actor collects data from publicly accessible Shopify product pages and their “.json” endpoints. It does not access private or authenticated areas.

Guidelines for compliance:

• Only collect public product information and respect each website’s terms of service.
• Be mindful of robots.txt and reasonable request rates.
• Ensure your usage complies with applicable laws and regulations (e.g., GDPR, CCPA).
• Do not use scraped data for spam or any unlawful activity.
• Consult your legal team for edge cases or jurisdiction-specific requirements.

Input parameters & output format

Example JSON input

{
  "startUrls": [
    "https://lootcrate.com",
    "https://www.decathlon.com"
  ],
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}

Input fields

• startUrls (array)

  • Description: List one or more Shopify store URLs (e.g., https://lootcrate.com, https://www.decathlon.com). Supports bulk input.
  • Default: ["https://lootcrate.com"]
  • Required: Yes

• proxyConfiguration (object)

  • Description: Choose which proxies to use. By default, no proxy is used. If the platform rejects or blocks the request, it will automatically fallback to datacenter proxy, then residential proxy with 3 retries.
  • Default: {"useApifyProxy": false}
  • Required: No

Example JSON output (dataset item)

{
  "store_url": "https://lootcrate.com",
  "product_url": "https://lootcrate.com/products/loot-crate",
  "json_url": "https://lootcrate.com/products/loot-crate.json",
  "product_id": 5083963261059,
  "title": "Loot Crate",
  "vendor": "Loot Crate Core",
  "product_type": "Subscription Box",
  "price": "29.99",
  "compare_at_price": "24.99",
  "tags": "Subscription, Collectibles, Pop Culture",
  "total_found": 5,
  "successful": 5,
  "full_data": {
    "product": {
      "id": 5083963261059,
      "title": "Loot Crate",
      "vendor": "Loot Crate Core",
      "product_type": "Subscription Box",
      "handle": "loot-crate",
      "variants": [
        {
          "id": 34197535719555,
          "price": "29.99",
          "compare_at_price": "24.99",
          "inventory_management": "shopify",
          "requires_shipping": true
        }
      ],
      "images": [
        {
          "id": 123456789,
          "src": "https://cdn.shopify.com/...",
          "width": 2000,
          "height": 2000,
          "alt": "Product image"
        }
      ],
      "tags": "Subscription, Collectibles, Pop Culture"
    }
  }
}

Notes:

• price and compare_at_price are derived from the first variant when variants exist; these may be null if no variants are present in the product JSON.
• In addition to dataset rows, the actor stores a store-level summary object in the default key-value store under the key OUTPUT, including method, total_found, successful, and an array of products with url and json.

FAQ

Do I need to list individual product URLs?

No. The actor scans store HTML for links containing “/products/” and, when Shopify is detected, paginates the “/products.json” endpoint. It automatically discovers products for you.

What happens if a store blocks my requests?

It implements proxy fallback. The run starts with no proxy, escalates to datacenter on 403/429, and then to residential proxies with retries. Once residential is used, it stays sticky for the remaining requests.

Can I scrape multiple Shopify stores at once?

Yes. Add multiple store homepages to startUrls. The run processes each store and writes all products to the dataset, with per-store discovery and success counts included in each row.

What data does the Shopify Products Scraper return?

Each dataset item includes store_url, product_url, json_url, product_id, title, vendor, product_type, price, compare_at_price, tags, total_found, successful, and full_data containing the complete Shopify product JSON (ideal for variants, images, and metadata).

Do I need login or cookies?

No. This Shopify product scraping tool targets public product pages and their JSON endpoints without login or session data.

How do I export the data?

Open the run’s Dataset and export to CSV or JSON. You can also access datasets via the Apify API for automated pipelines.

Can I use it as a Shopify product feed scraper or API?

Yes. Results are stored in Apify datasets (and a store-level summary in the key-value store), which you can access programmatically via the Apify API to build a Shopify product scraping API or product feed workflow.

Can I limit the number of products scraped?

The actor is designed to process discovered products comprehensively. You can filter or sample results downstream after export, or adapt the code to add custom limits for your workflow.

Closing CTA / Final thoughts

Shopify Products Scraper is built to discover and extract structured Shopify product data at scale. With automatic product discovery, concurrent fetching, intelligent proxy fallback, and live dataset saving, it streamlines catalog building, competitor tracking, and analysis for real-world workloads.

Whether you’re a marketer, developer, data analyst, or researcher, you can export clean product data (CSV/JSON), integrate via the Apify API, and automate end-to-end workflows. Start extracting smarter Shopify product data today and turn public storefronts into actionable insights.