Shopify Scraper

Shopify Scraper provide a way to crawl product details of site build on Shopify.

ChangeLogs

02/03/2025

• use new method to escape site gateway forbid
• added support of passthrough currency
• added support of parse InStock/OutOfStock status for single product (most cases are supported)

Features

1. Support to fetch all collections of Shopify site;
2. Fetch products of any collection or all the products of the site;
3. Fetch single product by product url;
4. Support of specify the currency of the product;

To check if this site is a Shopify site, open the url `https://

The price values are multiplied by 100 to avoiding floating point calculations.

TODO

1. Support tag filter for collection;

Input Parameters

The input of this scraper shoule be JSON formated. Fields are:

Here is the example:

1{
2  "startUrls": [
3    { "url": "https://uk.gymshark.com/collections/crop-tops" }
4  ],
5  "proxy": { "useApifyProxy": true }
6}

Input Examples

• fetch collections of shopify site

1{
2  "startUrls": [
3    { "url": "https://uk.gymshark.com/collections" }
4  ],
5  "proxy": { "useApifyProxy": true }
6}

• fetch all products of shopify site

1{
2  "startUrls": [
3    { "url": "https://uk.gymshark.com" }
4  ],
5  "proxy": { "useApifyProxy": true }
6}

• fetch products of one collection

1{
2  "startUrls": [
3    { "url": "https://uk.gymshark.com/collections/crop-tops?currency=USD" }
4  ],
5  "proxy": { "useApifyProxy": true }
6}

• fetch product detail

1{
2  "startUrls": [
3    { "url": "https://uk.gymshark.com/products/gymshark-vision-long-sleeve-crop-top-black-aw21" }
4  ],
5  "proxy": { "useApifyProxy": true }
6}

• fetch product detail with specified currency

1{
2  "startUrls": [
3    { "url": "https://www.fashionnova.com/products/von-dutch-short-sleeve-pj-boxer-pant-set-pink?currency=GBP" }
4  ],
5  "proxy": { "useApifyProxy": true }
6}

Data storage

Shopify site scraper stores the product data to default data set in JSON format.

1{
2  "source": {
3    "id": "4857453543626",
4    "canonicalUrl": "https://www.gymshark.com/products/gymshark-fraction-crop-top-light-green-white-logo",
5    "retailer": "Gymshark | Be a visionary.",
6    "language": "en",
7    "currency": "USD",
8    "createdUTC": 1613989955000,
9    "updatedUTC": 1640058905000,
10    "publishedUTC": 1614279501000
11  },
12  "title": "Gymshark Fraction Crop Top - Light Green",
13  "description": "<meta charset=\"utf-8\"><strong data-mce-fragment=\"1\">IN YOUR LOCKER</strong><span data-mce-fragment=\"1\"></span><br data-mce-fragment=\"1\"><br data-mce-fragment=\"1\"><span data-mce-fragment=\"1\">A slice of unique athleisure, the Fraction Crop Top is tasteful, charming and convenient. Available in a range of versatile colours, style any workout or rest day outfit with the finishing touch of this short sleeve crop top.</span><br data-mce-fragment=\"1\"><br data-mce-fragment=\"1\"><span data-mce-fragment=\"1\">- Cropped t-shirt</span><br data-mce-fragment=\"1\"><span data-mce-fragment=\"1\">- Slight dropped shoulder</span><br data-mce-fragment=\"1\"><span data-mce-fragment=\"1\">- Crew neck</span><br data-mce-fragment=\"1\"><span data-mce-fragment=\"1\">- Rolled hem to sleeve</span><br data-mce-fragment=\"1\"><span data-mce-fragment=\"1\">- Large cut off Gymshark logo at hem</span><br data-mce-fragment=\"1\"><span data-mce-fragment=\"1\">- 95% Cotton, 5% Elastane</span><br data-mce-fragment=\"1\"><span data-mce-fragment=\"1\">- We've cut down on our use of swing tags, so this product comes without one</span><br data-mce-fragment=\"1\"><span data-mce-fragment=\"1\">- Model is </span><meta charset=\"utf-8\"><span data-mce-fragment=\"1\">5'9\" and wears a size L<br></span>- Video model is 5'8\" and wears size XS<br data-mce-fragment=\"1\"><meta charset=\"utf-8\"><span data-mce-fragment=\"1\">- SKU: GLCT1844-CLM</span><br>",
14  "brand": "Gymshark | Be a visionary.",
15  "categories": [
16    "Womens Crop Top"
17  ],
18  "tags": [
19    "25/02/21",
20    "all-products",
21    "crop-tops",
22    "essentials",
23    "filter-colour: Green",
24    "filter-size:l",
25    "filter-size:m",
26    "filter-size:s",
27    "filter-size:xl",
28    "filter-size:xs",
29    "filter-size:xxl",
30    "home-workout-clothes",
31    "instock:s",
32    "instock:xs",
33    "outlet",
34    "retention-collection",
35    "short-sleeve",
36    "sizeguide:top",
37    "SS21",
38    "t-shirts-tops",
39    "Womens"
40  ],
41  "variants": [
42    {
43      "id": "32686177550538",
44      "title": "Extra Small",
45      "sku": "GLCT1844-CLM-XS",
46      "options": [
47        "Extra Small"
48      ],
49      "price": {
50        "current": 1500,
51        "previous": 2500,
52        "stockStatus": "InStock"
53      }
54    },
55    {
56      "id": "32686177583306",
57      "title": "Small",
58      "sku": "GLCT1844-CLM-S",
59      "options": [
60        "Small"
61      ],
62      "price": {
63        "current": 1500,
64        "previous": 2500,
65        "stockStatus": "InStock"
66      }
67    },
68    {
69      "id": "32686177616074",
70      "title": "Medium",
71      "sku": "GLCT1844-CLM-M",
72      "options": [
73        "Medium"
74      ],
75      "price": {
76        "current": 1500,
77        "previous": 2500,
78        "stockStatus": "OutOfStock"
79      }
80    },
81    {
82      "id": "32686177648842",
83      "title": "Large",
84      "sku": "GLCT1844-CLM-L",
85      "options": [
86        "Large"
87      ],
88      "price": {
89        "current": 1500,
90        "previous": 2500,
91        "stockStatus": "OutOfStock"
92      }
93    },
94    {
95      "id": "32686177681610",
96      "title": "Extra Large",
97      "sku": "GLCT1844-CLM-XL",
98      "options": [
99        "Extra Large"
100      ],
101      "price": {
102        "current": 1500,
103        "previous": 2500,
104        "stockStatus": "OutOfStock"
105      }
106    },
107    {
108      "id": "39785973285066",
109      "title": "Extra Extra Large",
110      "sku": "GLCT1844-CLM-XXL",
111      "options": [
112        "Extra Extra Large"
113      ],
114      "price": {
115        "current": 1500,
116        "previous": 2500,
117        "stockStatus": "OutOfStock"
118      }
119    }
120  ],
121  "medias": [
122    {
123      "id": "15928233361610",
124      "type": "Image",
125      "url": "https://cdn.shopifycdn.net/s/files/1/0156/6146/products/TRAININGCROPTEECOOLMINT.A-Edit_AS.jpg?v=1613989955",
126      "variantIds": [],
127      "alt": ""
128    },
129    {
130      "id": "15928233394378",
131      "type": "Image",
132      "url": "https://cdn.shopifycdn.net/s/files/1/0156/6146/products/TRAININGCROPTEECOOLMINT.B-Edit_AS.jpg?v=1613989955",
133      "variantIds": [],
134      "alt": ""
135    },
136    {
137      "id": "15928233427146",
138      "type": "Image",
139      "url": "https://cdn.shopifycdn.net/s/files/1/0156/6146/products/TRAININGCROPTEECOOLMINT.C-Edit_AS.jpg?v=1613989955",
140      "variantIds": [],
141      "alt": ""
142    },
143    {
144      "id": "15928233459914",
145      "type": "Image",
146      "url": "https://cdn.shopifycdn.net/s/files/1/0156/6146/products/TRAININGCROPTEECOOLMINT.D1-Edit_AS.jpg?v=1613997217",
147      "variantIds": [],
148      "alt": ""
149    },
150    {
151      "id": "15928233492682",
152      "type": "Image",
153      "url": "https://cdn.shopifycdn.net/s/files/1/0156/6146/products/TRAININGCROPTEECOOLMINT.D2-Edit_AS.jpg?v=1613997217",
154      "variantIds": [],
155      "alt": ""
156    },
157    {
158      "id": "15928233525450",
159      "type": "Image",
160      "url": "https://cdn.shopifycdn.net/s/files/1/0156/6146/products/TRAININGCROPTEECOOLMINT.D3-Edit_AS.jpg?v=1613997217",
161      "variantIds": [],
162      "alt": ""
163    }
164  ],
165  "options": [
166    {
167      "type": "Size",
168      "values": [
169        {
170          "id": "Extra Small",
171          "name": "Extra Small"
172        },
173        {
174          "id": "Small",
175          "name": "Small"
176        },
177        {
178          "id": "Medium",
179          "name": "Medium"
180        },
181        {
182          "id": "Large",
183          "name": "Large"
184        },
185        {
186          "id": "Extra Large",
187          "name": "Extra Large"
188        },
189        {
190          "id": "Extra Extra Large",
191          "name": "Extra Extra Large"
192        }
193      ]
194    }
195  ]
196}