Shopify Scraper

What does Shopify Scraper do?

Using this tool, you can automate monitoring prices on the most popular solution for building online stores and selling products online. Crawl arbitrary Shopify-powered online stores and extract a list of all products in a structured form, including product title, price, description, etc.

Need to find product pairs between Shopify and another online shop?

Use the AI Product Matcher. This AI model allows you to compare items from different web stores, identifying exact matches and comparing real-time data obtained via web scraping. With the AI Product Matcher, you can use scraped product data to monitor product matches across the industry, implement dynamic pricing for your website, replace or complement manual mapping, and obtain realistic estimates against your competition for upcoming promo campaigns.

Most importantly, it is relatively easy to get started with (just follow this guide) and it can match thousands of product pairs.

Extend Scraper and Output Function

Extend output function allows to filter the items that are output:

1async ({ item, customData }) => {
2    if (!item.title.includes('cuisine')) {
3        return null; // omit the output
4    }
5
6    delete item.additional; // remove data from output
7
8    item.requestId = customData.requestId; // add data from the outside
9
10    return item;
11}

Extend scraper function allows you to interact with scraper phases:

1async ({ label, url, filter, fns, filteredSitemapUrls, customData }) => {
2    switch (label) {
3        case 'FILTER_SITEMAP_URL': {
4            // product url, like .../products/cooking-for-dummies-2002-289854
5            filter(
6                url.includes('cooking') || url.includes(customData.filter)
7            );
8            break;
9        }
10        case 'SETUP': {
11            // filteredSitemapUrls is a `Set` instance and can be edited in-place
12            filteredSitemapUrls.add('https://example.com/secret-unlisted-sitemap.xml');
13            filteredSitemapUrls.forEach((sitemapURL) => {
14                if (!sitemapURL.includes('en-us')) {
15                    filteredSitemapUrls.delete(sitemapURL);
16                }
17            });
18            break;
19        }
20    }
21}

License

Apache 2.0