Actor - Total Wine & More Scraper

Total Wine & More scraper

Since Total Wine & More doesn't provide a good and free API, this actor should help you to retrieve data from it.

The Total Wine & More data scraper supports the following features:

• Search any keyword - You can search any keyword you would like to have and get the results

• Scrape lists - Scrape any list that you'd like to get from Total Wine & More

Bugs, fixes, updates and changelog

This scraper is under active development. If you have any feature requests you can create an issue from here.

Setup & Usage

User has to enter a search keyword or a start URL to use the scraper. Currently, actor only support product lists URLs as the startURL The scraper needs to be used with an US-based proxy. Using the input field called shippingInfo, you can set the store/location to get the exact product listing available.

Using Search

• User can enter a single search term and set maxItems/endPage if wanted
• Actor iterates through all pages of the search result until all conditions are satisfied

You can check the output of this example here.

Input

1{
2    "maxItems": 25,
3    "endPage": 3,
4    "proxy": {
5        "useApifyProxy": true,
6        "apifyProxyCountry": "US"
7    },
8    "search": "sparkling",
9    "shippingInfo": "ispStore~204:ifcStore~null@ifcStoreState~US-VA@method~INSTORE_PICKUP"
10}

Using Start URLs

• User can enter any number of startURLs maxItems/endPage if wanted
• For every URL, actor parses the products listed and also iterates through succeding pages per startURL given

URL Query Parameters

• The actor keeps the aty query parameter of the given URL for the following pages if it's provided. If it's not provided, actor uses aty=1,1,0,1 parameter to fetch the all available products. (example url: https://www.totalwine.com/wine/c/c0020?aty=1,1,0,1)
• The actor keeps the pageSize query parameter of the given URL for the following pages if it's provided. If it's not provided, actor uses pageSize=24 parameter to fetch 24 items in every page. (example url: https://www.totalwine.com/wine/red-wine/c/000009?pageSize=24)

You can check the output of this example here.

Input

1{
2    "maxItems": 48,
3    "endPage": 5,
4    "proxy": {
5        "useApifyProxy": true,
6        "apifyProxyCountry": "US"
7    },
8    "startUrls": ["https://www.totalwine.com/wine/red-wine/c/000009"],
9    "shippingInfo": "ispStore~701:ifcStore~null@ifcStoreState~US-SC@method~INSTORE_PICKUP"
10}

Input Parameters

The input of this scraper should be JSON containing the list of pages on Total Wine & More that should be visited. Required fields are:

Field	Type	Description
search	String	(optional) Keyword that you want to search on Total Wine & More.
startUrls	Array	(optional) List of Total Wine & More URLs. You should only provide product list URLs
endPage	Integer	(optional) Final number of page that you want to scrape. Default is Infinite. This is applies to all search request and startUrls individually.
shippingInfo	String	(optional) This will set the store/location preference for the totalwine.com website. Website uses this preference to show available products for the given store/location. If not provided, the actor uses the location of the first proxy address to fetch the products.
maxItems	Integer	(optional) You can limit scraped products. This should be useful when you search through the big subcategories.
proxy	Object	Proxy configuration
extendOutputFunction	String	(optional) Function that takes a JQuery handle ($) as argument and returns object with data
customMapFunction	String	(optional) Function that takes each objects handle as argument and returns object with executing the function

This solution requires the use of Proxy servers, either your own proxy servers or you can use Apify Proxy.

Tip

When you want to have a scrape over a specific listing URL, just copy and paste the link as one of the startUrl.

If you would like to scrape only the first page of a list then put the link for the page and have the endPage as 1.

With the last approach that explained above you can also fetch any interval of pages. If you provide the 5th page of a list and define the endPage parameter as 6 then you'll have the 5th and 6th pages only.

If you would like to scrape the entire products of the given section (red wine, white wine, spirits, etc.), you should use a pageSize query parameter to increase the number of products you would get per request. Any pageSize value bigger than 200 might lead the requests to fail more often.

You can use the Chrome Extension called EditThisCookie to extract the cookie value for the store/location selection. Details for the extension

• Go to any initial listing/search page you want to scrape.
• Edit the Pickup At section next to the Search bar
• After picking the store/location, use the EditThisCookie extension to lookup twm-userStoreInformation cookie.
• Copy the value for the given key and put it to the Shipping Info field

Total Wine & More Scraper input example

1{
2    "startUrls": [
3        "https://www.totalwine.com/wine/red-wine/c/000009?page=3&viewAll=true&pageSize=100&aty=1,0,0,1"
4    ],
5    "proxy": {
6        "useApifyProxy": true,
7        "apifyProxyCountry": "US"
8    },
9    "endPage": 5,
10    "maxItems": 100,
11    "shippingInfo": "ispStore~701:ifcStore~null@ifcStoreState~US-SC@method~INSTORE_PICKUP"
12}

During the Run

During the run, the actor will output messages letting you know what is going on. Each message always contains a short label specifying which page from the provided list is currently specified. When items are loaded from the page, you should see a message about this event with a loaded item count and total item count for each page.

If you provide incorrect input to the actor, it will immediately stop with failure state and output an explanation of what is wrong.

Total Wine & More Export

During the run, the actor stores results into a dataset. Each item is a separate item in the dataset.

You can manage the results in any languague (Python, PHP, Node JS/NPM). See our API reference to learn more about getting results from this Total Wine & More actor.

Scraped Total Wine & More Properties

The structure of each item in Total Wine & More looks like this:

Item Detail

1{
2    "bay": "",
3    "brand": {
4        "id": "Mascota Vineyards",
5        "name": "Mascota Vineyards"
6    },
7    "categories": [
8        {
9            "id": "000284",
10            "name": "Mendoza",
11            "type": "REGION",
12            "url": "https://www.totalwine.com/wine/argentina/mendoza/c/000284",
13            "storefrontUrl": "/wine/argentina/mendoza/c/000284"
14        },
15        {
16            "id": "000009",
17            "name": "Red Wine",
18            "type": "PRODUCT_TYPE",
19            "url": "https://www.totalwine.com/wine/red-wine/c/000009",
20            "storefrontUrl": "/wine/red-wine/c/000009"
21        },
22        {
23            "id": "000011",
24            "name": "Cabernet Sauvignon",
25            "type": "VARIETAL_TYPE",
26            "url": "https://www.totalwine.com/wine/red-wine/cabernet-sauvignon/c/000011",
27            "storefrontUrl": "/wine/red-wine/cabernet-sauvignon/c/000011"
28        }
29    ],
30    "containerType": "Bottle",
31    "customerAverageRating": 4.4,
32    "customerReviewsCount": 998,
33    "department": "c0020",
34    "directType": "Winery Direct",
35    "id": "130007750",
36    "images": [
37        {
38            "imageType": "DEFAULT",
39            "mobileOptimizedUrl": "https://www.totalwine.com/media/sys_master/twmmedia/h72/h2e/15949509165086.png",
40            "thumbnailUrl": "https://www.totalwine.com/media/sys_master/twmmedia/h72/h2e/15949509165086.png",
41            "url": "https://www.totalwine.com/media/sys_master/twmmedia/h72/h2e/15949509165086.png",
42            "zoomImageUrl": "https://www.totalwine.com/media/sys_master/twmmedia/h72/h2e/15949509165086.png",
43            "altText": "Mascota Vineyards Unanime, 2017"
44        }
45    ],
46    "location": "Aisle 02, Left",
47    "name": "Mascota Vineyards Unanime, 2017",
48    "packageDescription": "750ml",
49    "packageValue": "Single",
50    "price": [
51        {
52            "price": 25.99,
53            "type": "EDLP"
54        },
55        {
56            "price": 19.99,
57            "type": "LTSP"
58        }
59    ],
60    "productUrl": "/wine/gift-center/deals/red-wine/cabernet-sauvignon/mascota-vineyards-unanime/p/130007750",
61    "review": "James Suckling-Mendoza, Argentina - \"This is a rich, dense red with blackberry and blueberry aromas and flavors. Extremely well-crafted tannins. Hints of vanilla to the ripe fruit at the end. Incredible value...already beautiful to taste.\"",
62    "rating": 95,
63    "ratingSource": "James Suckling",
64    "salesStrategy": {
65        "name": "Winery Direct"
66    },
67    "shoppingOptions": [
68        {
69            "eligible": false,
70            "location": "US-TX",
71            "message": ["Unavailable"],
72            "type": "SHIPPING",
73            "selected": false
74        },
75        {
76            "eligible": true,
77            "location": "Oak Lawn, TX",
78            "type": "INSTORE_PICKUP",
79            "selected": true
80        },
81        {
82            "eligible": true,
83            "location": "Oak Lawn, TX",
84            "type": "DELIVERY",
85            "selected": false
86        }
87    ],
88    "skuId": "130007750-1",
89    "stockLevel": [
90        {
91            "purchaseLimit": 6,
92            "stock": 211
93        }
94    ],
95    "storeDistance": 0.0001798902,
96    "storeId": "531",
97    "storeName": "Oak Lawn",
98    "itemTasteProfile": "Blackberry, Chocolate",
99    "itemStyle": "Elegant",
100    "itemBody": "Full-bodied",
101    "transactional": true,
102    "type": "PRODUCT",
103    "volume": "",
104    "stockMessages": {
105        "messages": [
106            {
107                "shoppingMethod": "INSTORE_PICKUP",
108                "stockMessage": "In stock",
109                "addToCartMessage": "Add to cart",
110                "addToCartStatus": true
111            },
112            {
113                "shoppingMethod": "DELIVERY",
114                "stockMessage": "Available",
115                "addToCartMessage": "Add to cart",
116                "addToCartStatus": true
117            },
118            {
119                "shoppingMethod": "SHIPPING",
120                "stockMessage": "Unavailable",
121                "addToCartMessage": "View Product",
122                "addToCartStatus": false
123            }
124        ],
125        "digitalTransactional": true,
126        "digitalInventoryQuantity": 211,
127        "digitalSpecialOrder": false,
128        "digitalLongTermOOS": false,
129        "digitalLimitedStock": false,
130        "digitalInStock": true,
131        "digitalStoreQuantity": 213,
132        "shippingTransactional": false,
133        "shippingInventoryQuantity": 211,
134        "shippingSpecialOrder": false,
135        "shippingLongTermOOS": false,
136        "shippingLimitedStock": false,
137        "shippingInStock": true,
138        "shippingStoreQuantity": 213,
139        "digitalDeliveryEligible": true
140    }
141}