Shopee Api Scraper

What is Shopee API Scraper and how does it work?

Shopee API Scraper allows you to directly invoke the unofficial API of Shopee, the leading ecommerce marketplace in South East Asia and also present in Central and South America.

This unofficial API is used by Shopee for its frontend, that's why it provides all the data displayed on the website. In fact, it actually provides more information (e.g. sales and stock levels per product variant) and is more flexible (e.g. configurable page size).

The Shopee API Scraper takes care of all the "low-level" details such as setting the right HTTP headers, to allow you to easily and efficiently requests data.

Quick Start

Click on Try for free, copy the following JSON input and click Start:

1{
2  "requests": [
3    { "url": "https://shopee.sg/Pants-cat.11012963.11027757" },
4    { "url": "https://shopee.sg/search?keyword=tshirt" },
5    { "url": "https://shopee.sg/Japanese-Samurai-T-Shirt-New-Samurai-T-Shirt-Distro-T-Shirts-i.414243778.23600133176?sp_atk=43c5e72a-9613-4ee1-9cd7-e29ebcdbd4b9&xptdk=43c5e72a-9613-4ee1-9cd7-e29ebcdbd4b9" },
6    { "url": "https://shopee.sg/lorealparissg" },
7  ],
8  "proxy": {
9    "useApifyProxy": true,
10    "apifyProxyGroups": ["RESIDENTIAL"],
11    "apifyProxyCountry": "SG"
12  },
13}

This performs four distinct requests:

Request	URL
Search products in the category "Men's Wear > Pants"	https://shopee.sg/Pants-cat.11012963.11027757
Search for products with the keyword "tshirt"	https://shopee.sg/search?keyword=tshirt
Retrieve detailed product information	https://shopee.sg/Japanese-Samurai-T-Shirt-New-Samurai-T-Shirt-Distro-T-Shirts-i.414243778.23600133176?sp_atk=43c5e72a-9613-4ee1-9cd7-e29ebcdbd4b9&xptdk=43c5e72a-9613-4ee1-9cd7-e29ebcdbd4b9
Retrieve detailed information about a shop	https://shopee.sg/lorealparissg

After few seconds the Shopee API Scraper returns the following results:

• Product listing for the category "Men's Wear > Pants"
• Product listing for the "tshirt" keyword
• Detailed information for the product "Japanese Samurai T-Shirt | New Samurai T-Shirt | Distro T-Shirts"
• Detailed information for the shop "L'Oréal Paris Official Store"

While the example above demonstrates the ease of using page URLs directly from your browser, advanced users might find greater value in tapping into the Shopee unofficial API for enhanced control and access to extended features not available through the website interface.

For example, leveraging the API allows for:

• Retrieving up to 50 product ratings in a single request, bypassing the standard limit of 6 results per page.
• Accessing a broader range of up to 440 popular keyword suggestions, far exceeding the usual cap of 11.

For a comprehensive understanding of how to utilize the Shopee API Scraper effectively, including more examples and detailed explanations, please consult the How to Scrape Shopee? section below.

What data can you extract?

Shopee API Scraper seamlessly interacts with Shopee's unofficial API to extract a wide range of data, including:

• Product searches by keyword and category.
• Detailed product information (description, price, stock levels, sales data, and variants).
• Buyer ratings, complete with comments and media.
• Hierarchical category trees and detailed category information.
• Lists of official shops, filtered by category if needed.
• Searches for shops by keyword.
• Comprehensive shop details (description, opening date, product counts, and performance metrics).
• Shop products.
• Keyword suggestions and autocompletion hints.

Note: please visit the API reference section below for more details about each API.

Advanced users can execute other APIs as well, you can find them by using the DevTools "Network" tab. Please contact me if you encounter issues or need additional features.

Why scrape Shopee data?

Discover what you can achieve with the Shopee API Scraper through these example use cases:

• Market and Competitive Analysis: Track category performance and identify emerging trends by analyzing sales data, product reviews, and competitor strategies. This can reveal insights into market demands and areas for strategic advantage.
• Opportunity Discovery: Use aggregated product search results to uncover popular yet underserved keywords and niches in the market. This can highlight potential areas for new product development or market entry.
• Product Review Analysis: Employ AI, such as ChatGPT, to deeply analyze product reviews. Summarize buyer feedback to gain insights into consumer satisfaction, preferences, and areas for product improvement.
• Competitor Watch: Regularly scrape competitor data to monitor for critical changes like products going out of stock, pricing adjustments, or new product launches. This enables real-time alerts and swift response strategies.

How much will it cost you?

Here’s a breakdown to help you estimate your expenses:

• Execution Cost: This includes computing units (CPU and memory), data transfer, and storage operations. On Apify, running the Shopee API Scraper costs approximately $0.50 for every 1,000 requests.
• Residential Proxy Cost: Typically charged per gigabyte, the cost of crawling products with their details is around 0.07GB per 1,000 requests.

In total, scraping 1,000 requests on Apify with the Starter plan, which includes residential proxies, will be around $1.40 ($0.50 for execution and $0.90 for the proxy).

As you can see, residential proxy is the more expensive component, and the cost varies based on type and volume of requests. For alternative, potentially more cost-effective proxy providers, consider exploring resources like the Apify and The Web Scraping Club Discord servers. For personalized advice, feel free to start a discussion.

Tip for Cost Optimization: To minimize costs, try to execute multiple requests in a single execution. This approach helps to reduce resource wastage during the scraper's start-up phase.

For a comprehensive understanding of Apify’s pricing, refer to their usage documentation.

How to scrape Shopee?

Getting started

Get started with Shopee API Scraper by following these simple steps:

1. Start: Click on Try for free.
2. Prepare Requests: Formulate the API request URLs you wish to scrape. Refer to the API reference section for guidance. Example URLs include:
   • Search for popular products by keyword: https://shopee.sg/api/v4/search/search_items?keyword=tshirt
   • Retrieve top categories: https://shopee.sg/api/v4/pages/get_category_tree
   • List products by category: https://shopee.sg/api/v4/search/search_items?match_id=11027757
   • List all official shops: https://shopee.sg/api/v4/official_shop/get_shops_by_category
   • List products by shop: https://shopee.sg/api/v4/shop/rcmd_items?shop_id=178877065
   • Access product detail: https://shopee.sg/api/v4/pdp/get_pc?shop_id=414243778&item_id=23600133176
   • Access product ratings: https://shopee.sg/api/v2/item/get_ratings?shopid=414243778&itemid=23600133176
3. Set Up Proxy: Input the prepared URLs and configure the proxy to 'Residential' with your target country.
4. Default Parameters: Maintain other input parameters at their default settings for simplicity.
5. Execution: Click Start and wait for the results. The status will switch from Running to Succeeded upon completion.
6. Results: Access your data in the Storage tab, available in various formats like JSON, CSV, and more.

Example input:

1{
2  // Search products with the "tshirt" keyword
3  "requests": [
4    { "url": "https://shopee.sg/api/v4/search/search_items?keyword=tshirt" }
5  ],
6  
7  // Use a residential proxy matching the country of the API endpoint (e.g. "SG" for "shopee.sg", "MY" for "shopee.com.my", ...etc.)
8  "proxy": {
9    "useApifyProxy": true,
10    "apifyProxyGroups": [
11      "RESIDENTIAL"
12    ],
13    "apifyProxyCountry": "SG"
14  },
15}

Example JSON output from a scraping session:

1{
2  "url": "https://shopee.sg/api/v4/search/search_items?keyword=tshirt",
3  "enrichedUrl": "https://shopee.sg/api/v4/search/search_items?by=relevancy&extra_params=%7B%22global_search_session_id%22%3A%22gs-c44e90e2-6f01-4ed4-9fe3-fd38fe17f6e9%22%2C%22search_session_id%22%3A%22ss-59a709b9-72fe-4fdf-8a46-23edd0ff3255%22%7D&keyword=tshirt&limit=5&newest=0&order=desc&page_type=search&scenario=PAGE_GLOBAL_SEARCH&version=2&view_session_id=53f46a2f-80b1-45b4-afbb-17cca078a722",
4  "referrer": "https://shopee.sg/search?keyword=tshirt",
5  "knownApi": "PRODUCT_SEARCH",
6  "responseStatus": 200,
7  "responseBody": {
8    // ...
9    "total_count": 3000,
10    "items": [
11      {
12        "item_basic": {
13          "itemid": 22541793907,
14          "shopid": 391955424,
15          "name": "Casual T-shirts Short-sleeved Tshirt Korean Round Neck Youth T-shirt Plain Color Tshirt Fashion Popular T-shirt",
16          "image": "sg-11134201-7qvez-lgj50bviqvph8b",
17          "currency": "SGD",
18          "stock": 242,
19          "ctime": 1683532055,
20          "sold": 826,
21          "historical_sold": 3044,
22          "catid": 100011,
23          "price": 850000,
24          "price_before_discount": 1990000,
25          "discount": "57%",
26          "item_rating": {
27            "rating_star": 4.702664796633941,
28            "rating_count": [
29              713,
30              11,
31              9,
32              33,
33              75,
34              585
35            ],
36            "rcount_with_context": 254,
37            "rcount_with_image": 164
38          },
39          // ...
40        },
41        "itemid": 22541793907,
42        "shopid": 391955424,
43        // ...
44      },
45      // ...
46    ],
47    // ...
48  }
49}

This output includes fields like the original URL (url), the enriched URL sent to Shopee (enrichedUrl), and the HTTP response (responseStatus and responseBody), among others.

Tip: Enable 'Enrich URLs' and 'Generate referrers' for enhanced scraping. Check the input parameters for more details.

Input parameters

The Shopee API Scraper offers a range of input parameters, allowing you to tailor your scraping process according to your needs. These parameters can be broadly categorized into common parameters applicable to all APIs and specific parameters unique to certain APIs (detailed in the API reference section).

Key common parameters include:

1. API Requests: This is where you specify the list of API URLs you intend to scrape. Additionally, you have the option to include a Referer header with each request. If 'Generate Referrers' is enabled, the scraper automatically generates a referrer for each request. However, if you manually set a referrer, it will override the automatically generated one, even if 'Generate Referrers' is active. An example configuration is as follows:

   1{
   2  "requests": [
   3    { 
   4      "url": "https://shopee.sg/api/v4/shop/rcmd_items?shop_id=178877065",
   5      "userData": {
   6        "referrer": "https://shopee.sg/xiaomiofficialstore.sg"
   7      }
   8    }
   9  ]
   10}
   

2. Proxy Configuration: Use a residential proxy set to the target country for effective scraping.
3. Concurrency Settings (Optional): Define the minimum and maximum number of parallel requests. Align this with the number of CPU cores allocated. See Crawlee documentation for more.
4. Maximum Requests per Crawl (Optional): Set the maximum number of requests the crawler will execute. The process stops once this limit is reached.
5. Enrich URLs: Enabled by default, this option automatically adds missing parameters to URLs, such as result limits and session IDs.
6. Generate Referrers: Also enabled by default, this generates the referrer HTTP header for each API request to enhance authenticity.

For any queries or assistance, feel free to reach out.

Output fields

The data returned from the Shopee API Scraper includes several important fields. Below is a description of each:

• url: This is the original URL you submitted for scraping.

• enrichedUrl: Represents the actual URL sent to Shopee. If you enable Enrich URLs, this URL is automatically enhanced with default values. For more details, refer to the input parameters section.

• referrer: The Referer HTTP header sent to Shopee. It's generated automatically if Generate referrers is enabled. You have the option to set this manually as well (see input parameters for more).

• knownApi: Enumerated value corresponding to the requested API. This helps you identify which API endpoint was used:

  API	KnownApi
  /api/v4/search/search_items	PRODUCT_SEARCH
  /api/v4/pdp/get_pc	PRODUCT_DETAIL
  /api/v2/item/get_ratings	PRODUCT_RATINGS
  /api/v4/pages/get_category_tree	FE_CATEGORY_TREE
  /api/v4/search/get_fe_category_detail	FE_CATEGORY_DETAIL
  /api/v4/official_shop/get_shops_by_category	OFFICIAL_SHOPS_BY_CATEGORY
  /api/v4/search/search_user	SHOP_SEARCH
  /api/v4/shop/get_shop_base	SHOP_DETAIL
  /api/v4/shop/rcmd_items	SHOP_PRODUCTS
  /api/v4/search/search_suggestion	KEYWORD_SUGGESTIONS
  /api/v4/search/search_hint	KEYWORD_AUTOCOMPLETION
  -- Other API --	-- not set --

• responseStatus: Indicates the HTTP response status received from Shopee.

• responseBody: Contains the HTTP response body from Shopee in JSON format. The specific content varies depending on the requested API.

API reference

Product Search by Keyword or Category

• URL Path:

  • /api/v4/search/search_items

• URL Parameters:

  • keyword: Search keyword, e.g., "tshirt".
  • match_id: Category ID from /api/v4/pages/get_category_tree.

    Note: keyword and match_id are mutually exclusive.

  • by (default = "relevancy"): Sort by "relevancy", "ctime", "sales", or "price".
  • order (default = "desc"): Sort order, "asc" or "desc".
  • limit (default = set by scraper): Number of results per page.
  • newest (default = 0): Offset for results, typically a multiple of limit.
  • page_type, scenario, version: Additional parameters with default settings.
  • extra_params, view_session_id: Automatically generated, unless overridden.

• Scraper Input Parameters

  • productSearch_enrichUrlQuery_pageSize: Number of results per request (default: 60).
  • productSearch_crawlNextPages: Set to true to crawl subsequent pages.
  • productSearch_crawlNextPages_maxPages: Limit on number of pages to crawl (default: 50).
  • productSearch_crawlNextPages_maxResults: Maximum products to crawl (default: 3000).
  • productSearch_crawlProductDetails: Set to true to crawl details of each product.

• Example Input:

1{
2  // Search products with the "tshirt" keyword
3  "requests": [
4    { "url": "https://shopee.sg/api/v4/search/search_items?keyword=tshirt" }
5  ],
6  
7  // Use a residential proxy matching the country of the API endpoint (e.g. "SG" for "shopee.sg", "MY" for "shopee.com.my", ...etc.)
8  "proxy": {
9    "useApifyProxy": true,
10    "apifyProxyGroups": [
11      "RESIDENTIAL"
12    ],
13    "apifyProxyCountry": "SG"
14  },
15
16  // Set the "limit" URL parameter to 60
17  "productSearch_enrichUrlQuery_pageSize": 60,
18  // Automatically crawl the next pages
19  "productSearch_crawlNextPages": true,
20  // Crawl until the 3rd page
21  "productSearch_crawlNextPages_maxPages": 3,
22  // Alternatively, indicate the maximum number of products to crawl
23  "productSearch_crawlNextPages_maxResults": 180,
24  // Automatically crawl the detail of each listed product
25  "productSearch_crawlProductDetails": true
26}

• Example Response:

  • View Full Example Response

• Notable Response Fields:

  • items: Array of product summaries.
  • items/itemid and items/shopid: Essential for scraping product details and ratings.
  • items/item_basic/name: Product name.
  • items/item_basic/image: Image key. Convert to URL with https://down-${country}.img.susercontent.com/file/${imageKey}, for example the URL to download the image "sg-11134201-7qvez-lgj50bviqvph8b" is "https://down-sg.img.susercontent.com/file/sg-11134201-7qvez-lgj50bviqvph8b".
  • items/item_basic/ctime: Creation timestamp.
  • items/item_basic/sold: Monthly sales.
  • items/item_basic/historical_sold: Total sales.
  • items/item_basic/price: Divide by 100,000 to get actual price.

Product Details

• URL Path:

  • /api/v4/pdp/get_pc

• URL Parameters:

  • shop_id: The unique identifier of the shop.
  • item_id: The unique identifier of the product.

• Scraper Input Parameters

  • productDetail_crawlProductRatings: Define the types of product ratings to crawl. Options include "ALL", "ONE_STAR", "TWO_STARS" to "FIVE_STARS", "WITH_COMMENTS", "WITH_MEDIA", "LOCAL_REVIEWS". Combine multiple values for a broader range, e.g., ["ONE_STAR", "TWO_STARS"].

• Example Input:

1{
2  // Get the details of a product
3  "requests": [
4    { "url": "https://shopee.sg/api/v4/pdp/get_pc?shop_id=414243778&item_id=23600133176" }
5  ],
6  
7  // Use a residential proxy matching the country of the API endpoint (e.g. "SG" for "shopee.sg", "MY" for "shopee.com.my", ...etc.)
8  "proxy": {
9    "useApifyProxy": true,
10    "apifyProxyGroups": [
11      "RESIDENTIAL"
12    ],
13    "apifyProxyCountry": "SG"
14  },
15  
16  // Automatically crawl ratings of any stars
17  "productDetail_crawlProductRatings": ["ALL"],
18
19  // Set the rating "limit" URL parameter to 50
20  "productRatings_enrichUrlQuery_pageSize": 50,
21  // Automatically crawl the next rating pages
22  "productRatings_crawlNextPages": true,
23  // Crawl until the 3rd rating page
24  "productRatings_crawlNextPages_maxPages": 3,
25  // Alternatively, indicate the maximum number of ratings to crawl
26  "productRatings_crawlNextPages_maxResults": 150,
27}

• Example Response:

  • View Full Example Response

• Notable Response Fields:

  • data/item/item_id and data/item/shop_id: Essential identifiers for the product and shop.
  • data/item/title: Product name.
  • data/item/description: Product description.
  • data/item/image: Image key. Convert to URL using the provided format: https://down-${country}.img.susercontent.com/file/${imageKey}.
  • data/item/ctime: Product creation timestamp.
  • data/item/fe_categories: Category information including parent categories.
  • data/item/models: Detailed SKU information.
  • data/product_review: Aggregated rating statistics.
  • data/product_shipping: Shipping details.
  • data/shop_detailed: Comprehensive shop details including name, username (slug), and image.

Product Ratings

• URL Path:

  • /api/v2/item/get_ratings

• URL Parameters:

  • shopid & itemid: Unique identifiers for the shop and product.
  • limit (default = set by scraper): Number of results per page.
  • offset (default = 0): Offset for results, typically a multiple of limit.
  • type (default = 0, max = 5): Filter by number of stars (e.g. 2 = 2-stars ratings only). 0 = all ratings.
  • filter (default = 0): Filter by content: 1 = with comments, 3 = with media, 9 = local ratings, 0 = all ratings.
  • exclude_filter, filter_size, flag, fold_filter, relevant_reviews, request_source, tag_filter, variation_filters: Additional parameters with default settings.

• Scraper Input Parameters

  • productRatings_enrichUrlQuery_pageSize: Number of results per request (default: 6).
  • productRatings_crawlNextPages: Set to true to crawl subsequent pages.
  • productRatings_crawlNextPages_maxPages: Limit on number of pages to crawl (default: 500).
  • productRatings_crawlNextPages_maxResults: Maximum products to crawl (default: 3000).
  • productRatings_autofixError10002: Automatically corrects error 10002 by splitting requests (default: true).

• Example Input:

1{
2  // Get the ratings of a product
3  "requests": [
4    { "url": "https://shopee.sg/api/v2/item/get_ratings?shopid=414243778&itemid=23600133176" }
5  ],
6  
7  // Use a residential proxy matching the country of the API endpoint (e.g. "SG" for "shopee.sg", "MY" for "shopee.com.my", ...etc.)
8  "proxy": {
9    "useApifyProxy": true,
10    "apifyProxyGroups": [
11      "RESIDENTIAL"
12    ],
13    "apifyProxyCountry": "SG"
14  },
15
16  // Set the rating "limit" URL parameter to 50
17  "productRatings_enrichUrlQuery_pageSize": 50,
18  // Automatically crawl the next rating pages
19  "productRatings_crawlNextPages": true,
20  // Crawl until the 3rd rating page
21  "productRatings_crawlNextPages_maxPages": 3,
22  // Alternatively, indicate the maximum number of ratings to crawl
23  "productRatings_crawlNextPages_maxResults": 150,
24  // Automatically fix the error 10002
25  "productRatings_autofixError10002": true,
26}

• Example Response:

  • View Full Example Response

• Notable Response Fields:

  • data/ratings/cmtid: Unique identifier for each rating.
  • data/ratings/ctime: Timestamp of the rating.
  • data/ratings/rating_star: Star rating from 1 to 5.
  • data/ratings/comment: Buyer's comment.
  • data/ratings/author_username: Buyer's username (sometimes anonymized).
  • data/ratings/author_portrait: Buyer's profile image.
  • data/ratings/images & videos: Media associated with the rating.
  • data/ratings/product_items: SKU details of the rated product.

Data Privacy Note: Always be mindful of buyers' personal information and comply with local data protection laws. Avoid storing personal data unless necessary.

Shopee Category Tree

• URL Path:

  • /api/v4/pages/get_category_tree

• Example Input:

1{
2  // Get the category tree
3  "requests": [
4    { "url": "https://shopee.sg/api/v4/pages/get_category_tree" }
5  ],
6  
7  // Use a residential proxy matching the country of the API endpoint (e.g. "SG" for "shopee.sg", "MY" for "shopee.com.my", ...etc.)
8  "proxy": {
9    "useApifyProxy": true,
10    "apifyProxyGroups": [
11      "RESIDENTIAL"
12    ],
13    "apifyProxyCountry": "SG"
14  }
15}

• Example Response:

  • View Full Example Response

• Notable Response Fields:

  • data/category_list/catid: Unique identifier for each category.
  • data/category_list/name: Category name.
  • data/category_list/children: Sub-categories.

Note: this API only provides the two top category levels.

Category Details

• URL Path:

  • /api/v4/search/get_fe_category_detail

• URL Parameters:

  • catids: Unique identifiers for the category.

• Example Input:

1{
2  // Retrieve details for a specific category
3  "requests": [
4    { "url": "https://shopee.sg/api/v4/search/get_fe_category_detail?catids=11027710" }
5  ],
6  
7  // Use a residential proxy matching the country of the API endpoint (e.g. "SG" for "shopee.sg", "MY" for "shopee.com.my", ...etc.)
8  "proxy": {
9    "useApifyProxy": true,
10    "apifyProxyGroups": [
11      "RESIDENTIAL"
12    ],
13    "apifyProxyCountry": "SG"
14  }
15}

• Example Response:

  • View Full Example Response

• Notable Response Fields:

  • data/categories/catid: The unique identifier for the category.
  • data/categories/parent_cat_id: Identifier for the parent category, helping to understand category hierarchy.
  • data/categories/display_name: The category name as displayed in different languages across the platform.

Official Shops

• URL Path:

  • /api/v4/official_shop/get_shops_by_category

• URL Parameters:

  • category_id (default = -1): Specify a category ID to retrieve official shops within that category. Use -1 to list all official shops.
  • need_zhuyin: An additional parameter, typically set to default.

Note: Only shops associated with registered brands are considered "official" on Shopee.

• Example Input:

1{
2  // Retrieve a list of all official shops
3  "requests": [
4    { "url": "https://shopee.sg/api/v4/official_shop/get_shops_by_category" }
5  ],
6  
7  // Use a residential proxy matching the country of the API endpoint (e.g. "SG" for "shopee.sg", "MY" for "shopee.com.my", ...etc.)
8  "proxy": {
9    "useApifyProxy": true,
10    "apifyProxyGroups": [
11      "RESIDENTIAL"
12    ],
13    "apifyProxyCountry": "SG"
14  }
15}

• Example Response:

  • View Full Example Response

• Notable Response Fields:

  • data/brands/brand_ids/shopid: The unique identifier for each official shop.
  • data/brands/brand_ids/brand_name: The name of the brand associated with the shop.
  • data/brands/brand_ids/username: The shop's username or slug, useful for direct queries or referencing.

Shop Search By Keyword

• URL Path:

  • /api/v4/search/search_user

• URL Parameters:

  • keyword (minimum: 3 characters): Enter a search term, such as "adi", to find relevant shops.
  • limit (default = set by scraper): Number of results per page.
  • offset (default = 0): Offset for results, typically a multiple of limit.
  • page, with_search_cover: Additional parameters with default settings.

Note 1: This API covers all shops on Shopee, both official and unofficial.

Note 2: For a comprehensive shop list, consider searching with various trigrams.

• Scraper Input Parameters

  • shopSearch_enrichUrlQuery_pageSize: Number of results per request (default: 6).
  • shopSearch_crawlNextPages: Set to true to crawl subsequent pages.
  • shopSearch_crawlNextPages_maxPages: Limit on number of pages to crawl (default: 500).
  • shopSearch_crawlNextPages_maxResults: Maximum shops to crawl (default: 30000).
  • shopSearch_crawlShopDetails: Set to true to crawl details of each shop.

• Example Input:

1{
2  // Search shop with the "adi" keyword
3  "requests": [
4    { "url": "https://shopee.sg/api/v4/search/search_user?keyword=adi" }
5  ],
6  
7  // Use a residential proxy matching the country of the API endpoint (e.g. "SG" for "shopee.sg", "MY" for "shopee.com.my", ...etc.)
8  "proxy": {
9    "useApifyProxy": true,
10    "apifyProxyGroups": [
11      "RESIDENTIAL"
12    ],
13    "apifyProxyCountry": "SG"
14  },
15
16  // Set the "limit" URL parameter to 50
17  "shopSearch_enrichUrlQuery_pageSize": 50,
18  // Automatically crawl the next pages
19  "shopSearch_crawlNextPages": true,
20  // Crawl until the 3rd page
21  "shopSearch_crawlNextPages_maxPages": 3,
22  // Alternatively, indicate the maximum number of shops to crawl
23  "shopSearch_crawlNextPages_maxResults": 150,
24  // Automatically crawl the detail of each listed shop
25  "shopSearch_crawlShopDetails": true
26}

• Example Response:

  • View Full Example Response

• Notable Response Fields:

  • data/users/shopid: Unique identifier for the shop.
  • data/users/shopname: Name of the shop.
  • data/users/username: The shop's username or slug, useful for direct queries or referencing.
  • data/users/portrait: Shop image key. Convert to URL with https://down-${country}.img.susercontent.com/file/${imageKey}, for example the URL to download the image "6d4ff02a5978ee10a7b4527c25396b0b" is "https://down-sg.img.susercontent.com/file/6d4ff02a5978ee10a7b4527c25396b0b".

Shop Detail

• URL Path:

  • /api/v4/shop/get_shop_base

• URL Parameters:

  • username: Specify the shop's username (not the shop ID).
  • entry_point, need_cancel_rate, request_source, version: Additional parameters with default settings.

• Scraper Input Parameters

  • shopDetail_crawlShopProducts: Set to true to enable automatic crawling of the shop's products.

• Example Input:

1{
2  // Retrieve detailed information about a specific shop
3  "requests": [
4    { "url": "https://shopee.sg/api/v4/shop/get_shop_base?username=xiaomiofficialstore.sg" }
5  ],
6  
7  // Use a residential proxy matching the country of the API endpoint (e.g. "SG" for "shopee.sg", "MY" for "shopee.com.my", ...etc.)
8  "proxy": {
9    "useApifyProxy": true,
10    "apifyProxyGroups": [
11      "RESIDENTIAL"
12    ],
13    "apifyProxyCountry": "SG"
14  },
15  
16  // Automatically crawl shop products
17  "shopSearch_crawlShopDetails": true,
18
19  // Set the product "limit" URL parameter to 30
20  "shopProducts_enrichUrlQuery_pageSize": 30,
21  // Automatically crawl the next product pages
22  "shopProducts_crawlNextPages": true,
23  // Crawl until the 3rd product page
24  "shopProducts_crawlNextPages_maxPages": 3,
25  // Alternatively, indicate the maximum number of products to crawl
26  "shopProducts_crawlNextPages_maxResults": 90,
27}

• Example Response:

  • View Full Example Response

• Notable Response Fields:

  • data/shopid: The unique identifier for the shop.
  • data/name: Official name of the shop.
  • data/description: Brief description of the shop.
  • data/cover: Image key for the shop's profile picture. Convert to URL using the provided format: https://down-${country}.img.susercontent.com/file/${imageKey}.
  • data/account/username: The shop's unique username or slug.
  • data/ctime: Timestamp marking the shop's creation.
  • data/shop_rating: Aggregated rating details of the shop.
  • data/seller_metrics: Various metrics providing insights into the shop's performance.
  • data/item_count: Total number of products offered by the shop.

Shop Products

• URL Path:

  • api/v4/shop/rcmd_items

• URL Parameters:

  • shop_id: Unique identifier for the shop.
  • limit (default = set by scraper): Number of results per page.
  • offset (default = 0): Offset for results, typically a multiple of limit.
  • sort_type (default = 1): Sorting option for products. 1 = popular first, 2 = latest first, 13 = top sales first, 8 = price low to high, 4 = price high to low
  • cat_id (optional): Filter products by a category.
  • bundle, upstream: Additional parameters with default settings.

• Scraper Input Parameters

  • shopProducts_enrichUrlQuery_pageSize: Number of results per request (default: 30).
  • shopProducts_crawlNextPages: Set to true to crawl subsequent pages.
  • shopProducts_crawlNextPages_maxPages: Limit on number of pages to crawl (default: 100).
  • shopProducts_crawlNextPages_maxResults: Maximum products to crawl (default: 3000).
  • shopProducts_crawlNextPages_minSales: Minimum sales threshold for page crawling (optional). Crawling stops when a product with sales equal to or less than this value appears, assuming products are sorted by sales in descending order.
  • shopProducts_crawlProductDetails: Set to true to automatically crawl product details (default: false).

• Example Input:

1{
2  // Get the products of a shop
3  "requests": [
4    { "url": "https://shopee.sg/api/v4/shop/rcmd_items?shop_id=178877065" }
5  ],
6  
7  // Use a residential proxy matching the country of the API endpoint (e.g. "SG" for "shopee.sg", "MY" for "shopee.com.my", ...etc.)
8  "proxy": {
9    "useApifyProxy": true,
10    "apifyProxyGroups": [
11      "RESIDENTIAL"
12    ],
13    "apifyProxyCountry": "SG"
14  },
15
16  // Set the product "limit" URL parameter to 30
17  "shopProducts_enrichUrlQuery_pageSize": 30,
18  // Automatically crawl the next product pages
19  "shopProducts_crawlNextPages": true,
20  // Crawl until the 3rd product page
21  "shopProducts_crawlNextPages_maxPages": 3,
22  // Alternatively, indicate the maximum number of products to crawl
23  "shopProducts_crawlNextPages_maxResults": 150,
24  // Automatically crawl product details
25  "shopProducts_crawlProductDetails": true
26}

• Example Response:
  • View Full Example Response

Note: this API is scraped by using the POST method.

• Notable Response Fields:
  • data/items: Array of product summaries.
  • data/items/itemid and items/shopid: Essential for scraping product details and ratings.
  • data/items/name: Product name.
  • data/items/image: Image key. Convert to URL with https://down-${country}.img.susercontent.com/file/${imageKey}.
  • data/items/ctime: Creation timestamp.
  • data/items/stock: Stock level.
  • data/items/sold: Monthly sales.
  • data/items/historical_sold: Total sales.
  • data/items/price: Divide by 100,000 to get actual price.

Keyword Suggestion

• URL Path:
  • /api/v4/search/search_suggestion

Note: The keyword suggestions are normally displayed under the top search bar.

• URL Parameters:

  • limit (default = set by scraper): Number of results per page.
  • offset (default = 0): Offset for results, typically a multiple of limit.
  • bundle: An additional parameter, usually set to default.

• Scraper Input Parameters

  • keywordSuggestions_enrichUrlQuery_pageSize: Number of keyword suggestions per request (default: 8).
  • keywordSuggestions_crawlNextPages: Set to true to crawl subsequent pages.
  • keywordSuggestions_crawlNextPages_maxPages: Limit on number of pages to crawl (default: 55).
  • keywordSuggestions_crawlNextPages_maxResults: Maximum keyword suggestions to crawl (default: 440).

• Example Input:

1{
2  // Retrieve the 50 top suggested keywords
3  "requests": [
4    { "url": "https://shopee.sg/api/v4/search/search_suggestion?limit=50" }
5  ],
6  
7  // Use a residential proxy matching the country of the API endpoint (e.g. "SG" for "shopee.sg", "MY" for "shopee.com.my", ...etc.)
8  "proxy": {
9    "useApifyProxy": true,
10    "apifyProxyGroups": [
11      "RESIDENTIAL"
12    ],
13    "apifyProxyCountry": "SG"
14  },
15
16  // Set the "limit" URL parameter to 50
17  "keywordSuggestions_enrichUrlQuery_pageSize": 50,
18  // Automatically crawl the next pages
19  "keywordSuggestions_crawlNextPages": true,
20  // Crawl until the 3rd page
21  "keywordSuggestions_crawlNextPages_maxPages": 3,
22  // Alternatively, indicate the maximum number of keyword suggestions to crawl
23  "keywordSuggestions_crawlNextPages_maxResults": 150
24}

• Example Response:

  • View Full Example Response

• Notable Response Fields:

  • data/queries/text: Suggested keyword.

Keyword Autocompletion

• URL Path:

  • /api/v4/search/search_hint

• URL Parameters:

  • keyword: The initial characters of a potential search term.
  • extra_params, search_type, version: Additional parameters with default settings.

• Example Input:

1{
2  // Retrieve autocomplete suggestions for the keyword prefix "n"
3  "requests": [
4    { "url": "https://shopee.sg/api/v4/search/search_hint?keyword=n" }
5  ],
6  
7  // Use a residential proxy matching the country of the API endpoint (e.g. "SG" for "shopee.sg", "MY" for "shopee.com.my", ...etc.)
8  "proxy": {
9    "useApifyProxy": true,
10    "apifyProxyGroups": [
11      "RESIDENTIAL"
12    ],
13    "apifyProxyCountry": "SG"
14  }
15}

• Example Response:

  • View Full Example Response

• Notable Response Fields:

  • keywords/keyword: Suggested keyword based on the input characters.

Is it legal to scrape Shopee?

It is legal to scrape publicly available data such as product descriptions, prices, or ratings. Read Apify's blog post on the legality of web scraping to learn more.

Need to find product pairs between Shopee and another online shop?

Use the AI Product Matcher. This AI model allows you to compare items from different web stores, identifying exact matches and comparing real-time data obtained via web scraping. With the AI Product Matcher, you can use scraped product data to monitor product matches across the industry, implement dynamic pricing for your website, replace or complement manual mapping, and obtain realistic estimates against your competition for upcoming promo campaigns.

Most importantly, it is relatively easy to get started with (just follow this guide), and it can check thousands of pairs of products.

Integrations and Shopee API Scraper

Shopee API Scraper can be connected with almost any cloud service or web app thanks to integrations on the Apify platform. You can integrate with Make, Zapier, Slack, Airbyte, GitHub, Google Sheets, Google Drive, and more. Or you can use webhooks to carry out an action whenever an event occurs, e.g. get a notification whenever Shopee API Scraper successfully finishes a run.

Your feedback

Please don't hesitate to share your feedback (improvement ideas, bug, ...etc.). You can create an issue on the Actor’s issues tab in Apify Console.

In addition, feel free to chat with me on Discord (username "marcplouhinec").

Thanks

Shopee API Scraper is built with Crawlee, a great JavaScript framework to accelerate scraper development.

It also uses ungoogled-chromium via the Chrome DevTools Protocol. These powerful technologies have been instrumental in "opening" the website despite anti-bot protections.

This README file was inspired from other ecommerce scrapers such as the Amazon Product Scraper and Temu Scraper.

Finally, a lot of knowledge that enabled the development of Shopee API Scraper comes from The Web Scraping Club and its Discord Community.