Morrisons Scraper

Overview

This actor scrapes product data from the Morrisons grocery website (groceries.morrisons.com). It can extract detailed product information from both category pages and individual product URLs.

Features

• Scrapes products from Morrisons category pages
• Extracts detailed information for individual products
• Controls the maximum number of products to scrape per category
• Handles pagination through scrolling
• Robust error handling and retry mechanisms

Input

The actor accepts the following input parameters:

{
  "inputURL": [
    {
      "url": "https://groceries.morrisons.com/categories/meat-poultry/7fd143ec-3236-4177-94d1-aff4913c9a2e",
      "requestsFromUrl": ""
    }
  ],
  "maxItem": 100
}

Input Parameters Explanation

• inputURL (required): An array of objects containing URLs to scrape.
  • url (optional): Direct URL to a Morrisons category or product page.
  • requestsFromUrl (optional): URL to a text file containing a list of Morrisons URLs (one per line).
• maxItem (optional): Maximum number of products to scrape per category. Default is 100.

Output

The actor outputs product data in the following format:

{
  "id": "112560029",
  "title": "Muller Corner Vanilla Yogurt with Chocolate Balls 124g",
  "price": {
    "current": 0.85,
    "original": 0.85,
    "currency": "GBP"
  },
  "brand": "Muller",
  "description": "Vanilla Yogurt with Chocolate Balls Corner with Vanilla Yogurt (70%) with Chocolate Flavoured Coated Cereal Crisp Pieces (30%)",
  "imageUrl": "https://groceries.morrisons.com/productImages/112/112560029_0_640x640.jpg",
  "nutritionalInfo": {...},
  "ingredients": "...",
  "allergens": [...],
  "storage": "...",
  "category": "Dairy & Eggs",
  "subcategory": "Yogurts"
  .........more
}

How it Works

1. URL Processing:

   • The actor parses the input URLs and categorizes them as either category pages or product pages.

2. Category Scraping:

   • For category URLs, the actor uses Puppeteer to scroll through the page, triggering product loads.
   • It intercepts API requests to /api/webproductpagews/v6/products to extract product data.
   • Products are collected until the maximum limit is reached or all products are scraped.

3. Individual Product Scraping:

   • For individual product URLs, the actor extracts the product ID and fetches detailed data using the product API.

4. Data Processing:

   • All collected data is pushed to the default dataset in your Apify account.

Memory Requirements

The actor may require additional memory when scraping large categories. Set CRAWLEE_MEMORY_MBYTES environment variable to increase the memory allocation:

CRAWLEE_MEMORY_MBYTES=8192

Limitations

• The actor respects website structure and may be affected by changes to Morrisons' website.
• Some categories may have thousands of products, so use the maxItem parameter to limit scraping.
• Rate limiting may occur if scraping too many products at once.