ProductReview.com.au Search Scraper

Pricing

Pay per usage

Try for free

Go to Apify Store

ProductReview.com.au Search Scraper

Try for free

Scrape product listings and reviews from ProductReview.com.au by providing a search URL, with automatic pagination and comprehensive product information extraction.

Pricing

Pay per usage

Rating

5.0

(1)

Developer

PowerAI

Maintained by Community

Actor stats

Bookmarked

Total users

Monthly active users

4 days ago

Last modified

.actor/Dockerfile

# Specify the base Docker image. You can read more about
# the available images at https://docs.apify.com/sdk/js/docs/guides/docker-images
# You can also use any other image from Docker Hub.
FROM apify/actor-node-puppeteer-chrome

# Check preinstalled packages
RUN npm ls apify

# Copy just package.json and package-lock.json
# to speed up the build using Docker layer cache.
COPY package*.json ./

# Install NPM packages, skip optional and development dependencies to
# keep the image small. Avoid logging too much and print the dependency
# tree for debugging
RUN npm --quiet set progress=false \
    && npm install --omit=dev --omit=optional \
    && echo "Installed NPM packages:" \
    && (npm list --omit=dev --all || true) \
    && echo "Node.js version:" \
    && node --version \
    && echo "NPM version:" \
    && npm --version \
    && rm -r ~/.npm

# Next, copy the remaining files and directories with the source code.
# Since we do this after NPM install, quick build will be really fast
# for most source file changes.
COPY . ./


# Run the image.
CMD npm start --silent

.actor/actor.json

{
    "actorSpecification": 1,
    "name": "productreview-search-scraper",
    "title": "Empty JavaScript project",
    "description": "Empty project in JavaScript.",
    "version": "0.0",
    "buildTag": "latest",
    "meta": {
        "templateId": "js-empty"
    },
    "input": "./input_schema.json",
    "dockerfile": "./Dockerfile"
}

.actor/input_schema.json

{
  "type": "object",
  "schemaVersion": 1,
  "title": "ProductReview Search Input",
  "properties": {
    "searchUrl": {
      "title": "Search URL",
      "type": "string",
      "description": "The complete ProductReview search URL to scrape",
      "minLength": 1,
      "editor": "textfield",
      "default": "https://www.productreview.com.au/c/air-fryers",
      "prefill": "https://www.productreview.com.au/c/air-fryers"
    },
    "maxItems": {
      "title": "Max Items",
      "type": "integer",
      "description": "Maximum number of products to fetch",
      "default": 50,
      "minimum": 1,
      "maximum": 10000,
      "editor": "number",
      "prefill": 50
    },
    "proxyConfiguration": {
      "title": "Proxy Configuration",
      "type": "object",
      "description": "Proxy settings for the actor",
      "default": {
        "useApifyProxy": false,
        "apifyProxyGroups": [
          "RESIDENTIAL"
        ]
      },
      "editor": "proxy"
    }
  },
  "required": [
    "searchUrl"
  ]
}

src/main.js

1import { Actor } from 'apify';
2import { connect as connectRealBrowser } from 'puppeteer-real-browser';
3import * as cheerio from 'cheerio';
4
5// ============================================================================
6// 配置区域 - 可根据不同网站修改
7// ============================================================================
8const SCRAPER_CONFIG = {
9    // 无新数据时的最大等待次数
10    maxNoNewDataCount: 3,
11    // 无头浏览器
12    headless: false,
13    // 
14    turnstile: true,
15    // 过滤资源
16    filterResources: false,
17    // 最大数量
18    maxItems: 50,
19    // 请求url
20    requestUrl: 'https://www.productreview.com.au/c/air-fryers?brands=ninja%2Cphilips-domestic-appliances%2Csunbeam%2Ctefal',
21    // TODO4: 用于去重的唯一标识符字段名
22    uniqueIdField: 'uniqueId',
23    // TODO3: 等待的选择器（列表元素选择器）
24    waitSelector: '[data-ctx-namespace][data-ctx-influx]',
25    // TODO2: 自动翻页函数
26    autoScrollPage: async (page) => {
27        /**
28         * 翻页的 DOM 结构：
29         * 见 翻页DOM.html 文件
30         */
31        await page.evaluate(() => {
32            const button = document.querySelector('button[data-track-ga-click="{·category·:·load_more_results·}"]');
33            if (button) {
34                button.click();
35            }
36        });
37    }
38};
39
40// ============================================================================
41// 核心 Scraper 类 - 模板部分（一般不需要修改）
42// ============================================================================
43class BaseScraper {
44    /**
45     * 解析页面数据 - 需要自定义
46     * @param {CheerioAPI} $ - Cheerio 实例
47     * @returns {Array} - 解析后的数据数组
48     * 
49     * ⚠️ 自动生成的解析代码应该替换此方法的实现
50     */
51    parsePageItems($) {
52        // ==================== 开始：页面解析代码（可替换区域） ====================
53
54        const items = $(SCRAPER_CONFIG.waitSelector).map((i, el) => {
55            /**
56             * 数据列表的 DOM 结构：
57             * 见 数据列表DOM.html 文件
58             */
59            const $item = $(el);
60
61            // 提取uniqueId（从data-ctx-influx属性）
62            const influxData = $item.attr('data-ctx-influx') || '';
63            let uniqueId = '';
64            try {
65                const parsed = JSON.parse(influxData.replace(/·/g, '"'));
66                uniqueId = parsed.targetListingId || '';
67            } catch (e) {}
68
69            // 提取品牌（从data-ctx-ga属性）
70            const gaData = $item.attr('data-ctx-ga') || '';
71            let brand = '';
72            try {
73                const parsed = JSON.parse(gaData.replace(/·/g, '"'));
74                brand = parsed.dimensions?.targetBrand || '';
75            } catch (e) {}
76
77            // 提取产品标题和链接
78            const $titleLink = $item.find('a.dxgoRW').first();
79            const title = $titleLink.text().trim() || '';
80            const listingUrl = $titleLink.attr('href') || '';
81            const productUrl = listingUrl ? `https://www.productreview.com.au${listingUrl}` : '';
82
83            // 提取产品图片
84            const $image = $item.find('img#listing-header-main-picture').first();
85            const imageUrl = $image.attr('src') || '';
86            const imageAlt = $image.attr('alt') || '';
87
88            // 提取评分
89            const $rating = $item.find('span.Ydzmmx').first();
90            const ratingText = $rating.clone().children().remove().end().text().trim();
91            const ratingCount = $rating.attr('data-count') || '0';
92            const ratingValue = parseFloat(ratingText) || 0;
93
94            // 提取价格
95            const $priceItem = $item.find('span.MvDLkz:contains("Price")').parent();
96            const priceText = $priceItem.text().replace(/Price.*?:?/g, '').trim() || '';
97
98            // 提取容量
99            const $capacityItem = $item.find('span.MvDLkz:contains("Capacity")').parent();
100            const capacityText = $capacityItem.text().replace(/Capacity.*?:?/g, '').trim() || '';
101
102            if (uniqueId === '') {
103                return;
104            }
105            return {
106                uniqueId: uniqueId,
107                productTitle: title,
108                productUrl: productUrl,
109                imageUrl: imageUrl,
110                imageAlt: imageAlt,
111                brand: brand,
112                ratingValue: ratingValue,
113                ratingCount: parseInt(ratingCount) || 0,
114                priceText: priceText,
115                capacityText: capacityText,
116            };
117        }).get();
118
119        // ==================== 结束：页面解析代码（可替换区域） ====================
120
121        return items;
122    }
123
124    async run(input) {
125        const { searchUrl, maxItems = Infinity, proxyConfiguration } = input;
126        this.maxItems = maxItems;
127        this.searchUrl = searchUrl;
128        const proxyConfig = await Actor.createProxyConfiguration(proxyConfiguration);
129        const proxyUrl = proxyConfig ? await proxyConfig.newUrl() : undefined;
130        const proxyOptions = this.buildProxyOptions(proxyUrl);
131
132        const realBrowserOption = {
133            args: ["--start-maximized"],
134            turnstile: SCRAPER_CONFIG.turnstile,
135            headless: SCRAPER_CONFIG.headless,
136            // disableXvfb: true,
137            // ignoreAllFlags:true,
138            customConfig: {},
139            connectOption: {
140                defaultViewport: null
141            },
142            ...(proxyOptions ? { proxy: proxyOptions } : {}),
143            plugins: []
144        }
145
146        const { page, browser } = await connectRealBrowser(realBrowserOption);
147
148        try {
149            page.setDefaultNavigationTimeout(300 * 1000);
150            page.setDefaultTimeout(3600 * 1000);
151
152            // if (userAgent) {
153            //     await page.setUserAgent(userAgent);
154            // }
155
156            await this.handlePage(page, this.searchUrl);
157        } finally {
158            await browser?.close();
159        }
160    }
161
162    async handlePage(page, url) {
163        console.log(`Processing ${url}...`);
164        let count = 0;
165        let endOfResults = false;
166        let interceptedData = [];
167        let lastDataLength = 0;
168        let noNewDataCount = 0;
169
170        // await this.setupInterceptors(page, interceptedData);
171        await page.goto(url);
172
173        // 等待新内容加载
174        if (SCRAPER_CONFIG.waitSelector) {
175            await page.waitForSelector(SCRAPER_CONFIG.waitSelector, { timeout: 60000 }).catch(() => {
176                console.log(`data not found...`);
177            });
178        }
179
180        // 处理首页数据
181        const firstPageHtml = await page.content();
182        await this.processResponseData(firstPageHtml, interceptedData);
183
184        // 循环翻页获取数据
185        while (!endOfResults && interceptedData.length < this.maxItems) {
186            count++;
187            console.log(`data fetching...${count}`);
188            await this.randomDelay();
189
190            // 执行翻页操作
191            await SCRAPER_CONFIG.autoScrollPage(page);
192            await this.randomDelay(3000, 5000);
193
194            // 等待新内容加载
195            if (SCRAPER_CONFIG.waitSelector) {
196                await page.waitForSelector(SCRAPER_CONFIG.waitSelector, { timeout: 10000 }).catch(() => {
197                    console.log(`data not found...`);
198                });
199            }
200
201            // 处理新页面数据
202            const afterScrollHtml = await page.content();
203            await this.processResponseData(afterScrollHtml, interceptedData);
204
205            if (interceptedData.length === lastDataLength) {
206                noNewDataCount++;
207                if (noNewDataCount >= SCRAPER_CONFIG.maxNoNewDataCount) {
208                    endOfResults = true;
209                }
210            } else {
211                noNewDataCount = 0;
212                lastDataLength = interceptedData.length;
213            }
214        }
215
216        console.log(`Scraping completed. Total items: ${interceptedData.length}`);
217    }
218
219    // 设置请求拦截器（资源过滤）
220    async setupInterceptors(page, interceptedData) {
221        await page.setRequestInterception(true);
222        page.on('request', (request) => {
223            const resourceType = request.resourceType();
224            const blockedResourceTypes = [
225                'image',
226                'media',
227                'font',
228                'texttrack',
229                'stylesheet',
230                'ping'
231            ];
232
233            if (SCRAPER_CONFIG.filterResources && blockedResourceTypes.includes(resourceType)) {
234                request.abort();
235            } else {
236                request.continue();
237            }
238        });
239        page.on('response', async (response) => {
240            // 可选：拦截特定的 API 响应
241        });
242    }
243
244    // 处理响应数据（模板方法）
245    async processResponseData(htmlContent, interceptedData) {
246        let currentData = [];
247
248        if (!htmlContent) {
249            return;
250        }
251
252        const $ = cheerio.load(htmlContent);
253
254        // 调用自定义的解析方法
255        const items = this.parsePageItems($);
256
257        if (!items || items.length === 0) {
258            return;
259        }
260
261        // 去重处理
262        const uniqueField = SCRAPER_CONFIG.uniqueIdField;
263        const existingIds = new Set(interceptedData.map(item => item[uniqueField]));
264
265        let skippedCount = 0;
266        for (const listing of items) {
267            // 跳过已存在的项目
268            if (existingIds.has(listing[uniqueField])) {
269                skippedCount++;
270                continue;
271            }
272
273            const data = {
274                searchUrl: this.searchUrl,
275                ...listing,
276                scrapedAt: new Date().toISOString()
277            };
278
279            interceptedData.push(data);
280            currentData.push(data);
281            existingIds.add(listing[uniqueField]);
282
283            // 如果达到最大数量，停止处理
284            if (interceptedData.length >= this.maxItems) {
285                break;
286            }
287        }
288
289
290        // 保存数据
291        if (currentData.length > 0) {
292            console.log(`Saved ${currentData.length} items, total: ${interceptedData.length}`);
293            await Actor.pushData(currentData);
294        }
295    }
296
297    // 随机延迟
298    async randomDelay(min = 1000, max = 3000) {
299        const delay = Math.floor(Math.random() * (max - min + 1) + min);
300        await new Promise(resolve => setTimeout(resolve, delay));
301    }
302
303    // 可选：设置 cookies
304    async setCookies(page) {
305        // const cookies = [
306        //     {
307        //         name: 'cookie_name',
308        //         value: 'cookie_value',
309        //         domain: '.example.com',
310        //         path: '/',
311        //         expires: new Date('2025-12-31').getTime() / 1000,
312        //         secure: false
313        //     }
314        // ];
315        // await Promise.all(cookies.map(cookie => page.setCookie(cookie)));
316    }
317
318
319    buildProxyOptions(proxyUrl) {
320        if (!proxyUrl) {
321            return undefined;
322        }
323
324        try {
325            const parsed = new URL(proxyUrl);
326            if (!parsed.hostname || !parsed.port) {
327                return undefined;
328            }
329
330            return {
331                host: parsed.hostname,
332                port: Number(parsed.port),
333                username: parsed.username || undefined,
334                password: parsed.password || undefined,
335            };
336        } catch (error) {
337            console.warn(`Invalid proxy URL detected: ${proxyUrl}`, error);
338            return undefined;
339        }
340    }
341
342}
343
344
345// ============================================================================
346// Actor 入口
347// ============================================================================
348await Actor.init();
349
350Actor.main(async () => {
351    let input = await Actor.getInput();
352    if (!input) {
353        input = {
354            "searchUrl": SCRAPER_CONFIG.requestUrl,
355            "maxItems": SCRAPER_CONFIG.maxItems
356        };
357    }
358    const crawler = new BaseScraper();
359    await crawler.run(input);
360});

.dockerignore

# configurations
.idea

# crawlee and apify storage folders
apify_storage
crawlee_storage
storage

# installed files
node_modules

# git folder
.git

.editorconfig

root = true

[*]
indent_style = space
indent_size = 4
charset = utf-8
trim_trailing_whitespace = true
insert_final_newline = true
end_of_line = lf

.eslintrc

{
    "extends": "@apify",
    "root": true
}

.gitignore

# This file tells Git which files shouldn't be added to source control

.DS_Store
.idea
.zed
node_modules
storage

package.json

{
    "name": "js-scrape-single-page",
    "version": "0.0.1",
    "type": "module",
    "description": "This is an example of an Apify actor.",
    "engines": {
        "node": ">=18.0.0"
    },
    "dependencies": {
		"apify": "^3.4.4",
		"crawlee": "^3.11.5",
		"puppeteer": "^23.5.0",
		"puppeteer-core": "^23.5.0",
		"puppeteer-extra": "^3.3.6",
		"puppeteer-extra-plugin-stealth": "^2.11.2",
		"random-useragent": "^0.5.0",
        "rebrowser-puppeteer": "^23.10.3",
		"rebrowser-puppeteer-core": "^23.10.3",
        "puppeteer-real-browser": "^1.4.4",
        "ioredis": "^5.4.1",
        "marked": "^15.0.7",
		"node-fetch": "3",
        "uuid": "^11.1.0"
    },
    "scripts": {
        "start": "node ./src/main.js",
        "test": "echo \"Error: oops, the actor has no tests yet, sad!\" && exit 1"
    },
    "author": "It's not you it's me",
    "license": "ISC"
}

Product Reviews AU Scraper

abotapi/product-reviews-australia-scraper

Scrape and extract detailed product and business reviews from ProductReview.com.au with advanced filters, business info, and review insights. It supports location/category search, custom queries, proxy rotation, and flexible dataset outputs—ideal for research, analytics, and business intelligence.

AbotAPI

G2 Products Search Scraper

powerai/g2-products-search-scraper

Scrape product listings from G2.com by providing a search URL, with automatic pagination and comprehensive product information extraction.

PowerAI

5.0

Productreviews Reviews Spider

getdataforme/productreviews-reviews-spider

Extract comprehensive product reviews from ProductReview.com.au, including ratings, titles, and full bodies, to uncover consumer insights for market analysis and competitive intelligence. Scalable, error-resilient, and outputs structured JSON for seamless integration.

GetDataForMe

Action.com Search Scraper

powerai/action-com-search-scraper

Scrape products from Action.com by providing a search URL, with automatic pagination and comprehensive product information extraction.

PowerAI

5.0

Seek Job Listing Scraper

powerbox/seek-listing-scraper

Scrape job listings from Seek.com.au by providing a search URL, with automatic pagination and comprehensive job data extraction.

PowerBox

5.0

Tesco Products Scraper

powerai/tesco-products-scraper

Scrape product search results from Tesco.com by providing a search URL, with automatic pagination and comprehensive product information extraction.

PowerAI

5.0

Aldi Products Search Scraper

powerai/aldi-products-search-scraper

Scrape product listings from Aldi search results by providing a search URL, with automatic pagination and comprehensive product information extraction.

PowerAI

5.0

ASDA Products Scraper

powerai/asda-products-scraper

Scrape product listings from ASDA.com by providing a search URL, with automatic pagination and comprehensive product information extraction.

PowerAI

5.0

Argos Products Search Scraper

powerai/argos-products-search-scraper

Scrape product listings from Argos.co.uk by providing a search URL, with automatic pagination and comprehensive product information extraction.

PowerAI

5.0

Domain.com.au Scraper

scrapemind/Domaincomau-Scraper

Unlock Comprehensive Real Estate Data from Domain.com.au Discover a wealth of information from Domain.com.au's vast property listings. Effortlessly extract detailed descriptions, high-quality images, unique features, and essential property details.