John Lewis Products Search Scraper

Pricing

$19.99/month + usage

Try for free

Go to Apify Store

John Lewis Products Search Scraper

Try for free

Scrape product listings from John Lewis website by providing a search URL, with automatic pagination and comprehensive product information extraction.

Pricing

$19.99/month + usage

Rating

5.0

(1)

Developer

PowerAI

Maintained by Community

Actor stats

Bookmarked

Total users

Monthly active users

3 days ago

Last modified

.actor/Dockerfile

# Specify the base Docker image. You can read more about
# the available images at https://docs.apify.com/sdk/js/docs/guides/docker-images
# You can also use any other image from Docker Hub.
FROM apify/actor-node-puppeteer-chrome

# Check preinstalled packages
RUN npm ls apify

# Copy just package.json and package-lock.json
# to speed up the build using Docker layer cache.
COPY package*.json ./

# Install NPM packages, skip optional and development dependencies to
# keep the image small. Avoid logging too much and print the dependency
# tree for debugging
RUN npm --quiet set progress=false \
    && npm install --omit=dev --omit=optional \
    && echo "Installed NPM packages:" \
    && (npm list --omit=dev --all || true) \
    && echo "Node.js version:" \
    && node --version \
    && echo "NPM version:" \
    && npm --version \
    && rm -r ~/.npm

# Next, copy the remaining files and directories with the source code.
# Since we do this after NPM install, quick build will be really fast
# for most source file changes.
COPY . ./


# Run the image.
CMD npm start --silent

.actor/actor.json

{
    "actorSpecification": 1,
    "name": "johnlewis-products-search-scraper",
    "title": "Empty JavaScript project",
    "description": "Empty project in JavaScript.",
    "version": "0.0",
    "buildTag": "latest",
    "meta": {
        "templateId": "js-empty"
    },
    "input": "./input_schema.json",
    "dockerfile": "./Dockerfile"
}

.actor/input_schema.json

{
  "type": "object",
  "schemaVersion": 1,
  "title": "John Lewis Products Search Input",
  "properties": {
    "searchUrl": {
      "title": "Search URL",
      "type": "string",
      "description": "The complete John Lewis search URL to scrape",
      "minLength": 1,
      "editor": "textfield",
      "default": "https://www.johnlewis.com/search?search-term=coat&page=1",
      "prefill": "https://www.johnlewis.com/search?search-term=coat&page=1"
    },
    "maxItems": {
      "title": "Max Items",
      "type": "integer",
      "description": "Maximum number of products to fetch",
      "minimum": 1,
      "maximum": 10000,
      "editor": "number",
      "default": 100,
      "prefill": 100
    },
    "proxyConfiguration": {
      "title": "Proxy Configuration",
      "type": "object",
      "description": "Proxy settings for the actor",
      "default": {
        "useApifyProxy": false,
        "apifyProxyGroups": [
          "RESIDENTIAL"
        ]
      },
      "editor": "proxy"
    }
  },
  "required": [
    "searchUrl"
  ]
}

src/main.js

1import { Actor } from 'apify';
2import { connect as connectRealBrowser } from 'puppeteer-real-browser';
3import * as cheerio from 'cheerio';
4
5// ============================================================================
6// 配置区域 - 可根据不同网站修改
7// ============================================================================
8const SCRAPER_CONFIG = {
9    // 无新数据时的最大等待次数
10    maxNoNewDataCount: 3,
11    // 无头浏览器
12    headless: false,
13    // 
14    turnstile: true,
15    // 过滤资源
16    filterResources: false,
17    // 最大数量
18    maxItems: 50,
19    // 请求url
20    requestUrl: 'https://www.johnlewis.com/search?page=1&search-term=coat',
21    // 用于去重的唯一标识符字段名
22    uniqueIdField: 'uniqueId',
23    // 等待的选择器（列表元素选择器）
24    waitSelector: 'article[data-testid="product-card-layout"]',
25    // 自动翻页函数
26    autoScrollPage: async (page) => {
27        /**
28         * 翻页的 DOM 结构：
29         * 见 翻页DOM.html 文件
30         * 用原生js实现，放到await page.evaluate()中，举例如下：
31         * await page.evaluate(async () => {
32         *  const nextButton = document.querySelector('a[title="next page"]');
33            if (nextButton) {
34                await nextButton.click();
35            }
36           });
37         */
38        await page.evaluate(async () => {
39            const nextButton = document.querySelector('a[data-testid="next-btn"]');
40            if (nextButton) {
41                nextButton.click();
42            }
43        });
44    }
45};
46
47// ============================================================================
48// 核心 Scraper 类 - 模板部分（一般不需要修改）
49// ============================================================================
50class BaseScraper {
51    /**
52     * 解析页面数据 - 需要自定义
53     * @param {CheerioAPI} $ - Cheerio 实例
54     * @returns {Array} - 解析后的数据数组
55     * 
56     * ⚠️ 自动生成的解析代码应该替换此方法的实现
57     */
58    parsePageItems($) {
59        // ==================== 开始：页面解析代码（可替换区域） ====================
60
61        const items = $(SCRAPER_CONFIG.waitSelector).map((i, el) => {
62            /**
63             * 数据列表的 DOM 结构：
64             * 见 数据列表DOM.html 文件
65             */
66            const $item = $(el);
67            const productId = $item.attr('data-product-id') || '';
68            
69            // 获取产品图片
70            const imageEl = $item.find('img[data-testid="product-image"]').first();
71            const imageUrl = imageEl.attr('src') || imageEl.attr('srcset')?.split(',')[0]?.trim()?.split(' ')[0] || '';
72            
73            // 获取产品标题和URL
74            const titleEl = $item.find('h2[data-testid="product-title"]');
75            const titleText = titleEl.text().trim();
76            const productUrl = titleEl.find('a').attr('href') || '';
77            const brandEl = titleEl.find('span[class*="Brand_title__brand__"]');
78            const brand = brandEl.text().trim();
79            
80            // 获取价格
81            const prevPrice = $item.find('span[data-testid="product-card-price-prev"]').first().text().trim();
82            const nowPrice = $item.find('span[data-testid="product-card-price-now"]').last().text().trim();
83            
84            // 获取颜色选项
85            const colourButtons = $item.find('button[class*="ColourSwatches_button__"]');
86            const colours = colourButtons.map((i, btn) => {
87                return $(btn).attr('title');
88            }).get();
89            
90            return {
91                uniqueId: productId,
92                productTitle: titleText,
93                brand: brand,
94                prevPrice: prevPrice,
95                nowPrice: nowPrice,
96                imageUrl: imageUrl,
97                productUrl: productUrl ? `https://www.johnlewis.com${productUrl}` : '',
98                colours: colours
99            };
100        }).get();
101
102        // ==================== 结束：页面解析代码（可替换区域） ====================
103
104        return items;
105    }
106
107    async run(input) {
108        const { searchUrl, maxItems = Infinity, proxyConfiguration } = input;
109        this.maxItems = maxItems;
110        this.searchUrl = searchUrl;
111        const proxyConfig = await Actor.createProxyConfiguration(proxyConfiguration);
112        const proxyUrl = proxyConfig ? await proxyConfig.newUrl() : undefined;
113        const proxyOptions = this.buildProxyOptions(proxyUrl);
114
115        const realBrowserOption = {
116            args: ["--start-maximized"],
117            turnstile: SCRAPER_CONFIG.turnstile,
118            headless: SCRAPER_CONFIG.headless,
119            // disableXvfb: true,
120            // ignoreAllFlags:true,
121            customConfig: {},
122            connectOption: {
123                defaultViewport: null
124            },
125            ...(proxyOptions ? { proxy: proxyOptions } : {}),
126            plugins: []
127        }
128
129        const { page, browser } = await connectRealBrowser(realBrowserOption);
130
131        try {
132            page.setDefaultNavigationTimeout(300 * 1000);
133            page.setDefaultTimeout(3600 * 1000);
134
135            // if (userAgent) {
136            //     await page.setUserAgent(userAgent);
137            // }
138
139            await this.handlePage(page, this.searchUrl);
140        } finally {
141            await browser?.close();
142        }
143    }
144
145    async handlePage(page, url) {
146        console.log(`Processing ${url}...`);
147        let count = 0;
148        let endOfResults = false;
149        let interceptedData = [];
150        let lastDataLength = 0;
151        let noNewDataCount = 0;
152
153        // await this.setupInterceptors(page, interceptedData);
154        await page.goto(url);
155
156        // 等待新内容加载
157        if (SCRAPER_CONFIG.waitSelector) {
158            await page.waitForSelector(SCRAPER_CONFIG.waitSelector, { timeout: 60000 }).catch(() => {
159                console.log(`data not found...`);
160            });
161        }
162
163        // 处理首页数据
164        const firstPageHtml = await page.content();
165        await this.processResponseData(firstPageHtml, interceptedData);
166
167        // 循环翻页获取数据
168        while (!endOfResults && interceptedData.length < this.maxItems) {
169            count++;
170            console.log(`data fetching...${count}`);
171            await this.randomDelay();
172
173            // 执行翻页操作
174            await SCRAPER_CONFIG.autoScrollPage(page);
175            await this.randomDelay(3000, 5000);
176
177            // 等待新内容加载
178            if (SCRAPER_CONFIG.waitSelector) {
179                await page.waitForSelector(SCRAPER_CONFIG.waitSelector, { timeout: 10000 }).catch(() => {
180                    console.log(`data not found...`);
181                });
182            }
183
184            // 处理新页面数据
185            const afterScrollHtml = await page.content();
186            await this.processResponseData(afterScrollHtml, interceptedData);
187
188            if (interceptedData.length === lastDataLength) {
189                noNewDataCount++;
190                if (noNewDataCount >= SCRAPER_CONFIG.maxNoNewDataCount) {
191                    endOfResults = true;
192                }
193            } else {
194                noNewDataCount = 0;
195                lastDataLength = interceptedData.length;
196            }
197        }
198
199        console.log(`Scraping completed. Total items: ${interceptedData.length}`);
200    }
201
202    // 设置请求拦截器（资源过滤）
203    async setupInterceptors(page, interceptedData) {
204        await page.setRequestInterception(true);
205        page.on('request', (request) => {
206            const resourceType = request.resourceType();
207            const blockedResourceTypes = [
208                'image',
209                'media',
210                'font',
211                'texttrack',
212                'stylesheet',
213                'ping'
214            ];
215
216            if (SCRAPER_CONFIG.filterResources && blockedResourceTypes.includes(resourceType)) {
217                request.abort();
218            } else {
219                request.continue();
220            }
221        });
222        page.on('response', async (response) => {
223            // 可选：拦截特定的 API 响应
224        });
225    }
226
227    // 处理响应数据（模板方法）
228    async processResponseData(htmlContent, interceptedData) {
229        let currentData = [];
230
231        if (!htmlContent) {
232            return;
233        }
234
235        const $ = cheerio.load(htmlContent);
236
237        // 调用自定义的解析方法
238        const items = this.parsePageItems($);
239
240        if (!items || items.length === 0) {
241            return;
242        }
243
244        // 去重处理
245        const uniqueField = SCRAPER_CONFIG.uniqueIdField;
246        const existingIds = new Set(interceptedData.map(item => item[uniqueField]));
247
248        let skippedCount = 0;
249        for (const listing of items) {
250            // 跳过已存在的项目
251            if (existingIds.has(listing[uniqueField])) {
252                skippedCount++;
253                continue;
254            }
255
256            const data = {
257                searchUrl: this.searchUrl,
258                ...listing,
259                scrapedAt: new Date().toISOString()
260            };
261
262            interceptedData.push(data);
263            currentData.push(data);
264            existingIds.add(listing[uniqueField]);
265
266            // 如果达到最大数量，停止处理
267            if (interceptedData.length >= this.maxItems) {
268                break;
269            }
270        }
271
272
273        // 保存数据
274        if (currentData.length > 0) {
275            console.log(`Saved ${currentData.length} items, total: ${interceptedData.length}`);
276            await Actor.pushData(currentData);
277        }
278    }
279
280    // 随机延迟
281    async randomDelay(min = 1000, max = 3000) {
282        const delay = Math.floor(Math.random() * (max - min + 1) + min);
283        await new Promise(resolve => setTimeout(resolve, delay));
284    }
285
286    // 可选：设置 cookies
287    async setCookies(page) {
288        // const cookies = [
289        //     {
290        //         name: 'cookie_name',
291        //         value: 'cookie_value',
292        //         domain: '.example.com',
293        //         path: '/',
294        //         expires: new Date('2025-12-31').getTime() / 1000,
295        //         secure: false
296        //     }
297        // ];
298        // await Promise.all(cookies.map(cookie => page.setCookie(cookie)));
299    }
300
301
302    buildProxyOptions(proxyUrl) {
303        if (!proxyUrl) {
304            return undefined;
305        }
306
307        try {
308            const parsed = new URL(proxyUrl);
309            if (!parsed.hostname || !parsed.port) {
310                return undefined;
311            }
312
313            return {
314                host: parsed.hostname,
315                port: Number(parsed.port),
316                username: parsed.username || undefined,
317                password: parsed.password || undefined,
318            };
319        } catch (error) {
320            console.warn(`Invalid proxy URL detected: ${proxyUrl}`, error);
321            return undefined;
322        }
323    }
324
325}
326
327
328// ============================================================================
329// Actor 入口
330// ============================================================================
331await Actor.init();
332
333Actor.main(async () => {
334    let input = await Actor.getInput();
335    if (!input) {
336        input = {
337            "searchUrl": SCRAPER_CONFIG.requestUrl,
338            "maxItems": SCRAPER_CONFIG.maxItems
339        };
340    }
341    const crawler = new BaseScraper();
342    await crawler.run(input);
343});

.dockerignore

# configurations
.idea

# crawlee and apify storage folders
apify_storage
crawlee_storage
storage

# installed files
node_modules

# git folder
.git

.editorconfig

root = true

[*]
indent_style = space
indent_size = 4
charset = utf-8
trim_trailing_whitespace = true
insert_final_newline = true
end_of_line = lf

.eslintrc

{
    "extends": "@apify",
    "root": true
}

.gitignore

# This file tells Git which files shouldn't be added to source control

.DS_Store
.idea
.zed
node_modules
storage

package.json

{
    "name": "js-scrape-single-page",
    "version": "0.0.1",
    "type": "module",
    "description": "This is an example of an Apify actor.",
    "engines": {
        "node": ">=18.0.0"
    },
    "dependencies": {
		"apify": "^3.4.4",
		"crawlee": "^3.11.5",
		"puppeteer": "^23.5.0",
		"puppeteer-core": "^23.5.0",
		"puppeteer-extra": "^3.3.6",
		"puppeteer-extra-plugin-stealth": "^2.11.2",
		"random-useragent": "^0.5.0",
        "rebrowser-puppeteer": "^23.10.3",
		"rebrowser-puppeteer-core": "^23.10.3",
        "puppeteer-real-browser": "^1.4.4",
        "ioredis": "^5.4.1",
        "marked": "^15.0.7",
		"node-fetch": "3",
        "uuid": "^11.1.0"
    },
    "scripts": {
        "start": "node ./src/main.js",
        "test": "echo \"Error: oops, the actor has no tests yet, sad!\" && exit 1"
    },
    "author": "It's not you it's me",
    "license": "ISC"
}

John Lewis

mynewhome/john-lewis

John Lewis scraper

Aida Sarre

John lewis Categories Product Scaper

getdataforme/john-lewis-categories-product-scaper

The John Lewis Categories Product Scraper extracts product details from JohnLewis.com by brand, including names, prices, descriptions, and ratings. Perfect for e-commerce monitoring, market research, and price comparison.

GetDataForMe

Johnlewis Products Scraper

alkausari_mujahid/johnlewis-products-scraper

This scraper is designed to extract complete product and variant data from John Lewis product pages with high accuracy and structured output, ready for automation, analytics, or e-commerce sync.

Alkausari M

G2 Products Search Scraper

powerai/g2-products-search-scraper

Scrape product listings from G2.com by providing a search URL, with automatic pagination and comprehensive product information extraction.

PowerAI

5.0

Aldi Products Search Scraper

powerai/aldi-products-search-scraper

Scrape product listings from Aldi search results by providing a search URL, with automatic pagination and comprehensive product information extraction.

PowerAI

5.0

Johnlewis Parser Spider

getdataforme/johnlewis-parser-spider

Scrape detailed product data from John Lewis, including names, prices, ratings, and reviews, for e-commerce analysis and market research. Features brand filtering, configurable limits, proxy support for reliability, and fast JSON output. Ideal for price tracking and competitive intelligence.

GetDataForMe

Johnlewis Actor

getdataforme/my-Johnlewis-actor

The Johnlewis Actor lets you scrape product data from John Lewis effortlessly. Extract product names, prices, images, descriptions, and SKUs from any product page. Perfect for e-commerce insights or price comparisons. Boost your data collection with this efficient and proxy-ready solution!

GetDataForMe

5.0

Tesco Products Scraper

powerai/tesco-products-scraper

Scrape product search results from Tesco.com by providing a search URL, with automatic pagination and comprehensive product information extraction.

PowerAI

5.0

JL Parser

getdataforme/jl-parser

JL Parser is an Apify Actor that scrapes comprehensive product data from John Lewis, extracting prices, descriptions, reviews, and availability for targeted brands....

GetDataForMe

Argos Products Search Scraper

powerai/argos-products-search-scraper

Scrape product listings from Argos.co.uk by providing a search URL, with automatic pagination and comprehensive product information extraction.

PowerAI

5.0

John Lewis Products Search Scraper

John Lewis Products Search Scraper

.actor/Dockerfile

.actor/actor.json

.actor/input_schema.json

src/main.js

.dockerignore

.editorconfig

.eslintrc

.gitignore

package.json

You might also like

John Lewis

John lewis Categories Product Scaper

Johnlewis Products Scraper

G2 Products Search Scraper

Aldi Products Search Scraper

Johnlewis Parser Spider

Johnlewis Actor

Tesco Products Scraper

JL Parser

Argos Products Search Scraper

.actor/Dockerfile

.actor/actor.json

.actor/input_schema.json

src/main.js

.dockerignore

.editorconfig

.eslintrc

.gitignore

package.json