Under maintenance

Pricing

Pay per usage

Try for free

Go to Apify Store

Sotheby's Search Scraper

Under maintenance

Try for free

Search and scrape auction lots from Sotheby's by query, with automatic pagination and comprehensive lot data extraction.

Pricing

Pay per usage

Rating

5.0

(1)

Developer

PowerAI

Actor stats

Bookmarked

Total users

Monthly active users

7 days ago

Last modified

.actor/Dockerfile

# Specify the base Docker image. You can read more about
# the available images at https://docs.apify.com/sdk/js/docs/guides/docker-images
# You can also use any other image from Docker Hub.
FROM apify/actor-node-puppeteer-chrome

# Check preinstalled packages
RUN npm ls apify

# Copy just package.json and package-lock.json
# to speed up the build using Docker layer cache.
COPY package*.json ./

# Install NPM packages, skip optional and development dependencies to
# keep the image small. Avoid logging too much and print the dependency
# tree for debugging
RUN npm --quiet set progress=false \
    && npm install --omit=dev --omit=optional \
    && echo "Installed NPM packages:" \
    && (npm list --omit=dev --all || true) \
    && echo "Node.js version:" \
    && node --version \
    && echo "NPM version:" \
    && npm --version \
    && rm -r ~/.npm

# Next, copy the remaining files and directories with the source code.
# Since we do this after NPM install, quick build will be really fast
# for most source file changes.
COPY . ./


# Run the image.
CMD npm start --silent

.actor/actor.json

{
    "actorSpecification": 1,
    "name": "sothebys-search-scraper",
    "title": "Empty JavaScript project",
    "description": "Empty project in JavaScript.",
    "version": "0.0",
    "buildTag": "latest",
    "meta": {
        "templateId": "js-empty"
    },
    "input": "./input_schema.json",
    "dockerfile": "./Dockerfile"
}

.actor/input_schema.json

{
  "type": "object",
  "schemaVersion": 1,
  "title": "Sotheby's Search Input",
  "properties": {
    "searchUrl": {
      "title": "Search URL",
      "type": "string",
      "description": "The complete Sotheby's search URL with query parameters",
      "minLength": 1,
      "editor": "textfield",
      "default": "https://www.sothebys.com/en/search?query=chanel",
      "prefill": "https://www.sothebys.com/en/search?query=chanel"
    },
    "maxItems": {
      "title": "Max Items",
      "type": "integer",
      "description": "Maximum number of auction lots to fetch",
      "default": 50,
      "minimum": 1,
      "maximum": 1000,
      "editor": "number",
      "prefill": 50
    },
    "proxyConfiguration": {
      "title": "Proxy Configuration",
      "type": "object",
      "description": "Proxy settings for the actor",
      "default": {
        "useApifyProxy": false,
        "apifyProxyGroups": [
          "RESIDENTIAL"
        ]
      },
      "editor": "proxy"
    }
  },
  "required": [
    "searchUrl"
  ]
}

src/main.js

1import { Actor } from 'apify';
2import { connect as connectRealBrowser } from 'puppeteer-real-browser';
3import * as cheerio from 'cheerio';
4
5// ============================================================================
6// 配置区域 - 可根据不同网站修改
7// ============================================================================
8const SCRAPER_CONFIG = {
9    // 无新数据时的最大等待次数
10    maxNoNewDataCount: 3,
11    // 无头浏览器
12    headless: false,
13    // 
14    turnstile: false,
15    // 过滤资源
16    filterResources: false,
17    // 最大数量
18    maxItems: 50,
19    // 请求url
20    requestUrl: 'https://www.sothebys.com/en/search?query=chanel',
21    // TODO4: 用于去重的唯一标识符字段名
22    uniqueIdField: 'lotUrl',
23    // TODO3: 等待的选择器（列表元素选择器）
24    waitSelector: 'div[data-testid="results-search-item"]',
25    // TODO2: 自动翻页函数
26    autoScrollPage: async (page) => {
27        await page.evaluate(async () => {
28            const distance = 500;
29            const delay = 100;
30            while (document.scrollingElement.scrollTop + window.innerHeight < document.scrollingElement.scrollHeight) {
31                document.scrollingElement.scrollBy(0, distance);
32                await new Promise(resolve => setTimeout(resolve, delay));
33            }
34        });
35    }
36};
37
38// ============================================================================
39// 核心 Scraper 类 - 模板部分（一般不需要修改）
40// ============================================================================
41class BaseScraper {
42    /**
43     * 解析页面数据 - 需要自定义
44     * @param {CheerioAPI} $ - Cheerio 实例
45     * @returns {Array} - 解析后的数据数组
46     * 
47     * ⚠️ 自动生成的解析代码应该替换此方法的实现
48     */
49    parsePageItems($) {
50        // ==================== 开始：页面解析代码（可替换区域） ====================
51
52        const items = $(SCRAPER_CONFIG.waitSelector).map((i, el) => {
53            /**
54             * 数据列表的 DOM 结构：
55             * 见 数据列表DOM.html 文件
56             * Lot Card contains: title, image, description, estimate, URL, etc.
57             */
58            const $item = $(el);
59
60            // 提取标题和URL
61            const $titleLink = $item.find('a[class*="lotTitleLink"]').first();
62            const lotTitle = $titleLink.find('[class*="title"]').text().trim() ||
63                            $titleLink.text().split('\n')[0].trim();
64            const lotUrl = $titleLink.attr('href') || '';
65            const fullLotUrl = lotUrl ? 'https://www.sothebys.com' + lotUrl : '';
66
67            // 提取图片URL
68            const $img = $item.find('img[src*="sothebys-md"]').first();
69            const imageUrl = $img.attr('src') || '';
70
71            // 提取描述
72            const $description = $item.find('[class*="description"]').first();
73            const description = $description.text().trim().substring(0, 500);
74
75            // 提取估价格
76            const $estimate = $item.find('[class*="estimateContainer"]').first();
77            const estimateText = $estimate.text().trim();
78
79            // 提取拍卖信息
80            const $label = $item.find('[class*="labelContainer"]');
81            const labelText = $label.text().trim();
82
83            // 提取倒计时信息
84            const $countdown = $item.find('[data-testid="lot-card-countdown"]');
85            const countdownText = $countdown.text().trim();
86
87            // 提取竞标信息
88            const $bidInfo = $item.find('[class*="bidInfoWrapper"]');
89            const bidInfoText = $bidInfo.text().trim();
90
91            return {
92                lotTitle: lotTitle,
93                lotUrl: fullLotUrl,
94                imageUrl: imageUrl,
95                description: description,
96                estimate: estimateText,
97                label: labelText,
98                countdown: countdownText,
99                bidInfo: bidInfoText
100            };
101        }).get();
102
103        // ==================== 结束：页面解析代码（可替换区域） ====================
104
105        return items;
106    }
107
108    async run(input) {
109        const { searchUrl, maxItems = Infinity, proxyConfiguration } = input;
110        this.maxItems = maxItems;
111        this.searchUrl = searchUrl;
112        const proxyConfig = await Actor.createProxyConfiguration(proxyConfiguration);
113        const proxyUrl = proxyConfig ? await proxyConfig.newUrl() : undefined;
114        const proxyOptions = this.buildProxyOptions(proxyUrl);
115
116        const realBrowserOption = {
117            args: ["--start-maximized"],
118            turnstile: SCRAPER_CONFIG.turnstile,
119            headless: SCRAPER_CONFIG.headless,
120            // disableXvfb: true,
121            // ignoreAllFlags:true,
122            customConfig: {},
123            connectOption: {
124                defaultViewport: null
125            },
126            ...(proxyOptions ? { proxy: proxyOptions } : {}),
127            plugins: []
128        }
129
130        const { page, browser } = await connectRealBrowser(realBrowserOption);
131
132        try {
133            page.setDefaultNavigationTimeout(300 * 1000);
134            page.setDefaultTimeout(3600 * 1000);
135
136            // if (userAgent) {
137            //     await page.setUserAgent(userAgent);
138            // }
139
140            await this.handlePage(page, this.searchUrl);
141        } finally {
142            await browser?.close();
143        }
144    }
145
146    async handlePage(page, url) {
147        console.log(`Processing ${url}...`);
148        let count = 0;
149        let endOfResults = false;
150        let interceptedData = [];
151        let lastDataLength = 0;
152        let noNewDataCount = 0;
153
154        // await this.setupInterceptors(page, interceptedData);
155        await page.goto(url);
156
157        // 等待新内容加载
158        if (SCRAPER_CONFIG.waitSelector) {
159            await page.waitForSelector(SCRAPER_CONFIG.waitSelector, { timeout: 60000 }).catch(() => {
160                console.log(`data not found...`);
161            });
162        }
163
164        // 处理首页数据
165        const firstPageHtml = await page.content();
166        await this.processResponseData(firstPageHtml, interceptedData);
167
168        // 循环翻页获取数据
169        while (!endOfResults && interceptedData.length < this.maxItems) {
170            count++;
171            console.log(`data fetching...${count}`);
172            await this.randomDelay();
173
174            // 执行翻页操作
175            await SCRAPER_CONFIG.autoScrollPage(page);
176            await this.randomDelay(3000, 5000);
177
178            // 等待新内容加载
179            if (SCRAPER_CONFIG.waitSelector) {
180                await page.waitForSelector(SCRAPER_CONFIG.waitSelector, { timeout: 10000 }).catch(() => {
181                    console.log(`data not found...`);
182                });
183            }
184
185            // 处理新页面数据
186            const afterScrollHtml = await page.content();
187            await this.processResponseData(afterScrollHtml, interceptedData);
188
189            if (interceptedData.length === lastDataLength) {
190                noNewDataCount++;
191                if (noNewDataCount >= SCRAPER_CONFIG.maxNoNewDataCount) {
192                    endOfResults = true;
193                }
194            } else {
195                noNewDataCount = 0;
196                lastDataLength = interceptedData.length;
197            }
198        }
199
200        console.log(`Scraping completed. Total items: ${interceptedData.length}`);
201    }
202
203    // 设置请求拦截器（资源过滤）
204    async setupInterceptors(page, interceptedData) {
205        await page.setRequestInterception(true);
206        page.on('request', (request) => {
207            const resourceType = request.resourceType();
208            const blockedResourceTypes = [
209                'image',
210                'media',
211                'font',
212                'texttrack',
213                'stylesheet',
214                'ping'
215            ];
216
217            if (SCRAPER_CONFIG.filterResources && blockedResourceTypes.includes(resourceType)) {
218                request.abort();
219            } else {
220                request.continue();
221            }
222        });
223        page.on('response', async (response) => {
224            // 可选：拦截特定的 API 响应
225        });
226    }
227
228    // 处理响应数据（模板方法）
229    async processResponseData(htmlContent, interceptedData) {
230        let currentData = [];
231
232        if (!htmlContent) {
233            return;
234        }
235
236        const $ = cheerio.load(htmlContent);
237
238        // 调用自定义的解析方法
239        const items = this.parsePageItems($);
240
241        if (!items || items.length === 0) {
242            return;
243        }
244
245        // 去重处理
246        const uniqueField = SCRAPER_CONFIG.uniqueIdField;
247        const existingIds = new Set(interceptedData.map(item => item[uniqueField]));
248
249        let skippedCount = 0;
250        for (const listing of items) {
251            // 跳过已存在的项目
252            if (existingIds.has(listing[uniqueField])) {
253                skippedCount++;
254                continue;
255            }
256
257            const data = {
258                searchUrl: this.searchUrl,
259                ...listing,
260                scrapedAt: new Date().toISOString()
261            };
262
263            interceptedData.push(data);
264            currentData.push(data);
265            existingIds.add(listing[uniqueField]);
266
267            // 如果达到最大数量，停止处理
268            if (interceptedData.length >= this.maxItems) {
269                break;
270            }
271        }
272
273
274        // 保存数据
275        if (currentData.length > 0) {
276            console.log(`Saved ${currentData.length} items, total: ${interceptedData.length}`);
277            await Actor.pushData(currentData);
278        }
279    }
280
281    // 随机延迟
282    async randomDelay(min = 1000, max = 3000) {
283        const delay = Math.floor(Math.random() * (max - min + 1) + min);
284        await new Promise(resolve => setTimeout(resolve, delay));
285    }
286
287    // 可选：设置 cookies
288    async setCookies(page) {
289        // const cookies = [
290        //     {
291        //         name: 'cookie_name',
292        //         value: 'cookie_value',
293        //         domain: '.example.com',
294        //         path: '/',
295        //         expires: new Date('2025-12-31').getTime() / 1000,
296        //         secure: false
297        //     }
298        // ];
299        // await Promise.all(cookies.map(cookie => page.setCookie(cookie)));
300    }
301
302
303    buildProxyOptions(proxyUrl) {
304        if (!proxyUrl) {
305            return undefined;
306        }
307
308        try {
309            const parsed = new URL(proxyUrl);
310            if (!parsed.hostname || !parsed.port) {
311                return undefined;
312            }
313
314            return {
315                host: parsed.hostname,
316                port: Number(parsed.port),
317                username: parsed.username || undefined,
318                password: parsed.password || undefined,
319            };
320        } catch (error) {
321            console.warn(`Invalid proxy URL detected: ${proxyUrl}`, error);
322            return undefined;
323        }
324    }
325
326}
327
328
329// ============================================================================
330// Actor 入口
331// ============================================================================
332await Actor.init();
333
334Actor.main(async () => {
335    let input = await Actor.getInput();
336    if (!input) {
337        input = {
338            "searchUrl": SCRAPER_CONFIG.requestUrl,
339            "maxItems": SCRAPER_CONFIG.maxItems
340        };
341    }
342    const crawler = new BaseScraper();
343    await crawler.run(input);
344});

.dockerignore

# configurations
.idea

# crawlee and apify storage folders
apify_storage
crawlee_storage
storage

# installed files
node_modules

# git folder
.git

.editorconfig

root = true

[*]
indent_style = space
indent_size = 4
charset = utf-8
trim_trailing_whitespace = true
insert_final_newline = true
end_of_line = lf

.eslintrc

{
    "extends": "@apify",
    "root": true
}

.gitignore

# This file tells Git which files shouldn't be added to source control

.DS_Store
.idea
.zed
node_modules
storage

package.json

{
    "name": "js-scrape-single-page",
    "version": "0.0.1",
    "type": "module",
    "description": "This is an example of an Apify actor.",
    "engines": {
        "node": ">=18.0.0"
    },
    "dependencies": {
		"apify": "^3.4.4",
		"crawlee": "^3.11.5",
		"puppeteer": "^23.5.0",
		"puppeteer-core": "^23.5.0",
		"puppeteer-extra": "^3.3.6",
		"puppeteer-extra-plugin-stealth": "^2.11.2",
		"random-useragent": "^0.5.0",
        "rebrowser-puppeteer": "^23.10.3",
		"rebrowser-puppeteer-core": "^23.10.3",
        "puppeteer-real-browser": "^1.4.4",
        "ioredis": "^5.4.1",
        "marked": "^15.0.7",
		"node-fetch": "3",
        "uuid": "^11.1.0"
    },
    "scripts": {
        "start": "node ./src/main.js",
        "test": "echo \"Error: oops, the actor has no tests yet, sad!\" && exit 1"
    },
    "author": "It's not you it's me",
    "license": "ISC"
}

Sothebys International Realty Scraper

getdataforme/Sothebys-International-Realty-Scraper

Scrape comprehensive real estate listings from Sotheby's International Realty using location-based queries. Ideal for market analysis, investment research, and competitive intelligence....

GetDataForMe

Hibid Auction Details Scraper

ecomscrape/hibid-product-details-scraper

Automate HiBid.com auction data extraction with our comprehensive scraper. Extract lot details, bid information, auction data, and product specifications from thousands of live and online auctions efficiently and accurately.

ecomscrape

Hibid Auction Listings Scraper

ecomscrape/hibid-auction-listings-scraper

Extract comprehensive auction data from HiBid.com with our powerful scraper. Get product details, bid information, lot numbers, and seller data for market research, price tracking, and competitive analysis. Perfect for auction professionals and data analysts.

ecomscrape

Macy's Search Scraper

powerai/macys-search-scraper

Scrape products from Macy's.com with automatic pagination and comprehensive product data extraction including prices, ratings, and reviews.

PowerAI

5.0

Catawiki Search Scraper 🔍

easyapi/catawiki-search-scraper

Scrape Catawiki auction search results efficiently. Extract detailed lot information including titles, prices, descriptions and more from any Catawiki search URL. Perfect for auction data analysis and market research.

EasyApi

Aldi Products Search Scraper

powerai/aldi-products-search-scraper

Scrape product listings from Aldi search results by providing a search URL, with automatic pagination and comprehensive product information extraction.

PowerAI

5.0

Craigslist Jobs Search Scraper

powerai/craigslist-jobs-serach-scraper

Scrape job listings from Craigslist by providing a search URL, with automatic pagination and comprehensive job information extraction.

PowerAI

5.0

Action.com Search Scraper

powerai/action-com-search-scraper

Scrape products from Action.com by providing a search URL, with automatic pagination and comprehensive product information extraction.

PowerAI

5.0

Overstock Products Search Scraper

powerai/overstock-products-search-scraper

Scrape product listings from Overstock.com by providing a search URL, with automatic pagination and comprehensive product information extraction.

PowerAI

5.0

Otto Search Scraper

powerai/otto-search-scraper

Scrape product search results from Otto.de by providing a search URL, with automatic pagination and comprehensive product information extraction including prices, ratings, colors, and availability.

PowerAI

5.0