# Wayfair Products Scraper (`scrapyx/wayfair-products-scraper`) Actor

Product listings from Wayfair category and keyword pages: name, brand, price and was-price, rating, review count, stock message and promo flags. De-duplicated by SKU, with the resolved category reported.

- **URL**: https://apify.com/scrapyx/wayfair-products-scraper.md
- **Developed by:** [Ibnu Adzim](https://apify.com/scrapyx) (community)
- **Categories:** E-commerce
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.05 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Wayfair Products Scraper

Product listings from Wayfair category and keyword pages — name, brand, price
and was-price, rating, review count, stock message, promo flags and shipping
badge.

HTTP only, no browser, no login.

### Input

Category URLs are the precise option:

```json
{
  "mode": "urls",
  "listingUrls": ["https://www.wayfair.com/furniture/sb0/desks-c46129.html"],
  "maxPages": 3,
  "maxItems": 150
}
```

Keywords also work, with a caveat worth reading below:

```json
{ "mode": "search", "keywords": ["bookcase", "office chair"] }
```

### What you get

| `recordType` | One per | Carries |
| --- | --- | --- |
| `SEARCH_SUMMARY` | keyword / URL | requested vs **resolved** URL, whether it redirected, pages fetched, rows fetched vs returned, duplicates dropped |
| `PRODUCT` | product | SKU, name, brand, price, was-price, rating, review count, stock, badges — plus the raw card metadata |
| `ERROR` | failed input | why |

```jsonc
{
  "recordType": "PRODUCT",
  "cardSchema": "full",
  "sku": "W114696675",
  "name": "Modern Farmhouse Computer Desk with LED Lights and Power Outlets",
  "brand": "Millwood Pines",
  "currency": "USD", "price": 184.99, "strikethroughPrice": 465.00,
  "ratingAverage": 4.43, "reviewCount": 154,
  "stockMessage": "554 Left in Stock",
  "promoFlagText": "Labor Day Deal",
  "shippingBadgeText": "FREE Delivery"
}
```

### Known limits — read these before you rely on the output

**Keyword search is not a keyword search.** `keyword.php?keyword=desk`
redirects to whichever single *category* Wayfair picks for that term — in
testing, "desk" landed on **unfinished desks**. The products are real, but
the category may be narrower than your term implies. Every summary reports
`requestedUrl`, `resolvedUrl` and `wasRedirected` so you can see where you
ended up. For precise control, pass category URLs in `listingUrls`.

Wayfair also rewrites category slugs from the id: the URL
`…/sb0/desks-c46129.html` resolved to `…/sb0/kitchen-dining-tables-c46129.html`.
The **id** is authoritative, not the slug you typed.

**Wayfair rate-limits bursts with HTTP 429.** It is transient, not a block —
in testing a rapid burst earned one, and backing off to a request every 3–4
seconds cleared it on the very next request and then ran clean for four
consecutive pages. The actor retries 429 with backoff and defaults to one
request at a time. If you see repeated failures, raise
`minRequestInterval` rather than lowering it.

**Only Chrome TLS fingerprints are served.** On the identical URL,
`chrome124` and `chrome131` return the full 3.5 MB page while `safari17_0`,
`firefox133` and `edge101` all return 429. The actor uses Chrome profiles
only. This is worth knowing if you fork it.

**Cards repeat, so products are de-duplicated by SKU.** A 60-card page holds
roughly 48 distinct products; `duplicatesDropped` tells you how many were
removed, both within a page and across pages.

**Some products have no name.** Wayfair emits two card metadata schemas on
the same page. The common one (`cardSchema: "full"`, 48 of 60 in testing)
carries name and brand; a second one (`"priceOnly"`, 3 of 60) carries SKU,
prices, rating and review count but **no name or brand at all**. Those are
distinct products, not duplicates — measured zero SKU overlap with the named
ones — so they are kept rather than dropped, with `name` left `null` instead
of invented. Filter on `cardSchema` if you only want named rows.

**Prices come from Wayfair's own card tracking data**, which is what the site
uses for its analytics. It matches the rendered price at fetch time; it is
not a quote and does not include shipping or tax.

### Technical

Wayfair is a Next.js App Router app that streams RSC, but the product grid is
**not** in the flight stream (that carries only recommendation rails). The
grid is server-rendered HTML, and each card carries a `data-tracking-metadata`
JSON attribute holding the whole record — which is what this actor parses,
rather than depending on generated class names like `class="dq5rf45 dq5rf40"`
that change on every deploy.

`robots.txt` was checked at **path** level, not just the apex:
`/keyword.php?keyword=…` and `/furniture/sb0/…` are both ALLOWED, with no
`ClaudeBot`/`anthropic-ai` group and no blanket disallow.

Full recon trail is in `CRAWLING_METHOD.md`.

# Actor input Schema

## `mode` (type: `string`):

'search' sends your keyword to Wayfair's keyword.php, which REDIRECTS to whichever category Wayfair picks for that term — the summary reports where you landed. 'urls' takes category listing URLs directly and is the precise option.

## `keywords` (type: `array`):

Search terms for the 'search' mode. Note Wayfair maps a term to a single category rather than doing a keyword match, so 'desk' can land on 'unfinished desks'. Check 'resolvedUrl' in the summary.

## `listingUrls` (type: `array`):

Wayfair category listing URLs, e.g. https://www.wayfair.com/furniture/sb0/desks-c46129.html . Product detail pages (/pdp/…) are a different layout and are refused with an explanation.

## `maxPages` (type: `integer`):

Pages to fetch per keyword or URL, via ?curpage=N. Each page holds 60 cards, of which roughly 48 are new — the rest repeat and are de-duplicated by SKU.

## `maxItems` (type: `integer`):

Optional cap on total product rows across the run, shared across every keyword or URL. Counted after de-duplication.

## `maxConcurrency` (type: `integer`):

Left at 1 by default on purpose: Wayfair answers bursts with HTTP 429. The 429 is transient and is retried with backoff, but going wide mostly buys retries rather than speed.

## `minRequestInterval` (type: `number`):

The real speed control, and the one that matters here. A rapid burst earned a 429 during testing; one request every 3-4 seconds ran clean. Lower it only if you are seeing no 429s.

## `proxyConfiguration` (type: `object`):

Residential is the cloud default. Wayfair serves the US storefront in USD; the proxy country does not change the data. Note the actor uses Chrome TLS profiles only — Safari, Firefox and Edge fingerprints are refused with a 429 on this site.

## Actor input object example

```json
{
  "mode": "urls",
  "keywords": [],
  "listingUrls": [
    "https://www.wayfair.com/furniture/sb0/desks-c46129.html"
  ],
  "maxPages": 3,
  "maxConcurrency": 1,
  "minRequestInterval": 3,
  "proxyConfiguration": {
    "useApifyProxy": true,
    "apifyProxyGroups": [
      "RESIDENTIAL"
    ]
  }
}
```

# Actor output Schema

## `items` (type: `string`):

One row per scraped record. See the dataset's default view for field definitions.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {};

// Run the Actor and wait for it to finish
const run = await client.actor("scrapyx/wayfair-products-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {}

# Run the Actor and wait for it to finish
run = client.actor("scrapyx/wayfair-products-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{}' |
apify call scrapyx/wayfair-products-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,scrapyx/wayfair-products-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/47YvgimmVVCfQI8k9/builds/aQVIrAmoKfkyYUT3V/openapi.json