# Price Scraper for Bot-Protected Stores (`akhil_rajesh/bot-wall-price-scraper`) Actor

Compare live prices across major Canadian stores — just type a product name. Searches Amazon.ca, Best Buy, Walmart, Newegg, Vuugo and Canada Computers, then scrapes each match for price, stock and currency, cheapest first. Works on Cloudflare and bot-protected sites where other scrapers fail.

- **URL**: https://apify.com/akhil\_rajesh/bot-wall-price-scraper.md
- **Developed by:** [Akhil rajesh](https://apify.com/akhil_rajesh) (community)
- **Categories:** E-commerce, Other
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.00 / 1,000 results

This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Price Scraper for Bot-Protected Stores

Extract live product prices from **bot-protected retailers** — including sites where other scrapers hit a Cloudflare wall, a "Just a moment…" challenge, or an empty price field and give up.

**Just type a product name** (e.g. `RTX 5070 Ti`) and get a price comparison across every supported store — no store selection needed. Or paste exact URLs in Advanced mode.

Works out of the box on Amazon.ca, Best Buy Canada, Walmart.ca, Newegg.ca, Vuugo and Canada Computers — plus Memory Express and any product page URL you provide.

### Two ways to use it

#### 1. Search by product name (easiest)

Enter a product name and the Actor searches **every supported store automatically** — no store selection needed. It then scrapes each match for its live price. Results come back **sorted cheapest-first** — a ready-made price comparison.

```
Product name:  RTX 5070 Ti
```

That's the whole input. Stores, URLs and selectors are all optional and live under **Advanced** — leave them untouched for the default behaviour.

#### 2. Direct URLs (Advanced)

Paste product URLs (or a whole category/search page) for exact control. You can also limit the search to specific stores from the Advanced section.

### What you get

| Field | Example |
|---|---|
| `retailer` | newegg |
| `name` | ASUS PRIME GeForce RTX 5070 Ti 16GB |
| `price` | 1559.99 |
| `currency` | CAD |
| `inStock` | true |
| `url` | https://www.newegg.ca/... |
| `scrapedAt` | 2026-09-09T12:00:00Z |

### Why it works where others don't

- **Bypasses TLS fingerprinting walls** that block plain HTTP clients and most scrapers
- **Renders JavaScript pages** automatically — sites that load prices dynamically work too
- **Reads prices from embedded page data** when they're not in the visible HTML
- **Ignores subscription decoys** — a "$19.99/month membership" banner can't masquerade as a product price
- **Full category crawling** — point it at a category/search page and it finds and scrapes every product across result pages

### Input

| Field | Type | Description |
|-------|------|-------------|
| `searchQuery` | string | **Product name to search for** — all supported stores are searched automatically |
| `maxResultsPerStore` | int | Matches per store to scrape (default 3) |
| `retailers` | array | *Advanced:* limit to specific stores. Empty/omitted = every supported store |
| `productUrls` | array | *Advanced:* exact product/category URLs (takes priority over search) |
| `maxPagesPerUrl` | int | For category URLs: pages to crawl (default 1) |
| `proxy` | proxy | Optional. Try without first — enable if a store blocks your IP |
| `nameSelector`/`priceSelector`/`stockSelector` | string | *Advanced:* CSS overrides for unusual layouts |
| `useJavascript`, `debug` | bool | *Advanced:* force browser rendering / verbose logs |

### Example output

```json
{
  "retailer": "newegg",
  "url": "https://www.newegg.ca/p/N82E16814932771",
  "name": "GIGABYTE WindForce GeForce RTX 5070 Ti 16GB",
  "price": "1669.00",
  "currency": "CAD",
  "inStock": true,
  "engine": "standard",
  "scrapedAt": "2026-09-10T12:00:00Z"
}
```

If a URL fails entirely, you get a row with an `error` field — **one bad URL never kills a batch run**.

### Validated on (real runs)

Amazon.ca · Best Buy Canada · Walmart.ca · Newegg.ca · Vuugo · Canada Computers · Memory Express — tested on GPUs, phones, headphones and game consoles. Batch runs are safe: automatic politeness delays, per-URL time budgets, and retry on transient errors.

### Use cases

- **Price comparison** — type a product name, get every store's price side by side, cheapest first.
- **Competitor monitoring** — schedule a daily run and track how rival listings move.
- **Price-drop alerts** — feed the dataset into your own alerting to catch discounts.
- **Market research** — pull category pages to survey a whole product segment at once.
- **Procurement** — find the cheapest source for a spec before you buy.

### Troubleshooting

| Symptom | Fix |
|---|---|
| A store returned nothing | That store may not stock the item (e.g. phones are only sold by phone retailers). Check the run log — it reports per-store results. |
| Empty price on a row | Some marketplace listings hide the price until you pick a variant. The row returns `price: null` rather than guessing. |
| A store is rate-limiting you | Enable the proxy in Advanced, or reduce **Max results per store**. |
| Wrong or odd price | Send the URL and the expected price — price extraction is actively improved. |
| Searching an unusual site | Use a direct URL in Advanced; add a price CSS selector if the page uses non-standard markup. |
| "Always render JavaScript" gave worse results | Leave that option **off**. It's a troubleshooting fallback only — automatic detection already renders when a page needs it, and some stores react badly to browser sessions. |

### Sample output

```json
[
  {
    "retailer": "vuugo",
    "url": "https://www.vuugo.com/product/msi-geforce-rtx-g507t-16vgs/",
    "name": "MSI GeForce RTX 5070 Ti 16G VANGUARD SOC Graphics Card",
    "price": "1386.00",
    "currency": "CAD",
    "inStock": true,
    "engine": "standard",
    "searchQuery": "RTX 5070 Ti",
    "scrapedAt": "2026-09-10T15:34:51.209Z"
  },
  {
    "retailer": "bestbuy",
    "url": "https://www.bestbuy.ca/en-ca/product/...",
    "name": "MSI GeForce RTX 5070 Ti Gaming Trio OC 16GB GDDR7",
    "price": "1799.99",
    "currency": "CAD",
    "inStock": true,
    "engine": "standard",
    "searchQuery": "RTX 5070 Ti",
    "scrapedAt": "2026-09-10T15:34:58.881Z"
  }
]
```

### Limitations (honest)

- **Not every store carries every product.** For new phones (e.g. iPhone 17 Pro Max), Best Buy and Walmart stock them while component-focused stores (Newegg, Canada Computers) list only accessories — those accessory matches are filtered out automatically, so fewer stores may return results for such items.
- **Interactive image CAPTCHAs** (select-the-traffic-lights) cannot be automated — may need Apify proxy
- **Some stores rate-limit datacenter IPs** — if a store returns nothing repeatedly, enable the Apify proxy
- **Memory Express search** is heavily protected; use a direct product/category URL for it instead
- Some marketplace listings show no price until options are chosen — those return `price: null` rather than a guessed number
- Respect each site's Terms of Service and robots.txt — this actor is for accessing public data you're entitled to see

# Actor input Schema

## `searchQuery` (type: `string`):

What to look for, e.g. 'RTX 5070 Ti' or 'Sony WH-1000XM6'. All supported stores are searched automatically and each match is scraped for its live price — results come back cheapest-first.

## `maxResultsPerStore` (type: `integer`):

How many matching products to collect from each store, then scrape for price.

## `retailers` (type: `array`):

Which stores to search. Empty or omitted = all supported stores.

## `productUrls` (type: `array`):

Prefer exact pages? One URL per line. Product pages return one row each; category/search pages are crawled for all listed products. Takes priority over the product name above.

## `maxPagesPerUrl` (type: `integer`):

When a URL above is a category/search listing, how many result pages to crawl.

## `proxy` (type: `object`):

Only needed if a store blocks the request from your IP. Off by default.

## `nameSelector` (type: `string`):

CSS selector for the product name. Auto-detected if empty.

## `priceSelector` (type: `string`):

CSS selector for the price. Auto-detected if empty.

## `stockSelector` (type: `string`):

CSS selector that, when present, means the item is in stock. Auto-detected if empty.

## `useJavascript` (type: `boolean`):

Force full browser rendering for every URL. Normally left off — automatic detection gives better results, because some stores serve worse pages (or block follow-up requests) once a browser session is used.

## `debug` (type: `boolean`):

Verbose logging for troubleshooting.

## Actor input object example

```json
{
  "searchQuery": "RTX 5070 Ti",
  "maxResultsPerStore": 3,
  "maxPagesPerUrl": 1,
  "proxy": {
    "useApifyProxy": false
  },
  "useJavascript": false,
  "debug": false
}
```

# Actor output Schema

## `results` (type: `string`):

Product price records stored in the default dataset, one item per product page scraped, sorted cheapest-first in search mode.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "searchQuery": "RTX 5070 Ti",
    "proxy": {
        "useApifyProxy": false
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("akhil_rajesh/bot-wall-price-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "searchQuery": "RTX 5070 Ti",
    "proxy": { "useApifyProxy": False },
}

# Run the Actor and wait for it to finish
run = client.actor("akhil_rajesh/bot-wall-price-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "searchQuery": "RTX 5070 Ti",
  "proxy": {
    "useApifyProxy": false
  }
}' |
apify call akhil_rajesh/bot-wall-price-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,akhil_rajesh/bot-wall-price-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/xVmZK4iXQahaBYB9q/builds/SMe75Av9AJGZw9osR/openapi.json