# E-commerce Product Scraper for AI & RAG (`coolinbex/ecommerce-product-scraper-ai-rag`) Actor

Scrape product catalogs into clean, chunked data for AI search, RAG, embeddings, and vector databases.

- **URL**: https://apify.com/coolinbex/ecommerce-product-scraper-ai-rag.md
- **Developed by:** [coolinbex](https://apify.com/coolinbex) (community)
- **Stats:** 1 total users, 0 monthly users, 0.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $4.00 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## E-commerce RAG Ingestion Scraper

Turn an online product catalog into clean, structured data for AI search, product assistants, recommendations, and RAG applications.

This Actor crawls e-commerce websites, extracts product information, and prepares the result for embedding and storage in a vector database. It works with traditional server-rendered pages as well as modern React and Next.js storefronts.

### What this Actor does

The Actor can:

- Crawl product pages, category pages, collection pages, store homepages, and sitemap URLs.
- Render JavaScript-heavy storefronts when `renderJavaScript` is enabled.
- Extract product names, SKUs, brands, categories, descriptions, images, prices, variants, availability, ratings, and reviews.
- Read structured data such as JSON-LD and use page content and frontend framework data as fallbacks.
- Clean product content into Markdown or JSON.
- Split product content into overlapping retrieval chunks.
- Add stable chunk IDs and metadata for vector database ingestion.
- Continue processing when individual pages fail.
- Use Apify Proxy, browser sessions, retries, and backoff for larger or rate-limited crawls.

### Who should use it?

This Actor is useful for:

- **E-commerce teams** building AI shopping assistants or semantic product search.
- **RAG developers** creating chatbots that answer questions about product catalogs.
- **Data engineers** preparing product data for Pinecone, Qdrant, Weaviate, Chroma, pgvector, Elasticsearch, or other vector stores.
- **Agencies and consultants** building catalog search, recommendation, and knowledge-base solutions for clients.
- **Market researchers** collecting product, price, availability, and review information from permitted sources.
- **Retail and marketplace teams** consolidating product information from multiple storefronts.

### Typical workflow

```text
Product or catalog URLs
          ↓
Browser-based crawl and extraction
          ↓
Normalized product records
          ↓
Clean Markdown or JSON content
          ↓
Overlapping RAG chunks with metadata
          ↓
Embeddings and vector database
          ↓
AI assistant, semantic search, or recommendations
```

### How to use it

1. Add one or more URLs to `startUrls`. You can provide product URLs, category URLs, collection URLs, sitemap URLs, or a store homepage.
2. Set `maxItems` to control the maximum number of products saved.
3. Leave `renderJavaScript` enabled for React, Next.js, Shopify themes, and other dynamically rendered stores.
4. Keep `followLinks` enabled when starting from a category, collection, or store URL.
5. Choose `Markdown` for human-readable RAG content or `JSON` when your downstream pipeline expects machine-readable content.
6. Adjust `chunkSizeTokens` and `chunkOverlapTokens` to match the embedding model and retrieval strategy used by your application.
7. Enable Apify Proxy through `proxyConfiguration` when the target website rate-limits or blocks direct requests.

No custom code is required for the default use case.

### Input examples

#### Crawl a product page

```json
{
  "startUrls": [
    {
      "url": "https://example.com/products/example-product"
    }
  ],
  "maxItems": 1
}
```

#### Crawl a catalog

```json
{
  "startUrls": [
    {
      "url": "https://example.com/collections/all"
    }
  ],
  "maxItems": 500,
  "maxPages": 2000,
  "renderJavaScript": true,
  "followLinks": true,
  "includeReviews": true,
  "outputFormat": "Markdown",
  "chunkSizeTokens": 400,
  "chunkOverlapTokens": 50,
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}
```

### Dataset output

The Actor saves one dataset item per product. Each item contains:

- `canonicalUrl` — the preferred URL for the product.
- `product` — normalized product metadata, including SKU, brand, offers, variants, ratings, reviews, and images.
- `content` — cleaned product content in Markdown or JSON format.
- `chunks` — embedding-ready text segments with stable IDs and retrieval metadata.
- `scrapedAt` — timestamp showing when the product was collected.

Example chunk:

```json
{
  "id": "b7f4f8d7a3c54a9b1e4c2d10",
  "index": 0,
  "text": "# Example Product\n\nA lightweight product for everyday use.",
  "embeddingText": "# Example Product\n\nA lightweight product for everyday use.",
  "tokenCount": 12,
  "metadata": {
    "canonicalUrl": "https://example.com/products/example-product",
    "sku": "EX-001",
    "productName": "Example Product",
    "brand": "Example Brand",
    "category": "Accessories",
    "chunkIndex": 0
  }
}
```

These chunks can be sent directly to an embedding service. Store the returned embedding alongside the chunk `id` and `metadata` in your vector database.

### Recommended settings

| Goal | Recommended configuration |
| --- | --- |
| One product page | `maxItems: 1`, `followLinks: false` |
| Full catalog | `followLinks: true`, a suitable `maxPages`, and Apify Proxy |
| React or Next.js storefront | `renderJavaScript: true` |
| Low-cost static crawl | `renderJavaScript: false` |
| General RAG | `outputFormat: "Markdown"`, `chunkSizeTokens: 300–500` |
| Structured downstream processing | `outputFormat: "JSON"` |

### Important considerations

Use this Actor only on websites and data that you are authorized to access. Follow the target website’s terms, robots policies, applicable privacy rules, and rate limits. Apify Proxy and browser rendering improve crawl reliability, but no scraper can guarantee access to every anti-bot protected website.

Product layouts and review widgets vary between websites. The Actor uses multiple extraction strategies and continues after page-level failures, but highly customized stores may require site-specific selectors or configuration.

### Local development

```bash
npm install
npm run check
npm start
```

At least one valid HTTP or HTTPS `startUrls` entry is required.

# Actor input Schema

## `startUrls` (type: `array`):

Product pages, category pages, sitemaps, or store homepages to crawl.

## `maxItems` (type: `integer`):

Maximum number of product records to save.

## `maxPages` (type: `integer`):

Safety limit for pages fetched, including category and navigation pages.

## `outputFormat` (type: `string`):

Format used for the canonical product text and chunks.

## `chunkSizeTokens` (type: `integer`):

Approximate maximum tokens per retrieval chunk.

## `chunkOverlapTokens` (type: `integer`):

Approximate overlap between adjacent chunks.

## `renderJavaScript` (type: `boolean`):

Use a real browser so React, Next.js, Shopify, and other hydrated stores are parsed.

## `followLinks` (type: `boolean`):

Follow same-domain pagination, category, and product links from the supplied URLs.

## `includeReviews` (type: `boolean`):

Extract visible review text and aggregate ratings when available.

## `allowedDomains` (type: `array`):

Optional hostname allowlist. Empty means the hostnames in startUrls are allowed.

## `productUrlPatterns` (type: `array`):

Optional regular expressions used to identify product pages.

## `maxConcurrency` (type: `integer`):

Number of browser tabs used concurrently.

## `maxRequestRetries` (type: `integer`):

Retries per request before it is reported as failed.

## `proxyConfiguration` (type: `object`):

Apify Proxy settings. A fresh session is selected after blocked responses.

## Actor input object example

```json
{
  "maxItems": 100,
  "maxPages": 500,
  "outputFormat": "Markdown",
  "chunkSizeTokens": 400,
  "chunkOverlapTokens": 50,
  "renderJavaScript": true,
  "followLinks": true,
  "includeReviews": true,
  "allowedDomains": [],
  "productUrlPatterns": [],
  "maxConcurrency": 3,
  "maxRequestRetries": 5,
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}
```

# Actor output Schema

## `dataset` (type: `string`):

Dataset containing one normalized product record per item, including RAG chunks.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {};

// Run the Actor and wait for it to finish
const run = await client.actor("coolinbex/ecommerce-product-scraper-ai-rag").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {}

# Run the Actor and wait for it to finish
run = client.actor("coolinbex/ecommerce-product-scraper-ai-rag").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{}' |
apify call coolinbex/ecommerce-product-scraper-ai-rag --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,coolinbex/ecommerce-product-scraper-ai-rag"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/zQG0QUp2argbtnfXR/builds/1vn4cSW8j8OTugyqw/openapi.json