E-commerce Product Scraper for AI & RAG

Convert e-commerce websites into clean, structured, AI-ready product data.

This Actor crawls product catalogs and prepares product information for semantic search, AI shopping assistants, recommendation systems, embeddings, vector databases, and retrieval-augmented generation (RAG) applications.

Why use this Actor?

Product pages are difficult to use directly in an AI pipeline. They contain navigation, scripts, layout markup, duplicated text, variant selectors, review widgets, and other content that is not useful to an embedding model.

This Actor handles that preparation for you:

• Finds product pages from product, category, collection, sitemap, or store URLs.
• Supports static HTML and JavaScript-rendered React and Next.js storefronts.
• Extracts product identity, descriptions, brands, categories, SKUs, images, prices, variants, availability, ratings, and reviews.
• Reads JSON-LD structured data, page markup, and common frontend framework state.
• Removes page noise and produces clean Markdown or JSON content.
• Splits content into overlapping retrieval chunks.
• Adds stable chunk IDs and metadata for filtering, citations, and vector storage.
• Continues processing when individual pages fail.
• Supports Apify Proxy, browser sessions, request retries, and backoff for larger crawls.

Who is it for?

This Actor is designed for:

• E-commerce businesses building AI shopping assistants.
• Developers building product chatbots and semantic search.
• Data teams creating product knowledge bases for RAG.
• Agencies delivering AI search and recommendation projects.
• Retailers consolidating product information from multiple stores.
• Researchers collecting permitted product, pricing, availability, and review data.

How it works

Your product or catalog URLs
            ↓
Browser-based crawl
            ↓
Product extraction and normalization
            ↓
Clean Markdown or JSON content
            ↓
Overlapping RAG chunks with metadata
            ↓
Your embedding model and vector database

The Actor stores one dataset item per detected product. You can export the dataset as JSON, CSV, Excel, or use the Apify API to continue your pipeline automatically.

How to use the Actor

1. Open the Actor

Open the Actor in Apify Console and go to the Input tab.

2. Add at least one start URL

Use a product page for a single product, or use a category, collection, sitemap, or store homepage to discover multiple products.

Example:

{
  "startUrls": [
    {
      "url": "https://example.com/collections/all"
    }
  ]
}

The prefilled smoke input uses the public Books to Scrape demo product page, so Apify's automated health check can start without customer input and finish within the platform test window. Replace it with your authorized storefront URLs for real workloads.

The Actor requires at least one valid HTTP or HTTPS URL. It will not start from an empty URL list.

3. Select crawl settings

For most modern stores, keep the defaults:

{
  "maxItems": 100,
  "maxPages": 500,
  "renderJavaScript": true,
  "followLinks": true,
  "includeReviews": true,
  "outputFormat": "Markdown",
  "chunkSizeTokens": 400,
  "chunkOverlapTokens": 50
}

The Actor's prefilled health-check input intentionally uses one product page, one page, and no link traversal so Apify's automated test finishes quickly. For a catalog crawl, set followLinks to true and raise maxItems and maxPages deliberately.

4. Configure Proxy when needed

Enable Apify Proxy for websites that rate-limit direct requests:

{
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}

Proxy use does not guarantee access to every anti-bot protected website. Always crawl websites and data that you are authorized to access.

5. Start the run

Click Start in Apify Console. When the run completes, open the Dataset or Output tab to inspect the product records and RAG chunks.

Input specification

Input	Type	Default	Purpose
startUrls	URL list	Required	Product, category, collection, sitemap, or store URLs to crawl.
maxItems	Integer	100	Maximum number of product records to save.
maxPages	Integer	500	Maximum pages to fetch during discovery.
outputFormat	Markdown or JSON	Markdown	Format of the cleaned product content.
chunkSizeTokens	Integer	400	Approximate maximum size of each retrieval chunk.
chunkOverlapTokens	Integer	50	Context repeated between adjacent chunks.
renderJavaScript	Boolean	true	Render React, Next.js, and other hydrated storefronts.
followLinks	Boolean	true	Follow same-domain links to discover products.
includeReviews	Boolean	true	Extract visible reviews and aggregate ratings.
allowedDomains	String list	Start URL domains	Restrict crawling to selected hostnames.
productUrlPatterns	Regex list	Empty	Optional patterns for identifying product URLs.
maxConcurrency	Integer	3	Number of browser pages used concurrently.
maxRequestRetries	Integer	5	Retry count for failed requests.
proxyConfiguration	Object	Apify Proxy	Proxy settings for the crawl.

Ready-to-use input examples

One product page

{
  "startUrls": [
    {
      "url": "https://example.com/products/example-product"
    }
  ],
  "maxItems": 1,
  "followLinks": false
}

Full collection or category

{
  "startUrls": [
    {
      "url": "https://example.com/collections/headphones"
    }
  ],
  "maxItems": 250,
  "maxPages": 1000,
  "renderJavaScript": true,
  "followLinks": true,
  "includeReviews": true,
  "outputFormat": "Markdown",
  "chunkSizeTokens": 400,
  "chunkOverlapTokens": 50,
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}

Store search results

If the store has an internal search page, provide the search URL:

{
  "startUrls": [
    {
      "url": "https://example.com/search?q=wireless+headphones"
    }
  ],
  "maxItems": 50,
  "followLinks": true
}

Search URL formats vary by website. Use the format supported by the target store.

Output specification

Each product is saved as one dataset item with these top-level fields:

Field	Description
recordType	Record type, currently product.
source	Actor identifier that created the record.
canonicalUrl	Preferred canonical URL for the product page.
product	Normalized product metadata.
content	Cleaned Markdown or JSON product content.
chunks	Embedding-ready retrieval chunks.
scrapedAt	UTC timestamp for the record.

Product data

The product object may contain:

• name
• sku
• brand
• description
• category
• images
• offers and variants
• rating
• reviews

Fields that are not published by the source website may be null or empty.

RAG chunk data

Each item in chunks contains:

{
  "id": "b7f4f8d7a3c54a9b1e4c2d10",
  "index": 0,
  "text": "# Example Product\n\nA lightweight product for everyday use.",
  "embeddingText": "# Example Product\n\nA lightweight product for everyday use.",
  "tokenCount": 12,
  "metadata": {
    "canonicalUrl": "https://example.com/products/example-product",
    "sku": "EX-001",
    "productName": "Example Product",
    "brand": "Example Brand",
    "category": "Accessories",
    "chunkIndex": 0
  }
}

Send embeddingText to your embedding model and store the returned vector with id and metadata in Pinecone, Qdrant, Weaviate, Chroma, pgvector, Elasticsearch, or another vector database.

Recommended settings

Use case	Recommended settings
One known product	maxItems: 1, followLinks: false
Small catalog	maxItems: 100, followLinks: true
Large catalog	Increase maxPages, use Apify Proxy, and limit the allowed domains.
React or Next.js store	renderJavaScript: true
Simple static website	renderJavaScript: false to reduce browser usage.
General RAG	Markdown, chunkSizeTokens: 300–500, overlap 30–75.
Structured processing	outputFormat: "JSON".

Connecting the output to a RAG pipeline

1. Run the Actor and wait for the dataset to finish.
2. Read the chunks array from each dataset item.
3. Generate embeddings from each chunk’s embeddingText.
4. Store each vector using the chunk id as its identifier.
5. Store metadata with the vector for product filtering and source citations.
6. At query time, retrieve the most relevant chunks and provide them to your language model as context.

Important considerations

The Actor uses multiple extraction strategies, but websites differ in structure. A product page may not expose a SKU, price, reviews, or structured data; unavailable values cannot be inferred reliably.

Use this Actor only on websites and data you are authorized to access. Respect website terms, robots policies, applicable privacy requirements, and reasonable request rates. Proxy rotation and browser rendering improve reliability but cannot guarantee access to every anti-bot protected website.

Local development

npm install
npm run check
npm start

The deployed Actor is configured through Apify Console and requires at least one startUrls entry.