E-commerce RAG Ingestion Scraper

Turn an online product catalog into clean, structured data for AI search, product assistants, recommendations, and RAG applications.

This Actor crawls e-commerce websites, extracts product information, and prepares the result for embedding and storage in a vector database. It works with traditional server-rendered pages as well as modern React and Next.js storefronts.

What this Actor does

The Actor can:

• Crawl product pages, category pages, collection pages, store homepages, and sitemap URLs.
• Render JavaScript-heavy storefronts when renderJavaScript is enabled.
• Extract product names, SKUs, brands, categories, descriptions, images, prices, variants, availability, ratings, and reviews.
• Read structured data such as JSON-LD and use page content and frontend framework data as fallbacks.
• Clean product content into Markdown or JSON.
• Split product content into overlapping retrieval chunks.
• Add stable chunk IDs and metadata for vector database ingestion.
• Continue processing when individual pages fail.
• Use Apify Proxy, browser sessions, retries, and backoff for larger or rate-limited crawls.

Who should use it?

This Actor is useful for:

• E-commerce teams building AI shopping assistants or semantic product search.
• RAG developers creating chatbots that answer questions about product catalogs.
• Data engineers preparing product data for Pinecone, Qdrant, Weaviate, Chroma, pgvector, Elasticsearch, or other vector stores.
• Agencies and consultants building catalog search, recommendation, and knowledge-base solutions for clients.
• Market researchers collecting product, price, availability, and review information from permitted sources.
• Retail and marketplace teams consolidating product information from multiple storefronts.

Typical workflow

Product or catalog URLs
          ↓
Browser-based crawl and extraction
          ↓
Normalized product records
          ↓
Clean Markdown or JSON content
          ↓
Overlapping RAG chunks with metadata
          ↓
Embeddings and vector database
          ↓
AI assistant, semantic search, or recommendations

How to use it

1. Add one or more URLs to startUrls. You can provide product URLs, category URLs, collection URLs, sitemap URLs, or a store homepage.
2. Set maxItems to control the maximum number of products saved.
3. Leave renderJavaScript enabled for React, Next.js, Shopify themes, and other dynamically rendered stores.
4. Keep followLinks enabled when starting from a category, collection, or store URL.
5. Choose Markdown for human-readable RAG content or JSON when your downstream pipeline expects machine-readable content.
6. Adjust chunkSizeTokens and chunkOverlapTokens to match the embedding model and retrieval strategy used by your application.
7. Enable Apify Proxy through proxyConfiguration when the target website rate-limits or blocks direct requests.

No custom code is required for the default use case.

Input examples

Crawl a product page

{
  "startUrls": [
    {
      "url": "https://example.com/products/example-product"
    }
  ],
  "maxItems": 1
}

Crawl a catalog

{
  "startUrls": [
    {
      "url": "https://example.com/collections/all"
    }
  ],
  "maxItems": 500,
  "maxPages": 2000,
  "renderJavaScript": true,
  "followLinks": true,
  "includeReviews": true,
  "outputFormat": "Markdown",
  "chunkSizeTokens": 400,
  "chunkOverlapTokens": 50,
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}

Dataset output

The Actor saves one dataset item per product. Each item contains:

• canonicalUrl — the preferred URL for the product.
• product — normalized product metadata, including SKU, brand, offers, variants, ratings, reviews, and images.
• content — cleaned product content in Markdown or JSON format.
• chunks — embedding-ready text segments with stable IDs and retrieval metadata.
• scrapedAt — timestamp showing when the product was collected.

Example chunk:

{
  "id": "b7f4f8d7a3c54a9b1e4c2d10",
  "index": 0,
  "text": "# Example Product\n\nA lightweight product for everyday use.",
  "embeddingText": "# Example Product\n\nA lightweight product for everyday use.",
  "tokenCount": 12,
  "metadata": {
    "canonicalUrl": "https://example.com/products/example-product",
    "sku": "EX-001",
    "productName": "Example Product",
    "brand": "Example Brand",
    "category": "Accessories",
    "chunkIndex": 0
  }
}

These chunks can be sent directly to an embedding service. Store the returned embedding alongside the chunk id and metadata in your vector database.

Recommended settings

Goal	Recommended configuration
One product page	maxItems: 1, followLinks: false
Full catalog	followLinks: true, a suitable maxPages, and Apify Proxy
React or Next.js storefront	renderJavaScript: true
Low-cost static crawl	renderJavaScript: false
General RAG	outputFormat: "Markdown", chunkSizeTokens: 300–500
Structured downstream processing	outputFormat: "JSON"

Important considerations

Use this Actor only on websites and data that you are authorized to access. Follow the target website’s terms, robots policies, applicable privacy rules, and rate limits. Apify Proxy and browser rendering improve crawl reliability, but no scraper can guarantee access to every anti-bot protected website.

Product layouts and review widgets vary between websites. The Actor uses multiple extraction strategies and continues after page-level failures, but highly customized stores may require site-specific selectors or configuration.

Local development

npm install
npm run check
npm start

At least one valid HTTP or HTTPS startUrls entry is required.