Amazon Commerce Intelligence — Products, Reviews & Price Monitoring

Publisher: OptiRefine
Actor slug: amazon-commerce-intelligence
Release: 0.1.0 (tested starter implementation, not a verified live-Amazon release)
Runtime: Node.js 22, no npm dependencies

Observe publicly accessible Amazon product information, search cards and reviews across multiple marketplaces. Produce normalized datasets with ASIN identity, provenance, visibility/completeness metadata, buyer-facing prices, featured sellers and incremental price snapshots. Access-control failures generate explicit machine-readable error records instead of silently returning empty data.

Testing status: Node.js parser and workflow tests pass using curated HTML fixtures and local mock APIs. This project has not been tested against live Amazon pages, Apify's actual cloud deployment, real Apify Proxy traffic, or live pay-per-event billing. Amazon markup and access controls change frequently. Review the release blockers before charging customers.

Capabilities in v0.1.0

Mode	What this build does	Caveats
product	Extracts title, brand, ASIN, currency/price, list price, stock text, rating, review count, features, breadcrumbs, visible variants, main image, Buy Box seller.	Fields missing in public HTML stay null/empty; no browser rendering.
search	Keywords and Amazon search URLs, result card ASIN/title/price/rating, limited pagination; optional detail enrichment.	Public search HTML only; results, ranks and sponsorship labels depend on markup.
category	Takes Amazon category URLs or numeric browse-node IDs and parses public result cards.	No recursive category tree traversal.
reviews	Public review-page records, visible local filters, reviewer-PII minimization, visibility summary.	May yield zero reviews if sign-in is required; not all reviews.
monitor	Compares product observations against baselineDatasetId; emits changed product rows and price_snapshot rows.	Does not recreate historical prices before the first observation.
bestsellers	Can accept ranking/category URLs and attempt normal public result-card parsing.	Dedicated chart-specific ranking parser is not implemented.

All recordType values use schemaVersion: "1.0": product, search_result, review, review_coverage, offer, price_snapshot, error.

Known feature gaps relative to the source research: no Crawlee Request Queue or persistent in-run resume; no Playwright fallback; no multi-offer expansion; no actual delivery-postcode setting; no full cross-locale review-date support; no official SP-API/Creators API provider; no seller-page crawl; no advanced adaptive concurrency or 1,000-target live performance benchmark. Variants/featured offers rely on visible product HTML. Plan a second release for these before describing the Actor as feature-complete.

Install or deploy on Apify

1. Extract the ZIP into a local project directory.
2. Install Apify CLI separately if needed: npm install -g apify-cli.
3. Sign in using apify login, then run apify push from the extracted project folder. This uploads the Actor source and lets Apify build from the included Dockerfile. Alternatively connect a GitHub repo containing the extracted project as an Actor source.
4. Open the Actor in Apify Console, verify the Input and Output tabs, and run a tiny test using one ASIN and maxRequests: 5 before any broader workloads.
5. Examine the default Dataset rows and key-value store record OUTPUT. Verify with human eyes that actual price, rating, ASIN, locale and source coverage are correct, and note any ROBOTS_DISALLOWED, ACCESS_CHALLENGE or LOGIN_REQUIRED rows.
6. Do not publish a monetized Store listing until successful live runs and billing tests have been confirmed.

The Apify platform supplies the Actor token and default storage IDs automatically. Do not paste an Apify token into the Actor input or commit one to source.

Example: product details (Canada)

{
  "mode": "product",
  "marketplace": "CA",
  "asins": ["B08N5WRWNW"],
  "includeReviews": false,
  "includeOffers": true,
  "maxRequests": 5,
  "maxRequestsPerMinute": 20,
  "robotsPolicy": "respect",
  "privacyMode": "minimal"
}

Example: keyword search

{
  "mode": "search",
  "marketplace": "US",
  "keywords": ["usb-c charger"],
  "maxPages": 2,
  "maxProducts": 30,
  "includeProductDetails": false,
  "maxRequests": 50,
  "maxRequestsPerMinute": 20
}

Example: public reviews

{
  "mode": "reviews",
  "marketplace": "US",
  "asins": ["B08N5WRWNW"],
  "reviewRating": "all",
  "maxReviewsPerAsin": 20,
  "maxPages": 2,
  "privacyMode": "minimal"
}

The review limit is a maximum, not a guarantee: Amazon can limit what is publicly viewable. If data is not exposed publicly, expect a review_coverage record with a limitation or an access error, rather than fabricated reviews.

Example: price changes / history

1. Run product mode to create a baseline dataset; retain its ID.
2. Run monitor with the same ASINs and baselineDatasetId set to that dataset ID:

{
  "mode": "monitor",
  "marketplace": "US",
  "asins": ["B08N5WRWNW"],
  "baselineDatasetId": "REPLACE_WITH_PREVIOUS_DATASET_ID",
  "maxRequests": 10
}

3. For ongoing monitoring, use Apify Schedules and update the baseline dataset ID or implement a durable baseline manager. Each changed observation is recorded as a price_snapshot, not an inferred historical price. When nothing changed, the summary reports unchanged even though the dataset may be empty.

Important: A run without baselineDatasetId treats observed products as new. This MVP does not update its own baseline ID across scheduled runs.

Source access, privacy and behavior

• Only exact, allowlisted https:// Amazon marketplace domains are requested. Redirects to outside hosts are rejected. The included marketplace map covers US, CA, UK, DE, FR, IT, ES, JP, IN, AU, MX, BR, NL, PL, SE, SG, AE, SA and TR.
• Requests identify themselves as Agent/OptiRefineAmazonCommerceIntelligence in the User-Agent header.
• robotsPolicy: "respect" is the default. Robots rules are fetched for the relevant marketplace and the Actor fails closed when they cannot be read. Some product and review pages may therefore be inaccessible in default mode.
• robotsPolicy: "report_only" requires authorizedAccessConfirmed: true and is intended only for separately authorized access. Changing the setting is not evidence of legal permission and does not bypass access challenges.
• CAPTCHA, 403, 429, 503 and login-only conditions are classified, not bypassed. No CAPTCHA solver, stealth/human simulation, credential scraping or login flow is included.
• Reviewer display names and profile links are excluded by default (privacyMode: "minimal"). full allows these fields when publicly available. You are responsible for a lawful basis, retention and downstream privacy controls.
• Review rating, date, verified status and keyword filters apply to records actually observed, not to an imagined complete Amazon review archive. Some locales may not expose enough information to reliably apply every filter.
• deliveryLocation is only recorded as an annotation in price snapshots. It does not manipulate the retail website's shipping location.
• Optional proxyConfiguration accepts Apify Proxy for geo-routing. This uses an HTTP CONNECT tunnel; real residential proxy operation has not been verified here. Proxy charges accrue on the Apify user's plan and are separate from Actor event fees.

Optional Apify Proxy object

{
  "useApifyProxy": true,
  "apifyProxyGroups": ["RESIDENTIAL"],
  "apifyProxyCountry": "CA"
}

Use proxy settings only where allowed by the relevant service terms and actual authorization; this Actor does not rotate proxies around blocks or CAPTCHA challenges.

Dataset outputs

Each dataset row has recordType and schemaVersion to support downstream ingestion. Source metadata includes the public URL, observed timestamp, page type, HTTP status, locale, and access method. Products include a completeness object. Review runs emit review_coverage for every successfully parsed first review page. Errors include a target, stable code, category, retryability, HTTP status and captured timestamp. A run summary lives in the default key-value store under OUTPUT.

Example error codes: INVALID_INPUT, UNSUPPORTED_URL, UNSUPPORTED_MARKETPLACE, PRODUCT_NOT_FOUND, PAGE_LAYOUT_UNRECOGNIZED, ACCESS_CHALLENGE, LOGIN_REQUIRED, RATE_LIMITED, ROBOTS_DISALLOWED, ROBOTS_UNAVAILABLE, NETWORK_TIMEOUT, PROXY_ERROR, PARSE_PARTIAL, HTTP_ERROR, BILLING_ERROR.

The Console shows one Results and diagnostics dataset view for all row types. Filter by recordType in exports. This version does not expose separate physical datasets for reviews, sellers and errors.

Monetization setup (optional, not enabled out of the box)

This Actor includes an Apify pay-per-event charging hook, but charging is off until the owner configures it. To avoid unbudgeted billing and surprises, take these steps before making any listing paid:

1. In Apify Console, choose Pay Per Event and configure these exact event names. Example launch prices (USD, suggestions only):

   Event	Example USD / event	Typical record
   search_result	$0.0015	One search result card
   product_detail	$0.0060	One product record
   review	$0.0030	One review record
   offer	$0.0010	One visible featured offer
   history_snapshot	$0.0010	One changed observation

2. In the Actor owner's environment variables (not customer input), set PPE_ENABLED=1 and the JSON string below, keeping it synchronized with the actual Console event prices:

   PPE_EVENT_PRICES_JSON={"search_result":0.0015,"product_detail":0.006,"review":0.003,"offer":0.001,"history_snapshot":0.001}
   

3. The Actor uses Apify's ACTOR_MAX_TOTAL_CHARGE_USD run limit to prevent explicitly configured charges from going over the user's budget. Unknown event prices fail billing rather than charging at an unverified price. Verify a tiny paid test run and your revenue/usage dashboard before publishing.

4. Prices are examples, not profit guarantees. Benchmark Amazon access availability and proxy/network usage before setting final rates. The Actor may produce no paid records on blocked or robots-disallowed paths.

Billing caveat: Direct Apify REST charging and storage writes are not an atomic transaction. If a storage or billing call fails in the middle of a run, inspect the run and its event log before retrying. A fully production-hardened Actor should use Apify's SDK charging manager and persistent per-record checkpoints.

Local tests (offline, no Amazon traffic)

Node.js 22+ is required. There are no npm dependencies to install.

$npm test

The Node.js built-in test runner exercises the HTML parsers, currency formats, privacy, robots rules, marketplace validation, redirects, pagination, CAPTCHA classification, per-target errors, price changes, local dataset storage, a simulated Apify REST API, and the actual src/main.js entrypoint with mocked public pages. Mocked HTML is illustrative and is not proof of current live Amazon compatibility.

To run locally against live pages (subject to robots rules and terms), edit storage/key_value_stores/default/INPUT.json and execute:

$node src/main.js

Inspect storage/datasets/default/*.json and storage/key_value_stores/default/OUTPUT.json. Delete/recreate local default dataset storage between distinct runs to avoid confusing old records with current ones.

Before publishing to Apify Store

This is a tested MVP, not yet a feature-complete Store launch:

• Node.js syntax and offline automated tests
• Real local dataset/KV storage, mocked platform REST API, true entrypoint smoke test
• Product/search/review/monitor parsing and error classification on fixtures
• CAPTCHA-detection and robots respect on fixtures
• Successful actual Apify build from this Dockerfile
• Successful, authorized live Amazon US and CA product/search/review tests with manual accuracy verification
• Live Apify Proxy testing where authorized
• Live PPE event/budget/retry verification in a sandbox-priced Actor
• Live locale HTML fixtures for all advertised marketplaces
• Request-queue restart/resume and large-scale cost benchmark
• Full implementation of all advanced features in the research specification

Amazon's automated-agent rules require agent identification and prohibit circumventing CAPTCHA and similar access measures. This project follows those guardrails but that alone does not establish permission to extract or redistribute the content. The customer and Actor owner must review Amazon's terms and applicable laws before deployment.

Files

• src/main.js — Apify/local entrypoint
• src/actor.js — mode router, workflows, normalization, incremental diffs and errors
• src/parsers.js — product, search and public-review parsers
• src/robots.js — robots policy and challenge-safe behavior
• src/network.js — HTTP client, allowlisted redirects and optional proxy tunnel
• src/storage.js — local files and Apify REST storage/event adapters
• src/html.js — dependency-free basic HTML tree parser
• src/marketplaces.js — marketplace definitions and URL validation
• .actor/ — input/output/dataset schemas and deployment metadata
• tests/ — fixture-based unit/integration tests
• examples/ — starter inputs

References

• Apify Actor setup: https://docs.apify.com/actors/development/quick-start/locally
• Apify input schema: https://docs.apify.com/actors/development/actor-definition/input-schema/specification/v1
• Apify output schema: https://docs.apify.com/actors/development/actor-definition/output-schema
• Apify charge API: https://docs.apify.com/api/v2/post-charge-run
• Amazon Agent Policy: https://sellercentral.amazon.com/help/hub/reference/external/GS83KH2MA7HM69PH

No affiliation or endorsement by Amazon or Apify is implied.