Competitor Intelligence & Price Monitor

An Apify Actor that extracts structured product data, prices, availability, and promotions from public web pages. Track changes across repeated runs for competitor analysis.

Features

• Multi-source extraction: JSON-LD, OpenGraph, and semantic HTML parsing
• Change detection: Compare results across runs to identify price drops, availability changes, and promotions
• Keyword matching: Filter and tag pages based on relevant keywords
• Structured output: Clean JSON dataset ready for analysis or AI consumption
• Respectful crawling: Configurable request delays, duplicate URL handling, graceful error handling

Usage

Input Schema

Field	Type	Description	Default
startUrls	Array<{url: string}>	Required. List of public web page URLs to analyze	-
keywords	string[]	Optional keywords to match against page content	[]
maxPages	number	Maximum pages to process (0 = unlimited)	100
requestDelayMs	number	Delay between requests in ms (higher = gentler)	1000
extractMode	string	Extraction aggressiveness: auto, strict, or lenient	auto
proxyConfig	object	Proxy configuration (optional)	{}

Output Schema

Each record in the dataset contains:

Field	Type	Description
sourceUrl	string	URL that was requested
finalUrl	string	Final URL after redirects
domain	string	Source website domain
productName	string	Product name extracted from page
price	number	Current price
currency	string	ISO 4217 currency code
availability	string	in_stock, out_of_stock, preorder, or unknown
originalPrice	number	List/original price before discounts
discount	number	Discount percentage (0-100)
promotionText	string	Promotional text found on page
description	string	Product description
imageUrl	string	Main product image URL
keywordsMatched	string[]	Input keywords that matched
scrapedAt	string	ISO 8601 timestamp
changeDetected	boolean	Whether changes detected vs. previous run
changes	array	List of detected changes
extractionMethod	string	How data was extracted
error	string	Error message if extraction failed

Change Detection

The Actor stores historical data and compares new results. Changes are detected for:

• Price - Price increases or decreases
• Availability - Stock status changes
• Product name - Title/name changes
• Description - Description text changes
• Promotion text - Promotional messaging changes

For AI Agent Consumption

Output is designed for easy AI agent consumption:

• All fields are JSON-compatible
• changeDetected and changes fields enable "what changed?" queries
• availability field identifies out-of-stock items
• discount field identifies promotions
• keywordsMatched helps filter relevant results

Example queries AI agents can answer:

• "What changed since last run?" → Filter by changeDetected: true
• "Which competitor lowered price?" → Look for changes with field: 'price' and newValue < oldValue
• "Which products are unavailable?" → Filter by availability: 'out_of_stock'
• "Which competitors have promotions?" → Filter by discount > 0

Limitations

• Only extracts from publicly accessible pages
• JavaScript-heavy pages may need lenient mode or browser-based crawler
• Change detection requires previous runs against same URLs
• Extraction accuracy depends on page structure
• Does not bypass authentication, CAPTCHAs, or anti-bot measures

Compliance

This Actor only collects publicly accessible information. Users are responsible for:

• Complying with applicable website terms of service
• Respecting robots.txt where applicable
• Not exceeding reasonable request rates (use requestDelayMs)
• Complying with applicable data protection laws
• Not using extracted data for unauthorized purposes

Development

Install Dependencies

$npm install

Build

$npm run build

Run Tests

$node --test tests/test-utils.js tests/test-input-validation.js tests/test-change-detection.js

Run Locally (with Apify CLI)

# Set up input.json with your test URLs
apify run

Monetization

This Actor uses the Pay-per-event (PPE) pricing model:

• Primary event: product-analyzed - charged per successfully processed page
• Always pushes data to dataset BEFORE charging (critical for user trust)

License

ISC