🕵️ Smart Competitor Monitor (AI Self-Healing)

Stop fixing broken CSS selectors. The Smart Competitor Monitor uses Generative AI to track competitor websites, adapting automatically when layouts change. It filters out the noise and only alerts you on business-critical updates.

⚡ Why use this Actor?

Traditional scrapers are brittle. If a competitor changes a class name from .price-bold to .price-v2, your scraper breaks. Visual monitors are noisy; they alert you when a pixel moves or a timestamp updates.

This Actor is different:

1. 🧠 Self-Healing: It doesn't use hardcoded selectors. It generates extraction logic on the fly using Gemini. If the site changes, the Actor "re-architects" itself automatically.
2. 🤫 Zero Noise: It understands meaning. It ignores rotating testimonials or minor layout shifts and only reports changes that match your goal (e.g., "Price Increased").
3. 🗣️ Natural Language Setup: Just tell it what to track in plain English (e.g., "Track changes to the Enterprise Plan features").

🚀 Features

This actor is in beta release. There could be unexpected problems. In that case please open an issue and we will make sure to solve it asap.

• Semantic Schema Generation: No point-and-click. The AI analyzes the page and builds a schema based on your prompt.
• Smart Wait™ Technology: Intelligently waits for dynamic content (SPAs/React) by observing DOM mutations and probing for specific keywords before scraping.
• Significance Grading: The AI Analyst grades every change as Critical, Notable, or Insignificant so you can prioritize.
• Cost-Optimized: Uses Google's lowest Gemini model for high speed and low cost analysis.
• Webhook Ready: Perfect for integrating with n8n, Zapier, or Make to send alerts to Slack/Teams.

🛠️ How It Works

The Actor operates in a 3-Phase Cycle to ensure accuracy and efficiency:

1. Phase 1: The Architect 🏗️
   • Triggers on first run or website structure mismatch.
   • Analyzes the website structure and builds a custom extraction plan based on your User Goal.
2. Phase 2: The Worker 👷
   • Triggers on every run.
   • Uses the cached plan to extract data quickly. Compares data against the previous run using our advanced algorithm.
3. Phase 3: The Analyst 🧐
   • Triggers only when a change is detected.
   • Reviews the "Before" vs. "After" data. Decides if the change is relevant to your business goal and generates a human-readable summary, lists out old and new content and analyzes the impact these changes could give your competitors business and some suggestions for you.

📥 Input Configuration

Field	Type	Description	Example
run_name	String	Unique name for this run	"SaaS Pricing Tracker"
urls	Array	List of competitor URLs to monitor	https://competitor.com/pricing
prompt	String	What specifically should the AI look for?	"Notify me of any price increases"
email	String	Email address to receive alerts	alerts@company.com
proxyConfiguration	Object	Select proxies for advanced scraping with anti bot detection etc... (Defaults to Apify Proxy)	{ "useApifyProxy": true }
gemini_api_key	String	(Optional) Your Gemini API Key for cost savings	AIzaSy...
alert_webhook_url	String	(Optional) Custom URL for Change Alerts	https://n8n.webhook/alert
first_run_webhook_url	String	(Optional) Custom URL for First Run Summary	https://n8n.webhook/summary

📤 Output Data (Webhooks)

While this Actor stores data in the Apify Dataset, its primary power lies in real-time integrations- sent data anywhere. The Actor sends a POST request to any URL you provide (n8n, Zapier, Make, or your custom API) containing the structured analysis.

There are two types of events sent to your endpoint:

1. 🚨 Batched Competitor Alert (Run Complete)

Sent once at the end of the run. Contains a list of all relevant changes detected across all monitored URLs in a single run.

{  
  "type": "batch_competitor_alert",  
  "run_name": "SaaS Pricing Tracker",  
  "email": "alerts@company.com",  
  "timestamp": "2023-10-27T10:05:00.000Z",  
  "change_count": 2,
  "changes": [
    {
      "url": "https://competitor.com/pricing",
      "run_name": "SaaS Pricing Tracker",
      "timestamp": "2023-10-27T10:01:00.000Z",
      "analysis": {
        "is_relevant": true,  
        "significance": "high",  
        "observation": "The Enterprise plan price increased from $99 to $129.",  
        "changes": [  
          ["OLD: Pro: $99 values", "NEW: Pro: $129"]
        ],  
        "impact": "Competitor is moving upmarket; opportunity to undercut."
      }
    }
  ]  
}

2. ✨ First Run Summary (Monitoring Started)

Sent when new URLs are successfully analyzed for the first time, establishing your baseline.

{  
  "type": "first_run_summary",  
  "run_name": "SaaS Pricing Tracker",  
  "email": "alerts@company.com",  
  "timestamp": "2023-10-27T10:00:00.000Z",  
  "input_config": {  
    "urls": ["https://competitor.com/pricing"],  
    "prompt": "Track pricing and features",  
    "is_schedule": false,  
    "schedule_duration": null  
  },  
  "results": [  
    {  
      "url": "https://competitor.com/pricing",  
      "size_bytes": 10240  
    }  
  ]  
}

💰 Cost & Pricing

We aim for a transparent, pay-as-you-go pricing model that scales with your usage.

Total Cost = Actor Fee + AI Fee (Optional) + Apify Platform Usage

Cost Component	Price	Notes
Base Fee	$0.01	Per Actor run.
Usage Fee	$0.003	Per URL analyzed.
AI Usage Fee	$0.03 / URL	FREE if you provide your own Google Gemini API Key.
Apify Platform	Standard	Based on Compute Units (CU) consumed. Incured by Apify based on your plan.

• Pro Tip: Add your own Gemini API Key in the input configuration to waive the "AI Intelligence" fee completely!

Important Note

⚠️ Critical: Why you should use your own API Key

We charge a premium ($0.03/URL) for the built-in AI because managing Google's strict rate limits requires complex infrastructure. We strongly recommend providing your own Gemini API Key. The Risks of using the built-in key:

1. High Failure Rate: Due to strict usage limits, there is a >75% chance of the model overloading or blocking requests during peak times.
2. Data Corruption: If the analysis fails mid-stream, it may generate "garbage" content, potentially breaking your historical analysis baseline.
3. Financial Risk: If the AI fails, you will NOT be charged the $0.03 AI fee, BUT you will still be charged for the Apify Platform and Actor Usage fees. 🛡️ Security: Your API Key is injected directly into the secure container environment. It never leaves your Apify account or this Actor instance.

👉 Instructions to get your free Gemini API Key

🔌 Integrations & Webhooks

This Actor is built with an "Integration First" philosophy. It is designed to act as a trigger for your external automation workflows.

You can pipe the data to n8n, Zapier, Make, Slack, Discord, or your own custom API endpoints.

Recommended Setup:

1. Create an Endpoint: Set up a webhook receiver (e.g., an n8n webhook node or a custom API route) that accepts POST requests.
2. Configure Actor: Paste this URL into the Webhook URL field in the Actor's input.
3. Schedule: Set up an Apify Schedule to run the monitor (e.g., Daily at 9 AM).
4. Automate:
   • When a High Significance change is found, the Actor POSTs the JSON payload to your URL.
   • Your system can then process the request—send a Slack DM, update a Google Sheet, or trigger a marketing campaign.

❓ FAQ

Q: Do I need to know programming to use this?
A: Not at all. The setup works entirely with Natural Language. If you can write a sentence like "Watch for price changes", you can use this Actor.
Q: Can I monitor any website?
A: Yes. The Actor is capable of navigating complex websites, including those that require JavaScript rendering or have dynamic layouts.
Q: How do I stop getting spam/irrelevant alerts?
A: This is the core feature of our AI. Unlike other tools that alert on visual changes (pixels), this Actor alerts on meaning. If you ask for "Pricing changes," it will ignore typo fixes, footer updates, or rotating testimonials automatically.
Q: What happens if I don't provide an API Key?
A: The Actor will work perfectly using our built-in AI engine. A small surcharge ($0.03 per URL) will be applied to cover the intelligence costs.
Q: Can I track multiple competitors in one run?
A: Yes! You can add as many URLs as you like to the list. The Actor will process them and send individual alerts to your webhook for each relevant change detected. But the alert is sent per run not per url so that your inbox or your endpoint wont be spammed by our alerts. But it is recommended to set maximum 25 urls per run. Otherwise you should use large RAM size for your run. Also having too many urls may lead to your api key being blocked. We are working on how to fix this problem. This is because of google's strict usage limits.

🌍 Beyond Competitor Analysis: Universal Use Cases

Because this Actor uses Generative AI to understand the content of a page, it is perfect for any scenario where you need to filter out noise and track specific concepts.

🏛️ 1. Regulatory & Compliance Automation

The Problem: Government websites often change their "Last Updated" timestamp without changing the actual regulations. Traditional scrapers trigger false alarms constantly.
The Solution: Tell the AI to look for material changes in the text.

• Target: https://gdpr-info.eu/ or Government Policy Pages
• Prompt: "Monitor this page for any changes to data retention requirements or new compliance deadlines. Ignore minor phrasing updates."

💼 2. Recruitment & Headhunting Intelligence

The Problem: You want to know when a specific company is hiring for a specific role, but you don't want an alert for every new intern job post.
The Solution: Filter job postings by seniority or tech stack.

• Target: https://competitor.com/careers
• Prompt: "Notify me only if they post a new 'Senior' or 'Staff' engineer role related to AI or Machine Learning."

🗞️ 3. PR & Crisis Monitoring

The Problem: You need to know how the media is portraying a specific topic, but news homepages are dynamic and messy.
The Solution: Track specific narratives or sentiment shifts.

• Target: https://techcrunch.com/ or Industry Blogs
• Prompt: "Scan the headlines for any mentions of [Your Industry] and summarize if the sentiment is positive or negative."

🛍️ 4. Smart Inventory & Price Arbitrage

The Problem: E-commerce sites constantly change "Recommended Products" or "Flash Sales," breaking visual monitors.
The Solution: Focus strictly on the "Add to Cart" button or Price tag.

• Target: https://retailer.com/product/ps5-pro
• Prompt: "Check if the product status changes from 'Out of Stock' to 'In Stock'. Ignore changes to the 'People also bought' section."

🛡️ 5. Security & Bug Bounty Hunting

The Problem: Companies silently update their security.txt or terms of service to patch vulnerabilities or change liability clauses.
The Solution: Detect subtle legal language shifts.

• Target: https://example.com/.well-known/security.txt
• Prompt: "Track changes to the 'Disclosure Policy' or 'Safe Harbor' terms."

📉 6. Investment Research (Alpha Generation)

The Problem: Financial reports are long and complex.
The Solution: Extract specific financial health indicators.

• Target: https://ir.tesla.com/press-releases
• Prompt: "Monitor for new press releases mentioning 'Q4 Earnings', 'Guidance', or 'Deliveries'. Summary key numbers."

🛡️ AI for Social Good: The "Digital Watchdog" Initiative

While the commercial application of this Actor is "Competitor Monitoring," its social relevance lies in its ability to enforce Transparency and Accountability at scale.

In a "Social Problem Solver" context, this tool functions as an Automated Truth Defense System. Here is how it scores 100/100 on social impact:

🛑 1. Combating Misinformation & "Stealth Editing" (Score: 100/100)

The Problem: News outlets, blogs, and political figures often "stealth edit" articles or statements after publication to change the narrative without accountability. They might quietly remove a controversial quote or change a headline to be less incriminating.
How This Actor Solves It:

• The Watchdog: It creates an immutable record of the narrative.
• The Prompt: "Monitor this article for any changes to the quotes attributed to [Politician Name] or changes to the headline's sentiment."
• Social Impact: It holds media and public figures accountable to their original words, preventing "gaslighting" of the public record.

⚖️ 2. Consumer Protection: Anti-Price Gouging Defense (Score: 100/100)

The Problem: During crises (pandemics, natural disasters), bad actors artificially inflate prices of essential goods (masks, water, baby formula). Regulators cannot manually check millions of product pages every hour.
How This Actor Solves It::

• The Watchdog: NGOs and Regulators can deploy thousands of instances to monitor essential goods across e-commerce platforms.
• The Prompt: "Alert IMMEDIATELY if the price of [Essential Item] exceeds 20% of the baseline average."
• Social Impact: It protects vulnerable populations from exploitation during emergencies by providing regulators with real-time evidence of gouging.

🏛️ 3. Digital Democracy: Government Transparency (Score: 95/100)

The Problem: Governments sometimes quietly alter policy documents, eligibility requirements for welfare, or immigration rules without public announcement. Vulnerable citizens are often left unaware of rights they have lost.
How This Actor Solves It::

• The Watchdog: Civil Rights groups can monitor thousands of government URLs simultaneously.
• The Prompt: "Track changes to the 'Eligibility' section. Flag if any new requirements (like ID proofs or income limits) are added."
• Social Impact: It ensures that democracy happens in the light, preventing "bureaucratic violence" where rules are changed in the dark.

🌿 4. Corporate Accountability: Greenwashing Detector (Score: 90/100)

The Problem: Companies often make bold environmental claims on their "Sustainability" pages but quietly dilute them later. (e.g., changing "We commit to zero carbon by 2030" to "We aim to reduce carbon...").
How This Actor Solves It::

• The Watchdog: Environmental watchdogs can track the specific language of ESG commitments.
• The Prompt: "Monitor the 'Sustainability Commitments' section. Alert me if specific dates, hard numbers, or definitive promises are removed or softened."
• Social Impact: It prevents corporations from using "future promises" as marketing while quietly walking them back when no one is looking.

🔐 5. Digital Rights: Privacy Policy & ToS Surveillance (Score: 95/100)

The Problem: "I have read and agree to the Terms." No one reads them. Companies frequently update these legal contracts to strip users of rights (e.g., adding forced arbitration clauses or data selling rights) without users noticing.
How This Actor Solves It::

• The Watchdog: Privacy advocacy groups can maintain a "ToS Watch" list for major platforms.
• The Prompt: "Analyze changes to the 'Data Sharing' or 'Third Party Partners' section. Alert if they add new categories of data they are allowed to sell."
• Social Impact: It restores the balance of power between tech giants and individual users by making contract changes transparent.