# Competitor Content Monitoring Bot (`coolinbex/competitor-content-monitor`) Actor

Monitor competitor websites for new and changed content. Track blog posts, guides, landing pages, themes, keywords, and ready-to-use content ideas with optional AI analysis.

- **URL**: https://apify.com/coolinbex/competitor-content-monitor.md
- **Developed by:** [coolinbex](https://apify.com/coolinbex) (community)
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $7.00 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Competitor Content Monitor

**Competitor Content Monitor** watches competitor websites for new and changed content, extracts the topics they are publishing about, and turns those signals into practical content ideas.

Use it for competitor blogs, newsrooms, resource hubs, guides, case studies, changelogs, product pages, pricing pages, and other public website content.

### What you get

- New and changed competitor content across repeated runs
- Blog posts, guides, news, resources, case studies, updates, and optional marketing pages
- Title, description, author, publish date, modified date, image, summary, and word count
- Automatically extracted keywords and recurring themes
- Ready-to-use content ideas without requiring an AI API key
- Optional AI-enhanced competitive analysis using an OpenAI-compatible API
- RSS/Atom feed and XML sitemap discovery
- Persistent monitoring state for scheduled runs
- JSON, CSV, Excel, XML, RSS, and other standard Apify Dataset exports
- JSON run summary and a readable Markdown report
- Optional webhook delivery for workflow automation

### Quick start

1. Add the competitor URLs you want to monitor.
2. Run the Actor once to create a baseline.
3. Schedule the same Actor task to run daily, weekly, or at any interval you prefer.
4. Later runs identify content as **new**, **changed**, or **unchanged**.

The default input is immediately runnable and scans two public example blogs so you can see the output without configuring anything.

### Monitoring behavior

The default **Smart monitoring** mode is designed for recurring automation:

- **First run:** returns the detected baseline content.
- **Later runs:** returns only new or changed content.

The Actor stores a compact content fingerprint for each monitored page in a persistent key-value store. Navigation, footer, scripts, styles, and other common page chrome are excluded from the main content fingerprint where possible, reducing noisy change alerts.

#### Status values

| Status | Meaning |
| --- | --- |
| `baseline` | Content found while establishing a fresh monitoring baseline |
| `new` | A page not present in the previous baseline |
| `changed` | A previously known page whose meaningful content fingerprint changed |
| `unchanged` | A known page whose fingerprint did not change |

For changed pages, `changeReasons` identifies changes such as the title, description, publication date, modification date, or body content.

### Content discovery

The Actor combines several discovery methods:

- Links found on the supplied competitor pages
- RSS and Atom feeds advertised by the website
- XML sitemaps found through `robots.txt`
- The common `/sitemap.xml` location

URLs are deduplicated before crawling. Query parameters can be removed automatically to avoid tracking URLs and crawl traps.

### Content scope

#### Editorial content

Best for monitoring:

- Blog posts
- Articles
- News
- Guides
- Resources
- Insights
- Case studies
- Customer stories
- Changelogs and updates

This is the default.

#### Editorial + marketing pages

Also monitors substantial pages such as:

- Product pages
- Feature pages
- Pricing pages
- Solutions
- Enterprise pages
- Service pages

#### All substantial pages

Keeps any sufficiently substantial public page discovered within the configured crawl limits.

### Content ideas

Idea generation works without an external AI service. The Actor identifies important topics in competitor content and creates original angles such as:

- Practical guides
- Checklists
- Comparison frameworks
- Common mistakes
- Benchmarks
- Beginner explainers
- Trend and future-focused pieces
- Gaps and unanswered questions

The goal is competitive inspiration, not copying competitor titles or text.

Each dataset item contains `ideaSuggestions`, while the run summary contains cross-competitor `overallIdeas` and `topThemes`.

### Optional AI enhancement

Enable **Use AI enhancement** to add one AI analysis call per run. This keeps cost predictable while providing deeper cross-competitor analysis.

You can use OpenRouter or another OpenAI-compatible chat-completions endpoint by providing:

- `aiApiKey`
- `aiBaseUrl`
- `aiModel`

The API key is configured as a secret Actor input and is not written to the Dataset, summary, or report.

If the AI request fails, the monitoring run still completes and the built-in keyword, theme, and idea generation remains available.

### Webhook automation

Set `webhookUrl` to send the completed monitoring summary as JSON to an HTTPS endpoint.

This can connect the Actor to tools such as:

- Make
- Zapier
- n8n
- Slack/Discord bridge workflows
- Custom dashboards
- Internal automation APIs

A webhook failure does not discard successfully collected monitoring results.

### Inputs

| Input | Default | Purpose |
| --- | --- | --- |
| `competitorUrls` | Webflow Blog + Zapier Blog | Websites or content sections to monitor |
| `contentScope` | `editorial` | Choose editorial, marketing, or all substantial pages |
| `maxPagesPerCompetitor` | `30` | Crawl budget for each competitor |
| `maxDepth` | `2` | Maximum link depth from each supplied URL |
| `includeSubdomains` | `false` | Include matching subdomains |
| `discoverFeeds` | `true` | Find content through RSS/Atom feeds |
| `discoverSitemaps` | `true` | Find content through XML sitemaps |
| `respectRobotsTxt` | `true` | Respect website crawling rules |
| `stripQueryParameters` | `true` | Remove query parameters from discovered pages |
| `includePatterns` | empty | Optional wildcard allow-list |
| `excludePatterns` | common non-content paths | Optional wildcard block-list |
| `outputMode` | `smart` | Baseline first, changes later |
| `monitorId` | automatic | Optional stable state identifier |
| `resetState` | `false` | Start a fresh baseline |
| `generateIdeas` | `true` | Create deterministic content opportunities |
| `ideasPerPage` | `3` | Ideas attached to each item |
| `overallIdeas` | `10` | Cross-competitor ideas in the summary |
| `useAiEnhancement` | `false` | Add optional AI analysis |
| `webhookUrl` | empty | Send summary to an HTTPS webhook |
| `maxConcurrency` | `6` | Simultaneous requests |
| `requestTimeoutSecs` | `30` | Per-request timeout |
| `maxRequestRetries` | `2` | Retries for temporary failures |
| `proxyConfiguration` | direct connection | Optional Apify Proxy configuration |

### URL patterns

Use `*` as a wildcard.

Examples:

```text
*/blog/*
*/resources/*
*/guides/*
```

Exclude examples:

```text
*/careers/*
*/events/*
*/partners/*
```

When `includePatterns` is empty, the Actor uses automatic content detection.

### Dataset output

A typical record looks like:

```json
{
  "competitor": "Example",
  "domain": "example.com",
  "url": "https://example.com/blog/automation-guide",
  "canonicalUrl": "https://example.com/blog/automation-guide",
  "sourceUrl": "https://example.com/blog",
  "contentType": "article",
  "status": "new",
  "title": "Automation Guide",
  "description": "A practical guide to automation.",
  "author": "Jane Smith",
  "publishedDate": "2026-09-18T08:00:00.000Z",
  "modifiedDate": "",
  "summary": "A practical guide covering automation strategy...",
  "keywords": ["automation", "workflow", "operations"],
  "ideaSuggestions": [
    "A practical guide to Automation and Workflow: what to evaluate before getting started"
  ],
  "changeReasons": [],
  "imageUrl": "https://example.com/image.jpg",
  "wordCount": 1420,
  "depth": 1,
  "contentHash": "...",
  "firstSeenAt": "2026-09-18T10:00:00.000Z",
  "lastSeenAt": "2026-09-18T10:00:00.000Z",
  "scrapedAt": "2026-09-18T10:00:00.000Z"
}
```

The Dataset includes three views:

- **Content monitor** — status, competitor, content, topics, and changes
- **Content ideas** — summaries, themes, and content opportunities
- **Technical details** — hashes, timestamps, crawl depth, and source information

### Summary and report

Every successful run creates:

- `SUMMARY` — machine-readable JSON with statistics, competitor counts, themes, ideas, AI results, and webhook status
- `REPORT.md` — readable monitoring report for quick review

These are available from the Actor run's Output tab.

### Using schedules effectively

For recurring monitoring, create an Apify Task with your preferred competitor list and schedule that Task. Keeping the same input preserves the same automatic monitoring identity.

Use `monitorId` when you want to explicitly control the identity of a monitor. This is useful when you expect to adjust URLs or crawl settings but want the run to continue using the same saved baseline.

Use `resetState: true` whenever you intentionally want to rebuild the baseline from scratch.

### Reliability and safety

The Actor includes safeguards intended for public website monitoring:

- Public HTTP/HTTPS targets only
- Localhost and private-network targets blocked
- DNS resolution checked for private/non-routable addresses
- Redirect targets revalidated
- Embedded URL credentials blocked
- Response-size limits
- Crawl page/depth limits
- URL deduplication
- Retry handling for temporary network/server failures
- `robots.txt` support enabled by default
- Secret handling for AI keys and webhook URLs

### Limitations

The Actor uses fast HTTP-based extraction and does not execute page JavaScript in a browser. Websites whose meaningful content exists only after client-side rendering may provide limited results. Login-protected or private content is not supported.

Content classification and idea generation are automated signals. Review important competitive decisions against the original source pages.

# Changelog

This Actor's version history is a separate document: https://apify.com/coolinbex/competitor-content-monitor/changelog.md

# Actor input Schema

## `competitorUrls` (type: `array`):

Website, blog, news, resources, changelog, or content hub URLs to monitor. Add one URL per competitor or content section.

## `contentScope` (type: `string`):

Editorial focuses on blogs, news, guides, resources, case studies, and updates. Marketing also includes product, feature, pricing, and solution pages. All keeps any substantial public page.

## `maxPagesPerCompetitor` (type: `integer`):

Maximum number of normal website pages fetched for each competitor in one run. Feed and sitemap files do not count toward this limit.

## `maxDepth` (type: `integer`):

Maximum number of link levels followed from each competitor URL. A value of 0 scans only the supplied URLs plus optional feed and sitemap discovery.

## `includeSubdomains` (type: `boolean`):

Allow discovered pages on subdomains of each competitor domain.

## `discoverFeeds` (type: `boolean`):

Discover RSS or Atom feeds linked from competitor pages and use them to find recent content efficiently.

## `discoverSitemaps` (type: `boolean`):

Check robots.txt and common sitemap locations for additional relevant content URLs.

## `respectRobotsTxt` (type: `boolean`):

Skip URLs disallowed for the CompetitorContentMonitor user agent by each site's robots.txt rules.

## `stripQueryParameters` (type: `boolean`):

Remove query strings from discovered page URLs to reduce duplicates, tracking parameters, and crawl traps.

## `includePatterns` (type: `array`):

Optional wildcard patterns that discovered page URLs must match, for example */blog/* or */resources/*. Leave empty to use automatic content detection.

## `excludePatterns` (type: `array`):

Wildcard URL patterns to skip. Useful for login, account, legal, search, cart, and other irrelevant sections.

## `outputMode` (type: `string`):

Smart returns the complete baseline on the first run and only new or changed content later. All always returns every monitored content page. Changes only emits only new or changed pages after a baseline exists.

## `monitorId` (type: `string`):

Optional stable identifier for this monitoring setup. Leave blank to automatically derive state from the competitor URLs and crawl configuration.

## `resetState` (type: `boolean`):

Discard the saved baseline for this monitor and treat the current run as a fresh first scan.

## `generateIdeas` (type: `boolean`):

Generate practical content angles from discovered competitor themes without requiring an external AI API key.

## `ideasPerPage` (type: `integer`):

Maximum number of content ideas attached to each emitted page.

## `overallIdeas` (type: `integer`):

Maximum number of cross-competitor content opportunities included in the run summary.

## `useAiEnhancement` (type: `boolean`):

Optionally enhance the run summary with an OpenAI-compatible chat completion API. The Actor still works fully without AI.

## `aiApiKey` (type: `string`):

API key for the configured OpenAI-compatible endpoint. Required only when AI enhancement is enabled.

## `aiBaseUrl` (type: `string`):

Base URL of an OpenAI-compatible API. The Actor calls the /chat/completions endpoint.

## `aiModel` (type: `string`):

Model identifier accepted by your AI provider.

## `webhookUrl` (type: `string`):

Optional HTTPS endpoint that receives the completed monitoring summary. Useful for Slack bridges, Make, Zapier, n8n, or custom automations.

## `maxConcurrency` (type: `integer`):

Maximum number of pages fetched at the same time. Lower this value for sensitive or rate-limited sites.

## `requestTimeoutSecs` (type: `integer`):

Maximum time allowed for each HTTP request.

## `maxRequestRetries` (type: `integer`):

Number of retry attempts for temporary network, rate-limit, or server errors.

## `maxResponseBytes` (type: `integer`):

Maximum HTML, XML, or text response size accepted per page to protect memory usage.

## `proxyConfiguration` (type: `object`):

Optional Apify Proxy configuration. Direct requests are used by default and are sufficient for most public competitor sites.

## Actor input object example

```json
{
  "competitorUrls": [
    "https://webflow.com/blog",
    "https://zapier.com/blog"
  ],
  "contentScope": "editorial",
  "maxPagesPerCompetitor": 30,
  "maxDepth": 2,
  "includeSubdomains": false,
  "discoverFeeds": true,
  "discoverSitemaps": true,
  "respectRobotsTxt": true,
  "stripQueryParameters": true,
  "includePatterns": [],
  "excludePatterns": [
    "*/login*",
    "*/signup*",
    "*/account*",
    "*/privacy*",
    "*/terms*",
    "*/search*",
    "*/cart*",
    "*/checkout*"
  ],
  "outputMode": "smart",
  "monitorId": "",
  "resetState": false,
  "generateIdeas": true,
  "ideasPerPage": 3,
  "overallIdeas": 10,
  "useAiEnhancement": false,
  "aiBaseUrl": "https://openrouter.ai/api/v1",
  "aiModel": "openai/gpt-4.1-mini",
  "maxConcurrency": 6,
  "requestTimeoutSecs": 30,
  "maxRequestRetries": 2,
  "maxResponseBytes": 5000000,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}
```

# Actor output Schema

## `results` (type: `string`):

Structured monitored content and change records in the default dataset.

## `summary` (type: `string`):

JSON summary with run statistics, themes, overall ideas, and competitor-level counts.

## `report` (type: `string`):

Markdown report containing the most important changes, themes, and content opportunities.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {};

// Run the Actor and wait for it to finish
const run = await client.actor("coolinbex/competitor-content-monitor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {}

# Run the Actor and wait for it to finish
run = client.actor("coolinbex/competitor-content-monitor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{}' |
apify call coolinbex/competitor-content-monitor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,coolinbex/competitor-content-monitor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/oGTcDZHSopg7Sdwrz/builds/jlB66hf5xWtklclQW/openapi.json