Competitor Content Monitor

Competitor Content Monitor watches competitor websites for new and changed content, extracts the topics they are publishing about, and turns those signals into practical content ideas.

Use it for competitor blogs, newsrooms, resource hubs, guides, case studies, changelogs, product pages, pricing pages, and other public website content.

What you get

• New and changed competitor content across repeated runs
• Blog posts, guides, news, resources, case studies, updates, and optional marketing pages
• Title, description, author, publish date, modified date, image, summary, and word count
• Automatically extracted keywords and recurring themes
• Ready-to-use content ideas without requiring an AI API key
• Optional AI-enhanced competitive analysis using an OpenAI-compatible API
• RSS/Atom feed and XML sitemap discovery
• Persistent monitoring state for scheduled runs
• JSON, CSV, Excel, XML, RSS, and other standard Apify Dataset exports
• JSON run summary and a readable Markdown report
• Optional webhook delivery for workflow automation

Quick start

1. Add the competitor URLs you want to monitor.
2. Run the Actor once to create a baseline.
3. Schedule the same Actor task to run daily, weekly, or at any interval you prefer.
4. Later runs identify content as new, changed, or unchanged.

The default input is immediately runnable and scans two public example blogs so you can see the output without configuring anything.

Monitoring behavior

The default Smart monitoring mode is designed for recurring automation:

• First run: returns the detected baseline content.
• Later runs: returns only new or changed content.

The Actor stores a compact content fingerprint for each monitored page in a persistent key-value store. Navigation, footer, scripts, styles, and other common page chrome are excluded from the main content fingerprint where possible, reducing noisy change alerts.

Status values

Status	Meaning
baseline	Content found while establishing a fresh monitoring baseline
new	A page not present in the previous baseline
changed	A previously known page whose meaningful content fingerprint changed
unchanged	A known page whose fingerprint did not change

For changed pages, changeReasons identifies changes such as the title, description, publication date, modification date, or body content.

Content discovery

The Actor combines several discovery methods:

• Links found on the supplied competitor pages
• RSS and Atom feeds advertised by the website
• XML sitemaps found through robots.txt
• The common /sitemap.xml location

URLs are deduplicated before crawling. Query parameters can be removed automatically to avoid tracking URLs and crawl traps.

Content scope

Editorial content

Best for monitoring:

• Blog posts
• Articles
• News
• Guides
• Resources
• Insights
• Case studies
• Customer stories
• Changelogs and updates

This is the default.

Editorial + marketing pages

Also monitors substantial pages such as:

• Product pages
• Feature pages
• Pricing pages
• Solutions
• Enterprise pages
• Service pages

All substantial pages

Keeps any sufficiently substantial public page discovered within the configured crawl limits.

Content ideas

Idea generation works without an external AI service. The Actor identifies important topics in competitor content and creates original angles such as:

• Practical guides
• Checklists
• Comparison frameworks
• Common mistakes
• Benchmarks
• Beginner explainers
• Trend and future-focused pieces
• Gaps and unanswered questions

The goal is competitive inspiration, not copying competitor titles or text.

Each dataset item contains ideaSuggestions, while the run summary contains cross-competitor overallIdeas and topThemes.

Optional AI enhancement

Enable Use AI enhancement to add one AI analysis call per run. This keeps cost predictable while providing deeper cross-competitor analysis.

You can use OpenRouter or another OpenAI-compatible chat-completions endpoint by providing:

• aiApiKey
• aiBaseUrl
• aiModel

The API key is configured as a secret Actor input and is not written to the Dataset, summary, or report.

If the AI request fails, the monitoring run still completes and the built-in keyword, theme, and idea generation remains available.

Webhook automation

Set webhookUrl to send the completed monitoring summary as JSON to an HTTPS endpoint.

This can connect the Actor to tools such as:

• Make
• Zapier
• n8n
• Slack/Discord bridge workflows
• Custom dashboards
• Internal automation APIs

A webhook failure does not discard successfully collected monitoring results.

Inputs

Input	Default	Purpose
competitorUrls	Webflow Blog + Zapier Blog	Websites or content sections to monitor
contentScope	editorial	Choose editorial, marketing, or all substantial pages
maxPagesPerCompetitor	30	Crawl budget for each competitor
maxDepth	2	Maximum link depth from each supplied URL
includeSubdomains	false	Include matching subdomains
discoverFeeds	true	Find content through RSS/Atom feeds
discoverSitemaps	true	Find content through XML sitemaps
respectRobotsTxt	true	Respect website crawling rules
stripQueryParameters	true	Remove query parameters from discovered pages
includePatterns	empty	Optional wildcard allow-list
excludePatterns	common non-content paths	Optional wildcard block-list
outputMode	smart	Baseline first, changes later
monitorId	automatic	Optional stable state identifier
resetState	false	Start a fresh baseline
generateIdeas	true	Create deterministic content opportunities
ideasPerPage	3	Ideas attached to each item
overallIdeas	10	Cross-competitor ideas in the summary
useAiEnhancement	false	Add optional AI analysis
webhookUrl	empty	Send summary to an HTTPS webhook
maxConcurrency	6	Simultaneous requests
requestTimeoutSecs	30	Per-request timeout
maxRequestRetries	2	Retries for temporary failures
proxyConfiguration	direct connection	Optional Apify Proxy configuration

URL patterns

Use * as a wildcard.

Examples:

*/blog/*
*/resources/*
*/guides/*

Exclude examples:

*/careers/*
*/events/*
*/partners/*

When includePatterns is empty, the Actor uses automatic content detection.

Dataset output

A typical record looks like:

{
  "competitor": "Example",
  "domain": "example.com",
  "url": "https://example.com/blog/automation-guide",
  "canonicalUrl": "https://example.com/blog/automation-guide",
  "sourceUrl": "https://example.com/blog",
  "contentType": "article",
  "status": "new",
  "title": "Automation Guide",
  "description": "A practical guide to automation.",
  "author": "Jane Smith",
  "publishedDate": "2026-09-18T08:00:00.000Z",
  "modifiedDate": "",
  "summary": "A practical guide covering automation strategy...",
  "keywords": ["automation", "workflow", "operations"],
  "ideaSuggestions": [
    "A practical guide to Automation and Workflow: what to evaluate before getting started"
  ],
  "changeReasons": [],
  "imageUrl": "https://example.com/image.jpg",
  "wordCount": 1420,
  "depth": 1,
  "contentHash": "...",
  "firstSeenAt": "2026-09-18T10:00:00.000Z",
  "lastSeenAt": "2026-09-18T10:00:00.000Z",
  "scrapedAt": "2026-09-18T10:00:00.000Z"
}

The Dataset includes three views:

• Content monitor — status, competitor, content, topics, and changes
• Content ideas — summaries, themes, and content opportunities
• Technical details — hashes, timestamps, crawl depth, and source information

Summary and report

Every successful run creates:

• SUMMARY — machine-readable JSON with statistics, competitor counts, themes, ideas, AI results, and webhook status
• REPORT.md — readable monitoring report for quick review

These are available from the Actor run's Output tab.

Using schedules effectively

For recurring monitoring, create an Apify Task with your preferred competitor list and schedule that Task. Keeping the same input preserves the same automatic monitoring identity.

Use monitorId when you want to explicitly control the identity of a monitor. This is useful when you expect to adjust URLs or crawl settings but want the run to continue using the same saved baseline.

Use resetState: true whenever you intentionally want to rebuild the baseline from scratch.

Reliability and safety

The Actor includes safeguards intended for public website monitoring:

• Public HTTP/HTTPS targets only
• Localhost and private-network targets blocked
• DNS resolution checked for private/non-routable addresses
• Redirect targets revalidated
• Embedded URL credentials blocked
• Response-size limits
• Crawl page/depth limits
• URL deduplication
• Retry handling for temporary network/server failures
• robots.txt support enabled by default
• Secret handling for AI keys and webhook URLs

Limitations

The Actor uses fast HTTP-based extraction and does not execute page JavaScript in a browser. Websites whose meaningful content exists only after client-side rendering may provide limited results. Login-protected or private content is not supported.

Content classification and idea generation are automated signals. Review important competitive decisions against the original source pages.