# Seo Content Drift Penalty Radar (`muhammad-bilal/seo-content-drift-penalty-radar`) Actor

An Apify Actor that monitors SEO-critical web pages over time and detects content changes that explain ranking drops, such as content removal, heading drift, keyword dilution, and CTA loss.

- **URL**: https://apify.com/muhammad-bilal/seo-content-drift-penalty-radar.md
- **Developed by:** [Muhammad Bilal](https://apify.com/muhammad-bilal) (community)
- **Categories:** SEO tools, Automation, AI
- **Stats:** 5 total users, 1 monthly users, 100.0% runs succeeded, 2 bookmarks
- **User rating**: 5.00 out of 5 stars

## Pricing

from $6.00 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md):

```bash
npm install apify-client
```

In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md):

```bash
pip install apify-client
```

In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md):

````bash
# MacOS / Linux
curl -fsSL https://apify.com/install-cli.sh | bash
# Windows
irm https://apify.com/install-cli.ps1 | iex
```bash

In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md).

If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).


# README

## 🕵️ SEO Content Drift & Penalty Radar

**Enterprise-grade SEO monitoring system for detecting content changes that explain ranking drops.**

[![Apify SDK](https://img.shields.io/badge/Apify-SDK%20v3-green)](https://sdk.apify.com)
[![Crawlee](https://img.shields.io/badge/Crawlee-v3-blue)](https://crawlee.dev)
[![Node](https://img.shields.io/badge/Node-%3E%3D20-brightgreen)](https://nodejs.org)
[![TypeScript](https://img.shields.io/badge/TypeScript-5.0-blue)](https://www.typescriptlang.org/)

### 🎯 Overview

SEO Content Drift & Penalty Radar is a production-grade Apify Actor that monitors SEO-critical web pages over time and detects content changes that explain ranking drops. Built with enterprise security, scalability, and extensibility in mind.

#### Key Capabilities

- ✅ **SEO Risk Classification** - LOW/MEDIUM/HIGH risk assessment based on content changes
- ✅ **Content Change Detection** - Monitors text additions, removals, and modifications
- ✅ **SEO Element Analysis** - Tracks heading structure, keyword density, CTAs, and link profiles
- ✅ **Optional AI Analysis** - LLM-powered impact assessment and recommendations (OpenAI-compatible)
- ✅ **Configurable Monitoring** - Focus on specific keywords and SEO-critical elements
- ✅ **Backward Compatible** - Works as simple monitor or advanced intelligence system
- ✅ **Cloud-Safe** - No hardcoded secrets, graceful failures, input validation

---
### 🚨 Why SEO Content Drift & Penalty Radar?

SEO rankings can drop mysteriously — content updates, structural changes, or subtle modifications often go unnoticed until they impact search visibility.

SEO Content Drift & Penalty Radar automatically monitors webpages and detects:

📄 Content changes (text additions, removals, edits)

🧱 Structural changes (heading hierarchy, layout shifts)

🔍 SEO element changes (keyword dilution, CTA removal, link profile changes)

You get actionable SEO risk assessments, not raw HTML diffs.

### 🎯 Who is this for?

SEO teams monitoring ranking-critical pages

Digital marketing agencies tracking client sites

E-commerce teams watching product pages & category listings

Content teams ensuring SEO compliance

Enterprise SEO managers detecting penalty triggers

### ⚙️ How it works (3 steps)

Provide URLs of SEO-critical pages to monitor

Configure monitoring sensitivity and AI analysis

Run the Actor → receive structured SEO risk assessments

Each result includes:

SEO risk level (LOW/MEDIUM/HIGH)

Specific changes detected

AI impact analysis & recommendations

Timestamp & metadata

### 💰 Pricing example (transparent)

Monitoring 1,000 pages ≈ $0.20

Analyzing 1,000 changes ≈ $0.60

No monthly fees — pay only for what you use

### 🚀 Quick Start

#### Local Development

```bash
## Install dependencies
npm install

## Build TypeScript
npm run build

## Run Actor locally (preserves snapshots between runs)
npm start

## Or use Apify CLI (clears storage each run)
apify run

## Login to Apify platform
apify login

## Push to Apify cloud
apify push
````

#### Input Configuration

Create `.actor/INPUT.json` or `storage/key_value_stores/default/INPUT.json`:

```json
{
  "pageUrls": [
    "https://example.com/product-page",
    "https://example.com/blog/seo-critical-post"
  ],
  "enableAIExplanation": false,
  "focusKeywords": [
    "buy now",
    "contact us",
    "learn more"
  ]
}
```

***

### 📊 Output Format

Each monitored page produces structured JSON:

```json
{
  "url": "https://example.com",
  "seoRisk": "HIGH",
  "contentRemoved": true,
  "headingStructureChanged": false,
  "keywordDilution": true,
  "ctaRemoved": false,
  "linkProfileChanged": true,
  "aiSummary": "Significant content removal detected on product page...",
  "aiImpact": "High risk of ranking drop due to keyword dilution and content loss",
  "aiRecommendation": "Restore removed content immediately and monitor keyword density",
  "checkedAt": "2025-12-21T07:15:00.000Z"
}
```

#### Field Descriptions

| Field | Type | Description |
|-------|------|-------------|
| `url` | string | Monitored page URL |
| `seoRisk` | string | Classified SEO risk level: `LOW`, `MEDIUM`, `HIGH` |
| `contentRemoved` | boolean | Whether significant content was removed (>15% word loss) |
| `headingStructureChanged` | boolean | Whether heading structure (h1-h3) changed |
| `keywordDilution` | boolean | Whether keyword density dropped significantly |
| `ctaRemoved` | boolean | Whether call-to-action elements were removed |
| `linkProfileChanged` | boolean | Whether internal/external link profile changed |
| `aiSummary` | string|null | AI explanation summary (null if disabled) |
| `aiImpact` | string|null | AI predicted impact (null if disabled) |
| `aiRecommendation` | string|null | AI recommendation (null if disabled) |
| `checkedAt` | string | Timestamp of the check |

***

### ⚙️ Configuration Options

#### `pageUrls` (required)

Array of URLs to monitor. Each URL represents an SEO-critical page.

#### `enableAIExplanation` (default: false)

Enable AI-powered analysis for detected changes. Requires `OPENAI_API_KEY` environment variable.

#### `focusKeywords` (optional)

Array of primary keywords expected on the monitored pages. Used for keyword dilution analysis.

***

### 🔒 Security & Best Practices

#### API Keys

Never hardcode API keys. Use environment variables:

```bash
## Local development
export OPENAI_API_KEY="sk-..."

## Apify platform
## Set in Actor → Settings → Environment Variables
```

#### Input Validation

All inputs are validated:

- URLs are normalized and validated
- Arrays are checked for proper structure
- Missing fields have safe defaults

#### Graceful Failures

- Missing API keys → Warning + null AI results
- Network errors → Retry mechanism with backoff
- Malformed HTML → Logged + continues processing

***

### 🏗️ Architecture

#### Core Components

```
src/main.ts
├── Helper Functions
│   ├── crawlPage()          - Page fetching with CheerioCrawler
│   ├── extractSeoElements() - SEO element extraction
│   ├── normalizeSnapshot()  - Content normalization
│   ├── computeDiff()        - Change detection algorithm
│   ├── classifyRisk()       - SEO risk classification
│   └── getAIExplanation()   - LLM integration
│
└── Main Logic
    ├── Input validation
    ├── Page crawling loop
    ├── Change detection
    ├── Risk classification
    └── Dataset storage
```

#### Storage Strategy

**Key-Value Store** (`seo-snapshots`)

- Snapshot keys: `SNAPSHOT_{url_hash}`
- SEO element keys: `SEO_ELEMENTS_{url_hash}`
- Persistent across runs for historical comparison

**Dataset** (default)

- One record per monitored page
- Structured JSON with SEO risk assessment
- Overview view for easy inspection

***

### 🧪 Testing & Verification

#### Test SEO Risk Detection

```bash
## First run - establishes baseline
npm start

## Check output
cat storage/datasets/default/000000001.json
## Output: "seoRisk": "LOW" (first run baseline)

## Modify page content and run again
npm start

## Check output
cat storage/datasets/default/000000001.json
## Output: "seoRisk": "HIGH" (detected changes)
```

#### Test AI Analysis

```bash
export OPENAI_API_KEY="sk-..."
```

Update input:

```json
{
  "pageUrls": ["https://example.com"],
  "enableAIExplanation": true
}
```

#### Test Keyword Focus

Update input with focus keywords:

```json
{
  "pageUrls": ["https://example.com"],
  "focusKeywords": ["buy now", "contact us", "pricing"]
}
```

***

### 📈 Performance Characteristics

- **Memory**: ~50-100MB per 1000 pages
- **Speed**: ~30-60 pages/minute (network-dependent)
- **Storage**: ~2KB per page snapshot (includes SEO elements)
- **Scalability**: Handles 10,000+ pages efficiently

***

### 🔮 Future Enhancements

This Actor is designed as a foundational building block for:

- \[ ] **Historical SEO Tracking** - Time-series ranking correlation
- \[ ] **Alert System** - Webhooks for HIGH risk detections
- \[ ] **Google Search Console Integration** - Ranking change correlation
- \[ ] **Core Web Vitals Monitoring** - Performance impact analysis
- \[ ] **Competitor Monitoring** - Side-by-side SEO comparisons
- \[ ] **Custom SEO Rules** - XPath/CSS-based monitoring
- \[ ] **Multi-Agent Workflows** - Orchestration with other SEO tools

***

### 📚 Resources

- [Apify Documentation](https://docs.apify.com)
- [Apify SDK](https://sdk.apify.com)
- [Crawlee Documentation](https://crawlee.dev)
- [Actor Store](https://apify.com/store)

***

### 🎓 Technical Notes

#### Why CheerioCrawler?

- Lightweight (no browser overhead)
- Fast parsing for SEO analysis
- Sufficient for static HTML content
- Cost-effective at scale

#### Why TypeScript?

- Type safety for complex SEO logic
- Better IDE support and refactoring
- Self-documenting code with interfaces
- Production-ready reliability

#### Why Named KV Store?

- Persists between runs for historical comparison
- Enables SEO trend analysis over time
- Cloud-compatible storage
- Automatic cleanup policies

***

### 📜 License

This Actor follows Apify's standard terms of service.

***

### 🤝 Contributing

This Actor was built with extensibility in mind. Key extension points:

1. **Custom SEO extractors** - Modify `extractor/index.ts`
2. **Alternative risk models** - Update `classifier/index.ts`
3. **Additional LLM providers** - Modify `intelligence/index.ts`
4. **Custom change detection** - Update `diff/index.ts`

***

### 🏆 Enterprise-Grade Features

✅ Deterministic output\
✅ Structured and readable\
✅ No unnecessary dependencies\
✅ Reusable foundation\
✅ Code tells a story\
✅ Production-ready\
✅ Judge-friendly demo mode\
✅ Extensive documentation

***

**Built with ❤️ for the Apify ecosystem**

# Actor input Schema

## `pageUrls` (type: `array`):

SEO pages to monitor

## `enableAIExplanation` (type: `boolean`):

Enable AI explanation for changes

## `focusKeywords` (type: `array`):

Primary keywords expected on page

## Actor input object example

```json
{
  "pageUrls": [
    "https://fiverstack.com",
    "https://upwork.com/freelancers"
  ],
  "enableAIExplanation": false,
  "focusKeywords": [
    "freelance marketplace",
    "hire freelancers",
    "find work online"
  ]
}
```

# Actor output Schema

## `results` (type: `string`):

Dataset containing the SEO content drift analysis results

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {};

// Run the Actor and wait for it to finish
const run = await client.actor("muhammad-bilal/seo-content-drift-penalty-radar").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {}

# Run the Actor and wait for it to finish
run = client.actor("muhammad-bilal/seo-content-drift-penalty-radar").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"])
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{}' |
apify call muhammad-bilal/seo-content-drift-penalty-radar --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "command": "npx",
            "args": [
                "mcp-remote",
                "https://mcp.apify.com/?tools=muhammad-bilal/seo-content-drift-penalty-radar",
                "--header",
                "Authorization: Bearer <YOUR_API_TOKEN>"
            ]
        }
    }
}

```

## OpenAPI specification

```json
{
    "openapi": "3.0.1",
    "info": {
        "title": "Seo Content Drift Penalty Radar",
        "description": "An Apify Actor that monitors SEO-critical web pages over time and detects content changes that explain ranking drops, such as content removal, heading drift, keyword dilution, and CTA loss.",
        "version": "1.0",
        "x-build-id": "726cHQW07N28odZoE"
    },
    "servers": [
        {
            "url": "https://api.apify.com/v2"
        }
    ],
    "paths": {
        "/acts/muhammad-bilal~seo-content-drift-penalty-radar/run-sync-get-dataset-items": {
            "post": {
                "operationId": "run-sync-get-dataset-items-muhammad-bilal-seo-content-drift-penalty-radar",
                "x-openai-isConsequential": false,
                "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.",
                "tags": [
                    "Run Actor"
                ],
                "requestBody": {
                    "required": true,
                    "content": {
                        "application/json": {
                            "schema": {
                                "$ref": "#/components/schemas/inputSchema"
                            }
                        }
                    }
                },
                "parameters": [
                    {
                        "name": "token",
                        "in": "query",
                        "required": true,
                        "schema": {
                            "type": "string"
                        },
                        "description": "Enter your Apify token here"
                    }
                ],
                "responses": {
                    "200": {
                        "description": "OK"
                    }
                }
            }
        },
        "/acts/muhammad-bilal~seo-content-drift-penalty-radar/runs": {
            "post": {
                "operationId": "runs-sync-muhammad-bilal-seo-content-drift-penalty-radar",
                "x-openai-isConsequential": false,
                "summary": "Executes an Actor and returns information about the initiated run in response.",
                "tags": [
                    "Run Actor"
                ],
                "requestBody": {
                    "required": true,
                    "content": {
                        "application/json": {
                            "schema": {
                                "$ref": "#/components/schemas/inputSchema"
                            }
                        }
                    }
                },
                "parameters": [
                    {
                        "name": "token",
                        "in": "query",
                        "required": true,
                        "schema": {
                            "type": "string"
                        },
                        "description": "Enter your Apify token here"
                    }
                ],
                "responses": {
                    "200": {
                        "description": "OK",
                        "content": {
                            "application/json": {
                                "schema": {
                                    "$ref": "#/components/schemas/runsResponseSchema"
                                }
                            }
                        }
                    }
                }
            }
        },
        "/acts/muhammad-bilal~seo-content-drift-penalty-radar/run-sync": {
            "post": {
                "operationId": "run-sync-muhammad-bilal-seo-content-drift-penalty-radar",
                "x-openai-isConsequential": false,
                "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.",
                "tags": [
                    "Run Actor"
                ],
                "requestBody": {
                    "required": true,
                    "content": {
                        "application/json": {
                            "schema": {
                                "$ref": "#/components/schemas/inputSchema"
                            }
                        }
                    }
                },
                "parameters": [
                    {
                        "name": "token",
                        "in": "query",
                        "required": true,
                        "schema": {
                            "type": "string"
                        },
                        "description": "Enter your Apify token here"
                    }
                ],
                "responses": {
                    "200": {
                        "description": "OK"
                    }
                }
            }
        }
    },
    "components": {
        "schemas": {
            "inputSchema": {
                "type": "object",
                "required": [
                    "pageUrls"
                ],
                "properties": {
                    "pageUrls": {
                        "title": "Page URLs",
                        "type": "array",
                        "description": "SEO pages to monitor",
                        "items": {
                            "type": "string"
                        },
                        "default": [
                            "https://fiverstack.com",
                            "https://upwork.com/freelancers"
                        ]
                    },
                    "enableAIExplanation": {
                        "title": "Enable AI Explanation",
                        "type": "boolean",
                        "description": "Enable AI explanation for changes",
                        "default": false
                    },
                    "focusKeywords": {
                        "title": "Focus Keywords",
                        "type": "array",
                        "description": "Primary keywords expected on page",
                        "items": {
                            "type": "string"
                        },
                        "default": [
                            "freelance marketplace",
                            "hire freelancers",
                            "find work online"
                        ]
                    }
                }
            },
            "runsResponseSchema": {
                "type": "object",
                "properties": {
                    "data": {
                        "type": "object",
                        "properties": {
                            "id": {
                                "type": "string"
                            },
                            "actId": {
                                "type": "string"
                            },
                            "userId": {
                                "type": "string"
                            },
                            "startedAt": {
                                "type": "string",
                                "format": "date-time",
                                "example": "2025-01-08T00:00:00.000Z"
                            },
                            "finishedAt": {
                                "type": "string",
                                "format": "date-time",
                                "example": "2025-01-08T00:00:00.000Z"
                            },
                            "status": {
                                "type": "string",
                                "example": "READY"
                            },
                            "meta": {
                                "type": "object",
                                "properties": {
                                    "origin": {
                                        "type": "string",
                                        "example": "API"
                                    },
                                    "userAgent": {
                                        "type": "string"
                                    }
                                }
                            },
                            "stats": {
                                "type": "object",
                                "properties": {
                                    "inputBodyLen": {
                                        "type": "integer",
                                        "example": 2000
                                    },
                                    "rebootCount": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "restartCount": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "resurrectCount": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "computeUnits": {
                                        "type": "integer",
                                        "example": 0
                                    }
                                }
                            },
                            "options": {
                                "type": "object",
                                "properties": {
                                    "build": {
                                        "type": "string",
                                        "example": "latest"
                                    },
                                    "timeoutSecs": {
                                        "type": "integer",
                                        "example": 300
                                    },
                                    "memoryMbytes": {
                                        "type": "integer",
                                        "example": 1024
                                    },
                                    "diskMbytes": {
                                        "type": "integer",
                                        "example": 2048
                                    }
                                }
                            },
                            "buildId": {
                                "type": "string"
                            },
                            "defaultKeyValueStoreId": {
                                "type": "string"
                            },
                            "defaultDatasetId": {
                                "type": "string"
                            },
                            "defaultRequestQueueId": {
                                "type": "string"
                            },
                            "buildNumber": {
                                "type": "string",
                                "example": "1.0.0"
                            },
                            "containerUrl": {
                                "type": "string"
                            },
                            "usage": {
                                "type": "object",
                                "properties": {
                                    "ACTOR_COMPUTE_UNITS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATASET_READS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATASET_WRITES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "KEY_VALUE_STORE_READS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "KEY_VALUE_STORE_WRITES": {
                                        "type": "integer",
                                        "example": 1
                                    },
                                    "KEY_VALUE_STORE_LISTS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "REQUEST_QUEUE_READS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "REQUEST_QUEUE_WRITES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATA_TRANSFER_INTERNAL_GBYTES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATA_TRANSFER_EXTERNAL_GBYTES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "PROXY_RESIDENTIAL_TRANSFER_GBYTES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "PROXY_SERPS": {
                                        "type": "integer",
                                        "example": 0
                                    }
                                }
                            },
                            "usageTotalUsd": {
                                "type": "number",
                                "example": 0.00005
                            },
                            "usageUsd": {
                                "type": "object",
                                "properties": {
                                    "ACTOR_COMPUTE_UNITS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATASET_READS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATASET_WRITES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "KEY_VALUE_STORE_READS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "KEY_VALUE_STORE_WRITES": {
                                        "type": "number",
                                        "example": 0.00005
                                    },
                                    "KEY_VALUE_STORE_LISTS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "REQUEST_QUEUE_READS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "REQUEST_QUEUE_WRITES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATA_TRANSFER_INTERNAL_GBYTES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATA_TRANSFER_EXTERNAL_GBYTES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "PROXY_RESIDENTIAL_TRANSFER_GBYTES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "PROXY_SERPS": {
                                        "type": "integer",
                                        "example": 0
                                    }
                                }
                            }
                        }
                    }
                }
            }
        }
    }
}
```