# SiteVerdict — robots.txt & Sitemap Monitor (`siteverdict/siteverdict-robots-sitemap-monitor`) Actor

Catch SEO disasters before Google does: watches robots.txt, XML sitemaps and key pages for Disallow: /, noindex, canonical and status changes, sitemap URL drops and errors. Diffs vs last run, Slack/webhook alerts, HTML report.

- **URL**: https://apify.com/siteverdict/siteverdict-robots-sitemap-monitor.md
- **Developed by:** [Aman Thakran](https://apify.com/siteverdict) (community)
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

Pay per event

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## SiteVerdict — robots.txt & Sitemap Monitor

**Find out within the hour when a deploy blocks Googlebot, noindexes your best page or empties your sitemap.**

A single `Disallow: /` or a stray `noindex` left over from staging can remove a site from Google within days. This Actor watches robots.txt, your XML sitemaps and your key pages. It compares every run with the last one and alerts you in Slack or by webhook when something risky changes.

### Who it is for

- **SEO agencies.** Monitor every client site from one schedule and learn about a broken deploy before the client does.
- **In-house SEO and dev teams.** Use it as a post-deploy guard-rail for robots.txt, sitemaps, canonicals and noindex.
- **Site owners and e-commerce managers.** Get an alert when the sitemap loses half its products or a category page starts redirecting.

### What it checks

| Area | Checks |
|---|---|
| **robots.txt** | HTTP status (5xx = Google stops crawling), RFC 9309 rules, access for `*`, Googlebot, Bingbot and AI crawlers (GPTBot, Google-Extended, ClaudeBot, PerplexityBot, CCBot), blocked CSS/JS, size over 500 KiB, missing `Sitemap:` line, Crawl-delay, rule-level diff |
| **Sitemaps** | Discovery from robots.txt or /sitemap.xml, nested indexes, `.gz`, HTTP errors, HTML soft-404s, XML problems, namespace, 50,000-URL and 50 MB limits, URL count and drops, URLs added or removed, invalid or future lastmod, off-host and http:// URLs, duplicates |
| **Key URLs** | Status, redirects and final URL, noindex (meta robots, googlebot meta, X-Robots-Tag), canonical (tag or Link header), blocked by robots.txt for each search crawler |
| **Sitemap spot checks** | N URLs spread across the sitemap: 200, indexable, not redirected, self-canonical |

Each issue has a severity (critical, warning or info) and a plain-English fix. Each change shows its before and after values.

### Example: the alert you get

```json
{
  "site": "https://shop.example.com",
  "riskLevel": "critical",
  "summary": "5 change(s) since 2026-10-10 08:00: robots.txt now blocks the entire site for Googlebot. noindex appeared on https://shop.example.com/products/blue-shoe (meta robots). Sitemap URL count fell 50% (6 → 3).",
  "changes": [
    { "code": "robots-newly-blocks-all", "severity": "critical", "message": "robots.txt now blocks the entire site for Googlebot." },
    { "code": "key-noindex-added", "severity": "critical", "message": "noindex appeared on https://shop.example.com/products/blue-shoe (meta robots)." },
    { "code": "sitemap-count-drop", "severity": "critical", "message": "Sitemap URL count fell 50% (6 → 3)." }
  ],
  "robotsDiff": "+ User-agent: Googlebot\n+ Disallow: /"
}
```

Run with **Mode = Sample** to see this full output for free, without any network calls.

### How to use

1. Add your **sites** and your money pages under **Key URLs**.
2. Run it once to save the baseline.
3. Add an Apify **Schedule** (daily, or hourly with Output = "Only changes").
4. Optionally add a Slack or webhook URL and choose the alert threshold.

Each run writes:

- **Dataset:** one row per site. Views: Overview, Changes and alerts, Open issues and fixes, Crawler access.
- **Key-value store:** `REPORT` (HTML report), `REPORT_CSV` (one line per issue or change), `SUMMARY` (compact JSON for agents and integrations).
- **State:** the named store (default `siteverdict-robots-sitemap-monitor`) keeps each site's last snapshot and change history. Use one store name per client.

### Pricing

Pay per event: **$0.01 per site check**, **$0.001 per URL checked**, **$0.01 per delivered alert**, and **$0.02 per AI answer** if you turn that on. Sample mode is free. Every checked site is charged, including unchanged sites in "Only changes" mode.

- 1 site checked daily with 5 key URLs and 10 spot checks costs about **$0.75 a month**.
- 50 client sites checked daily cost about **$20 a month**.

See PRICING.md for details.

### Optional: AI fix suggestions (bring your own key)

Turn on **AI fix suggestions** and paste your own Anthropic or OpenAI key. For sites with warnings or critical issues, the findings are sent to your provider, never any personal data. You get a prioritised JSON list of fixes, each with an exact snippet (robots.txt lines, sitemap XML or HTML tags). Your key is stored encrypted and never logged or output. It is sent only to the provider you choose (Anthropic or OpenAI), and redirects are never followed, so it cannot be forwarded to another host.

### FAQ

**Does it crawl my whole site?** No. It reads robots.txt, the sitemap files and the key URLs, plus up to 100 spot-check URLs per site. Requests to one host are sequential, at most one per second, and slower if your robots.txt sets a Crawl-delay.

**What if robots.txt blocks the Actor itself?** The Actor obeys robots.txt for user-agent `SiteVerdict` (or `*`), so pages it may not fetch are reported as "Not fetched" and not charged. robots.txt access for Googlebot and others is evaluated from the rules alone; we never pretend to be Googlebot.

**Will a cosmetic robots.txt edit trigger an alert?** No. Comments, blank lines, whitespace and field-name case are ignored. Only real rule changes count.

**Can it check Search Console?** No. It uses public URLs only and needs no Google login.

### Run time, aborts and partial results

- **Run time** is mostly polite waiting: requests to one host are at most one per second (slower with a Crawl-delay), so one site with 5 key URLs and 10 spot checks usually takes well under a minute; a site with 50 large sitemap files takes several minutes. Up to 3 sites run in parallel by default. Memory does not make it faster.
- **Each site is saved as soon as it is done.** Its dataset row is pushed and its events are charged together (except in "Only changes" mode, see below), right after its checks, alert and AI step finish. If you abort the run or it times out, finished sites keep their rows and were charged; the site in progress has no row and is not charged (except a change alert that was already delivered), and sites not started yet are not charged.
- **"Only changes" mode:** a site that has not changed since the last run is still checked and charged (the site check and its URL checks), but it gets **no dataset row**. It still appears in `REPORT`, `REPORT_CSV` and `SUMMARY`. The first (baseline) run and every run with a change write a row as usual.
- **Saved only at the end:** the `REPORT`, `REPORT_CSV` and `SUMMARY` records are written when the run finishes, so an aborted run has none of them. The dataset rows are complete on their own.
- **Monitoring state after an abort:** a site's new snapshot is stored as soon as its checks finish, before its alert is sent. If the run is aborted in between, that change will not be alerted again next time; look at the aborted run's dataset (`changes`).
- **Partial success is normal:** a site that cannot be fetched, a blocked URL or one bad sitemap gives findings or an error row for that site; the run still succeeds for the others.

### Limits

- At most 50 sitemap files per site by default (configurable up to 500). The URL list stored for diffs is capped at 100,000 URLs per site (each sitemap file over the 50,000-URL protocol limit is reported as an error).
- Sites behind bot protection, a login or a CAPTCHA are reported as fetch errors. We never try to bypass them.
- JavaScript-injected meta robots or canonical tags are not seen, because pages are fetched as HTML without a browser.
- Allowed ports are 80, 443, 8080, 8443, 8000 and 8888. Private and internal addresses are refused.

### Sources and compliance

- **Data sources:** public robots.txt, sitemap and HTML responses of the sites you list. No third-party API and no personal data.
- **Standards:** robots.txt follows RFC 9309 (https://www.rfc-editor.org/rfc/rfc9309, checked 2026-10-11). Sitemaps follow the sitemaps.org protocol (https://www.sitemaps.org/protocol.html, checked 2026-10-11). Google limits (500 KiB robots.txt, 5xx handling) follow https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt (checked 2026-10-11).
- **Polite crawling:** honest User-Agent `SiteVerdict/robots-sitemap-monitor (+https://apify.com/siteverdict/siteverdict-robots-sitemap-monitor)`. robots.txt is respected on every redirect hop, and a 5xx robots.txt is treated as "disallow all". No proxy rotation and no fingerprinting.
- **Security:** SSRF guard on every connection and every redirect (private, loopback, link-local and metadata addresses are refused; the DNS answer is checked inside the connection itself), size caps and timeouts. Webhook and Slack URLs must be public `https://` addresses; unsafe ones are refused when the run starts, and delivery errors never repeat the URL or its host. Secrets (webhook URLs, AI key) are stored encrypted and redacted from logs and output.
- **UK GDPR / PECR:** the Actor processes no personal data. Only monitor sites you own or are authorised to monitor.

# Changelog

This Actor's version history is a separate document: https://apify.com/siteverdict/siteverdict-robots-sitemap-monitor/changelog.md

# Actor input Schema

## `executionMode` (type: `string`):

<b>Live</b> checks your sites. <b>Sample</b> replays two recorded days of a demo shop (a leaked 'Disallow: /' for Googlebot, a new noindex and a sitemap drop) so you can see the alert output for free, with no network calls.

## `sites` (type: `array`):

Homepages or any URL of each site; only the origin (scheme + host) is used. robots.txt and sitemaps are discovered automatically.

## `keyUrls` (type: `array`):

Important pages to watch for noindex, canonical, redirect, status and robots.txt-block changes, e.g. your homepage, top category and product pages. Their sites are added automatically.

## `sitemapUrls` (type: `array`):

Sitemaps to check in addition to those declared in robots.txt (or /sitemap.xml). .gz is supported.

## `sampleSitemapUrls` (type: `integer`):

This many URLs, spread across the sitemap, are fetched to check they return 200, are indexable, not redirected and self-canonical. 0 = off. Each is billed as a URL check.

## `maxSitemapFiles` (type: `integer`):

Safety cap on sitemap files read per site (sitemap indexes are followed up to 3 levels).

## `countDropWarnPct` (type: `integer`):

Warn when the number of sitemap URLs falls by at least this percentage since the last run.

## `countDropCriticalPct` (type: `integer`):

Critical alert when the sitemap URL count falls by at least this percentage.

## `outputMode` (type: `string`):

'All sites' writes one row per site every run. 'Only changes' writes a row only for the first (baseline) run and when something changed; ideal for hourly schedules.

## `alertMinSeverity` (type: `string`):

Send an alert when a change of at least this severity is detected. Critical = e.g. 'Disallow: /' for Googlebot, noindex on a key URL, sitemap 404, URL count down 50%+.

## `webhookUrl` (type: `string`):

We POST a JSON alert (site, changes, robots.txt diff) here, e.g. a Zapier/Make/n8n hook. Stored encrypted.

## `slackWebhookUrl` (type: `string`):

https://hooks.slack.com/services/... Stored encrypted.

## `stateStoreName` (type: `string`):

Named key-value store holding the last snapshot and change history of each site. Use one name per client/project.

## `historySize` (type: `integer`):

Past change events kept per site in the state store.

## `resetBaseline` (type: `boolean`):

Ignore saved snapshots and start fresh this run.

## `maxConcurrency` (type: `integer`):

Requests to the same host are always sequential, at most 1 per second (or the site's Crawl-delay).

## `timeoutSecs` (type: `integer`):

Per request.

## `aiEnabled` (type: `boolean`):

Off by default. When on, the findings (no personal data) are sent to your chosen LLM provider with YOUR API key to get concrete, prioritised fix snippets. Charged as one 'ai-fix' event per successful answer; your provider bills tokens separately.

## `aiProvider` (type: `string`):

Which API your key belongs to.

## `aiApiKey` (type: `string`):

Your Anthropic or OpenAI API key. Stored encrypted by Apify, never logged or written to the output.

## `aiModel` (type: `string`):

Leave empty for the default (Anthropic: claude-sonnet-5-5, OpenAI: gpt-5-mini).

## Actor input object example

```json
{
  "executionMode": "live",
  "sites": [
    {
      "url": "https://www.python.org/"
    }
  ],
  "keyUrls": [
    "https://www.python.org/downloads/"
  ],
  "sampleSitemapUrls": 10,
  "maxSitemapFiles": 50,
  "countDropWarnPct": 20,
  "countDropCriticalPct": 50,
  "outputMode": "all",
  "alertMinSeverity": "warning",
  "stateStoreName": "siteverdict-robots-sitemap-monitor",
  "historySize": 30,
  "resetBaseline": false,
  "maxConcurrency": 3,
  "timeoutSecs": 30,
  "aiEnabled": false,
  "aiProvider": "anthropic"
}
```

# Actor output Schema

## `results` (type: `string`):

No description

## `report` (type: `string`):

No description

## `summary` (type: `string`):

No description

## `files` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "sites": [
        {
            "url": "https://www.python.org/"
        }
    ],
    "keyUrls": [
        "https://www.python.org/downloads/"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("siteverdict/siteverdict-robots-sitemap-monitor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "sites": [{ "url": "https://www.python.org/" }],
    "keyUrls": ["https://www.python.org/downloads/"],
}

# Run the Actor and wait for it to finish
run = client.actor("siteverdict/siteverdict-robots-sitemap-monitor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "sites": [
    {
      "url": "https://www.python.org/"
    }
  ],
  "keyUrls": [
    "https://www.python.org/downloads/"
  ]
}' |
apify call siteverdict/siteverdict-robots-sitemap-monitor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,siteverdict/siteverdict-robots-sitemap-monitor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/U7nvcCRtqvyRU1Aza/builds/kt0Ecayhro9pRpRTH/openapi.json