# Monitor robots.txt changes to prevent site de-indexing

**Use case:** 

Track shifts in crawl directives, user-agent rules, and disallow flags. Prevent staging errors from blocking search bots and tanking organic rankings.

## Input

```json
{
  "robotsUrl": "https://www.google.com/robots.txt",
  "followRedirects": true,
  "timeoutMs": 10000
}
```

## Output

```json
{
  "baseline": {
    "label": "Baseline Status",
    "format": "boolean"
  },
  "unchanged": {
    "label": "Unchanged Flag",
    "format": "boolean"
  },
  "changes.type": {
    "label": "Modification Type",
    "format": "text"
  },
  "changes.severity": {
    "label": "Severity",
    "format": "text"
  },
  "changes.description": {
    "label": "Change Details",
    "format": "text"
  },
  "summary.critical": {
    "label": "Critical Alert Count",
    "format": "integer"
  },
  "summary.warning": {
    "label": "Warning",
    "format": "integer"
  }
}
```

## About this Actor

This example demonstrates how to use [Robots.txt Monitor](https://apify.com/datawinder/robots-txt-monitor) with a specific input configuration. Visit the [Actor detail page](https://apify.com/datawinder/robots-txt-monitor) to learn more, explore other use cases, and run it yourself.