# Sitemap & Robots.txt Auditor (`zaden/my-actor-2`) Actor

Checks robots.txt and XML sitemaps for a site: crawlability issues, sitemap validity, and broken sitemap URLs. Fast, no browser needed.

- **URL**: https://apify.com/zaden/my-actor-2.md
- **Developed by:** [Zaden](https://apify.com/zaden) (community)
- **Categories:** SEO tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $4.00 / 1,000 site auditeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Sitemap & Robots.txt Auditor

Checks whether search engines can actually crawl and index your site. Fetches `robots.txt`, follows every declared sitemap (including sitemap indexes), and spot-checks a sample of the URLs inside them -- no browser needed. Get back a **letter grade**, a 0-100 crawlability score, a severity tag, and a plain-English summary.

### What you get, per site

- A **letter grade (A+ to F)** and 0-100 crawlability score at a glance
- A one-line **executive summary** ("robots.txt blocks all crawlers -- this site will not be indexed.")
- A **severity** tag -- `critical` (robots.txt blocks all crawlers, no usable sitemap could be found, or every sampled URL is broken), `warning` (other issues), or `ok` -- for instant triage
- **robots.txt status**: found or missing, whether it blocks all crawlers, whether it declares any sitemaps
- **Sitemap discovery**: every sitemap referenced in `robots.txt` (falls back to `/sitemap.xml`), following sitemap index files to their child sitemaps
- **Sitemap validity**: well-formed XML, `<urlset>` vs `<sitemapindex>`, total URL count, presence of `<lastmod>` dates
- **Broken URL sampling**: a configurable number of sitemap URLs are checked live, concurrently, for a working response, so you catch sitemaps full of 404s or redirects
- Sites are audited **in parallel** (configurable), so a batch of 20 client sites finishes in a fraction of the time a one-by-one audit would take

### Why this instead of checking manually

| | Manual spot-checks | Sitemap & Robots.txt Auditor |
|---|---|---|
| Coverage | Whatever site you remember to test | Every site you list, automatically |
| Speed | Open robots.txt, open sitemap.xml, click a few links | Seconds, via parallel audits and sampling |
| Triage | You read the raw files yourself | Pre-classified severity + letter grade |
| Repeatability | Forgotten until rankings drop | One click, or schedule it |
| Output | Notes in a doc | Structured dataset, exportable to CSV/JSON/BI tools |

### Who this is for

- SEO consultants and agencies auditing client sites
- Developers verifying a new sitemap/robots.txt setup before launch
- Anyone who wants to know if their sitemap is stale, broken, or simply not being read by crawlers

### How to use it

1. Add one or more **sites to audit** (e.g. `https://example.com`).
2. Optionally tune how many sitemap URLs get spot-checked, how many child sitemaps to follow, and **concurrency** -- these control run time and cost on very large sites.
3. Run it once, or schedule it to catch regressions after deploys.

### Example output (one item)

```json
{
  "site": "https://example.com",
  "score": 100,
  "grade": "A+",
  "severity": "ok",
  "summary": "No issues found -- robots.txt and sitemap look healthy.",
  "robotsFound": true,
  "robotsDisallowsAll": false,
  "sitemapsDeclaredInRobots": ["https://example.com/sitemap.xml"],
  "sitemapsChecked": ["https://example.com/sitemap.xml"],
  "totalUrlsInSitemaps": 342,
  "urlsSampled": 15,
  "brokenSampledUrls": [],
  "issueCount": 0,
  "issues": [],
  "checkedAt": "2026-08-22T00:00:00.000Z"
}
```

### Pricing

This Actor uses pay-per-event pricing:

- **Site audited** -- charged once per site processed, regardless of how many sitemaps or sampled URLs it involved.

Parallel auditing doesn't cost more -- it just finishes faster.

### FAQ

**Does this crawl my whole site?**
No. It only reads `robots.txt` and the XML sitemap files it references (plus, optionally, a sample of the individual URLs listed inside them). It never follows on-page links.

**What if my sitemap is huge?**
Sitemap index files are followed up to the "max child sitemaps" limit you set, and URL sampling is capped by "URLs to spot-check" -- both are configurable so you can trade thoroughness for speed and cost on very large sites.

**What makes something "critical" vs. a "warning"?**
`critical` covers the failures that mean a site effectively can't be indexed: robots.txt blocking all crawlers, no usable sitemap found at all, or every sampled sitemap URL failing. `warning` covers everything else that lowers the score -- worth fixing, but not indexing-breaking.

**Does it respect robots.txt itself while auditing?**
This Actor reads `robots.txt` as data to audit it, and only ever fetches the sitemap files and sample URLs it explicitly discovers -- it does not perform a general crawl.

**Will higher concurrency get me rate-limited?**
It can, on servers with aggressive rate limiting, though each site is a different host so cross-site concurrency is usually gentle. Start at the default (5) and lower it if you see timeouts spike.

# Actor input Schema

## `startUrls` (type: `array`):

One or more site URLs (e.g. https://example.com). robots.txt and any referenced sitemaps are fetched from each site's origin.

## `maxUrlsToSample` (type: `integer`):

How many URLs listed in the sitemap(s) to check with a live request to detect broken/dead entries. Set to 0 to skip spot-checking. Checked concurrently within each site.

## `maxChildSitemaps` (type: `integer`):

If robots.txt or the sitemap points to a sitemap index, how many child sitemap files to fetch and aggregate (largest sites split into many files).

## `requestTimeoutSecs` (type: `integer`):

How long to wait for each HTTP request before giving up.

## `concurrency` (type: `integer`):

How many sites to audit in parallel. Higher values finish faster at the same cost; lower values are gentler if you're auditing many sites on the same shared host.

## Actor input object example

```json
{
  "startUrls": [
    "https://example.com"
  ],
  "maxUrlsToSample": 15,
  "maxChildSitemaps": 5,
  "requestTimeoutSecs": 15,
  "concurrency": 5
}
```

# Actor output Schema

## `results` (type: `string`):

Robots.txt and sitemap audit findings for each site.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        "https://example.com"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("zaden/my-actor-2").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "startUrls": ["https://example.com"] }

# Run the Actor and wait for it to finish
run = client.actor("zaden/my-actor-2").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    "https://example.com"
  ]
}' |
apify call zaden/my-actor-2 --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,zaden/my-actor-2"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/YiW6yW5oaWyWcpQHl/builds/wdhX173AKAZWPcnZ3/openapi.json