# Robots.txt, Sitemap & Schema Validator + AI Crawler Check (`appropriate_songbird/structured-data-sitemap-validator`) Actor

Validate robots.txt, XML sitemaps and JSON-LD schema markup in one run and see which AI crawlers (GPTBot, ClaudeBot, Googlebot) are allowed. Scored JSON, from $0.02.

- **URL**: https://apify.com/appropriate_songbird/structured-data-sitemap-validator.md
- **Developed by:** [Puneet](https://apify.com/appropriate_songbird) (community)
- **Categories:** SEO tools, Developer tools, Marketing
- **Security:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

Pay per event

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Robots.txt, Sitemap & Schema Markup Validator (AI Crawler Check)

**Validate a site's robots.txt, XML sitemaps and JSON-LD schema markup in one run, and see whether GPTBot, ClaudeBot, PerplexityBot and Googlebot are allowed in. Scored JSON report, about $0.03 for a first site check, no subscription.**

**Try it in 10 seconds:** press *Start* with the prefilled input. It checks one site and 3 sample pages for about 3 cents. Then swap in your own home page.

Give the Actor the home page of a site you own (or are allowed to test). It reads `robots.txt`, finds and validates every sitemap (following sitemap indexes), then fetches a sample of pages and checks them for broken or incomplete structured data. You get one record per site and one per page, each with plain-language issue codes you can filter, diff or feed to CI.

No browser, no login, no API key, no third-party SEO database. **robots.txt is always obeyed** when fetching pages, and there is no switch to turn that off.

### What people use it for

- **Technical SEO and agencies**: a quick pre-launch or post-release check across client sites.
- **AI-visibility audits**: see in one table whether GPTBot, ClaudeBot, CCBot, PerplexityBot, Google-Extended and other crawlers are allowed, blocked or partly restricted, and from which rule group.
- **Site migrations**: confirm sitemaps list only live, indexable, canonical URLs.
- **E-commerce and publishers**: catch Product, Article, FAQ, Event, Recipe and Breadcrumb markup that lost required properties after a template change.
- **CI and monitoring**: schedule a run, then alert when `errorCount` rises.

### What it checks

| Area | Examples of issue codes |
|---|---|
| robots.txt | `robots-missing`, `robots-server-error`, `robots-served-as-html`, `robots-blocks-all`, `robots-directive-typo`, `robots-sitemap-not-absolute`, `robots-rule-before-user-agent`, `robots-too-large` (over 500 KiB), `robots-blocks-css-js`, `robots-no-sitemap-directive` |
| AI and search crawlers | 16 crawler tokens, each `allowed`, `partly-restricted` or `blocked`, with the rule group that decided it (RFC 9309 matching: longest rule wins, `*` and `$` supported) |
| Sitemaps | `sitemap-xml-invalid`, `sitemap-namespace-wrong`, `sitemap-too-many-urls` (over 50,000), `sitemap-too-large` (over 50 MB), `sitemap-loc-not-absolute`, `sitemap-offsite-url`, `sitemap-duplicate-url`, `sitemap-lastmod-invalid`, `sitemap-lastmod-future`, `sitemap-priority-invalid`, `sitemap-changefreq-invalid`, `sitemap-is-html`, `sitemap-fetch-failed`, gzip files supported |
| Sampled sitemap URLs | `sitemap-url-error` (404/5xx), `sitemap-url-redirects`, `sitemap-url-noindex`, `sitemap-url-canonical-elsewhere`, `sitemap-url-blocked-by-robots` |
| Structured data | `json-ld-parse-error` (with line and column), `json-ld-missing-context`, `missing-required-property`, `missing-recommended-property`, `invalid-date-format`, `url-not-absolute`, `not-a-number`, `product-needs-offers-review-or-rating` |
| Types covered | Article family, Product and Offer, AggregateRating, Review, Organization, LocalBusiness family, WebSite, BreadcrumbList, FAQPage, Event, Recipe, VideoObject, JobPosting, SoftwareApplication, Course. Microdata and RDFa types are listed. |

Each site and page gets a simple `score` (100 minus 15 per error and 4 per warning, capped), so you can sort the worst first.

### Sample output (site record, shortened)

```json
{
  "recordType": "site",
  "siteUrl": "https://example.org/",
  "score": 70,
  "errorCount": 1,
  "warningCount": 3,
  "robots": {
    "httpStatus": 200,
    "groups": 3,
    "sitemapDirectives": ["https://example.org/sitemap_index.xml"],
    "aiCrawlersBlocked": ["GPTBot", "CCBot"],
    "aiCrawlers": [{"agent": "GPTBot", "purpose": "AI training", "access": "blocked", "ruleSource": "own group", "disallowRules": 1}]
  },
  "sitemapSummary": {"filesRead": 4, "urlsListed": 1830, "uniqueUrls": 1830},
  "structuredDataSummary": {"pagesValidated": 10, "pagesWithStructuredData": 8, "errors": 2, "warnings": 6},
  "issues": [{"area": "sitemap", "code": "sitemap-lastmod-invalid", "severity": "warning", "count": 12}]
}
```

A page record carries `status`, `inSitemap`, `noindex`, `canonical`, the schema `structuredDataTypes`, and a list of issues that name the type and the property (for example *Product is missing "offers"*).

### Inputs

| Input | What it does |
|---|---|
| Site URLs | 1 to 20 home pages. Only that host is contacted |
| Extra page URLs | Up to 100 specific pages (one per template is a good habit); must share a host with a site URL |
| Check robots.txt / sitemaps / structured data | Switch each area on or off |
| Pages to sample per site | Home page plus an even spread of sitemap URLs (0 to 200, default 10). 0 = robots.txt and sitemaps only |
| Max sitemap files per site | 1 to 100, default 25 |
| Delay between page requests | 500 to 10000 ms. A higher Crawl-delay in robots.txt wins |

Bad input (no URL, not an http(s) address, a page URL on a different host, too many sites) stops the run at once with a plain message and no charge beyond the start event. If no site can be reached the run fails and explains why.

### Pricing

Pay per event: **$0.005 per run**, **$0.015 per site validated** and **$0.002 per page validated**. One site with 10 sampled pages costs about $0.04. Unreachable sites, error pages and URLs blocked by robots.txt are free, and nothing is charged past the *Maximum cost per run* you set.

### How much does it cost?

You pay per event: $0.005 per run, $0.015 per site and $0.002 per page validated. Unreachable sites, error pages and robots.txt-blocked URLs are free.

| What you run | Sites | Pages sampled | Approx. cost |
|---|---|---|---|
| Quick robots.txt and sitemap check | 1 | 0 | $0.02 |
| Pre-launch check of one site | 1 | 10 | $0.04 |
| Agency batch, 10 client sites | 10 | 10 each | $0.36 |
| Template coverage, 1 site plus 30 extra page URLs | 1 | 40 | $0.10 |
| Weekly monitor, 5 sites x 20 pages | 5 | 20 each | $0.28 per run |

Set *Maximum cost per run* in the run options and the Actor stops at your cap.

### FAQ

**Which AI crawlers does it report on?** 16 tokens including GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot, Bytespider, Applebot-Extended and Googlebot, each marked allowed, partly restricted or blocked, with the deciding rule group.

**Can I test a site I do not own?** Only sites you own or have permission to test. Only that host is contacted, one request at a time, and robots.txt is always obeyed for page fetches.

**Does it replace Google's Rich Results Test?** No. It is a fast bulk pre-check that finds missing required properties, parse errors and bad dates across many pages and sites; confirm key templates with the search engine's own tools.

**Can I run it from CI or on a schedule?** Yes. Use an Apify schedule or the API and alert when `errorCount` goes up.

**Why is a page not validated?** It returned an error, was blocked by robots.txt, is not HTML, or is outside *Pages to sample per site* and *Extra page URLs*.

**Related Actors by the same developer:** [Website SEO Audit & Broken Link Checker](https://apify.com/appropriate_songbird/site-seo-audit-link-checker) for a full crawl with broken links and duplicate titles, and [PDF to Text, Tables & RAG Chunks](https://apify.com/appropriate_songbird/pdf-text-tables-chunker).

### Good to know

- **Polite by design.** One request at a time per host, at least 500 ms apart, a declared user agent (`SitemapSchemaValidator`), retries with back-off on 429/5xx. Sitemap files are fetched only from your own host; sitemap directives pointing elsewhere are reported, not followed.
- **Pages are read as the server delivers them**, without running JavaScript. Markup injected by scripts after load is not seen.
- **A sample, not a crawl.** Structured data is checked on the pages you sample or name; use *Extra page URLs* to cover every template.
- **Property checks are practical guidance**, based on the open schema.org vocabulary and common search-engine documentation. They are not a guarantee of rich-result eligibility. Always confirm important templates with the search engine's own testing tools.
- **Access refusals are reported, never bypassed.** The Actor never signs in, solves captchas or retries around a 401/403.
- **Private networks are refused**, redirects included (localhost, internal IPs, cloud metadata addresses). XML is parsed with entity resolution and network access disabled.
- Sitemap files over 25 MB are flagged but not analysed in full.

### Privacy and rights

Test only sites you own or have permission to test. The Actor does not extract email addresses, phone numbers or profile data. For structured data it reports types and the *names* of missing or malformed properties, never property values; Person entities are counted, not read. Nothing is kept after the run: results live only in your own run's dataset and key-value store, plus run-health counters (runs started, succeeded, failed, last error) in a named store in your account.

Built on httpx, BeautifulSoup and lxml. This is an independent tool and is not affiliated with schema.org, any search engine or any AI company; crawler names appear only as the user-agent tokens used in robots.txt files.

# Actor input Schema

## `siteUrls` (type: `array`):

Home page of each site you own or have permission to test (1 to 20). Only that host is contacted. robots.txt is always obeyed for page fetches.

## `pageUrls` (type: `array`):

Specific pages to validate, for example one URL per page template (product, article, event). Each must be on the same host as one of your site URLs. Up to 100.

## `checkRobots` (type: `boolean`):

Validate syntax and rules, and report which AI and search crawlers (GPTBot, ClaudeBot, CCBot, Googlebot and others) the file allows or blocks.

## `checkSitemaps` (type: `boolean`):

Find sitemaps via robots.txt (or /sitemap.xml), follow sitemap indexes and validate protocol limits, URLs, lastmod, duplicates and off-host entries.

## `checkStructuredData` (type: `boolean`):

Fetch sampled pages and check JSON-LD (plus microdata/RDFa types) for parse errors and missing required or recommended properties.

## `pagesPerSite` (type: `integer`):

How many pages to fetch per site: the home page plus an even spread of sitemap URLs (0 to 200). Set 0 to check only robots.txt and sitemaps.

## `maxSitemapsPerSite` (type: `integer`):

Cap on sitemap files read per site, including those listed in sitemap indexes (1 to 100).

## `requestDelayMs` (type: `integer`):

Pause between requests to the same host (500 to 10000). A higher Crawl-delay in robots.txt wins.

## Actor input object example

```json
{
  "siteUrls": [
    "https://example.com/"
  ],
  "checkRobots": true,
  "checkSitemaps": true,
  "checkStructuredData": true,
  "pagesPerSite": 3,
  "maxSitemapsPerSite": 25,
  "requestDelayMs": 500
}
```

# Actor output Schema

## `results` (type: `string`):

Dataset with one 'site' record per site (robots.txt, AI-crawler access, sitemaps, issues, score) and one 'page' record per sampled page (status, indexability, structured-data findings).

## `runReport` (type: `string`):

Score, error and warning counts per site.

## `runSummary` (type: `string`):

Run statistics and health counters.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "siteUrls": [
        "https://example.com/"
    ],
    "pagesPerSite": 3
};

// Run the Actor and wait for it to finish
const run = await client.actor("appropriate_songbird/structured-data-sitemap-validator").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "siteUrls": ["https://example.com/"],
    "pagesPerSite": 3,
}

# Run the Actor and wait for it to finish
run = client.actor("appropriate_songbird/structured-data-sitemap-validator").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "siteUrls": [
    "https://example.com/"
  ],
  "pagesPerSite": 3
}' |
apify call appropriate_songbird/structured-data-sitemap-validator --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,appropriate_songbird/structured-data-sitemap-validator"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/tmrC5i6JkQfpLQCu4/builds/Rtq94VlmGxNRaVJQk/openapi.json