Robots.txt, Sitemap & Schema Markup Validator (AI Crawler Check)

Validate a site's robots.txt, XML sitemaps and JSON-LD schema markup in one run, and see whether GPTBot, ClaudeBot, PerplexityBot and Googlebot are allowed in. Scored JSON report, about $0.03 for a first site check, no subscription.

Try it in 10 seconds: press Start with the prefilled input. It checks one site and 3 sample pages for about 3 cents. Then swap in your own home page.

Give the Actor the home page of a site you own (or are allowed to test). It reads robots.txt, finds and validates every sitemap (following sitemap indexes), then fetches a sample of pages and checks them for broken or incomplete structured data. You get one record per site and one per page, each with plain-language issue codes you can filter, diff or feed to CI.

No browser, no login, no API key, no third-party SEO database. robots.txt is always obeyed when fetching pages, and there is no switch to turn that off.

What people use it for

• Technical SEO and agencies: a quick pre-launch or post-release check across client sites.
• AI-visibility audits: see in one table whether GPTBot, ClaudeBot, CCBot, PerplexityBot, Google-Extended and other crawlers are allowed, blocked or partly restricted, and from which rule group.
• Site migrations: confirm sitemaps list only live, indexable, canonical URLs.
• E-commerce and publishers: catch Product, Article, FAQ, Event, Recipe and Breadcrumb markup that lost required properties after a template change.
• CI and monitoring: schedule a run, then alert when errorCount rises.

What it checks

Area	Examples of issue codes
robots.txt	robots-missing, robots-server-error, robots-served-as-html, robots-blocks-all, robots-directive-typo, robots-sitemap-not-absolute, robots-rule-before-user-agent, robots-too-large (over 500 KiB), robots-blocks-css-js, robots-no-sitemap-directive
AI and search crawlers	16 crawler tokens, each allowed, partly-restricted or blocked, with the rule group that decided it (RFC 9309 matching: longest rule wins, * and $ supported)
Sitemaps	sitemap-xml-invalid, sitemap-namespace-wrong, sitemap-too-many-urls (over 50,000), sitemap-too-large (over 50 MB), sitemap-loc-not-absolute, sitemap-offsite-url, sitemap-duplicate-url, sitemap-lastmod-invalid, sitemap-lastmod-future, sitemap-priority-invalid, sitemap-changefreq-invalid, sitemap-is-html, sitemap-fetch-failed, gzip files supported
Sampled sitemap URLs	sitemap-url-error (404/5xx), sitemap-url-redirects, sitemap-url-noindex, sitemap-url-canonical-elsewhere, sitemap-url-blocked-by-robots
Structured data	json-ld-parse-error (with line and column), json-ld-missing-context, missing-required-property, missing-recommended-property, invalid-date-format, url-not-absolute, not-a-number, product-needs-offers-review-or-rating
Types covered	Article family, Product and Offer, AggregateRating, Review, Organization, LocalBusiness family, WebSite, BreadcrumbList, FAQPage, Event, Recipe, VideoObject, JobPosting, SoftwareApplication, Course. Microdata and RDFa types are listed.

Each site and page gets a simple score (100 minus 15 per error and 4 per warning, capped), so you can sort the worst first.

Sample output (site record, shortened)

{
  "recordType": "site",
  "siteUrl": "https://example.org/",
  "score": 70,
  "errorCount": 1,
  "warningCount": 3,
  "robots": {
    "httpStatus": 200,
    "groups": 3,
    "sitemapDirectives": ["https://example.org/sitemap_index.xml"],
    "aiCrawlersBlocked": ["GPTBot", "CCBot"],
    "aiCrawlers": [{"agent": "GPTBot", "purpose": "AI training", "access": "blocked", "ruleSource": "own group", "disallowRules": 1}]
  },
  "sitemapSummary": {"filesRead": 4, "urlsListed": 1830, "uniqueUrls": 1830},
  "structuredDataSummary": {"pagesValidated": 10, "pagesWithStructuredData": 8, "errors": 2, "warnings": 6},
  "issues": [{"area": "sitemap", "code": "sitemap-lastmod-invalid", "severity": "warning", "count": 12}]
}

A page record carries status, inSitemap, noindex, canonical, the schema structuredDataTypes, and a list of issues that name the type and the property (for example Product is missing "offers").

Inputs

Input	What it does
Site URLs	1 to 20 home pages. Only that host is contacted
Extra page URLs	Up to 100 specific pages (one per template is a good habit); must share a host with a site URL
Check robots.txt / sitemaps / structured data	Switch each area on or off
Pages to sample per site	Home page plus an even spread of sitemap URLs (0 to 200, default 10). 0 = robots.txt and sitemaps only
Max sitemap files per site	1 to 100, default 25
Delay between page requests	500 to 10000 ms. A higher Crawl-delay in robots.txt wins

Bad input (no URL, not an http(s) address, a page URL on a different host, too many sites) stops the run at once with a plain message and no charge beyond the start event. If no site can be reached the run fails and explains why.

Pricing

Pay per event: $0.005 per run, $0.015 per site validated and $0.002 per page validated. One site with 10 sampled pages costs about $0.04. Unreachable sites, error pages and URLs blocked by robots.txt are free, and nothing is charged past the Maximum cost per run you set.

How much does it cost?

You pay per event: $0.005 per run, $0.015 per site and $0.002 per page validated. Unreachable sites, error pages and robots.txt-blocked URLs are free.

What you run	Sites	Pages sampled	Approx. cost
Quick robots.txt and sitemap check	1	0	$0.02
Pre-launch check of one site	1	10	$0.04
Agency batch, 10 client sites	10	10 each	$0.36
Template coverage, 1 site plus 30 extra page URLs	1	40	$0.10
Weekly monitor, 5 sites x 20 pages	5	20 each	$0.28 per run

Set Maximum cost per run in the run options and the Actor stops at your cap.

FAQ

Which AI crawlers does it report on? 16 tokens including GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot, Bytespider, Applebot-Extended and Googlebot, each marked allowed, partly restricted or blocked, with the deciding rule group.

Can I test a site I do not own? Only sites you own or have permission to test. Only that host is contacted, one request at a time, and robots.txt is always obeyed for page fetches.

Does it replace Google's Rich Results Test? No. It is a fast bulk pre-check that finds missing required properties, parse errors and bad dates across many pages and sites; confirm key templates with the search engine's own tools.

Can I run it from CI or on a schedule? Yes. Use an Apify schedule or the API and alert when errorCount goes up.

Why is a page not validated? It returned an error, was blocked by robots.txt, is not HTML, or is outside Pages to sample per site and Extra page URLs.

Related Actors by the same developer: Website SEO Audit & Broken Link Checker for a full crawl with broken links and duplicate titles, and PDF to Text, Tables & RAG Chunks.

Good to know

• Polite by design. One request at a time per host, at least 500 ms apart, a declared user agent (SitemapSchemaValidator), retries with back-off on 429/5xx. Sitemap files are fetched only from your own host; sitemap directives pointing elsewhere are reported, not followed.
• Pages are read as the server delivers them, without running JavaScript. Markup injected by scripts after load is not seen.
• A sample, not a crawl. Structured data is checked on the pages you sample or name; use Extra page URLs to cover every template.
• Property checks are practical guidance, based on the open schema.org vocabulary and common search-engine documentation. They are not a guarantee of rich-result eligibility. Always confirm important templates with the search engine's own testing tools.
• Access refusals are reported, never bypassed. The Actor never signs in, solves captchas or retries around a 401/403.
• Private networks are refused, redirects included (localhost, internal IPs, cloud metadata addresses). XML is parsed with entity resolution and network access disabled.
• Sitemap files over 25 MB are flagged but not analysed in full.

Privacy and rights

Test only sites you own or have permission to test. The Actor does not extract email addresses, phone numbers or profile data. For structured data it reports types and the names of missing or malformed properties, never property values; Person entities are counted, not read. Nothing is kept after the run: results live only in your own run's dataset and key-value store, plus run-health counters (runs started, succeeded, failed, last error) in a named store in your account.

Built on httpx, BeautifulSoup and lxml. This is an independent tool and is not affiliated with schema.org, any search engine or any AI company; crawler names appear only as the user-agent tokens used in robots.txt files.