Website Robots & Sitemap Indexability Audit

Checks a public HTTPS site you are authorized to audit for contradictions between what the sitemap promises and what the pages and robots.txt say, then reports what changed since the previous run. It is a bounded sampling tool, not a full-site crawler.

What it checks

1. Reads robots.txt first and obeys it (longest-match rules, * and $ wildcards) for its own user agent.
2. Reads sitemaps declared in robots.txt (or /sitemap.xml) within strict file, size and URL limits.
3. Requests a limited number of listed pages and flags, per URL:
   • SITEMAP_URL_NOINDEX (meta robots / googlebot or X-Robots-Tag)
   • SITEMAP_URL_CANONICAL_MISMATCH, SITEMAP_URL_CANONICAL_OFFSITE, MULTIPLE_CANONICALS
   • SITEMAP_URL_DISALLOWED_BY_ROBOTS (not fetched) and SITEMAP_URL_DISALLOWED_FOR_GOOGLEBOT
   • SITEMAP_URL_REDIRECTS, SITEMAP_URL_UNSAFE_REDIRECT, SITEMAP_URL_ERROR_STATUS
   • Site-level: SITEMAP_NOT_FOUND, SITEMAP_BROKEN_XML, SITEMAP_TOO_LARGE, SITEMAP_EMPTY, SITEMAP_BLOCKED_BY_ROBOTS
4. Compares with the stored baseline for that domain and emits CHANGE records: URLs added/removed, issues appeared/resolved, status or canonical changed.

Sample input

{
    "startUrl": "https://www.your-own-site.com",
    "maxPages": 25,
    "maxSitemapUrls": 1000,
    "maxSitemapFiles": 5,
    "requestDelayMs": 1000
}

Only startUrl is required. Use a site you own or may audit. The default https://example.com is a placeholder and will mostly report a missing sitemap.

Run statuses

Status	Meaning	Charged
BASELINE_CREATED	First run for this domain; baseline stored, current findings listed	No
CHANGES_DETECTED	Differences versus the baseline were found	Event change-report once (price set by the publisher, if any)
NO_CHANGES	Nothing differs in the audited sample	No
BLOCKED_BY_ROBOTS	robots.txt forbids this crawler (or returns 401/403); nothing fetched	No
INVALID_INPUT, FAILED_*	Bad input, unreadable robots.txt, or sitemap network error; run fails	No

Output

• Dataset rows with recordType = FINDING, CHANGE or SUMMARY, each with sourceUrl and checkedAt. Export as JSON or CSV from the Console/API.
• Key-value store OUTPUT (JSON) and REPORT_CSV (CSV) in the run's default store.
• The baseline lives in a named key-value store (snapshotStoreName, default pal-indexability-snapshots) with one key per domain, so it survives between runs. The latest run replaces the baseline.

Scheduling

Create an Apify Task with your input and schedule it (for example weekly). Keep snapshotStoreName identical across runs and tasks that should share a baseline. Use distinct names to keep separate baselines for the same domain.

Safety rules built in

• HTTPS site root only; rejects credentials, custom ports, IP addresses, localhost/internal names, and hosts resolving to non-public addresses.
• Redirects are followed only within the same origin, up to 3 hops, and only to robots-permitted URLs. Others are reported and not followed.
• Page requests are hard-capped by maxPages (redirect hops count); at least 500 ms between requests; no proxies, cookies or logins; 401/403/blocked pages are reported, never retried or bypassed.
• Response sizes are capped (page 256 KB, sitemap 2 MB, robots.txt 500 KB). Sitemaps with DTD/entity declarations are rejected.
• Only URLs, status codes and indexability directives are stored. Page text, emails and phone numbers are never extracted; URLs containing @ are skipped.

Limitations and sample-size caveats

• Sampling: pages are checked in sitemap order up to maxPages. A clean result covers only that sample; it does not prove the whole site is clean. Check pagesNotRequestedDueToLimit in the summary.
• If a sitemap limit is reached, additions/removals are not compared for that run.
• Only the document head is parsed: JavaScript-rendered tags, <base>, HTTP Link canonical headers and .xml.gz sitemaps are not evaluated.
• Canonical and noindex are reported as signals; search engines may decide differently. Googlebot disallow is evaluated from the same robots.txt and is informational.
• Robots matching is a careful reimplementation, not an official parser, and does not normalize percent-encoding. DNS is checked before each request, but a determined DNS rebinding race cannot be fully excluded.
• Changes caused by A/B tests, geo-dependent or flaky responses may appear as changes.
• It does not measure actual index coverage, rankings or traffic.

Local development

$python -m unittest discover -s tests -t .