Seo Robots Txt Checker avatar

Seo Robots Txt Checker

Pricing

$0.01 / actor start

Go to Apify Store
Seo Robots Txt Checker

Seo Robots Txt Checker

Pricing

$0.01 / actor start

Rating

0.0

(0)

Developer

Eichan Jolly

Eichan Jolly

Maintained by Community

Actor stats

0

Bookmarked

2

Total users

1

Monthly active users

10 days ago

Last modified

Categories

Features

Limited permissions

Share

Robots.txt Rules Checker

One small SEO job: request a website's public /robots.txt and show readable wildcard (*) and requested named-bot Allow / Disallow path rules plus declared Sitemap URLs.

Who this is for

SEO specialists, site owners, and content teams who want a quick inspection of published crawler directives without a full-site audit or login.

Input (copy into the Actor input form)

{
"siteUrl": "https://www.python.org/",
"namedBot": "Nutch"
}

This is a real public target, checked on 2026-11-01. The Python site publishes wildcard and Nutch-specific rules. A domain like www.python.org (without scheme) also works and defaults to HTTPS. The path on siteUrl is ignored, since robots.txt is always read from the website root. namedBot defaults to Nutch in the form and to Googlebot if omitted from a direct API input.

Output (example record)

One row for * and one for the requested named bot (unless that bot is *). Field shape is stable:

{
"robotsUrl": "https://www.python.org/robots.txt",
"userAgent": "Nutch",
"groupFound": true,
"allowPaths": [],
"disallowPaths": ["/"],
"effectiveAllowPaths": [],
"effectiveDisallowPaths": ["/"],
"effectiveRuleSource": "Nutch",
"sitemaps": [],
"httpStatus": 200,
"status": "ok",
"message": "Robots.txt rules parsed.",
"error": null
}

disallowPaths/allowPaths reflect directives explicitly written for that agent. If no agent-specific group exists, the effective...Paths fields fall back to the * group, and effectiveRuleSource is *. Repeated matching groups are combined. Sitemap directives are collected across the file, deduplicated, and repeated on each row.

A 404 or 410 is a valid result, not a failed run:

{
"robotsUrl": "https://site.example/robots.txt",
"userAgent": "*",
"groupFound": false,
"allowPaths": [],
"disallowPaths": [],
"effectiveAllowPaths": [],
"effectiveDisallowPaths": [],
"effectiveRuleSource": null,
"sitemaps": [],
"httpStatus": 404,
"status": "no_robots_txt",
"message": "No robots.txt found (no restrictions).",
"error": null
}

A network error or HTTP denial returns a plain-language status: error record and does not claim unrestricted crawling.

Operation and limits

  • Standard library only. No Apify SDK, web browser, search engine scraping, authentication, or third-party packages.
  • Fetches only the specified host's root /robots.txt; nothing else is crawled. Follows at most four same-host redirects, with a 10-second timeout and a 500 KB response cap.
  • Understands case-insensitive User-agent, Allow, Disallow, and Sitemap directive names; supports multiple agents in a group and multiple groups. Empty Disallow: does not mean blocked; it is omitted from blocked paths.
  • This is a directive inspector, not a complete RFC 9309 permission evaluator. Patterns are shown as written; it does not apply longest-rule matching, prefix-based user-agent selection, query matching, crawl-delay semantics, or bot-vendor-specific interpretation. An empty list can mean no matching directive, not necessarily permission in every crawler implementation.
  • HTTP 404/410 is treated as absent; other HTTP errors and connection problems are status: error. Robots.txt may be served dynamically, so readings are point-in-time.
  • Rejects credentials in URLs, unsafe/private IP destinations, nonstandard ports, cross-host redirects, and HTTPS downgrade. SEO_TEST_ALLOW_LOOPBACK=1 is only for local fixtures and must never be used in production.

Use locally / test

$ACTOR_INPUT_JSON='{"siteUrl":"https://www.python.org/","namedBot":"Nutch"}' python src/main.py

Local JSON items are written to storage/datasets/default/. Alternatively set storage/key_value_stores/default/INPUT.json. On Apify, the Actor reads input and pushes dataset items through Apify REST APIs using platform-provided token and default storage IDs (no Apify SDK).

Offline script-mode fixture tests and captured evidence:

python tests/run_smoke.py
python -m unittest discover -s tests -p 'test_*.py' -v

tests/evidence/ contains actual python src/main.py output and dataset records. Local fixtures are used because public DNS was unavailable during packaging; live Apify or external HTTP operation was not run.

Store configuration suggestion (NOT applied)

  • Categories (max 3): SEO tools, Marketing, Developer tools.
  • Pricing: $0.01 per run, regardless of whether a wildcard group exists. No billing or publication changes were made.
  • See store_metadata.json for the proposal; this file is documentation, not a platform billing configuration.