Seo Robots Txt Checker
Pricing
$0.01 / actor start
Seo Robots Txt Checker
Pricing
$0.01 / actor start
Rating
0.0
(0)
Developer
Eichan Jolly
Maintained by CommunityActor stats
0
Bookmarked
2
Total users
1
Monthly active users
10 days ago
Last modified
Categories
Features
Share
Robots.txt Rules Checker
One small SEO job: request a website's public /robots.txt and show readable wildcard (*) and requested named-bot Allow / Disallow path rules plus declared Sitemap URLs.
Who this is for
SEO specialists, site owners, and content teams who want a quick inspection of published crawler directives without a full-site audit or login.
Input (copy into the Actor input form)
{"siteUrl": "https://www.python.org/","namedBot": "Nutch"}
This is a real public target, checked on 2026-11-01. The Python site publishes wildcard and Nutch-specific rules. A domain like www.python.org (without scheme) also works and defaults to HTTPS. The path on siteUrl is ignored, since robots.txt is always read from the website root. namedBot defaults to Nutch in the form and to Googlebot if omitted from a direct API input.
Output (example record)
One row for * and one for the requested named bot (unless that bot is *). Field shape is stable:
{"robotsUrl": "https://www.python.org/robots.txt","userAgent": "Nutch","groupFound": true,"allowPaths": [],"disallowPaths": ["/"],"effectiveAllowPaths": [],"effectiveDisallowPaths": ["/"],"effectiveRuleSource": "Nutch","sitemaps": [],"httpStatus": 200,"status": "ok","message": "Robots.txt rules parsed.","error": null}
disallowPaths/allowPaths reflect directives explicitly written for that agent. If no agent-specific group exists, the effective...Paths fields fall back to the * group, and effectiveRuleSource is *. Repeated matching groups are combined. Sitemap directives are collected across the file, deduplicated, and repeated on each row.
A 404 or 410 is a valid result, not a failed run:
{"robotsUrl": "https://site.example/robots.txt","userAgent": "*","groupFound": false,"allowPaths": [],"disallowPaths": [],"effectiveAllowPaths": [],"effectiveDisallowPaths": [],"effectiveRuleSource": null,"sitemaps": [],"httpStatus": 404,"status": "no_robots_txt","message": "No robots.txt found (no restrictions).","error": null}
A network error or HTTP denial returns a plain-language status: error record and does not claim unrestricted crawling.
Operation and limits
- Standard library only. No Apify SDK, web browser, search engine scraping, authentication, or third-party packages.
- Fetches only the specified host's root
/robots.txt; nothing else is crawled. Follows at most four same-host redirects, with a 10-second timeout and a 500 KB response cap. - Understands case-insensitive
User-agent,Allow,Disallow, andSitemapdirective names; supports multiple agents in a group and multiple groups. EmptyDisallow:does not mean blocked; it is omitted from blocked paths. - This is a directive inspector, not a complete RFC 9309 permission evaluator. Patterns are shown as written; it does not apply longest-rule matching, prefix-based user-agent selection, query matching, crawl-delay semantics, or bot-vendor-specific interpretation. An empty list can mean no matching directive, not necessarily permission in every crawler implementation.
- HTTP 404/410 is treated as absent; other HTTP errors and connection problems are
status: error. Robots.txt may be served dynamically, so readings are point-in-time. - Rejects credentials in URLs, unsafe/private IP destinations, nonstandard ports, cross-host redirects, and HTTPS downgrade.
SEO_TEST_ALLOW_LOOPBACK=1is only for local fixtures and must never be used in production.
Use locally / test
$ACTOR_INPUT_JSON='{"siteUrl":"https://www.python.org/","namedBot":"Nutch"}' python src/main.py
Local JSON items are written to storage/datasets/default/. Alternatively set storage/key_value_stores/default/INPUT.json. On Apify, the Actor reads input and pushes dataset items through Apify REST APIs using platform-provided token and default storage IDs (no Apify SDK).
Offline script-mode fixture tests and captured evidence:
python tests/run_smoke.pypython -m unittest discover -s tests -p 'test_*.py' -v
tests/evidence/ contains actual python src/main.py output and dataset records. Local fixtures are used because public DNS was unavailable during packaging; live Apify or external HTTP operation was not run.
Store configuration suggestion (NOT applied)
- Categories (max 3): SEO tools, Marketing, Developer tools.
- Pricing: $0.01 per run, regardless of whether a wildcard group exists. No billing or publication changes were made.
- See
store_metadata.jsonfor the proposal; this file is documentation, not a platform billing configuration.