Robots Txt Inspector
Pricing
$0.01 / actor start
Robots Txt Inspector
Pricing
$0.01 / actor start
Rating
0.0
(0)
Developer
Eichan Jolly
Maintained by CommunityActor stats
0
Bookmarked
2
Total users
1
Monthly active users
5 days ago
Last modified
Categories
Features
Share
Robots.txt Inspector
One job: Retrieve the root /robots.txt from a single public website and output a structured record for each user-agent group. This is a parser, not a sitemap comparison tool and not a crawler-permissions simulation.
Who is it for?
Website owners, technical SEO practitioners, developers, and QA teams who need a faithful, auditable summary of rules in one public robots.txt file.
Input
{"siteUrl":"https://www.python.org"}
siteUrl accepts either a public http:// / https:// site URL or a bare domain such as www.python.org. The Actor fetches only the /robots.txt at that site's origin. It ignores any input page path. No login, key, cookies or custom headers are needed.
The editor prefill is https://www.python.org, which has a nonempty public robots.txt with user-agent groups, Allow and Disallow directives. Public availability was checked at preparation time, but can change.
Output
One dataset item per consecutive user-agent group. userAgent is deliberately a list to preserve a group shared by multiple crawler names without duplicating rules. Examples below are illustrative, not a claim about the live python.org rules:
{"userAgent": ["ExampleBot", "OtherBot"],"allowRules": ["/public/"],"disallowRules": ["/private/"],"crawlDelaySeconds": 2.5,"sitemaps": ["https://example.org/sitemap.xml"],"sourceUrl": "https://example.org/robots.txt","httpStatus": 200,"otherDirectives": [],"issues": [],"status": "ok","note": null,"error": null}
allowRules and disallowRules preserve directive values in their original order and do not try to rank or merge them. otherDirectives preserves unrecognized named directives as {name, value} objects. Global Sitemap: URLs are repeated on every group item. crawlDelaySeconds is a numeric value (including decimals) or null. issues contains plain-English line-numbered warnings; status is ok or warning. If no groups can be parsed, one invalid item is emitted. HTTP errors, redirects and network errors each generate one structured error item instead of a traceback.
A missing file (HTTP 404) produces one item with status: "not_found", empty rule arrays, and the note: No robots.txt file was found. A missing robots.txt imposes no crawling restrictions by this file. It does not mean other access controls or policies do not exist.
How to run
On Apify, provide siteUrl in the input editor. Apify's default key-value store INPUT is fetched over Apify REST using its automatically supplied runtime token, and dataset items are pushed to the default dataset. The Actor never asks the buyer to provide a token.
Locally (from this folder):
$ACTOR_INPUT_JSON='{"siteUrl":"https://www.python.org"}' python src/main.py
Alternatively put the input at storage/key_value_stores/default/INPUT.json; output appears as JSON item files in storage/datasets/default/. ACTOR_INPUT_JSON takes precedence over the Apify REST key-value-store input and the local file.
Limits and safety
- Exactly one public site per run. HTTP and HTTPS default ports only. Resolved IP addresses must all be public.
- One GET to
/robots.txt, 12-second timeout, maximum 512,000 bytes. No redirect is followed (even if same-host); a redirect is reported instead. - Does not access or test paths listed in rules, does not retrieve sitemaps, and does not discover subdomains.
- Recognizes
User-agent,Allow,Disallow,Crawl-delay,Sitemap(case insensitive). Comments are removed, unrecognized directive names are retained, malformed lines are noted, not fatal. - Robots exclusion behavior is crawler-specific; this Actor reports the file's contents rather than asserting Googlebot, Bingbot, etc. obey any particular rule. It does not merge noncontiguous groups by user-agent.
- TLS failures and other connection errors are plain-language dataset errors. Neither live cloud builds nor live Apify storage writes have been tested in the provided offline smoke run.
Reproducible verification
python tests/run_smoke.pypython -m unittest discover -s tests -p 'test_*.py' -v
The tests/ smoke runner executes python src/main.py as a subprocess with valid prefilled-style input and deterministic fixture network responses. See verification/script-mode-smoke.txt and verification/dataset-items.json. The smoke transport is simulated, not a claim of live python.org HTTP execution.
Store listing
Category: DEVELOPER_TOOLS (only). Suggested price: $0.003 per successful run, advisory only. No publishing or billing changes were performed.