Robots.txt Checker & Parser - Crawl Rules API

robots.txt Validator & Analyzer

Fetch, parse, and analyze robots.txt files for any domain in bulk. Built for SEO professionals, developers, and crawler operators who need to audit site access rules at scale.

What It Does

For each domain you supply, the actor:

1. Fetches /robots.txt from the domain root over HTTPS (falls back gracefully on 404 or network errors)
2. Parses all User-agent, Allow, Disallow, Crawl-delay, and Sitemap directives
3. Reports structured rules grouped by user-agent
4. Optionally checks whether specific paths are allowed or blocked for your chosen user-agent

Input

Field	Type	Required	Description
urls	string[]	Yes	Domains or full URLs (e.g. google.com, https://openai.com/blog)
userAgent	string	No	User-agent to evaluate rules for. Defaults to *
checkPaths	string[]	No	Specific paths to test for allow/disallow (e.g. /admin, /api/)
maxResults	integer	No	Cap on domains to process. Defaults to 100

Output

One record per domain:

Field	Description
domain	Domain name
robotsTxtUrl	Full URL of the fetched robots.txt
robotsTxtFound	true if HTTP 200 was returned
robotsTxtContent	Raw robots.txt text
userAgentRules	Parsed rule blocks, each with userAgent and rules array of {directive, path}
sitemapUrls	All Sitemap URLs declared in the file
crawlDelay	Crawl-delay in seconds for the requested user-agent (null if not set)
analyzedPaths	Per-path results: {path, allowed} for each path in checkPaths
fetchError	Error message if the file could not be fetched

Example Use Cases

• SEO audit: Check which bots can access which parts of your site
• Crawler compliance: Verify your spider respects Disallow rules before running at scale
• Competitive research: Understand what paths competitors block from indexing
• Security review: Identify paths hidden from crawlers (admin panels, staging URLs)
• Sitemap discovery: Extract all declared sitemap URLs without manual inspection

Pricing

$0.10 per 1,000 domains checked. Typical run of 100 domains costs less than $0.02.

How to use

1. Open Robots.txt Checker & Parser - Crawl Rules API in Apify Console and click Start (or Try for free).
2. Fill in the input form - start with Domains / URLs, User-Agent, Max Results. Every field is optional unless marked required, and the form is prefilled with a working example.
3. Set Maximum results to cap your cost, then run the Actor.
4. Download the results as JSON, CSV, Excel, XML or HTML, or pull them with the Apify API.

Example input

{
  "urls": [
    "https://google.com",
    "https://github.com",
    "https://amazon.com"
  ],
  "userAgent": "*",
  "maxResults": 100
}

Output example

One record from a real run (long values shortened, contact details partly masked):

{
  "domain": "google.com",
  "robotsTxtUrl": "https://google.com/robots.txt",
  "robotsTxtFound": true,
  "robotsTxtContent": "User-agent: *\nUser-agent: Yandex\nDisallow: /search\nAllow: /search/about\nAllow: /search/howsearchworks\nDisallow: /sdch\nDisallow: /groups\nDisallow: /index.html...",
  "userAgentRules": [
    {
      "userAgent": "*",
      "rules": [
        {
          "directive": "Disallow",
          "path": "/search"
        },
        {
          "directive": "Allow",
          "path": "/search/about"
        },
        {
          "directive": "Allow",
          "path": "/search/howsearchworks"
        }
      ]
    },
    {
      "userAgent": "Yandex",
      "rules": [
        {
          "directive": "Disallow",
          "path": "/search"
        },
        {
          "directive": "Allow",
          "path": "/search/about"
        },
        {
          "directive": "Allow",
          "path": "/search/howsearchworks"
        }
      ]
    },
    {
      "userAgent": "AdsBot-Google",
      "rules": [
        {
          "directive": "Disallow",
          "path": "/maps/api/js/"
        },
        {
          "directive": "Allow",
          "path": "/maps/api/js"
        },
        {
          "directive": "Disallow",
          "path": "/maps/api/place/js/"
        }
      ]
    }
  ],
  "sitemapUrls": [
    "https://www.google.com/sitemap.xml"
  ]
}

FAQ

How much does Robots.txt Checker & Parser - Crawl Rules API cost?

$0.001 per result plus $0.0001 per run. For example, 1,000 results cost about $1.00. Higher Apify plans get a Store discount.

Do I need an API key or login?

No. Just configure the input and run it.

Can I run it on a schedule or from code?

Yes. Use Apify Schedules for recurring runs, or call it from the Apify API, the JavaScript or Python client, Make, Zapier, n8n, or any MCP-compatible AI agent.

Why did my run return no results?

Your filters are probably too narrow. Remove a filter or broaden the search, then run again. You only pay the result price for records that are actually saved.

Is the data accurate and up to date?

Each run fetches live data from the source described above, so results are as current as that source. Verify critical records with the original source before acting on them.

Use with the API

curl -X POST "https://api.apify.com/v2/acts/pink_comic~robots-txt-validator/run-sync-get-dataset-items?token=YOUR_APIFY_TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"urls": ["https://google.com", "https://github.com", "https://amazon.com"], "userAgent": "*", "maxResults": 100}'