AI Crawler Checker — Robots.txt Policy for Six Bots

AI crawler checker reports declared robots.txt permissions for GPTBot, ClaudeBot, Google-Extended, PerplexityBot, CCBot and Amazonbot at each supplied public URL path, without impersonating bots or fetching the tested page. AI agents can call and pay through Apify MCP. List event rates are $0.0015 per path; start is extra, usage is included and failed results are uncharged.

Example result

Example record for the minimal input under How to use it, as this Actor returned it on 2026-10-31 in a run of its production image; the record is kept unchanged with the release.

{
  "robotsStatus": 200,
  "robotsSha256": "e62876aa4593da0024f896868ea789201cf929606aebcc56dcda88592017209f",
  "crawlers": [
    {
      "crawler": "GPTBot",
      "allowed": true
    },
    {
      "crawler": "ClaudeBot",
      "allowed": true
    },
    {
      "crawler": "Google-Extended",
      "allowed": true
    },
    {
      "crawler": "PerplexityBot",
      "allowed": true
    },
    {
      "crawler": "CCBot",
      "allowed": true
    },
    {
      "crawler": "Amazonbot",
      "allowed": true
    }
  ],
  "scope": "Declared robots.txt policy only; not a live bot access test or legal permission.",
  "url": "input-1",
  "ok": true
}

How to use it

Start a run in Apify Console, by API or from an agent with an input like this one — it is the input Apify runs every day to test this Actor:

{"urls": ["https://docs.apify.com/"]}

Field	Type	Meaning
urls	array	One record for each input; only nonpersonal technical metadata is returned.

Events and prices

Event	Title	Charged when	FREE / BRONZE	SILVER	GOLD / PLATINUM / DIAMOND
crawler-policy-checked	Crawler policy checked	One successful input with a readable result written to the dataset.	$0.0015	$0.00135	$0.0012

Platform usage is not passed through. Apify's automatic actor-start event remains $0.00005 per started GB (one GB or less counts once). The synthetic dataset-item event is absent. Charge limits stop further paying work; unprocessed inputs receive uncharged errors. Completed readable pending events are settled on abort.

Worked cost example: ten successful crawler policy checked events cost $0.015 at list, plus actor start. Failures are not included in that count.

Call it from an agent

Through Apify's MCP server (https://mcp.apify.com) use call-actor:

{
  "actor": "hermeskd/ai-crawler-access-check",
  "input": {
    "urls": [
      "https://docs.apify.com/"
    ]
  },
  "callOptions": {
    "maxTotalChargeUsd": 0.01,
    "timeout": 600
  },
  "waitSecs": 30
}

Then call get-dataset-items with the returned defaultDatasetId. Dedicated tool URL: https://mcp.apify.com?tools=hermeskd/ai-crawler-access-check

Output fields

Field	Meaning
url	One-based input reference (input-1, input-2, ...); never the supplied value.
ok	True for a returned result; a failure is never charged.
robotsStatus	HTTP status from robots.txt; 404/410 means no declared policy.
robotsSha256	Hash of a valid policy body, or null if absent.
crawlers	Fixed named crawler tokens and declared allowed booleans.
scope	Declared robots policy only, not a live bot access test.
error	Failure, always uncharged.

Limits and run time

1–50 public URLs, each at most 2048 characters. Only the origin robots.txt is fetched, at most 64 KiB/512 allow-disallow rules/2048 lines. Six fixed crawler tokens; unavailable/throttled/invalid policy refuses uncharged. Missing 404/410 policy permits by RFC convention. This checks declared policy, not actual bot HTTP access, training rights, page content or legal permission.

The default run timeout is 600 seconds. Local production-image tests at 128 MB and one CPU: three cold typical calls took 7.88–9.37 seconds wall time, peak typical RSS 82.5 MiB; the ten-event benchmark took 13.78 seconds. Apify cold startup, billing and the deploy pricing floor remain unverified.

Inputs are correlated by their one-based list position. Caller strings, URL paths, queries, credentials and exception text are never copied into identities or errors. Duplicate entries retain separate positions.

Errors

Code	Meaning
http_error	The source answered an unsuccessful HTTP status.
download_failed	The request or connection failed within its bounded deadline.
source_invalid	The source response is malformed or outside the documented supported shape.
not_found	The requested technical record is unavailable.
robots_disallowed	robots.txt forbids fetching the URL; no request was made to it.
blocked_host	The host is on the deny-list or resolves to a private or local address; no request was made.
too_large	The response exceeded the size cap.
timeout	The input did not finish within the run's time budget.
invalid_input	The run input passed the input schema but cannot be used; the message names the field.
internal_error	Unexpected internal failure; private exception details are excluded.
charge_limit_reached	The run reached its maximum total charge; this input was not processed.

No failure record is charged. A run fails only when all input outcomes are Actor-side failures; caller errors and a mixture of caller/Actor failures do not fail the run.

Sources and terms

Only process content you have the right to process. URL fetches obey HermesBot robots-group selection, the shared deny-list, public-address checks on connections and redirects, honest identity, deadlines and rate limits. No caller credentials, cookies, CAPTCHA solving, residential proxies or evasion. Caller-authorized public website robots.txt under the shared fetch policy, based on RFC 9309. The prefill is Apify Apache-2.0 documentation with permissive robots verified on 2026-10-30. No page content, raw rules, sitemap links or contacts returned.

Not affiliated

Not affiliated with or endorsed by the source publishers or Apify.

Related Hermes Actors

• Article Extractor — Main Text, Dates & Markdown from URLs — an article's clean text, title and publish date, without the rest of the page.
• Career Site Job Search — Verified Greenhouse & Lever Boards — search open jobs in the Hermes directory of Greenhouse and Lever boards, each confirmed live.
• Speech to Text — File Transcripts, Speaker Labels & Subtitles — speech to text with SRT and VTT subtitles.
• Video Converter & MP4 to MP3 — Trim, GIF; Audio $0.001 a Minute — audio and video conversion: MP4 to MP3, trim, GIF.
• Broken Links Checker — Crawl Pages, Assets & Redirect Chains — broken links, 404s and redirects on a site.
• Company ATS Finder — Which Hiring System & Job Board It Uses — which hiring system and job board a company uses.
• SEC EDGAR — Recent Company Filing Index by CIK — Read recent company financial filing metadata from the official SEC JSON API
• Crossref DOI Lookup — Titles, Venues, Dates & Citation Counts — crossref doi metadata.
• CVE Vulnerability Lookup — CVSS Scores & NVD Status — look up NVD CVE dates, status, numeric CVSS scores and severity.
• Merge JSON Datasets — Deduplicate Rows & Export Stable CSV — json dataset cleaner.
• DNS Lookup — Bulk SPF, DKIM, DMARC & Mail Policy Findings — DNS records and SPF, DKIM and DMARC checks.
• DOCX to Markdown — Word, Excel, Slides & Document Tables — Word, Excel, PowerPoint and other office files to Markdown.
• Domain Availability — Bulk RDAP Status, Dates & Nameservers — whether domain names are available, from registry RDAP.
• Bulk File Downloader — URL Lists to Storage, Links & SHA-256 — files and images from URLs, saved with a public link each.
• Greenhouse Job Board Scraper — Open Jobs, Live, Any Company — every open job on a company's Greenhouse board.
• Hacker News Metrics — Story Scores & Comment Counts — read numeric Hacker News story scores, timestamps and comment counts by ID.
• HTTP GET Request — Bulk Text Responses, Status & SHA-256 — public http get.
• Image Background Remover — Small U2NetP, PNG Base64 — Remove image backgrounds with the small U2NetP model and return transparent PNG pixels as base64
• Image Converter — Static JPEG, PNG & WebP Resize/Compress — image format converter.
• Image Metadata Remover — EXIF/GPS Scrub with Pixel Checks — GPS location, camera serial numbers and other private metadata removed from photos, losslessly.
• Image to Text — OCR Lines, Word Boxes & Confidence Scores — OCR: the text in images.
• Image Upscaler — Real-ESRGAN Graphics & Photo Modes, 2x/4x — image upscaling 2x or 4x, and photo restoration.
• Lever Jobs Scraper — Postings from Any Lever Site, US & EU — every published job on a company's Lever site, global or EU.
• Lighthouse — Mobile/Desktop Lab Audits, Scores & Fix Hints — Lighthouse scores and Core Web Vitals.
• npm Package Metadata — Versions, Licenses & Counts — read latest npm package versions, license tokens and dependency counts.
• Website Change Monitor — Text Diffs & Stored Page Snapshots — what changed on a page since the last check.
• PDF to Markdown / PDF to Text — PDF Text Extractor + OCR — PDF to Markdown and text, with OCR for scanned pages.
• Public Holidays Calendar — Five Countries, 2000–2035 — build offline public-holiday calendars for five countries and bounded years.
• Remote Jobs Scraper — Remote OK Listings, Salary Filter, Links — current remote jobs from Remote OK, with salary and listing links.
• Research Paper Search — OpenAlex Titles, DOIs & Citations — Search OpenAlex for work-level paper metadata without authors, affiliations, abstracts or full text
• RSS Feed Reader — RSS/Atom/JSON Feed Items & Enclosures — every item of an RSS, Atom or JSON feed.
• SEO Audit — Supplied-Page Scores, Meta Tags & Schema Fixes — an on-page SEO audit: meta tags, schema, headings and links.
• Sitemap Extractor — Unique URLs, Indexes, Gzip & RSS Maps — the URLs in a site's sitemaps.
• Text to Speech — Generic Formant WAV, English/German/Spanish — Convert short nonpersonal texts to offline formant WAV speech for agents
• URL to Markdown — Clean Page Text, Selectors & Source Links — a public job page as clean Markdown where the site allows it (deny-listed job sites are refused).
• Extract Video Frames — Scene Cuts, Intervals & Timestamps — frames, thumbnails and scenes from a video.
• Website Color Palette — Static CSS Literals, Alpha & Counts — Extract declared website CSS color literals and counts from public HTML and bounded linked stylesheets
• Favicon Extractor — Declared Icons, Manifests & Theme Colors — declared website icons, theme colors and manifest links.
• Website Screenshot — Full Pages, Elements, HTML & PDF Prints — screenshots and PDF prints of web pages.
• Tech Stack Detector — CMS, Framework & CDN with Evidence — the technologies a website runs, detected live.
• Website Crawler — Markdown Pages, Sitemaps & RAG Provenance — bounded whole-site Markdown with robots and crawl-delay enforced
• GitHub Repository Metrics — Stars, Forks & Issue Counts — Read numeric metrics for known public GitHub repositories.
• HTTP Header Policy Check — HSTS, CSP & Frame Declarations — Check seven declared HTTP security header families without returning raw values.
• ECB Currency Reference Rates — Latest Daily Cross Pairs — Read latest ECB daily currency reference rates and calculated cross pairs.
• USGS Earthquake Metrics — Magnitude, Coordinates & Times — Read numeric USGS earthquake metadata for known event identifiers.
• Timezone Conversion — IANA Offsets, DST & Repeated Hours — Convert one offset-qualified instant to named IANA time zones using pinned data.

Changelog

• 1.0 (2026-10-30) — 2026-10-30: first factory-built version..