# Check Googlebot access in robots.txt

**Use case:** 

Test whether Googlebot and standard crawlers are allowed to fetch selected URLs according to your robots.txt rules.

## Input

```json
{
  "domains": [
    "apify.com"
  ],
  "urls": [
    "https://example.com/"
  ],
  "maxUrlsPerSite": 25,
  "includeNestedSitemaps": true,
  "sameHostOnly": true,
  "checkHttpStatus": false,
  "checkIndexability": false,
  "checkRobotsTxt": true,
  "checkAiCrawlerAccess": false,
  "aiCrawlerUserAgents": [
    "GPTBot",
    "ChatGPT-User",
    "ClaudeBot",
    "PerplexityBot",
    "Google-Extended",
    "CCBot"
  ],
  "requestsPerSecond": 2,
  "timeoutSecs": 20,
  "customUserAgent": "Sitemap Indexability AI Crawler Audit (+https://apify.com)"
}
```

## Output

```json
{
  "recordType": {
    "label": "Record type",
    "format": "string"
  },
  "domain": {
    "label": "Domain",
    "format": "string"
  },
  "url": {
    "label": "URL",
    "format": "string"
  },
  "httpStatus": {
    "label": "HTTP status",
    "format": "integer"
  },
  "indexable": {
    "label": "Indexable",
    "format": "boolean"
  },
  "robotsAllowed": {
    "label": "Robots allowed",
    "format": "boolean"
  },
  "googlebotAllowed": {
    "label": "Googlebot allowed",
    "format": "boolean"
  },
  "aiCrawlerAllowed": {
    "label": "AI crawlers allowed",
    "format": "boolean"
  },
  "noindex": {
    "label": "Noindex",
    "format": "boolean"
  },
  "canonicalUrl": {
    "label": "Canonical URL",
    "format": "string"
  },
  "auditedUrlCount": {
    "label": "Audited URL count",
    "format": "integer"
  },
  "indexableUrlCount": {
    "label": "Indexable URL count",
    "format": "integer"
  },
  "error.code": {
    "label": "Error code",
    "format": "string"
  },
  "error.message": {
    "label": "Error",
    "format": "string"
  }
}
```

## About this Actor

This example demonstrates how to use [Sitemap Indexability & AI Crawler Audit API](https://apify.com/dobus/sitemap-indexability-ai-crawler-audit) with a specific input configuration. Visit the [Actor detail page](https://apify.com/dobus/sitemap-indexability-ai-crawler-audit) to learn more, explore other use cases, and run it yourself.


## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
This Task's input is already configured above — use it as-is rather than inventing a new one.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For full API examples (JavaScript, Python, CLI, MCP, OpenAPI), see this Task's Actor page: https://apify.com/dobus/sitemap-indexability-ai-crawler-audit.md

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).