# llms.txt and AI Crawler Policy Audit

**Use case:** 

Audit public llms.txt, llms-full.txt, and robots.txt rules for nine AI crawlers with evidence, confidence, gaps, and review actions.

## Input

```json
{
  "items": [
    "apify.com",
    "nytimes.com"
  ],
  "maxConcurrency": 2
}
```

## Output

```json
{
  "entityId": {
    "label": "Website entity ID",
    "format": "text"
  },
  "input": {
    "label": "Domain",
    "format": "text"
  },
  "found": {
    "label": "Found",
    "format": "boolean"
  },
  "hasLlmsTxt": {
    "label": "Has llms.txt",
    "format": "boolean"
  },
  "hasLlmsFullTxt": {
    "label": "Has llms-full.txt",
    "format": "boolean"
  },
  "llmsTxtBytes": {
    "label": "llms.txt bytes",
    "format": "number"
  },
  "summary": {
    "label": "Summary",
    "format": "text"
  },
  "confidenceBasis": {
    "label": "Confidence basis",
    "format": "text"
  },
  "confidenceScore": {
    "label": "Confidence",
    "format": "number"
  },
  "confidenceBand": {
    "label": "Confidence band",
    "format": "text"
  },
  "confidenceRisks": {
    "label": "Confidence risks",
    "format": "array"
  },
  "recommendedAction": {
    "label": "Recommended action",
    "format": "text"
  },
  "actionPriority": {
    "label": "Priority",
    "format": "text"
  },
  "interpretationBoundary": {
    "label": "Interpretation boundary",
    "format": "text"
  },
  "safeToAutomate": {
    "label": "Automation safe",
    "format": "boolean"
  },
  "observedAt": {
    "label": "Observed at",
    "format": "date"
  },
  "freshness": {
    "label": "Evidence freshness",
    "format": "object"
  },
  "dataGaps": {
    "label": "Data gaps",
    "format": "array"
  },
  "negativeSignals": {
    "label": "Negative signals",
    "format": "array"
  },
  "partial": {
    "label": "Partial",
    "format": "boolean"
  },
  "failureType": {
    "label": "Failure type",
    "format": "text"
  },
  "retryable": {
    "label": "Retryable",
    "format": "boolean"
  },
  "failureDiagnostics": {
    "label": "Failure diagnostics",
    "format": "object"
  },
  "billing": {
    "label": "Delivery and billing semantics",
    "format": "object"
  },
  "domain": {
    "label": "Audited origin",
    "format": "link"
  },
  "aiCrawlers": {
    "label": "AI crawler root policies",
    "format": "object"
  },
  "sourceEvidence": {
    "label": "Policy-file source evidence",
    "format": "array"
  }
}
```

## About this Actor

This example demonstrates how to use [llms.txt Auditor & AI Crawler Policy Checker](https://apify.com/zinin/llms-txt-auditor) with a specific input configuration. Visit the [Actor detail page](https://apify.com/zinin/llms-txt-auditor) to learn more, explore other use cases, and run it yourself.


## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
This Task's input is already configured above — use it as-is rather than inventing a new one.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For full API examples (JavaScript, Python, CLI, MCP, OpenAPI), see this Task's Actor page: https://apify.com/zinin/llms-txt-auditor.md

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).