# llms.txt Auditor & AI Crawler Policy Checker (`zinin/llms-txt-auditor`) Actor

Audit public llms.txt, llms-full.txt, and root robots.txt policy for nine named AI crawlers, with live evidence, explicit missing-versus-unreachable semantics, and actions.

- **URL**: https://apify.com/zinin/llms-txt-auditor.md
- **Developed by:** [Tim Zinin](https://apify.com/zinin) (community)
- **Categories:** SEO tools, AI
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $8.50 / 1,000 site auditeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## llms.txt Auditor — Presence + Per-AI-Crawler Policy

Give it a domain and get back whether it publishes a valid `llms.txt` or `llms-full.txt`,
and whether its `robots.txt` quietly blocks the AI crawlers that matter right now —
GPTBot, ClaudeBot, PerplexityBot and six more, all checked in parallel with a single fetch
each.

### What you get

- **llms.txt presence, not just a 200.** Confirms `/llms.txt` and `/llms-full.txt` are
  real, non-empty, non-HTML-error-page files, and reports their byte size.
- **Per-crawler robots.txt verdict.** Nine AI user-agents (`GPTBot`, `ChatGPT-User`,
  `ClaudeBot`, `anthropic-ai`, `Google-Extended`, `PerplexityBot`, `CCBot`, `Bytespider`,
  `Amazonbot`) parsed per user-agent group, most-specific match wins, reported `blocked` /
  `allowed` / `unspecified`.
- **A one-line verdict** you can hand off as-is: *"no llms.txt, blocks 8/9: GPTBot,
  ChatGPT-User, ClaudeBot…"*.
- **Narrower and complementary to AI Crawler Access Checker**, which audits a broader
  16-bot `robots.txt` matrix with path-level detail — this Actor's job is llms.txt
  presence plus a focused crawler-policy summary.
- **Internal and private addresses are refused, including through a redirect** — only the
  public domain you asked about is ever fetched.
- Runs on Apify: schedule it, monitor it, call it from the API or the MCP server, export
  to JSON, CSV or Excel, or push results straight into your own pipeline.

### How to run it

1. Click **Try for free** — no card needed on the free plan.
2. Paste your domains into **Domains**, one per line.
3. Press **Start**. Results appear in the dataset — read them in the UI, pull them from
   the API, or have a webhook push them onward.

### Pricing

Pay-per-event: **$0.005 per run start + $0.01 per domain audited**. No monthly seat, no
minimum. 100 domains cost about **$1.01**; 1,000 domains about **$10.01**.

A domain that could not be reached at all is still returned, with `found: false` and the
reason — and it is **not** charged for. You pay for answers, not for attempts.

### Input

| Field | Required | What it does |
|---|---|---|
| `items` | yes | Domains to audit for llms.txt presence and AI-crawler robots.txt policy. Up to 100 per run. |
| `maxConcurrency` | no | How many domains to audit at once, 1–50 (default 10). |

```json
{
    "items": ["apify.com", "nytimes.com"]
}
```

### Output

One row per domain. This is a real row from a real run:

```json
{
    "input": "nytimes.com",
    "domain": "https://nytimes.com",
    "found": true,
    "hasLlmsTxt": false,
    "hasLlmsFullTxt": false,
    "llmsTxtBytes": 0,
    "aiCrawlers": {
        "GPTBot": "blocked",
        "ChatGPT-User": "blocked",
        "ClaudeBot": "blocked",
        "anthropic-ai": "blocked",
        "Google-Extended": "blocked",
        "PerplexityBot": "blocked",
        "CCBot": "blocked",
        "Bytespider": "blocked",
        "Amazonbot": "allowed"
    },
    "summary": "no llms.txt, blocks 8/9: GPTBot, ChatGPT-User, ClaudeBot, anthropic-ai, Google-Extended, PerplexityBot, CCBot, Bytespider.",
    "scrapedAt": "2026-07-26T13:20:22.136Z"
}
```

| Field | What it means |
|---|---|
| `found` | Whether the domain was reachable at all — a plain 404 on `llms.txt` is still `found: true` |
| `hasLlmsTxt` / `hasLlmsFullTxt` | Whether a valid, non-empty, non-HTML file exists at that path |
| `llmsTxtBytes` | Byte size of `llms.txt` (0 if absent) |
| `aiCrawlers` | Per-bot verdict: `blocked` (robots.txt disallows `/` for that user-agent or the wildcard group), `allowed`, or `unspecified` (no robots.txt, or no matching group) |

#### Related tools

Related tools for adjacent workflows in AI and search visibility.

| Actor | What it does |
|---|---|
| [AI Crawler Access Checker](https://apify.com/zinin/ai-crawler-access-checker) | Pair it in the AI and search visibility workflow: Check which AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended & more) can access your website |
| [AI Answer Change Alert](https://apify.com/zinin/ai-answer-change-alert) | Pair it in the AI and search visibility workflow: Track whether an AI assistant's answer to a query you care about changed since last time — new sources... |
| [LLM Brand Visibility Tracker](https://apify.com/zinin/llm-brand-visibility) | Pair it in the AI and search visibility workflow: For each query that matters, check whether AI assistants recommend YOUR brand — and which competitors they... |
| [AI Overview Citation Tracker](https://apify.com/zinin/ai-overview-tracker) | Pair it in the AI and search visibility workflow: For each query that matters, see which sources and domains AI assistants cite in their answer — grounded... |
| [Domain Health Checker](https://apify.com/zinin/domain-health-checker) | Pair it in the AI and search visibility workflow: Bulk-audit domains: DNS records, SSL certificate expiry, SPF & DMARC email authentication |

### FAQ

**Does it need an API key or login?** No — public `llms.txt` and `robots.txt` fetches
only.

**How fresh is the data?** Live at run time — every run re-fetches both files, there is no
cache.

**Does a 404 on llms.txt count as `found: false`?** No — `found` only turns `false` when
the domain itself is unreachable (DNS or connection failure). A missing `llms.txt` is a
normal, billed result with `hasLlmsTxt: false`.

**Can I call it from an AI agent?** Yes — a standard Apify Actor, callable from the Apify
API, the SDK, or the Apify MCP server.

**What this is NOT.** It does not check crawler access beyond these 9 bots or beyond the
root path, and it does not validate the *content* of `llms.txt` against a spec — it
confirms the file exists, is real, and reports what `robots.txt` says about it.

Found a wrong result, or need a check we don't run? Open an issue on this Actor's page.

***

Built by [zinin](https://apify.com/zinin). Questions? Telegram [@timzinin](https://t.me/timzinin).

## Commercial guide: llms.txt Auditor — AI Crawler Policy and File Presence

> Audit public llms.txt, llms-full.txt, and root robots.txt policy for nine curated AI crawlers, with live evidence, explicit missing-versus-unreachable semantics, and safe remediation routing.

This guide is written for buyers, operators, analysts, and automation builders. It explains what the Actor observes, how to turn the Dataset into a controlled workflow, and where human verification remains mandatory.

### The decision this product supports

**Does each submitted public domain expose a valid non-empty llms.txt file, and what root robots.txt policy is explicitly observed for the curated AI crawler set?**

The Actor reduces collection and first-pass triage work. It does not remove responsibility for source verification or authorize an external business action. The commercial value comes from a structured, repeatable evidence layer: stable identity, observation time, source evidence, confidence, gaps, recommended action, and failure semantics travel with the raw facts.

### Who uses it

| User | Value |
| --- | --- |
| Small-business website owners | Check whether AI-readable documentation and crawler policy are visible without reading policy files manually. |
| SEO and GEO agencies | Audit a client-approved domain list and deliver a focused llms.txt plus crawler-policy report. |
| Web studios | Identify a missing llms.txt publication opportunity and explicit robots-policy questions for human review. |
| Content operations | Monitor policy files after site releases or infrastructure migrations. |
| Technical marketers | Compare nine named crawler rules while retaining allowed, blocked, and unspecified separately. |
| Automation builders | Export one decision-ready row per domain into a remediation or monitoring queue. |

### Input contract

| Input field | How to use it |
| --- | --- |
| items | Required list of up to 100 public domains. Each domain triggers bounded llms.txt, llms-full.txt, and robots.txt checks. |
| maxConcurrency | Parallel domain audits from 1 to 50. Start conservatively and increase only after confirming source reliability. |

#### Recommended first Input

```json
{
  "items": [
    "apify.com",
    "nytimes.com"
  ],
  "maxConcurrency": 10
}
```

Start with this bounded example, inspect every Dataset field, and only then expand the scope. Input limits are product controls, not inconveniences: they make cost, completeness, and error handling visible.

### Field dictionary

| Field or group | Meaning |
| --- | --- |
| entityId, input, domain | Stable normalized website identity, original input, and audited origin. |
| found | Whether the domain-level audit was reachable enough to support a result; a normal llms.txt 404 is still found=true. |
| hasLlmsTxt, hasLlmsFullTxt, llmsTxtBytes | Validity/presence and bounded byte evidence for non-empty non-HTML policy files. |
| aiCrawlers | Per-user-agent root policy: blocked, allowed, or unspecified. |
| summary | Plain-language llms.txt and blocked-crawler overview. |
| observedAt, firstSeenAt, lastSeenAt, freshness | Live policy-file observation timing. |
| confidenceScore, confidenceBand, confidenceReasons | Evidence support for the precise audit statement. |
| sourceEvidence, dataGaps | Policy-file responses and any unspecified crawler or incomplete-response gaps. |
| recommendedAction, actionPriority, actionReason, safeToAutomate | Review, publish, or monitor routing; automatic site modification remains false. |
| failureType, retryable | Invalid-input versus incomplete-source recovery semantics. |

#### Common decision fields

| Field | Operational meaning |
| --- | --- |
| recordType | The semantic row family. Use it to distinguish a business result from an advisory or terminal record. |
| schemaVersion | Version of the additive decision-intelligence contract. Pin or validate it in strict consumers. |
| entityId | Stable entity identity for deduplication and joins. It is not necessarily a legal identifier. |
| inputRef | The relevant submitted input reference after normalization. |
| observedAt | When the Actor observed or finalized the evidence. It is not necessarily the source publication time. |
| firstSeenAt and lastSeenAt | Always-emitted observation boundaries. Stateful monitors use the compatible baseline/current boundary. Stateless rows set both equal to observedAt for the current run; that equality does not establish historical tenure. |
| freshness | A structured statement about evidence age or availability, not a prediction. Its basis and age unit follow the source-specific field definition. |
| eventId | For monitors, the stable identity of one observed transition or monitor outcome. It is distinct from entityId. |
| before and after | For monitors, the bounded comparable snapshots used for the decision. Null means that side of a comparison was not honestly available. |
| changedFields and changeFlags | Machine-readable monitor deltas and normalized change labels. Empty arrays mean no supported changed field was established, not that every possible real-world fact stayed constant. |
| materialityScore and materialityBand | Magnitude of an observed monitor change when the Actor can calculate it. Materiality is separate from evidence confidence and may be unknown when the source lacks the required facts. |
| confidenceScore | Evidence support on a 0–100 scale. It is separate from materiality, lead score, or business value. |
| confidenceBand | Readable high/medium/low/unknown grouping of evidence support. |
| confidenceReasons | Observed facts that raise confidence. |
| confidenceRisks | Missing, partial, ambiguous, inferred, or conflicting aspects that reduce confidence. |
| confidenceConflict | Explicit consistency warning when structured evidence does not reconcile. |
| sourceEvidence | Source-linked observations supporting the row. Preserve this during export. |
| dataGaps | Important evidence the Actor did not observe or cannot establish. Keep these gaps visible in CRM, spreadsheet, and automation exports. |
| negativeSignals | Machine-readable risks or gaps. A negative signal is not automatically a negative business outcome. |
| recommendedAction | Bounded review label produced from the available evidence. |
| actionPriority | Suggested queue priority, not urgency guaranteed by the source. |
| actionReason | Plain-language explanation for the recommended action. |
| safeToAutomate | Whether the narrow recommended action is deterministic enough for automation. Organizational policy still applies. |
| failureType | Normalized terminal or partial failure classification. Null means no classified failure. |
| retryable | Whether a later retry may legitimately change an operationally incomplete result. |
| recommendation | Human-readable handling guidance, especially for terminal rows. |

### Evidence, confidence, and honest boundaries

#### What the evidence supports

- The Actor fetches exactly the normalized origin policy paths for llms.txt, llms-full.txt, and robots.txt.
- A valid llms file must be non-empty and not an HTML error page; a mere HTTP success is insufficient.
- The curated crawler set contains GPTBot, ChatGPT-User, ClaudeBot, anthropic-ai, Google-Extended, PerplexityBot, CCBot, Bytespider, and Amazonbot.
- robots.txt is parsed by user-agent group with the most-specific applicable rule for the root path.
- Private, loopback, metadata, and other non-public targets are refused, including after redirect resolution.

#### What this Actor never claims

- The Actor does not validate llms.txt content against a legally binding or universally adopted specification.
- It does not prove that an AI provider actually crawled, indexed, trained on, cited, or obeyed the website.
- allowed or unspecified robots policy does not guarantee access through firewalls, bot protection, authentication, or other controls.
- It does not audit crawler paths beyond the documented root-policy interpretation or bots outside the curated nine.
- It does not modify policy files, provide legal advice, or guarantee search or AI visibility.

#### Reading data gaps correctly

A data gap is part of the result. Nulls, partial flags, confidence risks, source failures, and unavailable fields must survive export. Removing these fields makes the remaining facts look more complete than they are. When two sources conflict or a required identity cannot be proven, lower confidence and keep `safeToAutomate=false`.

#### Source evidence is not permission

A public source proves only that a value or statement was observable at the recorded time and URL. It does not establish consent, contractual rights, legal status, accuracy after observation, or authorization for a downstream action. Your organization remains responsible for source terms, privacy rules, outreach policy, retention, and human review.

### Decision policy and action routing

| Action | How to use it |
| --- | --- |
| REVIEW\_AND\_PUBLISH\_LLMS\_TXT | Review site documentation goals and the emerging convention, then author and publish an appropriate file through an authorized website workflow. |
| REVIEW\_AI\_CRAWLER\_POLICY | Confirm the observed robots rules match the organization's intentional policy before editing them. |
| MONITOR\_POLICY\_FILES | Keep the completed observation and schedule another audit after meaningful site or policy changes. |
| RETRY\_SOURCE\_AUDIT | Retry only an incomplete public-source audit; correct invalid inputs rather than looping. |

#### Confidence is not attractiveness

`confidenceScore` answers “how strongly does the available evidence support this factual classification?” It does not answer “how valuable is this lead, property, account, or address?” A high-confidence negative fact may be commercially uninteresting; a low-confidence positive signal may deserve research but not action. Keep the concepts separate in dashboards, exports, and CRM fields.

#### Why safeToAutomate is conservative

`safeToAutomate` is intentionally false whenever the next step could amplify an uncertain inference. It may be true only for narrow deterministic actions explicitly supported by the row, such as suppressing an email with invalid syntax. A true value does not waive legal, privacy, consent, contractual, or organizational rules.

#### Retry policy

- Retry when `retryable=true` and the failure is operational, such as a temporary source or DNS problem.
- Do not endlessly retry deterministic invalid input, policy refusal, or confirmed absence.
- A retry must preserve the original input reference and must not create duplicate downstream actions.
- Budget exhaustion is not negative evidence about the entity. Resume only the unprocessed scope with an authorized budget.
- A failed Actor run is an operational event. Never transform it into “no listing,” “no contact,” “bad lead,” or “invalid email.”

### Commercial use-case playbooks

#### 1. Agency client audit

**Goal.** Submit approved domains, deliver presence, bytes, per-bot policy, evidence confidence, and direct remediation questions.

**Recommended runbook.**

1. Define the submitted cohort and write down why it is in scope.
2. Start with the smallest useful Input and preserve the exact run ID.
3. Inspect the Dataset overview before exporting anything.
4. Check `failureType`, `retryable`, completeness indicators, and `confidenceBand`.
5. Open the relevant `sourceEvidence` or source URL for material rows.
6. Apply the recommended action as a review label, not as an instruction to contact, buy, delete, accuse, or publish.
7. Record the analyst's final disposition in the destination system.

**Do not skip.** A found row proves the stated policy-file presence and root robots.txt interpretation at observation time. It does not prove crawler behavior, indexing, AI citation, compliance, adoption, visibility, or business impact.

#### 2. Website launch checklist

**Goal.** Check llms.txt and robots policy after deployment and manually inspect the published file content.

**Recommended runbook.**

1. Define the submitted cohort and write down why it is in scope.
2. Start with the smallest useful Input and preserve the exact run ID.
3. Inspect the Dataset overview before exporting anything.
4. Check `failureType`, `retryable`, completeness indicators, and `confidenceBand`.
5. Open the relevant `sourceEvidence` or source URL for material rows.
6. Apply the recommended action as a review label, not as an instruction to contact, buy, delete, accuse, or publish.
7. Record the analyst's final disposition in the destination system.

**Do not skip.** A found row proves the stated policy-file presence and root robots.txt interpretation at observation time. It does not prove crawler behavior, indexing, AI citation, compliance, adoption, visibility, or business impact.

#### 3. AI crawler policy review

**Goal.** Compare named bot states with the approved organization policy; keep unspecified distinct from allowed.

**Recommended runbook.**

1. Define the submitted cohort and write down why it is in scope.
2. Start with the smallest useful Input and preserve the exact run ID.
3. Inspect the Dataset overview before exporting anything.
4. Check `failureType`, `retryable`, completeness indicators, and `confidenceBand`.
5. Open the relevant `sourceEvidence` or source URL for material rows.
6. Apply the recommended action as a review label, not as an instruction to contact, buy, delete, accuse, or publish.
7. Record the analyst's final disposition in the destination system.

**Do not skip.** A found row proves the stated policy-file presence and root robots.txt interpretation at observation time. It does not prove crawler behavior, indexing, AI citation, compliance, adoption, visibility, or business impact.

#### 4. GEO documentation project

**Goal.** Use missing llms.txt as a project prompt, not proof that current content is invisible to AI systems.

**Recommended runbook.**

1. Define the submitted cohort and write down why it is in scope.
2. Start with the smallest useful Input and preserve the exact run ID.
3. Inspect the Dataset overview before exporting anything.
4. Check `failureType`, `retryable`, completeness indicators, and `confidenceBand`.
5. Open the relevant `sourceEvidence` or source URL for material rows.
6. Apply the recommended action as a review label, not as an instruction to contact, buy, delete, accuse, or publish.
7. Record the analyst's final disposition in the destination system.

**Do not skip.** A found row proves the stated policy-file presence and root robots.txt interpretation at observation time. It does not prove crawler behavior, indexing, AI citation, compliance, adoption, visibility, or business impact.

#### 5. Portfolio monitoring

**Goal.** Schedule a bounded domain list and alert only on independently computed changes in your downstream state layer.

**Recommended runbook.**

1. Define the submitted cohort and write down why it is in scope.
2. Start with the smallest useful Input and preserve the exact run ID.
3. Inspect the Dataset overview before exporting anything.
4. Check `failureType`, `retryable`, completeness indicators, and `confidenceBand`.
5. Open the relevant `sourceEvidence` or source URL for material rows.
6. Apply the recommended action as a review label, not as an instruction to contact, buy, delete, accuse, or publish.
7. Record the analyst's final disposition in the destination system.

**Do not skip.** A found row proves the stated policy-file presence and root robots.txt interpretation at observation time. It does not prove crawler behavior, indexing, AI citation, compliance, adoption, visibility, or business impact.

#### 6. CMS migration verification

**Goal.** Audit before and after a migration and confirm redirects did not expose internal or unintended policy targets.

**Recommended runbook.**

1. Define the submitted cohort and write down why it is in scope.
2. Start with the smallest useful Input and preserve the exact run ID.
3. Inspect the Dataset overview before exporting anything.
4. Check `failureType`, `retryable`, completeness indicators, and `confidenceBand`.
5. Open the relevant `sourceEvidence` or source URL for material rows.
6. Apply the recommended action as a review label, not as an instruction to contact, buy, delete, accuse, or publish.
7. Record the analyst's final disposition in the destination system.

**Do not skip.** A found row proves the stated policy-file presence and root robots.txt interpretation at observation time. It does not prove crawler behavior, indexing, AI citation, compliance, adoption, visibility, or business impact.

#### 7. Client report enrichment

**Goal.** Join entityId with existing website records and preserve source evidence and observation time.

**Recommended runbook.**

1. Define the submitted cohort and write down why it is in scope.
2. Start with the smallest useful Input and preserve the exact run ID.
3. Inspect the Dataset overview before exporting anything.
4. Check `failureType`, `retryable`, completeness indicators, and `confidenceBand`.
5. Open the relevant `sourceEvidence` or source URL for material rows.
6. Apply the recommended action as a review label, not as an instruction to contact, buy, delete, accuse, or publish.
7. Record the analyst's final disposition in the destination system.

**Do not skip.** A found row proves the stated policy-file presence and root robots.txt interpretation at observation time. It does not prove crawler behavior, indexing, AI citation, compliance, adoption, visibility, or business impact.

#### 8. Failure exception queue

**Goal.** Separate invalid domain, unreachable origin, and policy absence so technical failures never become false remediation claims.

**Recommended runbook.**

1. Define the submitted cohort and write down why it is in scope.
2. Start with the smallest useful Input and preserve the exact run ID.
3. Inspect the Dataset overview before exporting anything.
4. Check `failureType`, `retryable`, completeness indicators, and `confidenceBand`.
5. Open the relevant `sourceEvidence` or source URL for material rows.
6. Apply the recommended action as a review label, not as an instruction to contact, buy, delete, accuse, or publish.
7. Record the analyst's final disposition in the destination system.

**Do not skip.** A found row proves the stated policy-file presence and root robots.txt interpretation at observation time. It does not prove crawler behavior, indexing, AI citation, compliance, adoption, visibility, or business impact.

### Integration recipes

All examples use placeholders. Keep the Apify token in a secret manager and never write it into a Dataset, README, screenshot, or client-side application.

#### cURL: start a run and wait briefly

```bash
curl -sS -X POST 'https://api.apify.com/v2/acts/zinin~llms-txt-auditor/runs?waitForFinish=60' \
  -H "Authorization: Bearer $APIFY_TOKEN" \
  -H 'Content-Type: application/json' \
  --data '{"items":["apify.com","nytimes.com"],"maxConcurrency":10}'
```

The run response includes `defaultDatasetId`. Read clean JSON rows with:

```bash
curl -sS "https://api.apify.com/v2/datasets/$DEFAULT_DATASET_ID/items?clean=true&format=json" \
  -H "Authorization: Bearer $APIFY_TOKEN"
```

#### JavaScript with apify-client

```javascript
import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const input = {
  "items": [
    "apify.com",
    "nytimes.com"
  ],
  "maxConcurrency": 10
};
const run = await client.actor('zinin/llms-txt-auditor').call(input);
const { items } = await client.dataset(run.defaultDatasetId).listItems({ clean: true });

for (const row of items) {
    console.log({
        entityId: row.entityId,
        confidenceBand: row.confidenceBand,
        recommendedAction: row.recommendedAction,
        safeToAutomate: row.safeToAutomate,
        failureType: row.failureType,
    });
}
```

#### Python with apify-client

```python
import os
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])
run = client.actor("zinin/llms-txt-auditor").call(run_input={
    "items": [
        "apify.com",
        "nytimes.com"
    ],
    "maxConcurrency": 10
})

for row in client.dataset(run["defaultDatasetId"]).iterate_items(clean=True):
    print({
        "entityId": row.get("entityId"),
        "confidenceBand": row.get("confidenceBand"),
        "recommendedAction": row.get("recommendedAction"),
        "safeToAutomate": row.get("safeToAutomate"),
        "failureType": row.get("failureType"),
    })
```

#### Apify MCP call

```json
{
  "name": "call-actor",
  "arguments": {
    "actor": "zinin/llms-txt-auditor",
    "input": {
      "items": [
        "apify.com",
        "nytimes.com"
      ],
      "maxConcurrency": 10
    }
  }
}
```

#### Generic webhook consumer policy

1. Trigger on a terminal Actor run event.
2. Confirm the run status is `SUCCEEDED` before reading business rows.
3. Retrieve rows from `defaultDatasetId`.
4. Reject or quarantine rows whose `failureType` is non-null unless your policy explicitly handles that failure.
5. Send `safeToAutomate=false` rows to a human-review queue.
6. Store `entityId`, `observedAt`, `sourceEvidence`, confidence, action, and the Apify run ID together.
7. Make retries idempotent by keying the destination on the stable entity ID plus the intended observation or event identity.

### Where this fits in a practical stack

| Destination | Recommended pattern |
| --- | --- |
| Apify Console | Use the visual Input form, start the run, then open the default Dataset overview. This is the fastest path for a one-off review and the best place to inspect evidence before automating anything. |
| Apify API | POST JSON input to the Actor run endpoint, wait or poll for completion, then read the default Dataset through the URL returned by the run object. |
| JavaScript client | Use apify-client from a Node.js service, pass the same JSON object as the Console Input, and preserve the returned run and Dataset IDs in your own audit log. |
| Python client | Use apify-client in a Python enrichment job, iterate Dataset items, and route rows by recommendedAction, confidenceBand, failureType, and retryable. |
| Make | Start the Actor from a scenario, wait for the run, retrieve Dataset items, filter unsafe or low-confidence rows, then insert review-ready rows into the destination application. |
| Zapier | Use an Apify run action or webhook trigger, fetch Dataset items, apply a Filter step, and send only review-approved fields into the next sales or operations step. |
| n8n | Use HTTP Request or Apify nodes, branch on failureType and retryable, keep a manual-review lane for safeToAutomate=false, and write sourceEvidence together with the business fields. |
| Google Sheets | Export the Dataset directly or append rows from an automation. Keep stable entityId as a hidden key so reruns update the correct record instead of creating ambiguous duplicates. |
| Airtable | Map entityId to a primary or deduplication field, store confidence and evidence in separate columns, and expose recommendedAction as the triage view. |
| Webhook | Configure an Apify webhook for terminal run states, retrieve the Dataset after SUCCEEDED, and treat FAILED or TIMED-OUT runs as operational events rather than negative business evidence. |

#### A safe automation shape

The Actor is a collection and decision-support component. A production workflow should keep raw evidence, decision metadata, and business action in distinct layers:

1. **Collect:** run the Actor with explicit bounded input.
2. **Validate:** require a successful run and schema-valid Dataset rows.
3. **Triage:** branch on `failureType`, `retryable`, `confidenceBand`, and `safeToAutomate`.
4. **Review:** open source evidence for rows that may affect a person, campaign, investment, compliance decision, or customer record.
5. **Act:** execute only the action approved by your own policy and authorized operator.
6. **Audit:** retain run ID, Dataset ID, observation time, input reference, source evidence, and the final human decision.

This separation prevents a common automation error: turning “data was observed” into “a business action is justified.”

### Operating guide

#### Before the first production run

1. Write the business question in one sentence: **Does each submitted public domain expose a valid non-empty llms.txt file, and what root robots.txt policy is explicitly observed for the curated AI crawler set?**
2. Confirm every submitted input is within your authorized scope.
3. Use the prefilled small example and review all returned row types.
4. Map stable identifiers, confidence, evidence, actions, gaps, failure, and retry fields into the destination.
5. Establish a human owner for review exceptions.
6. Set a run budget and output bound appropriate to the test.
7. Verify that secrets are stored only in the platform or workflow secret manager.

#### After every scheduled run

1. Check terminal run status and logs.
2. Compare the number of submitted entities, produced business rows, and advisory rows.
3. Review partial, unknown, conflict, and low-confidence buckets.
4. Inspect a sample of source evidence, including at least one positive and one negative result.
5. Confirm the destination deduplicated on the intended stable key.
6. Verify that no downstream action was triggered from an error row.
7. Track cost per useful reviewed row rather than cost per raw request alone.

#### Production monitoring signals

Monitor source-unavailable rate, partial-row rate, low-confidence share, missing evidence, retry volume, run duration, Dataset row count, and spend. A sudden shift may indicate source drift, input drift, or an upstream outage. Stop automation and investigate before accepting a new pattern as business truth.

#### Cost control

Start with two domains that have different expected policies. Inspect the three source responses and per-bot states, then increase concurrency and batch size. Verify current result pricing in the live Apify panel.

Use `maxTotalChargeUsd` when calling a monetized Actor if your workflow supports it. Treat a buyer-set cap as a hard safety boundary. If the cap stops work, the unfinished items remain unprocessed; they do not become negative results.

### Review templates and quality reporting

#### Row-review worksheet

For every material row, an analyst should be able to answer the following without relying on memory or an unstated assumption:

1. What submitted entity or query does this row refer to?
2. Is it a business result, a baseline/advisory row, a partial observation, or a failure?
3. Which exact source evidence supports the headline fact?
4. When was the evidence observed, and is there a different source publication time?
5. Which fields are direct observations, which are normalized, and which are deterministic derivations?
6. What important evidence is null, missing, partial, ambiguous, or conflicting?
7. Does confidence describe evidence support only, or has someone incorrectly treated it as business value?
8. What recommended action is present, and what additional verification does its reason require?
9. Is the narrow action marked safe to automate? If yes, does organizational policy also permit it?
10. What final human disposition was made, by whom, and from which run and Dataset item?

#### Field-group review prompts

##### 1. `entityId, input, domain`

**Contract meaning:** Stable normalized website identity, original input, and audited origin.

**Reviewer prompts:** Is the value present? Does its type match the schema? Is it supported by sourceEvidence or a documented deterministic transformation? Is any null being silently converted into a default? Would the value still mean the same thing after CSV export? Does the destination preserve the related confidence and gap fields?

##### 2. `found`

**Contract meaning:** Whether the domain-level audit was reachable enough to support a result; a normal llms.txt 404 is still found=true.

**Reviewer prompts:** Is the value present? Does its type match the schema? Is it supported by sourceEvidence or a documented deterministic transformation? Is any null being silently converted into a default? Would the value still mean the same thing after CSV export? Does the destination preserve the related confidence and gap fields?

##### 3. `hasLlmsTxt, hasLlmsFullTxt, llmsTxtBytes`

**Contract meaning:** Validity/presence and bounded byte evidence for non-empty non-HTML policy files.

**Reviewer prompts:** Is the value present? Does its type match the schema? Is it supported by sourceEvidence or a documented deterministic transformation? Is any null being silently converted into a default? Would the value still mean the same thing after CSV export? Does the destination preserve the related confidence and gap fields?

##### 4. `aiCrawlers`

**Contract meaning:** Per-user-agent root policy: blocked, allowed, or unspecified.

**Reviewer prompts:** Is the value present? Does its type match the schema? Is it supported by sourceEvidence or a documented deterministic transformation? Is any null being silently converted into a default? Would the value still mean the same thing after CSV export? Does the destination preserve the related confidence and gap fields?

##### 5. `summary`

**Contract meaning:** Plain-language llms.txt and blocked-crawler overview.

**Reviewer prompts:** Is the value present? Does its type match the schema? Is it supported by sourceEvidence or a documented deterministic transformation? Is any null being silently converted into a default? Would the value still mean the same thing after CSV export? Does the destination preserve the related confidence and gap fields?

##### 6. `observedAt, firstSeenAt, lastSeenAt, freshness`

**Contract meaning:** Live policy-file observation timing.

**Reviewer prompts:** Is the value present? Does its type match the schema? Is it supported by sourceEvidence or a documented deterministic transformation? Is any null being silently converted into a default? Would the value still mean the same thing after CSV export? Does the destination preserve the related confidence and gap fields?

##### 7. `confidenceScore, confidenceBand, confidenceReasons`

**Contract meaning:** Evidence support for the precise audit statement.

**Reviewer prompts:** Is the value present? Does its type match the schema? Is it supported by sourceEvidence or a documented deterministic transformation? Is any null being silently converted into a default? Would the value still mean the same thing after CSV export? Does the destination preserve the related confidence and gap fields?

##### 8. `sourceEvidence, dataGaps`

**Contract meaning:** Policy-file responses and any unspecified crawler or incomplete-response gaps.

**Reviewer prompts:** Is the value present? Does its type match the schema? Is it supported by sourceEvidence or a documented deterministic transformation? Is any null being silently converted into a default? Would the value still mean the same thing after CSV export? Does the destination preserve the related confidence and gap fields?

##### 9. `recommendedAction, actionPriority, actionReason, safeToAutomate`

**Contract meaning:** Review, publish, or monitor routing; automatic site modification remains false.

**Reviewer prompts:** Is the value present? Does its type match the schema? Is it supported by sourceEvidence or a documented deterministic transformation? Is any null being silently converted into a default? Would the value still mean the same thing after CSV export? Does the destination preserve the related confidence and gap fields?

##### 10. `failureType, retryable`

**Contract meaning:** Invalid-input versus incomplete-source recovery semantics.

**Reviewer prompts:** Is the value present? Does its type match the schema? Is it supported by sourceEvidence or a documented deterministic transformation? Is any null being silently converted into a default? Would the value still mean the same thing after CSV export? Does the destination preserve the related confidence and gap fields?

#### Weekly quality report

Create a recurring internal report with these measures. The report is about pipeline health, not market demand unless the source contract explicitly measures demand.

| Metric | Why it matters | Investigate when |
| --- | --- | --- |
| Submitted inputs | Defines the actual denominator and scope of the run. | The count differs from the approved batch or schedule. |
| Business result rows | Shows how many usable observations were produced. | The rate changes sharply without an input explanation. |
| Advisory/failure rows | Prevents operational failures from disappearing in a results-only dashboard. | Any terminal class grows or is unmapped. |
| Partial-result rate | Measures incomplete source coverage or configured truncation. | It rises, or analysts stop seeing the partial warning. |
| Low-confidence rate | Shows the share of rows requiring more evidence. | It rises by source, cohort, or input pattern. |
| Retryable failure rate | Distinguishes temporary operational issues from deterministic outcomes. | Retries repeat without improving evidence. |
| Evidence-link coverage | Confirms material facts remain traceable after export. | Links or evidence objects are missing from delivered records. |
| Safe-automation share | Shows how little or much of the workflow can be deterministic. | A mapping change makes unsafe actions appear safe. |
| Manual-review backlog | Measures whether human verification capacity matches collection volume. | Rows age beyond the campaign or decision window. |
| Duplicate destination writes | Tests idempotency and stable identity mapping. | The same entity/run creates multiple external actions. |
| Cost per reviewed useful row | Relates platform spend to approved, decision-useful output. | Raw volume rises but reviewed utility falls. |
| Source-drift exceptions | Detects changed markup, response shape, policy, or source availability. | A new unknown pattern survives more than one bounded check. |

#### Client-facing delivery note template

Use a note like this when delivering exports to a client or another team:

> This Dataset contains bounded public-source observations produced by the Apify Actor for the submitted Input. Each row includes observation time, evidence confidence, recommended review action, and explicit gaps where available. A positive row is not proof of buyer intent, permission, legal status, future outcome, or any fact listed in the Actor's “never claims” section. Partial and failure rows are included so coverage is not overstated. Validate material rows at their source before acting.

Add the Actor URL, run URL, Dataset URL, build/version, exact Input scope, observation window, pricing model observed for the run, reviewer name, and date of approval.

#### CRM disposition vocabulary

Keep collection results and sales dispositions separate. A practical downstream vocabulary is:

- `needs_evidence_review`: useful signal exists but a reviewer has not approved it.
- `needs_identity_review`: entity or ownership association is not sufficiently proven.
- `needs_policy_review`: contact, privacy, suppression, legal, or contractual policy must be checked.
- `approved_for_research`: an analyst may perform more research; this is not approval for outreach.
- `approved_for_authorized_action`: a named operator approved one specific action under the organization's policy.
- `retry_operational_failure`: the source or infrastructure failed and a bounded retry is appropriate.
- `closed_no_supported_signal`: the completed bounded check found no supported signal; this is not a universal negative fact.
- `closed_out_of_scope`: the input should not have entered this workflow.

Never overwrite `recommendedAction` with the CRM disposition. The first is Actor-produced decision support; the second is your organization's accountable decision.

#### Sampling plan

For a new workflow, review every row in the first small run. When the contract is understood, sample all failure and partial rows plus a representative set of high-, medium-, and low-confidence results. Re-expand to full review whenever the source changes, the schema version changes, a new input cohort is introduced, the error distribution shifts, or a downstream user reports an unexplained result.

#### Change-management record

When you change field mappings or automation policy, record:

1. Previous mapping or rule.
2. New mapping or rule.
3. Actor build/version and schemaVersion used for validation.
4. Test run and Dataset URLs.
5. Positive, negative, partial, retry, and budget fixtures inspected.
6. Security and privacy review outcome.
7. Approver and activation time.
8. Rollback condition and responsible operator.

This makes a commercial data workflow supportable. Without the record, a later operator cannot distinguish a real source change from an undocumented mapping change.

### Delivery patterns for marketing and small-business teams

#### One-off research

Run the Actor in Console, inspect the overview table, open evidence for each material row, and export only the approved subset. Record the run URL in the client or campaign notes.

#### Recurring watch or hygiene job

Use an Apify schedule. Write rows into a staging table keyed by `entityId`. Compare current and previous observations only when the Actor supplies valid state or your own pipeline implements an explicit comparable baseline. Never infer a change from a failed run.

#### Agency client delivery

Deliver three views: business results, evidence/quality exceptions, and operational failures. Include the run URL, observation time, configured scope, and a plain-language statement of what the Actor does not prove. This makes the deliverable auditable and reduces disputes caused by overclaiming.

#### CRM enrichment

Write into staging fields first. A human or approved policy promotes values into canonical CRM fields. Keep raw source values separate from normalized and decision fields, and do not replace a verified value with a lower-confidence observation.

#### AI-assisted review

An LLM can summarize rows, but it must receive the evidence, confidence risks, negative signals, and limitations. Require citations to sourceEvidence and prohibit invented identity, intent, legal, funding, mailbox, valuation, or availability facts.

### Buyer and operator acceptance checklist

Use this checklist before calling the workflow production-ready.

#### Product fit

- \[ ] The business question matches: **Does each submitted public domain expose a valid non-empty llms.txt file, and what root robots.txt policy is explicitly observed for the curated AI crawler set?**
- \[ ] The submitted entities were selected through an authorized process.
- \[ ] A human owner understands the positive, negative, partial, and failure row types.
- \[ ] The team accepts the boundaries listed in “What this Actor never claims.”
- \[ ] The destination keeps evidence confidence separate from business scoring.

#### Input and run controls

- \[ ] `items` is explicitly reviewed and bounded.
- \[ ] `maxConcurrency` is explicitly reviewed and bounded.
- \[ ] The first production-like run uses a small representative sample.
- \[ ] A maximum charge or internal spend alert is configured where appropriate.
- \[ ] The workflow records Actor ID, build/version, run ID, Dataset ID, and input hash.

#### Data handling

- \[ ] `entityId` is mapped to an idempotent destination key.
- \[ ] `observedAt` and source-specific time fields remain distinct.
- \[ ] `sourceEvidence`, gaps, and nulls are preserved.
- \[ ] Advisory and failure rows cannot enter the positive-results lane.
- \[ ] Low-confidence and partial rows have a visible manual-review view.
- \[ ] Retention and deletion rules match the type of data collected.

#### Action safety

- \[ ] `recommendedAction` is treated as a review label.
- \[ ] `safeToAutomate=false` blocks automatic external action.
- \[ ] Consent, suppression, legal, contractual, and platform rules are evaluated downstream.
- \[ ] A reviewer can trace a material action back to source evidence and run metadata.
- \[ ] Retry logic cannot duplicate a downstream action.

#### Ongoing quality

- \[ ] The team monitors failure, retry, partial, low-confidence, and empty-result rates.
- \[ ] A source-drift threshold pauses the workflow for inspection.
- \[ ] Sample evidence is manually reviewed on a recurring basis.
- \[ ] Cost per useful reviewed row is measured.
- \[ ] Documentation and field mappings are updated when schemaVersion changes.

### Frequently asked questions

#### Is this a database?

No. It is an on-demand observation tool. Each run collects or evaluates the submitted scope and records evidence at that time.

#### Does a found row prove commercial interest?

No. A found row proves only the factual observation described by its fields. Buyer intent is never inferred.

#### Can I automatically contact every result?

No. Use recommendedAction as triage, verify the evidence and identity, and apply your own consent, privacy, suppression, and outreach rules.

#### Why is safeToAutomate often false?

Because a useful observation can still require identity, context, legal, or source verification before action. Conservative routing prevents false certainty from scaling.

#### What should I do with low confidence?

Open confidenceRisks and sourceEvidence, close the important gap, or keep the row in a manual queue. Do not hide the confidence field.

#### What does partial mean?

The Actor obtained some usable evidence but could not support a complete observation of the configured scope. Partial is not the same as empty.

#### What is a confirmed zero?

Only an explicit source or deterministic rule can support a confirmed absence. An outage, truncation, or unreadable response is not a zero.

#### Should I retry every failure?

No. Retry only when retryable is true. Invalid input, policy refusal, or deterministic classification should be corrected or handled, not looped.

#### Can I delete failure rows?

You can exclude them from a business-results view, but retain them in operational logs so Dataset completeness and retry decisions stay explainable.

#### How should I deduplicate?

Use entityId for the entity and, for stateful monitors, eventId for the observed transition. Also retain the Apify run ID.

#### Can I treat confidence as conversion probability?

No. Confidence measures evidence support, not purchase probability, revenue, suitability, or expected return.

#### Can I change the recommended action?

Yes. It is an explainable default. Your downstream policy can be stricter, and should encode organization-specific authorization and risk tolerance.

#### How do I estimate cost?

Run the smallest representative input, inspect live event prices and run usage in Apify, then model the number of billable result events. The live pricing panel is authoritative.

#### Why use a small prefill?

It produces a cheap, fast, inspectable first run and reduces the chance of scaling a wrong input or workflow assumption.

#### Can I schedule it?

Yes. Use an Apify schedule, but make the destination idempotent and review changes in failure, partial, and confidence rates.

#### Can I export CSV or Excel?

Yes. Apify Datasets support common export formats. JSON is recommended when you need nested evidence and decision fields.

#### Can I send results to Sheets or Airtable?

Yes. Preserve entityId, confidence, evidence, gaps, actions, and failure fields instead of mapping only the headline value.

#### Can I use it from Make, Zapier, or n8n?

Yes. Start the Actor, wait for a successful terminal state, read Dataset items, then branch on decision and failure fields.

#### Can an LLM consume the output?

Yes, but pass the structured evidence and limitations together. Instruct the model not to invent missing facts and to cite sourceEvidence.

#### What happens when a source changes?

The run may become partial, unavailable, or fail validation. Monitor these rates and inspect logs before treating changed output as a real-world shift.

#### Does public mean unrestricted?

No. Public visibility does not remove source terms, privacy obligations, retention rules, or the need for a legitimate downstream purpose.

#### Is a source URL permanent?

Not necessarily. Store observation time and material facts because web content can change or disappear.

#### Can I rely on one row for a high-stakes decision?

No. High-stakes legal, financial, employment, compliance, safety, or personal decisions require appropriate primary evidence and qualified review.

#### How do I report a suspected parsing issue?

Provide the Actor run ID, a redacted input, affected field, expected source evidence, and whether the issue reproduces. Never include tokens or private data.

#### What does success mean?

A found row proves the stated policy-file presence and root robots.txt interpretation at observation time. It does not prove crawler behavior, indexing, AI citation, compliance, adoption, visibility, or business impact.

### Support information to include with an issue

Provide the public Actor name, Apify run ID, Dataset item index or stable entity ID, a redacted Input, the relevant source URL, expected behavior, observed behavior, and whether retrying produced the same result. Do not include an Apify token, API key, private customer record, or unnecessary personal data.

### Final interpretation rule

A found row proves the stated policy-file presence and root robots.txt interpretation at observation time. It does not prove crawler behavior, indexing, AI citation, compliance, adoption, visibility, or business impact.

# Actor input Schema

## `items` (type: `array`):

List of domains to audit for llms.txt presence and AI-crawler robots.txt policy.

## `maxConcurrency` (type: `integer`):

How many domains to audit in parallel. Start conservatively and increase only after source reliability is understood.

## Actor input object example

```json
{
  "items": [
    "vercel.com",
    "nytimes.com"
  ],
  "maxConcurrency": 10
}
```

# Actor output Schema

## `results` (type: `string`):

API URL for the default dataset items produced by this run.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "items": [
        "vercel.com",
        "nytimes.com"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("zinin/llms-txt-auditor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "items": [
        "vercel.com",
        "nytimes.com",
    ] }

# Run the Actor and wait for it to finish
run = client.actor("zinin/llms-txt-auditor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "items": [
    "vercel.com",
    "nytimes.com"
  ]
}' |
apify call zinin/llms-txt-auditor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,zinin/llms-txt-auditor"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/gfNJ5SGq6UbTgn3ir/builds/fyCzjwHGueM8CnxTf/openapi.json