# AI Crawler Policy Auditor (`junipr/ai-crawler-policy-auditor`) Actor
Audit robots and AI crawler policy signals for AI crawler access, blocking, and site-readiness.
- **URL**: https://apify.com/junipr/ai-crawler-policy-auditor.md
- **Developed by:** [junipr](https://apify.com/junipr) (community)
- **Categories:** Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet
## Pricing
from $4.90 / 1,000 page auditeds
This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event
## What's an Apify Actor?
Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".
## How to integrate an Actor?
If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.
In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js/docs.md):
```bash
npm install apify-client
```
In Python projects, use official [Python client library](https://docs.apify.com/api/client/python/docs.md):
```bash
pip install apify-client
```
In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md):
````bash
# MacOS / Linux
curl -fsSL https://apify.com/install-cli.sh | bash
# Windows
irm https://apify.com/install-cli.ps1 | iex
```bash
In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/integrations/mcp.md).
If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md).
For usage examples, see the [API](#api) section below.
For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).
# README
## AI Crawler Policy Auditor
### Store Positioning
**Store title:** AI Crawler Policy Auditor
**Short description:** Audit robots and AI crawler policy signals for AI crawler access, blocking, and site-readiness.
**SEO title:** AI Crawler Policy Auditor — technical SEO, web, and domain audit
**SEO description:** Audit robots and AI crawler policy signals for AI crawler access, blocking, and site-readiness. Use it to find crawlability, indexability, security, metadata, and page-quality issues with evidence-backed rows and audit reports.
**Categories:** SEO_TOOLS
**Keywords:** ai, crawler, policy, auditor, web/domain audit
### Fixed-Inclusive PPE Pricing
This actor uses pay-per-event pricing. Event prices include Apify platform usage; users are not expected to pay a separate platform-usage pass-through charge for the configured pricing model.
- Tier: W1 — Web/domain audit
- Primary event: `page-audited` at $0.00490 base
- Default max charge: $10.00
- Store discounts: FREE/BRONZE base, SILVER discounted, GOLD deepest approved discount
Event set:
- `actor-start`: base $0.00500, GOLD $0.00400. Ai Crawler Policy Auditor: charged when actor start is completed. The price includes Apify platform usage; no separate usage pass-through is intended.
- `page-audited`: base $0.00490, GOLD $0.00392. Ai Crawler Policy Auditor: charged when page audited is completed. The price includes Apify platform usage; no separate usage pass-through is intended.
- `rule-parsed`: base $0.00372, GOLD $0.00298. Ai Crawler Policy Auditor: charged when rule parsed is completed. The price includes Apify platform usage; no separate usage pass-through is intended.
- `finding-emitted`: base $0.00372, GOLD $0.00298. Ai Crawler Policy Auditor: charged when finding emitted is completed. The price includes Apify platform usage; no separate usage pass-through is intended.
- `audit-report-generated`: base $0.05000, GOLD $0.04000. Ai Crawler Policy Auditor: charged when audit report generated is completed. The price includes Apify platform usage; no separate usage pass-through is intended.
### Public Task Concepts
- Audit AI Crawler Policy controls with capped supplied targets
- Find high-priority AI Crawler Policy issues before release
- Validate AI Crawler Policy evidence from supplied pages
- Prioritize AI Crawler Policy fixes with severity and proof
- Export AI Crawler Policy QA rows for client review
Audit robots.txt, llms.txt, X-Robots-Tag, and meta robots signals for AI crawler access, blocking, and site-readiness.
### What This Actor Does
AI Crawler Policy Auditor checks whether major AI and search crawler user agents are allowed, blocked, partially blocked, or unmanaged for a target path. It parses supplied robots.txt snapshots by default, can compare previous and current policies, detects page-level robots restrictions such as `noindex` and `noai`, and writes one structured dataset row per audited target.
The default run is local-first: it uses an inline fixture, makes no network calls, and writes summary diagnostics to the key-value store. Optional live robots fetching is available only when `fetchUrls` is explicitly set to `true`, with strict target, timeout, and byte caps.
### What This Actor Does Not Do
- It does not guarantee search ranking, AI citation, traffic, revenue, or compliance outcomes.
- It does not provide legal, medical, or financial advice.
- It does not collect sensitive personal data.
- It does not crawl full websites or render pages in a browser.
- It does not fetch live URLs unless `fetchUrls` is explicitly enabled.
### Operational Status
- Package status: local-green.
- Cloud status: cloud-deferred until the main actor publication workflow pushes and verifies the actor remotely.
- Default input: inline robots fixture with `fetchUrls: false`.
- Store icon workflow: actor-local `icon.png` is included; shared landing icon manifests and live Console upload are intentionally left to the main agent.
### Best Use Cases
- Confirm whether GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended, CCBot, Bytespider, and similar agents can access key pages.
- Audit an intentional AI crawler opt-out policy before a launch or client delivery.
- Catch robots.txt changes that newly block or unblock AI crawler agents.
- Detect mixed policy signals where robots.txt allows crawlers but page-level tags say `noindex`, `noai`, or `noimageai`.
- Produce a concise KVS report for SEO, content, platform, or AI search readiness QA.
### Why Use This Actor
| Option | Strength | Weakness | Best fit |
| --- | --- | --- | --- |
| Manual robots.txt review | Fast for one site | Easy to miss wildcard behavior, path specificity, and changed blocked agents | One-off spot checks |
| Generic SEO crawler | Broad page crawl | Usually not focused on AI crawler user-agent policy or supplied examples testing | Full-site SEO diagnostics |
| AI Crawler Policy Auditor | Focused AI crawler access decisions, deterministic defaults, KVS report, PPE billing controls | Does not render pages or crawl deep paths | AI crawler policy QA and repeatable robots snapshots |
### How To Use
Run the actor with the default input for a zero-config local policy audit. To audit your own policy without network calls, paste robots.txt content into `targets[].robotsTxt` and keep `fetchUrls` set to `false`.
```json
{
"targets": [
{
"sourceId": "homepage-policy",
"sourceUrl": "https://example.com",
"targetPath": "/",
"robotsTxt": "User-agent: *\nAllow: /\n\nUser-agent: GPTBot\nAllow: /\n\nUser-agent: ClaudeBot\nDisallow: /\nSitemap: https://example.com/sitemap.xml",
"llmsTxt": "# AI crawler guidance\nAllowed for documentation indexing.",
"headers": {
"x-robots-tag": "index, follow"
},
"html": ""
}
],
"expectedAccess": "allow",
"fetchUrls": false,
"includeReport": true,
"maxTargets": 10
}
````
#### AI Readiness Check
Use `expectedAccess: "allow"` when the goal is discoverability or AI search readiness. Blocked AI crawler agents become warnings or failures depending on scope.
#### AI Opt-Out Check
Use `expectedAccess: "block"` when the goal is to verify an opt-out policy. The actor flags agents that remain allowed, partially blocked, or unmanaged.
#### Inventory-Only Review
Use `expectedAccess: "neutral"` when you only want a policy inventory. Mixed policies are still surfaced, but the actor avoids assuming whether blocking is good or bad.
### Input Configuration
| Field | Type | Default | Description |
| --- | --- | --- | --- |
| `targets` | array | Inline fixture | Robots, llms.txt, header, and HTML snapshots to audit. |
| `urls` | string array | `[]` | Optional site URLs used only when `fetchUrls` is true. |
| `fetchUrls` | boolean | `false` | Fetch `/robots.txt` and `/llms.txt` with timeout and byte caps. |
| `expectedAccess` | string | `allow` | `allow`, `block`, or `neutral`. |
| `auditedAgents` | string array | Major AI and search crawlers | User-agent tokens to evaluate against robots.txt. |
| `comparePrevious` | boolean | `true` | Compare `previousRobotsTxt` when supplied. |
| `requireLlmsTxt` | boolean | `false` | Flag targets without llms.txt content. |
| `includeReport` | boolean | `true` | Write JSON and Markdown report artifacts to KVS. |
| `maxTargets` | integer | `25` | Maximum targets to audit; hard cap is `250`. |
| `fetchTimeoutMs` | integer | `10000` | Timeout for optional live fetches. |
| `maxRobotsBytes` | integer | `200000` | Maximum bytes to read from fetched robots.txt. |
| `maxChargeUsd` | number | `10` | Hard PPE cap; stops before work or withholds uncharged reports when the next event would exceed it. |
| `debug` | boolean | `false` | Enable debug logs. |
### Output Format
Each dataset row represents one audited target. Important fields include:
- `status`, `severity`, and `aiCrawlerReadinessScore`: Overall result.
- `expectedAccess` and `policySummary`: Goal and plain-English outcome.
- `allowedAgents`, `blockedAgents`, `partiallyBlockedAgents`, and `unknownAgents`: Per-agent access buckets.
- `agentPolicies`: Detailed matched robots group and rule for each audited user agent.
- `hasRobotsTxt`, `robotsGroupCount`, `hasSitemap`, `hasLlmsTxt`, and byte counts: Policy inventory.
- `xRobotsTag`, `metaRobotsDirectives`, `hasNoindexSignal`, and `hasNoaiSignal`: Page-level policy signals.
- `changedRobotsPolicy`, `addedBlockedAgents`, and `removedBlockedAgents`: Previous/current diff results.
- `issues` and `recommendation`: Structured findings and next action.
The key-value store also contains:
- `AI_CRAWLER_POLICY_RESULTS.json`: Full result array.
- `AI_CRAWLER_POLICY_SUMMARY.json`: Run-level counts, scores, blocked agents, and top issues.
- `AI_CRAWLER_POLICY_REPORT.md`: Human-readable report.
### Example Output
```json
{
"sourceId": "sample-ai-ready-site",
"sourceUrl": "https://example.com",
"targetPath": "/",
"status": "pass",
"severity": "none",
"aiCrawlerReadinessScore": 100,
"expectedAccess": "allow",
"policySummary": "Audited crawlers are allowed for the target path.",
"allowedAgents": ["Bingbot", "Bytespider", "CCBot", "ChatGPT-User", "Claude-User", "ClaudeBot", "GPTBot", "Google-Extended", "GoogleOther", "Googlebot", "Meta-ExternalAgent", "OAI-SearchBot", "PerplexityBot", "anthropic-ai"],
"blockedAgents": [],
"unknownAgents": [],
"hasRobotsTxt": true,
"hasSitemap": true,
"hasLlmsTxt": true,
"hasNoindexSignal": false,
"hasNoaiSignal": false,
"issueCount": 0,
"recommendation": "AI crawler policy signals are aligned for the supplied target."
}
```
### Integration Examples
#### Node.js
```js
import { ApifyClient } from 'apify-client';
const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('junipr/ai-crawler-policy-auditor').call({
targets: [{ sourceId: 'site', robotsTxt: 'User-agent: *\nAllow: /' }],
expectedAccess: 'allow',
fetchUrls: false
});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items[0].blockedAgents);
```
#### Python
```python
from apify_client import ApifyClient
client = ApifyClient(token="YOUR_APIFY_TOKEN")
run = client.actor("junipr/ai-crawler-policy-auditor").call(run_input={
"targets": [{"sourceId": "site", "robotsTxt": "User-agent: *\nAllow: /"}],
"expectedAccess": "allow",
"fetchUrls": False,
})
items = client.dataset(run["defaultDatasetId"]).list_items().items
print(items[0]["recommendation"])
```
### Cost-Control Tips
- Keep `fetchUrls` false for deterministic CI, QA, and client snapshot reviews.
- Use `maxTargets` between `1` and `10` for initial live checks.
- Prefer inline `robotsTxt` snapshots for repeatable regression tests.
- Disable `includeReport` if you only need dataset rows.
- Use `expectedAccess: "neutral"` for inventory-only reports where blocking may be intentional.
### Public Task Examples
This package includes five prepared task concepts:
- AI crawler allow-list check.
- AI crawler block audit.
- Mixed AI crawler policy review.
- Page-level noai signal scan.
- Optional live robots fetch.
### FAQ
#### Does this actor make network calls by default?
No. The default input is fully inline and `fetchUrls` defaults to `false`.
#### Can it fetch live robots.txt files?
Yes. Set `fetchUrls` to `true` and provide `urls`. The actor fetches `/robots.txt` and optionally `/llms.txt` with strict caps.
#### Does blocking an AI crawler always mean the site is wrong?
No. That depends on your policy goal. Use `expectedAccess: "block"` for opt-out audits or `neutral` for inventory-only checks.
#### Does this actor replace legal review?
No. It reports technical policy signals and does not provide legal advice.
#### Are diagnostics billed as dataset rows?
### Troubleshooting
- `robots-txt-missing`: Supply inline robots content or enable live fetching for authorized URLs.
- `ai-crawlers-blocked`: Review blocked agents against your AI search or opt-out goal.
- `mixed-policy-signals`: Check for conflicts between robots.txt, meta robots, X-Robots-Tag, and selected expected access.
- `robots-policy-changed`: Review the previous/current diff before treating the current policy as intentional.
### Limitations
- Robots matching is designed for practical policy QA, not a full crawler implementation.
- Live fetching can fail due auth, redirects, firewalls, DNS, or network conditions.
- The actor audits a target path, not every URL on a site.
- Page-level signals are parsed from supplied HTML and headers; the actor does not render JavaScript.
### Source And Safety Notes
Audit only URLs and policy snapshots you are authorized to inspect. Avoid uploading sensitive internal policy content unless you are allowed to process it in Apify.
### Changelog
- `1.0.0`: Initial package build with local-first robots auditing, optional capped fetches, previous/current diffing, KVS reports, PPE billing, examples, and fixture tests.
# Actor input Schema
## `targets` (type: `array`):
Robots.txt and page-signal snapshots to audit. Defaults to a local fixture that makes no network calls.
## `urls` (type: `array`):
Optional site URLs to fetch only when fetchUrls is true. The actor fetches /robots.txt and /llms.txt with strict byte and timeout caps.
## `fetchUrls` (type: `boolean`):
Fetch robots.txt and llms.txt over HTTP. Default is false to keep zero-config runs local, deterministic, and cheap.
## `expectedAccess` (type: `string`):
Use allow for AI crawler readiness checks, block for opt-out audits, or neutral for inventory-only reporting.
## `auditedAgents` (type: `array`):
Crawler user-agent tokens to evaluate against robots.txt.
## `comparePrevious` (type: `boolean`):
Compare previousRobotsTxt against current robotsTxt when a previous snapshot is supplied.
## `requireLlmsTxt` (type: `boolean`):
Flag targets that do not include llms.txt content.
## `maxItems` (type: `integer`):
Alias for Max Targets/Maximum Records used by the Junipr actor baseline. When both are provided, actor-specific limits still apply.
## `includeReport` (type: `boolean`):
Write summary JSON, full result JSON, and Markdown report artifacts to the key-value store.
## `maxTargets` (type: `integer`):
Maximum robots or URL targets to audit.
## `fetchTimeoutMs` (type: `integer`):
HTTP timeout for optional robots.txt and llms.txt fetches.
## `maxRobotsBytes` (type: `integer`):
Maximum bytes to read from a fetched robots.txt file.
## `maxChargeUsd` (type: `number`):
Hard local spending cap for actor-start, per-item, and report events. The actor stops before work or withholds uncharged output when the next event would exceed this amount.
## `dryRun` (type: `boolean`):
Validate input and write a dry-run summary without PPE charges or dataset output.
## `debug` (type: `boolean`):
Enable debug logs.
## Actor input object example
```json
{
"targets": [
{
"sourceId": "sample-ai-ready-site",
"sourceUrl": "https://example.com",
"targetPath": "/",
"robotsTxt": "User-agent: *\nAllow: /\n\nUser-agent: GPTBot\nAllow: /\n\nUser-agent: ChatGPT-User\nAllow: /\n\nUser-agent: OAI-SearchBot\nAllow: /\n\nUser-agent: ClaudeBot\nAllow: /\n\nUser-agent: Claude-User\nAllow: /\n\nUser-agent: anthropic-ai\nAllow: /\n\nUser-agent: PerplexityBot\nAllow: /\n\nUser-agent: Google-Extended\nAllow: /\n\nUser-agent: GoogleOther\nAllow: /\n\nUser-agent: CCBot\nAllow: /\n\nUser-agent: Bytespider\nAllow: /\n\nUser-agent: Meta-ExternalAgent\nAllow: /\n\nSitemap: https://example.com/sitemap.xml",
"llmsTxt": "# Example AI crawler guidance\n\nThis fixture represents a site that allows major AI crawlers and exposes a sitemap.",
"headers": {
"x-robots-tag": "index, follow"
},
"html": "ExampleExample"
}
],
"urls": [],
"fetchUrls": false,
"expectedAccess": "allow",
"auditedAgents": [
"GPTBot",
"ChatGPT-User",
"OAI-SearchBot",
"ClaudeBot",
"Claude-User",
"anthropic-ai",
"PerplexityBot",
"Google-Extended",
"GoogleOther",
"CCBot",
"Bytespider",
"Meta-ExternalAgent",
"Googlebot",
"Bingbot"
],
"comparePrevious": true,
"requireLlmsTxt": false,
"maxItems": 25,
"includeReport": true,
"maxTargets": 25,
"fetchTimeoutMs": 10000,
"maxRobotsBytes": 200000,
"maxChargeUsd": 10,
"dryRun": false,
"debug": false
}
```
# Actor output Schema
## `results` (type: `string`):
Each dataset row is one audited robots or site policy target with crawler access decisions, findings, and recommendations.
## `resultsJson` (type: `string`):
Full JSON array of audit results.
## `summaryJson` (type: `string`):
Run-level counts, scores, blocked agents, and top issues.
## `summaryReport` (type: `string`):
Markdown report describing AI crawler policy audit results.
# API
You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.
## JavaScript example
```javascript
import { ApifyClient } from 'apify-client';
// Initialize the ApifyClient with your Apify API token
// Replace the '' with your token
const client = new ApifyClient({
token: '',
});
// Prepare Actor input
const input = {};
// Run the Actor and wait for it to finish
const run = await client.actor("junipr/ai-crawler-policy-auditor").call(input);
// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
console.dir(item);
});
// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs
```
## Python example
```python
from apify_client import ApifyClient
# Initialize the ApifyClient with your Apify API token
# Replace '' with your token.
client = ApifyClient("")
# Prepare the Actor input
run_input = {}
# Run the Actor and wait for it to finish
run = client.actor("junipr/ai-crawler-policy-auditor").call(run_input=run_input)
# Fetch and print Actor results from the run's dataset (if there are any)
print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"])
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
print(item)
# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start
```
## CLI example
```bash
echo '{}' |
apify call junipr/ai-crawler-policy-auditor --silent --output-dataset
```
## MCP server setup
```json
{
"mcpServers": {
"apify": {
"command": "npx",
"args": [
"mcp-remote",
"https://mcp.apify.com/?tools=junipr/ai-crawler-policy-auditor",
"--header",
"Authorization: Bearer "
]
}
}
}
```
## OpenAPI specification
```json
{
"openapi": "3.0.1",
"info": {
"title": "AI Crawler Policy Auditor",
"description": "Audit robots and AI crawler policy signals for AI crawler access, blocking, and site-readiness.",
"version": "1.0",
"x-build-id": "80vNFOXNuAAgowY5A"
},
"servers": [
{
"url": "https://api.apify.com/v2"
}
],
"paths": {
"/acts/junipr~ai-crawler-policy-auditor/run-sync-get-dataset-items": {
"post": {
"operationId": "run-sync-get-dataset-items-junipr-ai-crawler-policy-auditor",
"x-openai-isConsequential": false,
"summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.",
"tags": [
"Run Actor"
],
"requestBody": {
"required": true,
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/inputSchema"
}
}
}
},
"parameters": [
{
"name": "token",
"in": "query",
"required": true,
"schema": {
"type": "string"
},
"description": "Enter your Apify token here"
}
],
"responses": {
"200": {
"description": "OK"
}
}
}
},
"/acts/junipr~ai-crawler-policy-auditor/runs": {
"post": {
"operationId": "runs-sync-junipr-ai-crawler-policy-auditor",
"x-openai-isConsequential": false,
"summary": "Executes an Actor and returns information about the initiated run in response.",
"tags": [
"Run Actor"
],
"requestBody": {
"required": true,
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/inputSchema"
}
}
}
},
"parameters": [
{
"name": "token",
"in": "query",
"required": true,
"schema": {
"type": "string"
},
"description": "Enter your Apify token here"
}
],
"responses": {
"200": {
"description": "OK",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/runsResponseSchema"
}
}
}
}
}
}
},
"/acts/junipr~ai-crawler-policy-auditor/run-sync": {
"post": {
"operationId": "run-sync-junipr-ai-crawler-policy-auditor",
"x-openai-isConsequential": false,
"summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.",
"tags": [
"Run Actor"
],
"requestBody": {
"required": true,
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/inputSchema"
}
}
}
},
"parameters": [
{
"name": "token",
"in": "query",
"required": true,
"schema": {
"type": "string"
},
"description": "Enter your Apify token here"
}
],
"responses": {
"200": {
"description": "OK"
}
}
}
}
},
"components": {
"schemas": {
"inputSchema": {
"type": "object",
"properties": {
"targets": {
"title": "Policy Targets",
"type": "array",
"description": "Robots.txt and page-signal snapshots to audit. Defaults to a local fixture that makes no network calls.",
"items": {
"type": "object"
},
"default": [
{
"sourceId": "sample-ai-ready-site",
"sourceUrl": "https://example.com",
"targetPath": "/",
"robotsTxt": "User-agent: *\nAllow: /\n\nUser-agent: GPTBot\nAllow: /\n\nUser-agent: ChatGPT-User\nAllow: /\n\nUser-agent: OAI-SearchBot\nAllow: /\n\nUser-agent: ClaudeBot\nAllow: /\n\nUser-agent: Claude-User\nAllow: /\n\nUser-agent: anthropic-ai\nAllow: /\n\nUser-agent: PerplexityBot\nAllow: /\n\nUser-agent: Google-Extended\nAllow: /\n\nUser-agent: GoogleOther\nAllow: /\n\nUser-agent: CCBot\nAllow: /\n\nUser-agent: Bytespider\nAllow: /\n\nUser-agent: Meta-ExternalAgent\nAllow: /\n\nSitemap: https://example.com/sitemap.xml",
"llmsTxt": "# Example AI crawler guidance\n\nThis fixture represents a site that allows major AI crawlers and exposes a sitemap.",
"headers": {
"x-robots-tag": "index, follow"
},
"html": "ExampleExample"
}
]
},
"urls": {
"title": "URLs To Fetch",
"type": "array",
"description": "Optional site URLs to fetch only when fetchUrls is true. The actor fetches /robots.txt and /llms.txt with strict byte and timeout caps.",
"items": {
"type": "string"
},
"default": []
},
"fetchUrls": {
"title": "Fetch URLs",
"type": "boolean",
"description": "Fetch robots.txt and llms.txt over HTTP. Default is false to keep zero-config runs local, deterministic, and cheap.",
"default": false
},
"expectedAccess": {
"title": "Expected AI Crawler Access",
"enum": [
"allow",
"block",
"neutral"
],
"type": "string",
"description": "Use allow for AI crawler readiness checks, block for opt-out audits, or neutral for inventory-only reporting.",
"default": "allow"
},
"auditedAgents": {
"title": "Audited Crawler Agents",
"type": "array",
"description": "Crawler user-agent tokens to evaluate against robots.txt.",
"items": {
"type": "string"
},
"default": [
"GPTBot",
"ChatGPT-User",
"OAI-SearchBot",
"ClaudeBot",
"Claude-User",
"anthropic-ai",
"PerplexityBot",
"Google-Extended",
"GoogleOther",
"CCBot",
"Bytespider",
"Meta-ExternalAgent",
"Googlebot",
"Bingbot"
]
},
"comparePrevious": {
"title": "Compare Previous Robots",
"type": "boolean",
"description": "Compare previousRobotsTxt against current robotsTxt when a previous snapshot is supplied.",
"default": true
},
"requireLlmsTxt": {
"title": "Require llms.txt",
"type": "boolean",
"description": "Flag targets that do not include llms.txt content.",
"default": false
},
"maxItems": {
"title": "Max Items",
"minimum": 1,
"maximum": 250,
"type": "integer",
"description": "Alias for Max Targets/Maximum Records used by the Junipr actor baseline. When both are provided, actor-specific limits still apply.",
"default": 25
},
"includeReport": {
"title": "Include Report Artifacts",
"type": "boolean",
"description": "Write summary JSON, full result JSON, and Markdown report artifacts to the key-value store.",
"default": true
},
"maxTargets": {
"title": "Max Targets",
"minimum": 1,
"maximum": 250,
"type": "integer",
"description": "Maximum robots or URL targets to audit.",
"default": 25
},
"fetchTimeoutMs": {
"title": "Fetch Timeout Milliseconds",
"minimum": 1000,
"maximum": 30000,
"type": "integer",
"description": "HTTP timeout for optional robots.txt and llms.txt fetches.",
"default": 10000
},
"maxRobotsBytes": {
"title": "Max Robots Bytes",
"minimum": 1000,
"maximum": 1000000,
"type": "integer",
"description": "Maximum bytes to read from a fetched robots.txt file.",
"default": 200000
},
"maxChargeUsd": {
"title": "Maximum Charge (USD)",
"minimum": 0,
"maximum": 1000,
"type": "number",
"description": "Hard local spending cap for actor-start, per-item, and report events. The actor stops before work or withholds uncharged output when the next event would exceed this amount.",
"default": 10
},
"dryRun": {
"title": "Dry Run",
"type": "boolean",
"description": "Validate input and write a dry-run summary without PPE charges or dataset output.",
"default": false
},
"debug": {
"title": "Debug Logs",
"type": "boolean",
"description": "Enable debug logs.",
"default": false
}
}
},
"runsResponseSchema": {
"type": "object",
"properties": {
"data": {
"type": "object",
"properties": {
"id": {
"type": "string"
},
"actId": {
"type": "string"
},
"userId": {
"type": "string"
},
"startedAt": {
"type": "string",
"format": "date-time",
"example": "2025-01-08T00:00:00.000Z"
},
"finishedAt": {
"type": "string",
"format": "date-time",
"example": "2025-01-08T00:00:00.000Z"
},
"status": {
"type": "string",
"example": "READY"
},
"meta": {
"type": "object",
"properties": {
"origin": {
"type": "string",
"example": "API"
},
"userAgent": {
"type": "string"
}
}
},
"stats": {
"type": "object",
"properties": {
"inputBodyLen": {
"type": "integer",
"example": 2000
},
"rebootCount": {
"type": "integer",
"example": 0
},
"restartCount": {
"type": "integer",
"example": 0
},
"resurrectCount": {
"type": "integer",
"example": 0
},
"computeUnits": {
"type": "integer",
"example": 0
}
}
},
"options": {
"type": "object",
"properties": {
"build": {
"type": "string",
"example": "latest"
},
"timeoutSecs": {
"type": "integer",
"example": 300
},
"memoryMbytes": {
"type": "integer",
"example": 1024
},
"diskMbytes": {
"type": "integer",
"example": 2048
}
}
},
"buildId": {
"type": "string"
},
"defaultKeyValueStoreId": {
"type": "string"
},
"defaultDatasetId": {
"type": "string"
},
"defaultRequestQueueId": {
"type": "string"
},
"buildNumber": {
"type": "string",
"example": "1.0.0"
},
"containerUrl": {
"type": "string"
},
"usage": {
"type": "object",
"properties": {
"ACTOR_COMPUTE_UNITS": {
"type": "integer",
"example": 0
},
"DATASET_READS": {
"type": "integer",
"example": 0
},
"DATASET_WRITES": {
"type": "integer",
"example": 0
},
"KEY_VALUE_STORE_READS": {
"type": "integer",
"example": 0
},
"KEY_VALUE_STORE_WRITES": {
"type": "integer",
"example": 1
},
"KEY_VALUE_STORE_LISTS": {
"type": "integer",
"example": 0
},
"REQUEST_QUEUE_READS": {
"type": "integer",
"example": 0
},
"REQUEST_QUEUE_WRITES": {
"type": "integer",
"example": 0
},
"DATA_TRANSFER_INTERNAL_GBYTES": {
"type": "integer",
"example": 0
},
"DATA_TRANSFER_EXTERNAL_GBYTES": {
"type": "integer",
"example": 0
},
"PROXY_RESIDENTIAL_TRANSFER_GBYTES": {
"type": "integer",
"example": 0
},
"PROXY_SERPS": {
"type": "integer",
"example": 0
}
}
},
"usageTotalUsd": {
"type": "number",
"example": 0.00005
},
"usageUsd": {
"type": "object",
"properties": {
"ACTOR_COMPUTE_UNITS": {
"type": "integer",
"example": 0
},
"DATASET_READS": {
"type": "integer",
"example": 0
},
"DATASET_WRITES": {
"type": "integer",
"example": 0
},
"KEY_VALUE_STORE_READS": {
"type": "integer",
"example": 0
},
"KEY_VALUE_STORE_WRITES": {
"type": "number",
"example": 0.00005
},
"KEY_VALUE_STORE_LISTS": {
"type": "integer",
"example": 0
},
"REQUEST_QUEUE_READS": {
"type": "integer",
"example": 0
},
"REQUEST_QUEUE_WRITES": {
"type": "integer",
"example": 0
},
"DATA_TRANSFER_INTERNAL_GBYTES": {
"type": "integer",
"example": 0
},
"DATA_TRANSFER_EXTERNAL_GBYTES": {
"type": "integer",
"example": 0
},
"PROXY_RESIDENTIAL_TRANSFER_GBYTES": {
"type": "integer",
"example": 0
},
"PROXY_SERPS": {
"type": "integer",
"example": 0
}
}
}
}
}
}
}
}
}
}
```