Site QA Indexability AI Crawler Report Scraper

Open Input (leave defaults → Start) Paid first-run JSON ≈ USD 0.42 (1 URL, initialRunMode: "emit_backfill": 2 AI crawler policy rows USD 0.03 + 3 indexability issues USD 0.12 for example.com; report/export OFF). Only audit URLs you own or are authorized to audit. Keep reports, exports, and movement options bounded for the first run. Open Console Input (deep link above) and run this JSON.

{
  "urls": [
    "https://example.com/?siteQaCanary=indexability-p"
  ],
  "maxPages": 1,
  "checkRobotsTxt": true,
  "checkLlmsTxt": true,
  "authorizedUseConfirmed": true,
  "aiCrawlerUserAgents": [
    "GPTBot",
    "Google-Extended"
  ],
  "emitPageRows": false,
  "generateReport": false,
  "emitExport": false,
  "emitUnchanged": false,
  "dryRun": false,
  "initialRunMode": "emit_backfill",
  "snapshotKey": "indexability-p"
}

Scheduled runs: turn on emitExport

If you run this Actor on a schedule, set emitExport: true to add one evidence export row per run. Leave it false (default) when you only need the usual report/rows. See Actor pricing for the export event.

Paste a URL and Start.

Site owners, SEO agencies, and content teams use this actor to audit public indexability and AI crawler access signals. Provide public URLs and optional AI crawler user-agent names. The actor returns source-linked policy observations, indexability issues, reports, and export rows.

The primary deliverable is the site-level indexability report or export. Policy observations and raw page rows are supporting evidence, not the main sales path.

Store Quickstart

Run with dryRun=false and public URLs that you own or are allowed to audit.

{
  "urls": ["https://example.com/?siteQaCanary=indexability-ai-crawler-v1"],
  "aiCrawlerUserAgents": ["GPTBot", "Google-Extended", "PerplexityBot", "ClaudeBot"],
  "checkRobotsTxt": true,
  "checkLlmsTxt": true,
  "authorizedUseConfirmed": true,
  "generateReport": true,
  "emitExport": true,
  "emitUnchanged": false,
  "dryRun": false
}

Use with Apify MCP

Exact actor: taroyamada/site-qa-indexability-ai-crawler-report-scraper

Sample input JSON:

{
  "urls": [
    "https://example.com/?siteQaCanary=indexability-ai-crawler-v1"
  ],
  "maxPages": 1
}

Report result preview

{
  "rowType": "ai_crawler_indexability_report",
  "billingEventName": "ai-crawler-indexability-report",
  "sourceUrl": "https://example.com/robots.txt",
  "checkedUrlCount": 3,
  "issueCount": 1,
  "reportStatus": "action_needed"
}

Use indexability-export-generated when a source-linked checklist must move into an implementation workflow. The policy event is the low-cost evidence step; it is not a ranking or citation guarantee.

Run the next report

• Generate ai-crawler-indexability-report when robots.txt, llms.txt, canonical, or noindex findings need a site-level handoff.
• Generate indexability-export-generated when SEO, content, or AI-readiness teams need a source-linked implementation checklist.
• Pair the findings with Site QA Content Report Scraper for content issues on the same URL set.
• Pair the findings with Site QA Broken Link Report Scraper for link-health issues on the same URL set.
• Audit public support and policy paths with Site QA Contact Page Audit Scraper when crawler findings expose missing response channels.

Input Examples

Audit one page and origin policies

{
  "urls": ["https://example.com/blog/launch"],
  "aiCrawlerUserAgents": ["GPTBot", "ClaudeBot"],
  "checkRobotsTxt": true,
  "checkLlmsTxt": true,
  "authorizedUseConfirmed": true,
  "dryRun": false
}

Batch audit a site section

{
  "urls": [
    "https://example.com/",
    "https://example.com/pricing",
    "https://example.com/docs"
  ],
  "maxPages": 25,
  "emitPageRows": false,
  "generateReport": true,
  "authorizedUseConfirmed": true,
  "dryRun": false
}

Generate a handoff export

{
  "urls": ["https://example.com/landing-page"],
  "aiCrawlerUserAgents": ["GPTBot", "Google-Extended", "PerplexityBot"],
  "emitExport": true,
  "emitUnchanged": false,
  "authorizedUseConfirmed": true,
  "dryRun": false
}

Sample Output

The complete sample starts with report and export rows, followed by issue and policy rows:

{
  "actorName": "site-qa-indexability-ai-crawler-report-scraper",
  "rowType": "ai_crawler_indexability_report",
  "billingEventName": "ai-crawler-indexability-report",
  "sourceUrl": "https://example.com/robots.txt",
  "issueCount": 1,
  "reportStatus": "action_needed"
}

Output Fields

• rowType: ai_crawler_policy_observation, indexability_issue, ai_crawler_indexability_report, or indexability_export.
• billingEventName: PAY_PER_EVENT event name used for the row.
• sourceUrl: public URL or policy file that supports the row.
• issueType: detected source-linked issue when applicable.
• blockedUserAgents: crawler names with broad robots.txt blocks when detected.

Pricing And No-Change Runs

• ai-crawler-policy-checked: $0.030 per public robots.txt or llms.txt policy observation.
• indexability-issue-detected: $0.120 per source-linked indexability issue.
• ai-crawler-indexability-report: $6.000 per site-level report.
• indexability-export-generated: $8.000 per generated export.

When emitUnchanged=false, repeated unchanged runs emit zero dataset rows and zero charges after state is saved.

For recurring audits, the value path is indexability-issue-detected, ai-crawler-indexability-report, and indexability-export-generated. An unchanged poll does not create a report/export charge.

Compliance Guardrails

• Public pages, robots.txt, and llms.txt only.
• No login, paywall, CAPTCHA, private session, credentialed API, or bypass behavior.
• Non-dry runs require authorizedUseConfirmed=true; use this only for sites you own, manage, or are allowed to audit.
• This is an unofficial audit tool and is not affiliated with any crawler, search engine, or AI provider.
• No ranking guarantee, AI citation guarantee, legal conclusion, or compliance certification.

Bundle Paths

• Site QA / AI Crawler Readiness: run a low-cost policy check first, then generate ai-crawler-indexability-report or indexability-export-generated for client delivery.
• Pair with Site QA Content Report Scraper for page content issue reports.
• Pair with Site QA Broken Link Report Scraper for link health reports.

See Also

• Site QA Content Report Scraper for content QA issue reports.
• Site QA Broken Link Report Scraper for broken link reports.