ChatGPT Ads & AI Crawler Policy Auditor

Audit public websites in bulk for AI crawler robots.txt policies, llms.txt validity, sitemap availability, and machine-readable homepage signals. The Actor returns one fixed Dataset record per successful website. It does not predict rankings, citations, or AI visibility.

Who it is for

SEO/GEO agencies, website owners, release QA teams, and n8n/Make/Zapier/API builders that need a repeatable technical inventory rather than a manual browser check.

Three steps

1. Paste public website URLs.
2. Set an optional URL cap, concurrency, and timeout.
3. Run and export the Dataset as JSON, CSV, or Excel.

Input

{
  "startUrls": [{ "url": "https://example.com" }],
  "maxUrls": 25,
  "maxConcurrency": 5,
  "requestTimeoutSecs": 15,
  "simulateOaiAdsBotRequest": false
}

Output

{
  "url": "https://example.com/",
  "finalUrl": "https://example.com/",
  "checkedAt": "2026-07-16T13:00:00.000Z",
  "httpStatus": 200,
  "chargeable": true,
  "responseTimeMs": 83,
  "score": 85,
  "grade": "B",
  "robotsTxt": {
    "exists": true,
    "blockedBots": ["ClaudeBot"],
    "purposeSummaries": {
      "model-training": { "blockedBots": ["ClaudeBot"] },
      "ai-search-and-grounding": { "allowedBots": ["OAI-SearchBot"] }
    },
    "landingPagePolicy": {
      "path": "/",
      "oaiAdsBotAccess": "allowed",
      "policyProfiles": [
        {
          "bot": "OAI-AdsBot",
          "access": "allowed",
          "matchedRule": {
            "directive": "allow",
            "path": "/",
            "userAgentScope": "exact"
          }
        }
      ],
      "purposeSummaries": {
        "ad-landing-page-review": { "allowedBots": ["OAI-AdsBot"] }
      }
    }
  },
  "llmsTxt": { "exists": true, "valid": true, "linkCount": 1 },
  "sitemap": { "exists": true },
  "page": { "title": "Fixture Site", "jsonLdTypes": ["Organization"] },
  "oaiAdsBotSimulation": {
    "attempted": false,
    "userAgent": "OAI-AdsBot",
    "httpStatus": null,
    "finalUrl": null,
    "responseTimeMs": null,
    "error": null,
    "note": "Not requested. This Actor does not use OpenAI IP ranges and does not prove actual OAI-AdsBot access."
  },
  "comparison": {
    "status": "compared",
    "previousAuditSource": "pasted-previous-audit",
    "changed": true,
    "changes": [
      {
        "field": "robotsTxt.landingPagePolicy.oaiAdsBotAccess",
        "previous": "allowed",
        "current": "blocked"
      }
    ],
    "note": "Observed values changed since the supplied previous audit JSON. Review with the site owner before changing any access policy."
  },
  "comparisonStatus": "compared",
  "comparisonChanged": true,
  "comparisonSummary": "Observed values changed since the supplied previous audit JSON. Review with the site owner before changing any access policy.",
  "oaiAdsBotAccess": "allowed",
  "oaiAdsBotRuleDirective": "allow",
  "oaiAdsBotRulePath": "/",
  "oaiAdsBotRuleScope": "exact",
  "oaiAdsBotPolicySummary": "OAI-AdsBot is allowed by exact allow rule: /.",
  "chatgptAdsPreflight": {
    "status": "manual-controls-review-required",
    "summary": "Robots policy and the standard request were observed. Owner review of real crawler controls is still required.",
    "officialGuidanceUrl": "https://help.openai.com/en/articles/20001243-advertiser-guidance-for-allowing-openai-web-crawlers",
    "observedProtectionSignals": ["cloudflare"],
    "manualChecks": [
      "Inspect the observed provider signal(s): cloudflare.",
      "Review WAF, CDN, firewall, and bot-mitigation rules with the site owner.",
      "Review CAPTCHA or JavaScript challenges, authentication, geo restrictions, and rate limits with the site owner."
    ]
  },
  "chatgptAdsPreflightStatus": "manual-controls-review-required",
  "chatgptAdsPreflightSummary": "Robots policy and the standard request were observed. Owner review of real crawler controls is still required.",
  "issues": [
    {
      "code": "AI_BOTS_BLOCKED",
      "severity": "warning",
      "message": "1 audited AI crawler(s) are blocked at the site root."
    }
  ],
  "recommendations": [
    {
      "issueCode": "AI_BOTS_BLOCKED",
      "priority": "high",
      "action": "Review each blocked AI crawler with the site owner before changing the documented access policy."
    }
  ]
}

recommendations[] turns observed issues into deterministic technical next steps. It does not promise rankings, citations, or AI visibility. The Actor stores only technical metadata, policies, counts, issue messages, and recommendations; it does not store raw homepage content.

oaiAdsBotAccess, oaiAdsBotRuleDirective, oaiAdsBotRulePath, oaiAdsBotRuleScope, and oaiAdsBotPolicySummary duplicate the current landing-page policy evidence as simple top-level fields. Use the OAI-AdsBot landing-page policy evidence Dataset view when the primary goal is a direct owner review of the observed rule rather than a broad technical overview or week-over-week comparison.

chatgptAdsPreflight turns the observable parts of OpenAI's published advertiser checklist into a conservative handoff: the path-specific robots.txt outcome, the standard HTTP result, optional simulated User-Agent result, detected CDN or web-protection response signals, and the manual controls an owner still needs to review. The ChatGPT Ads landing-page preflight Dataset view keeps that handoff compact. It never reports that an ad is approved or that an actual OpenAI crawler was able to access the site.

robotsTxt.policyProfiles[] and purposeSummaries separate observed root-policy controls into model training, AI search/grounding, user-requested page fetches, ad landing-page review, and open-web data. They are a policy inventory, not a visibility measurement. In particular, Google-Extended is a control token rather than a separate HTTP crawler and does not affect Google Search. Review every policy change with the site owner before deployment.

ChatGPT Ads landing-page preflight (policy-only)

For every submitted public landing-page URL, check robotsTxt.landingPagePolicy.oaiAdsBotAccess for the observed OAI-AdsBot policy at that page path, then compare robotsTxt.landingPagePolicy.purposeSummaries["ai-search-and-grounding"] for OAI-SearchBot. Each bot profile now includes matchedRule when a matching Allow or Disallow directive determined the result, including whether it came from that bot's exact group or a wildcard group. Root-wide policy profiles remain under robotsTxt.policyProfiles[] and purposeSummaries. If this Actor observes a non-2xx landing-page response, it stores a no-charge LANDING_PAGE_HTTP_STATUS diagnostic with chargeable: false; that response does not simulate or prove actual OAI-AdsBot access. This is useful as a first technical inventory before a team reviews its landing pages. It does not fetch with OpenAI user agents, test WAF/CDN configuration, CAPTCHA, geo restrictions, ad acceptance, or campaign performance. OpenAI's advertiser guidance should remain the authority for an actual ad submission.

60-second first preflight

1. Add one public landing-page URL, set maxUrls and maxConcurrency to 1, and leave the timeout at 15 seconds.
2. In Run options, set the maximum run charge to $0.005. This caps the run at one successfully stored audit.
3. Optionally set simulateOaiAdsBotRequest to true when the site owner wants an extra User-Agent signal from this Actor's IP. Leave it off for the lowest-impact baseline policy check.
4. Export the single Dataset item and select the ChatGPT Ads landing-page preflight Dataset view. A path-specific OAI-AdsBot blocked finding is a technical handoff for the site owner; a non-2xx standard response is stored without a charge. Neither outcome is ad approval or an instruction to alter access controls.

Reading a rule-evidence result

For a submitted landing page, find the OAI-AdsBot entry in robotsTxt.landingPagePolicy.policyProfiles[].

• access: "blocked" with matchedRule means the output names the exact observed Disallow directive and whether it came from the bot-specific or wildcard group. Forward that object with the landing-page URL to the person responsible for the deployed robots.txt; do not change the rule unless allowing the crawler matches the site's documented policy.
• access: "allowed" with an Allow matchedRule means an observed rule allows that page path. access: "allowed" with matchedRule: null means no applicable rule was found in a declared group, so review the remaining real-world controls separately.
• chargeable: false and LANDING_PAGE_HTTP_STATUS means this Actor received a non-2xx response. The diagnostic is stored without a website-audit event, so resolve the response, redirect, or access-control problem before treating robots policy as a useful preflight.

This output is a copyable technical handoff, not an instruction to change a website's crawler policy and not evidence of ad approval.

ChatGPT Ads preflight statuses

• blocked-by-robots-policy: the observed robots.txt rule blocks OAI-AdsBot at the submitted landing-page path.
• non-2xx-standard-request: this Actor's standard request did not receive a successful HTTP response, so resolve basic reachability first.
• simulation-non-2xx: the optional request from this Actor's IP using the OAI-AdsBot User-Agent was non-2xx. Inspect web controls; this is not an OpenAI-origin request.
• manual-controls-review-required: the observable HTTP and robots checks are recorded, but the owner must still review WAF/CDN, bot mitigation, CAPTCHA or JavaScript challenges, authentication, geo restrictions, and rate limits.

Optional OAI-AdsBot User-Agent simulation

Set simulateOaiAdsBotRequest to true when a site owner wants an additional technical signal. After a successful standard landing-page request, the Actor sends one simulated request flow (with normal redirect handling and no retries) using User-Agent: OAI-AdsBot, then records oaiAdsBotSimulation.httpStatus, final URL, duration, and any error. A non-2xx response becomes an advisory warning; it does not change the score or make an otherwise reachable audit non-chargeable.

This originates from the Actor's IP address, not OpenAI infrastructure. It is therefore not a verified OpenAI IP-range test, proof that OAI-AdsBot can access the page, proof of ad acceptance, or a reason to loosen a site's access policy. Use it only with the site owner's authorization and inspect WAF, CDN, bot mitigation, authentication, geo, and rate-limit controls with that owner. OpenAI's advertiser guidance remains authoritative for actual ads eligibility.

Pricing and free test

Active PPE pricing is $0.005 per successfully stored, reachable website-audit result: about $0.50 for 100 successful sites. A non-2xx landing page is stored as a chargeable: false diagnostic and is not charged; network failures are neither stored nor charged. Duplicate normalized URLs are processed and charged once per run. URLs beyond the user's maximum run charge are skipped before network work begins.

For a small free-plan test, run the sample input with one URL and set a small maximum run charge. Platform free-tier handling remains subject to the user's Apify plan.

API

curl -X POST "https://api.apify.com/v2/acts/x2ppi~ai-crawler-readiness-auditor/runs?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"startUrls":[{"url":"https://example.com"}],"maxUrls":1}'

import { ApifyClient } from "apify-client";
const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor("x2ppi/ai-crawler-readiness-auditor").call({
  startUrls: [{ url: "https://example.com" }],
  maxUrls: 1,
});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items);

from apify_client import ApifyClient
client = ApifyClient("YOUR_APIFY_TOKEN")
run = client.actor("x2ppi/ai-crawler-readiness-auditor").call(run_input={
    "startUrls": [{"url": "https://example.com"}], "maxUrls": 1
})
print(client.dataset(run["defaultDatasetId"]).list_items().items)

MCP and no-code automation

• MCP: connect the official Apify MCP server, select this Actor, and pass the same JSON input; the fixed output schema makes the Dataset usable by an agent.
• n8n: use the single-request setup below for small audits, or the asynchronous Actor-run endpoint plus a webhook for longer jobs.
• Make: Make an API call to start the Actor → retrieve Dataset items → map each result.
• Zapier: Webhooks POST to the Actor API → Delay/Poll → Webhooks GET Dataset items.

The documented request and response JSON above are the public, ready-to-copy integration examples.

n8n quick start: one HTTP Request node

For a small audit that finishes within five minutes, create an n8n HTTP Request node with a Header Auth credential whose Authorization value is Bearer YOUR_APIFY_TOKEN. Do not place the token in the workflow JSON or URL.

• Method: POST
• URL: https://api.apify.com/v2/actors/x2ppi~ai-crawler-readiness-auditor/run-sync-get-dataset-items?timeout=300&maxTotalChargeUsd=0.005
• Send JSON body:

{
  "startUrls": [{ "url": "https://example.com" }],
  "maxUrls": 1,
  "maxConcurrency": 1,
  "requestTimeoutSecs": 15
}

The node returns the Dataset items directly. Add an n8n Filter or IF node when robotsTxt.landingPagePolicy.oaiAdsBotAccess equals blocked to route the finding for owner review; do not automate changes to robots.txt. For larger or slower batches, use Apify's asynchronous endpoint and a webhook instead of holding an HTTP connection open.

Weekly scheduled policy review

For an agency or release-QA workflow, save a task with a fixed list of public sites and run that task weekly from Schedules in the Apify Console. After the first run, export its Dataset item as JSON (or a JSON array of items) and paste it into previousAuditJson before the next run. Each matching normalized URL then receives a compact comparison object for HTTP status, score/grade, path-specific OAI-AdsBot policy, llms.txt/sitemap flags, title/canonical, and optional simulation status. A missing matching URL reports no-previous-result; malformed JSON reports unavailable without failing the current audit.

Set the task's maximum run charge to the number of intended reachable audits × $0.005 before enabling a schedule. The Actor does not automatically select the prior Dataset, alter a saved task, send notifications, or make crawler-policy changes; the workflow owner chooses the previous Dataset and reviews every difference.

Use the complete exported Dataset item, not only its comparison object. The value may be one JSON object or an array of up to 1,000 previous items. This keeps the recurring comparison token-free: the Actor reads only the JSON supplied for that run and never accesses another customer's Dataset.

Current version note: comparison uses the supplied previousAuditJson; it does not read a prior Dataset or customer storage. The workflow owner exports the earlier result, supplies the JSON, and reviews every difference before deciding on any policy change.

Limits and errors

• Public HTTP(S) only; no authentication, cookies, CAPTCHA, paywalls, or access-control bypass.
• Server-rendered homepage metadata only; no JavaScript rendering.
• 429 and 5xx responses are retried twice. A persistent non-2xx landing-page response becomes a no-charge diagnostic; timeouts and network failures produce no result and no Actor event charge.
• The optional OAI-AdsBot User-Agent simulation has no retries and is only attempted after the standard landing-page response is 2xx. Redirects are still followed safely. Its result is advisory and does not prove a request from OpenAI infrastructure.
• Missing component files are valid findings. They reduce the score but do not fail the audit.
• Robots analysis is scoped to root access for named bots and common Allow/Disallow syntax.
• Non-public literal and DNS-resolved targets are blocked. Every redirect is revalidated, and each socket is pinned to its checked IP to prevent DNS rebinding.

Privacy and affiliation

The Actor accesses public pages supplied by the user and stores no raw page body or secrets. It is not affiliated with OpenAI, Anthropic, Google, Perplexity, Apple, Common Crawl, or the audited websites. Bot names are used solely to identify public user-agent policies.

Support and changelog

Report reproducible issues with the input (redacted if needed), run ID, expected behavior, and observed result. Best-effort maintenance target: acknowledge valid issues within three business days. See CHANGELOG.md for releases.