# Weekly OAI-AdsBot Policy Change Monitor **Use case:** Compare a public landing page with a prior audit, then show the current OAI-AdsBot policy change and exact robots.txt rule evidence for owner review. ## Input ```json { "startUrls": [ { "url": "https://www.nytimes.com/ads/" } ], "maxUrls": 1, "maxConcurrency": 1, "requestTimeoutSecs": 15, "simulateOaiAdsBotRequest": true, "previousAuditJson": "{\"url\":\"https://www.nytimes.com/ads/\",\"finalUrl\":\"https://www.nytimes.com/ads/\",\"checkedAt\":\"2026-07-21T05:14:55.516Z\",\"httpStatus\":200,\"responseTimeMs\":117,\"score\":40,\"grade\":\"D\",\"robotsTxt\":{\"url\":\"https://www.nytimes.com/robots.txt\",\"status\":200,\"exists\":true,\"evaluatedPath\":\"/\",\"policies\":{\"GPTBot\":false,\"ChatGPT-User\":false,\"OAI-SearchBot\":false,\"OAI-AdsBot\":true,\"ClaudeBot\":false,\"Claude-SearchBot\":false,\"PerplexityBot\":false,\"Google-Extended\":false,\"Applebot-Extended\":false,\"CCBot\":false},\"policyProfiles\":[{\"bot\":\"GPTBot\",\"access\":\"blocked\",\"purposes\":[\"model-training\"],\"matchedRule\":{\"directive\":\"disallow\",\"path\":\"/\",\"userAgentScope\":\"exact\"}},{\"bot\":\"ChatGPT-User\",\"access\":\"blocked\",\"purposes\":[\"user-requested-page-fetch\"],\"matchedRule\":{\"directive\":\"disallow\",\"path\":\"/\",\"userAgentScope\":\"exact\"}},{\"bot\":\"OAI-SearchBot\",\"access\":\"blocked\",\"purposes\":[\"ai-search-and-grounding\"],\"matchedRule\":{\"directive\":\"disallow\",\"path\":\"/\",\"userAgentScope\":\"exact\"}},{\"bot\":\"OAI-AdsBot\",\"access\":\"allowed\",\"purposes\":[\"ad-landing-page-review\"],\"matchedRule\":null},{\"bot\":\"ClaudeBot\",\"access\":\"blocked\",\"purposes\":[\"model-training\"],\"matchedRule\":{\"directive\":\"disallow\",\"path\":\"/\",\"userAgentScope\":\"exact\"}},{\"bot\":\"Claude-SearchBot\",\"access\":\"blocked\",\"purposes\":[\"ai-search-and-grounding\"],\"matchedRule\":{\"directive\":\"disallow\",\"path\":\"/\",\"userAgentScope\":\"exact\"}},{\"bot\":\"PerplexityBot\",\"access\":\"blocked\",\"purposes\":[\"ai-search-and-grounding\"],\"matchedRule\":{\"directive\":\"disallow\",\"path\":\"/\",\"userAgentScope\":\"exact\"}},{\"bot\":\"Google-Extended\",\"access\":\"blocked\",\"purposes\":[\"model-training\",\"ai-search-and-grounding\"],\"matchedRule\":{\"directive\":\"disallow\",\"path\":\"/\",\"userAgentScope\":\"exact\"}},{\"bot\":\"Applebot-Extended\",\"access\":\"blocked\",\"purposes\":[\"model-training\"],\"matchedRule\":{\"directive\":\"disallow\",\"path\":\"/\",\"userAgentScope\":\"exact\"}},{\"bot\":\"CCBot\",\"access\":\"blocked\",\"purposes\":[\"open-web-data\"],\"matchedRule\":{\"directive\":\"disallow\",\"path\":\"/\",\"userAgentScope\":\"exact\"}}],\"purposeSummaries\":{\"model-training\":{\"allowedBots\":[],\"blockedBots\":[\"GPTBot\",\"ClaudeBot\",\"Google-Extended\",\"Applebot-Extended\"],\"unspecifiedBots\":[]},\"ai-search-and-grounding\":{\"allowedBots\":[],\"blockedBots\":[\"OAI-SearchBot\",\"Claude-SearchBot\",\"PerplexityBot\",\"Google-Extended\"],\"unspecifiedBots\":[]},\"user-requested-page-fetch\":{\"allowedBots\":[],\"blockedBots\":[\"ChatGPT-User\"],\"unspecifiedBots\":[]},\"ad-landing-page-review\":{\"allowedBots\":[\"OAI-AdsBot\"],\"blockedBots\":[],\"unspecifiedBots\":[]},\"open-web-data\":{\"allowedBots\":[],\"blockedBots\":[\"CCBot\"],\"unspecifiedBots\":[]}},\"allowedBots\":[\"OAI-AdsBot\"],\"blockedBots\":[\"GPTBot\",\"ChatGPT-User\",\"OAI-SearchBot\",\"ClaudeBot\",\"Claude-SearchBot\",\"PerplexityBot\",\"Google-Extended\",\"Applebot-Extended\",\"CCBot\"],\"unspecifiedBots\":[],\"sitemaps\":[\"https://www.nytimes.com/sitemaps/new/news.xml.gz\",\"https://www.nytimes.com/sitemaps/new/sitemap.xml.gz\",\"https://www.nytimes.com/sitemaps/new/collections.xml.gz\",\"https://www.nytimes.com/sitemaps/new/video.xml.gz\",\"https://www.nytimes.com/sitemaps/new/cooking.xml.gz\",\"https://www.nytimes.com/sitemaps/new/recipe-collects.xml.gz\",\"https://www.nytimes.com/sitemaps/new/regions.xml\",\"https://www.nytimes.com/sitemaps/new/best-sellers.xml\",\"https://www.nytimes.com/sitemaps/new/subscription-landing-pages.xml\",\"https://www.nytimes.com/sitemaps/new/weather.xml.gz\",\"https://www.nytimes.com/sitemaps/new/espanol.xml.gz\",\"https://www.nytimes.com/sitemaps/new/espanol-collects.xml.gz\",\"https://www.nytimes.com/wirecutter/sitemapindex.xml\",\"https://www.nytimes.com/athletic/sitemap-videos-index.xml\",\"https://www.nytimes.com/athletic/sitemap-authors.xml\",\"https://www.nytimes.com/athletic/sitemap-verticals.xml\",\"https://www.nytimes.com/athletic/sitemap-teams.xml\",\"https://www.nytimes.com/athletic/sitemap-cities.xml\",\"https://www.nytimes.com/athletic/sitemap-players.xml\",\"https://www.nytimes.com/athletic/sitemap-tags.xml\",\"https://www.nytimes.com/athletic/sitemap-stats.xml\",\"https://www.nytimes.com/athletic/sitemap-schedule.xml\",\"https://www.nytimes.com/athletic/sitemap-roster.xml\",\"https://www.nytimes.com/athletic/sitemap.xml\",\"https://www.nytimes.com/games-assets/v2/assets/sitemap/games.xml\"],\"landingPagePolicy\":{\"path\":\"/ads/\",\"oaiAdsBotAccess\":\"blocked\",\"policyProfiles\":[{\"bot\":\"GPTBot\",\"access\":\"blocked\",\"purposes\":[\"model-training\"],\"matchedRule\":{\"directive\":\"disallow\",\"path\":\"/\",\"userAgentScope\":\"exact\"}},{\"bot\":\"ChatGPT-User\",\"access\":\"blocked\",\"purposes\":[\"user-requested-page-fetch\"],\"matchedRule\":{\"directive\":\"disallow\",\"path\":\"/\",\"userAgentScope\":\"exact\"}},{\"bot\":\"OAI-SearchBot\",\"access\":\"blocked\",\"purposes\":[\"ai-search-and-grounding\"],\"matchedRule\":{\"directive\":\"disallow\",\"path\":\"/\",\"userAgentScope\":\"exact\"}},{\"bot\":\"OAI-AdsBot\",\"access\":\"blocked\",\"purposes\":[\"ad-landing-page-review\"],\"matchedRule\":{\"directive\":\"disallow\",\"path\":\"/ads/\",\"userAgentScope\":\"wildcard\"}},{\"bot\":\"ClaudeBot\",\"access\":\"blocked\",\"purposes\":[\"model-training\"],\"matchedRule\":{\"directive\":\"disallow\",\"path\":\"/\",\"userAgentScope\":\"exact\"}},{\"bot\":\"Claude-SearchBot\",\"access\":\"blocked\",\"purposes\":[\"ai-search-and-grounding\"],\"matchedRule\":{\"directive\":\"disallow\",\"path\":\"/\",\"userAgentScope\":\"exact\"}},{\"bot\":\"PerplexityBot\",\"access\":\"blocked\",\"purposes\":[\"ai-search-and-grounding\"],\"matchedRule\":{\"directive\":\"disallow\",\"path\":\"/\",\"userAgentScope\":\"exact\"}},{\"bot\":\"Google-Extended\",\"access\":\"blocked\",\"purposes\":[\"model-training\",\"ai-search-and-grounding\"],\"matchedRule\":{\"directive\":\"disallow\",\"path\":\"/\",\"userAgentScope\":\"exact\"}},{\"bot\":\"Applebot-Extended\",\"access\":\"blocked\",\"purposes\":[\"model-training\"],\"matchedRule\":{\"directive\":\"disallow\",\"path\":\"/\",\"userAgentScope\":\"exact\"}},{\"bot\":\"CCBot\",\"access\":\"blocked\",\"purposes\":[\"open-web-data\"],\"matchedRule\":{\"directive\":\"disallow\",\"path\":\"/\",\"userAgentScope\":\"exact\"}}],\"purposeSummaries\":{\"model-training\":{\"allowedBots\":[],\"blockedBots\":[\"GPTBot\",\"ClaudeBot\",\"Google-Extended\",\"Applebot-Extended\"],\"unspecifiedBots\":[]},\"ai-search-and-grounding\":{\"allowedBots\":[],\"blockedBots\":[\"OAI-SearchBot\",\"Claude-SearchBot\",\"PerplexityBot\",\"Google-Extended\"],\"unspecifiedBots\":[]},\"user-requested-page-fetch\":{\"allowedBots\":[],\"blockedBots\":[\"ChatGPT-User\"],\"unspecifiedBots\":[]},\"ad-landing-page-review\":{\"allowedBots\":[],\"blockedBots\":[\"OAI-AdsBot\"],\"unspecifiedBots\":[]},\"open-web-data\":{\"allowedBots\":[],\"blockedBots\":[\"CCBot\"],\"unspecifiedBots\":[]}}}},\"llmsTxt\":{\"url\":\"https://www.nytimes.com/llms.txt\",\"status\":404,\"exists\":false,\"valid\":false,\"linkCount\":0},\"sitemap\":{\"url\":\"https://www.nytimes.com/sitemap.xml\",\"status\":403,\"exists\":false},\"page\":{\"title\":null,\"description\":null,\"canonical\":null,\"robotsMeta\":null,\"jsonLdTypes\":[],\"hasMainElement\":false},\"oaiAdsBotSimulation\":{\"attempted\":true,\"userAgent\":\"OAI-AdsBot\",\"httpStatus\":200,\"finalUrl\":\"https://www.nytimes.com/ads/\",\"responseTimeMs\":10,\"error\":null,\"note\":\"One simulated request flow used the documented OAI-AdsBot User-Agent from this Actor\\u0027s IP. It is a simulation, not proof of actual OpenAI crawler access.\"},\"comparison\":{\"status\":\"unavailable\",\"previousDatasetId\":\"LnBh1u0LORkm0kJNW\",\"changed\":null,\"changes\":[],\"note\":\"The previous Dataset could not be read; the current audit completed without comparison.\"},\"issues\":[{\"code\":\"AI_BOTS_BLOCKED\",\"severity\":\"warning\",\"message\":\"9 audited AI crawler(s) are blocked at the site root.\"},{\"code\":\"AD_LANDING_PAGE_BOT_BLOCKED\",\"severity\":\"warning\",\"message\":\"OAI-AdsBot is blocked for the audited landing-page path /ads/.\"},{\"code\":\"SITEMAP_MISSING\",\"severity\":\"warning\",\"message\":\"A valid root sitemap.xml was not found.\"},{\"code\":\"LLMS_MISSING\",\"severity\":\"info\",\"message\":\"A valid llms.txt with a title and absolute Markdown link was not found. This is optional and not a ranking guarantee.\"},{\"code\":\"TITLE_MISSING\",\"severity\":\"warning\",\"message\":\"The homepage has no title.\"},{\"code\":\"DESCRIPTION_MISSING\",\"severity\":\"info\",\"message\":\"The homepage has no meta description.\"},{\"code\":\"CANONICAL_MISSING\",\"severity\":\"info\",\"message\":\"The homepage has no canonical link.\"},{\"code\":\"JSONLD_MISSING\",\"severity\":\"info\",\"message\":\"No valid JSON-LD type was detected on the homepage.\"}],\"recommendations\":[{\"issueCode\":\"AI_BOTS_BLOCKED\",\"priority\":\"high\",\"action\":\"Review each blocked AI crawler with the site owner. Change robots.txt only when allowing that crawler matches the documented content and access policy.\"},{\"issueCode\":\"AD_LANDING_PAGE_BOT_BLOCKED\",\"priority\":\"high\",\"action\":\"Before submitting an ad, review the path-specific OAI-AdsBot rule with the site owner and remove the block only when it matches the documented access policy.\"},{\"issueCode\":\"SITEMAP_MISSING\",\"priority\":\"high\",\"action\":\"Publish a valid XML sitemap at /sitemap.xml or reference its actual URL from /robots.txt, then rerun the audit.\"},{\"issueCode\":\"LLMS_MISSING\",\"priority\":\"medium\",\"action\":\"Optionally publish /llms.txt with a clear H1 and absolute Markdown links to important public pages. This is not a ranking or citation guarantee.\"},{\"issueCode\":\"TITLE_MISSING\",\"priority\":\"medium\",\"action\":\"Add one descriptive \\u003ctitle\\u003e element to the homepage and confirm it is present in the deployed HTML.\"},{\"issueCode\":\"DESCRIPTION_MISSING\",\"priority\":\"low\",\"action\":\"Add a concise homepage meta description that accurately describes the public page.\"},{\"issueCode\":\"CANONICAL_MISSING\",\"priority\":\"low\",\"action\":\"Add an absolute canonical link for the preferred public homepage URL.\"},{\"issueCode\":\"JSONLD_MISSING\",\"priority\":\"low\",\"action\":\"Add valid JSON-LD that accurately represents the page, such as Organization or WebSite where appropriate.\"}],\"chargeable\":true}" } ``` ## Output ```json { "url": { "label": "Url", "format": "string" }, "comparisonStatus": { "label": "Comparison status", "format": "string" }, "comparisonChanged": { "label": "Comparison changed", "format": "boolean" }, "comparisonSummary": { "label": "Comparison summary", "format": "string" }, "checkedAt": { "label": "Checked at", "format": "string" } } ``` ## About this Actor This example demonstrates how to use [ChatGPT Ads & AI Crawler Policy Auditor](https://apify.com/x2ppi/ai-crawler-readiness-auditor) with a specific input configuration. Visit the [Actor detail page](https://apify.com/x2ppi/ai-crawler-readiness-auditor) to learn more, explore other use cases, and run it yourself.