OAI AdsBot Landing Page Auditor

Audit public HTTPS landing-page reachability and declared robots policy for OAI-AdsBot. This independent tool is not affiliated with or endorsed by OpenAI.

What this Actor helps you do

Check whether public landing pages are reachable and whether their declared robots.txt rules allow OAI-AdsBot before you investigate advertising or crawler-delivery problems. Each URL produces one structured evidence row with an explicit limitation statement.

Features

• Audits one to fifty unique public HTTPS landing-page URLs per run.
• Evaluates declared robots policy for OAI-AdsBot and optionally OAI-SearchBot.
• Follows bounded, revalidated public HTTPS redirects and blocks private-network targets.
• Returns typed readiness evidence and machine-readable reason codes in the default dataset.

Good fits

• Preflight a campaign landing-page list before a human advertising review.
• Find declared robots exclusions or public reachability failures across a small URL batch.
• Feed bounded technical evidence into monitoring, QA, or incident-response workflows.

Not a fit

• It does not approve ads, predict ranking, citations, traffic, revenue, or campaign performance.
• It does not impersonate an OpenAI crawler or prove crawler source addresses and WAF allowlists.
• It is not a legal, privacy, accessibility, security, or regulatory compliance certification.

Tutorial

1. Open the input form and keep the safe example.com URL for your first test run.
2. Replace or extend the URL list with up to fifty public HTTPS landing pages; optionally enable the OAI-SearchBot companion check.
3. Run the Actor with limited permissions and review the Overview table in the default dataset.
4. Use outcome, reason_codes, HTTP evidence, and the limitation field together; do not treat a ready row as ad approval or crawler-delivery proof.

Example input

The Actor input form is prefilled with this safe example. API users must still send an input object explicitly.

{
  "include_search_bot": false,
  "max_redirects": 2,
  "urls": [
    "https://example.com/"
  ]
}

Representative output

This illustrative row explains the output shape; live values depend on the requested public page and its declared robots policy.

{
  "adsbot_matched_rule": "",
  "adsbot_policy": "default_allow",
  "bytes_read": 1256,
  "content_type": "text/html",
  "evidence_summary": "The page was publicly reachable and no explicit OAI-AdsBot robots exclusion was observed.",
  "final_url": "https://example.com/",
  "input_url": "https://example.com/",
  "limitation": "This checks public reachability and declared robots policy; it does not verify crawler source addresses, WAF allowlisting, ad approval, ranking, citation, revenue, or compliance.",
  "outcome": "ready",
  "page_http_status": 200,
  "page_reachable": true,
  "reason_codes": [
    "robots_missing_default_allow",
    "ready_for_oai_adsbot_public_preflight",
    "crawler_network_identity_not_verified"
  ],
  "redirect_count": 0,
  "robots_http_status": 404,
  "robots_url": "https://example.com/robots.txt",
  "searchbot_checked": false,
  "searchbot_matched_rule": "",
  "searchbot_policy": "not_checked"
}

Pricing

Proposed pay-per-event pricing includes platform usage: USD 0.00005 when a run starts plus USD 0.00200 for each validated dataset row. These are draft prices and do not take effect until the exact release, benchmark, and pricing settings receive signed approval.

• One persisted URL row: approximately USD 0.00205.
• Ten persisted URL rows: approximately USD 0.02005.
• Fifty persisted URL rows: approximately USD 0.10005.
• A failed input that produces no row must not incur a dataset-item event charge.

Data and privacy

Inputs must contain only public HTTPS URLs and never credentials or private customer data. The Actor connects to each supplied public host and its robots.txt endpoint, does not call a model provider, and writes only the documented rows to the default Apify dataset. Apify workspace storage and retention settings continue to apply to input and results.

Limitations

• Robots.txt is publisher-declared policy and is not proof that a crawler will visit a page.
• Public HTTP evidence can change immediately after the run completes.
• A reachable page may still be blocked by systems or policies this Actor cannot observe.
• The Actor intentionally rejects private, local, credential-bearing, malformed, and non-HTTPS targets.

Support and maintenance

Report reproducible product issues through the Actor issue channel with a minimal public URL fixture and relevant reason codes. Do not include credentials, private URLs, billing details, or customer records. Routine reports target a two-business-day response; urgent security or source-visibility reports target one business day.

Releases use semantic versions and content-addressed packages. Health, issue age, cost, and quality are reviewed on a governed schedule. Breaking changes require a new reviewed release; planned deprecation preserves the listing, gives public notice, and identifies a replacement when one exists.

Input reference

• urls (array): One to fifty unique public HTTPS landing-page URLs.
• include_search_bot (boolean, optional): Also evaluate declared robots policy for OAI-SearchBot.
• max_redirects (integer, optional): Maximum zero to five validated redirects per request.

Dataset output reference

• adsbot_matched_rule (string): Longest matched OAI-AdsBot robots rule when present.
• adsbot_policy (string): Declared robots decision for OAI-AdsBot.
• bytes_read (integer): Landing-page response bytes read within the fixed ceiling.
• content_type (string): Bounded landing-page response content type when available.
• evidence_summary (string): Concise explanation of observed public evidence.
• final_url (string): Final public HTTPS URL after bounded redirects.
• input_url (string): Validated landing-page URL supplied by the customer.
• limitation (string): Explicit statement of claims this audit cannot verify.
• outcome (string): Bounded technical readiness outcome.
• page_http_status (integer): Observed landing-page HTTP status, or zero without HTTP evidence.
• page_reachable (boolean): Whether the landing page returned a successful public HTTP status.
• reason_codes (array): Machine-readable evidence and limitation reason codes.
• redirect_count (integer): Number of validated landing-page redirects followed.
• robots_http_status (integer): Observed robots HTTP status, or zero without HTTP evidence.
• robots_url (string): Robots policy URL evaluated for the final page origin.
• searchbot_checked (boolean): Whether the optional OAI-SearchBot companion check ran.
• searchbot_matched_rule (string): Longest matched OAI-SearchBot robots rule when present.
• searchbot_policy (string): Declared robots decision for OAI-SearchBot or not_checked.

Operational boundary

Generated from CAP-OAI-ADSBOT-LANDING-PAGE-AUDITOR version 0.4.0. The Actor fails closed on invalid input and writes only the typed dataset rows described above. No credential is embedded in this package.