Pricing

from $4.00 / 1,000 robots meta audit results

Try for free

Go to Apify Store

PAL | Robots Noindex Directive Checker

Try for free

Check public HTML robots directives including noindex and Googlebot tags for website launch and migration QA. Does not verify actual search-engine indexing. Try a prefilled Example task, export CSV/JSON, and check Pricing before larger runs. Static public HTML only.

Pricing

from $4.00 / 1,000 robots meta audit results

Rating

0.0

(0)

Developer

Practical AI Lab

Actor stats

Bookmarked

Total users

Monthly active users

a day ago

Last modified

Categories

Developer tools

You can access the PAL | Robots Noindex Directive Checker programmatically from your own applications by using the Apify API. You can also choose the language preference from below. To use the Apify API, you’ll need an Apify account and your API token, found in API & Integrations in Apify Console.

Python

JavaScript

CLI

OpenAPI

HTTP

MCP

{
  "openapi": "3.0.1",
  "info": {
    "version": "0.0",
    "x-build-id": "2r558VMgvtpx917bz"
  },
  "servers": [
    {
      "url": "https://api.apify.com/v2"
    }
  ],
  "paths": {
    "/acts/practical_ai_lab_jp~my-actor-14/run-sync-get-dataset-items": {
      "post": {
        "operationId": "run-sync-get-dataset-items-practical_ai_lab_jp-my-actor-14",
        "x-openai-isConsequential": false,
        "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.",
        "tags": [
          "Run Actor"
        ],
        "requestBody": {
          "required": true,
          "content": {
            "application/json": {
              "schema": {
                "$ref": "#/components/schemas/inputSchema"
              }
            }
          }
        },
        "parameters": [
          {
            "name": "token",
            "in": "query",
            "required": true,
            "schema": {
              "type": "string"
            },
            "description": "Enter your Apify token here"
          }
        ],
        "responses": {
          "200": {
            "description": "OK"
          }
        }
      }
    },
    "/acts/practical_ai_lab_jp~my-actor-14/runs": {
      "post": {
        "operationId": "runs-sync-practical_ai_lab_jp-my-actor-14",
        "x-openai-isConsequential": false,
        "summary": "Executes an Actor and returns information about the initiated run in response.",
        "tags": [
          "Run Actor"
        ],
        "requestBody": {
          "required": true,
          "content": {
            "application/json": {
              "schema": {
                "$ref": "#/components/schemas/inputSchema"
              }
            }
          }
        },
        "parameters": [
          {
            "name": "token",
            "in": "query",
            "required": true,
            "schema": {
              "type": "string"
            },
            "description": "Enter your Apify token here"
          }
        ],
        "responses": {
          "200": {
            "description": "OK",
            "content": {
              "application/json": {
                "schema": {
                  "$ref": "#/components/schemas/runsResponseSchema"
                }
              }
            }
          }
        }
      }
    },
    "/acts/practical_ai_lab_jp~my-actor-14/run-sync": {
      "post": {
        "operationId": "run-sync-practical_ai_lab_jp-my-actor-14",
        "x-openai-isConsequential": false,
        "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.",
        "tags": [
          "Run Actor"
        ],
        "requestBody": {
          "required": true,
          "content": {
            "application/json": {
              "schema": {
                "$ref": "#/components/schemas/inputSchema"
              }
            }
          }
        },
        "parameters": [
          {
            "name": "token",
            "in": "query",
            "required": true,
            "schema": {
              "type": "string"
            },
            "description": "Enter your Apify token here"
          }
        ],
        "responses": {
          "200": {
            "description": "OK"
          }
        }
      }
    }
  },
  "components": {
    "schemas": {
      "inputSchema": {
        "type": "object",
        "required": [
          "urls"
        ],
        "properties": {
          "urls": {
            "title": "Public HTTPS page URLs",
            "minItems": 1,
            "maxItems": 18,
            "type": "array",
            "description": "Each URL yields an independent audit row. Static HTML only. No authentication.",
            "default": [
              "https://scrapeme.live/shop/Pikachu/"
            ],
            "items": {
              "type": "string"
            }
          }
        }
      },
      "runsResponseSchema": {
        "type": "object",
        "properties": {
          "data": {
            "type": "object",
            "properties": {
              "id": {
                "type": "string"
              },
              "actId": {
                "type": "string"
              },
              "userId": {
                "type": "string"
              },
              "startedAt": {
                "type": "string",
                "format": "date-time",
                "example": "2025-01-08T00:00:00.000Z"
              },
              "finishedAt": {
                "type": "string",
                "format": "date-time",
                "example": "2025-01-08T00:00:00.000Z"
              },
              "status": {
                "type": "string",
                "example": "READY"
              },
              "meta": {
                "type": "object",
                "properties": {
                  "origin": {
                    "type": "string",
                    "example": "API"
                  },
                  "userAgent": {
                    "type": "string"
                  }
                }
              },
              "stats": {
                "type": "object",
                "properties": {
                  "inputBodyLen": {
                    "type": "integer",
                    "example": 2000
                  },
                  "rebootCount": {
                    "type": "integer",
                    "example": 0
                  },
                  "restartCount": {
                    "type": "integer",
                    "example": 0
                  },
                  "resurrectCount": {
                    "type": "integer",
                    "example": 0
                  },
                  "computeUnits": {
                    "type": "integer",
                    "example": 0
                  }
                }
              },
              "options": {
                "type": "object",
                "properties": {
                  "build": {
                    "type": "string",
                    "example": "latest"
                  },
                  "timeoutSecs": {
                    "type": "integer",
                    "example": 300
                  },
                  "memoryMbytes": {
                    "type": "integer",
                    "example": 1024
                  },
                  "diskMbytes": {
                    "type": "integer",
                    "example": 2048
                  }
                }
              },
              "buildId": {
                "type": "string"
              },
              "defaultKeyValueStoreId": {
                "type": "string"
              },
              "defaultDatasetId": {
                "type": "string"
              },
              "defaultRequestQueueId": {
                "type": "string"
              },
              "buildNumber": {
                "type": "string",
                "example": "1.0.0"
              },
              "containerUrl": {
                "type": "string"
              },
              "usage": {
                "type": "object",
                "properties": {
                  "ACTOR_COMPUTE_UNITS": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATASET_READS": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATASET_WRITES": {
                    "type": "integer",
                    "example": 0
                  },
                  "KEY_VALUE_STORE_READS": {
                    "type": "integer",
                    "example": 0
                  },
                  "KEY_VALUE_STORE_WRITES": {
                    "type": "integer",
                    "example": 1
                  },
                  "KEY_VALUE_STORE_LISTS": {
                    "type": "integer",
                    "example": 0
                  },
                  "REQUEST_QUEUE_READS": {
                    "type": "integer",
                    "example": 0
                  },
                  "REQUEST_QUEUE_WRITES": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATA_TRANSFER_INTERNAL_GBYTES": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATA_TRANSFER_EXTERNAL_GBYTES": {
                    "type": "integer",
                    "example": 0
                  },
                  "PROXY_RESIDENTIAL_TRANSFER_GBYTES": {
                    "type": "integer",
                    "example": 0
                  },
                  "PROXY_SERPS": {
                    "type": "integer",
                    "example": 0
                  },
                  "PROXY_UNBLOCKER_UNITS": {
                    "type": "integer",
                    "example": 0
                  }
                }
              },
              "usageTotalUsd": {
                "type": "number",
                "example": 0.00005
              },
              "usageUsd": {
                "type": "object",
                "properties": {
                  "ACTOR_COMPUTE_UNITS": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATASET_READS": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATASET_WRITES": {
                    "type": "integer",
                    "example": 0
                  },
                  "KEY_VALUE_STORE_READS": {
                    "type": "integer",
                    "example": 0
                  },
                  "KEY_VALUE_STORE_WRITES": {
                    "type": "number",
                    "example": 0.00005
                  },
                  "KEY_VALUE_STORE_LISTS": {
                    "type": "integer",
                    "example": 0
                  },
                  "REQUEST_QUEUE_READS": {
                    "type": "integer",
                    "example": 0
                  },
                  "REQUEST_QUEUE_WRITES": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATA_TRANSFER_INTERNAL_GBYTES": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATA_TRANSFER_EXTERNAL_GBYTES": {
                    "type": "integer",
                    "example": 0
                  },
                  "PROXY_RESIDENTIAL_TRANSFER_GBYTES": {
                    "type": "integer",
                    "example": 0
                  },
                  "PROXY_SERPS": {
                    "type": "integer",
                    "example": 0
                  },
                  "PROXY_UNBLOCKER_UNITS": {
                    "type": "integer",
                    "example": 0
                  }
                }
              }
            }
          }
        }
      }
    }
  }
}

Robots Noindex and Googlebot Meta Tag Checker | PAL OpenAPI definition

OpenAPI is a standard for designing and describing RESTful APIs, allowing developers to define API structure, endpoints, and data formats in a machine-readable way. It simplifies API development, integration, and documentation.

OpenAPI is effective when used with AI agents and GPTs by standardizing how these systems interact with various APIs, for reliable integrations and efficient communication.

By defining machine-readable API specifications, OpenAPI allows AI models like GPTs to understand and use varied data sources, improving accuracy. This accelerates development, reduces errors, and provides context-aware responses, making OpenAPI a core component for AI applications.

You can download the OpenAPI definitions for PAL | Robots Noindex Directive Checker from the options below:

OpenAPI.json

If you’d like to learn more about how OpenAPI powers GPTs, read our blog post.

You can also check out our other API clients:

PAL | Robots Noindex Directive Checker API in Python

PAL | Robots Noindex Directive Checker API in JavaScript

PAL | Robots Noindex Directive Checker API through CLI

PAL | Robots Noindex Directive Checker API

Expired Domains Finder — GoDaddy Auctions with SEO Metrics

hipersoft/godaddy-auctions-scraper

Find expired and auction domains from GoDaddy Auctions with price, bids, age, valuation, Majestic Trust/Citation Flow, backlinks, referring domains, SEMrush Authority Score, keyword search volume and CPC. Filter by TLD, keyword, price, age and SEO metrics. Export CSV/JSON.

hiper soft

Google Index Checker API

tugelbay/google-serp-checker

Compare a page sitemap with observed first-page Google site: results. Export sampled links, sitemap matches and missing-from-sample candidates. This is not a complete indexation check. Guide: https://konabayev.com/tools/google-serp-checker/

Tugelbay Konabayev

XML Sitemap Finder & Extractor API

andok/find-sitemap-from-url

Find and extract all XML sitemaps for any domain. Automatically parses robots.txt, scans HTML tags, and recursively follows indexes. Perfect for SEO & web scraping.

Andok

✨LLMO & AI Search Auditor - /llms.txt [ChatGPT/Perplexity]

invaluable_rondeau/llmo-ai-search-readiness-auditor

✅ Score any domain (0-100) for ChatGPT Search, Perplexity, Claude & Gemini citation readiness ($15/1k domains). ✅ Audits /llms.txt, robots.txt AI crawlers (GPTBot/ClaudeBot), Schema.org JSON-LD & auto-generates a ready-to-deploy /llms.txt file.

PROOFNEXA

PAL Pagination Link Audit

practical_ai_lab_jp/pal-pagination-link-checker

Inspect rel=next and rel=prev navigation tags. Static HTML and JSON-LD only; public HTTPS URLs.

Practical AI Lab

PAL | Internal Link & URL Inventory

practical_ai_lab_jp/pal-internal-link-auditor

Export internal navigation links and duplicate or fragment patterns from public HTML pages. Try the Example task before checking your pages. This is not a full-site crawler.

Practical AI Lab

PAL HTML Language Audit

practical_ai_lab_jp/pal-html-language-checker

Detect missing or malformed html lang attributes. Static HTML and JSON-LD only; public HTTPS URLs.

Practical AI Lab

PAL | Bulk Meta Description Checker for SEO

practical_ai_lab_jp/my-actor-12

Bulk meta description checker for SEO teams: inspect up to 18 public HTML URLs for missing, empty, repeated and unusually short or long descriptions. Export page-level findings as CSV or JSON. Try an Example task before running larger checks.

Practical AI Lab

Shopify Store Auditor: SEO & Product Catalog Audit

siftwright_data/shopify-store-auditor

Audit any Shopify store's catalog: thin or missing descriptions, images without alt text, duplicate titles, zero prices, missing product types and tags. 0-100 score per store plus every flagged product. No login or app needed.