Check llms.txt and AI crawler rules (GPTBot) in robots.txt

Created by

Lindenwerk Data

Actor

SEO Audit & Site Audit: Broken Links & AI Bot Check

Check whether websites publish an llms.txt file and which AI crawlers (GPTBot, ClaudeBot, Google-Extended, PerplexityBot, CCBot and others) their robots.txt allows or blocks, alongside a quick SEO check of each homepage. One site summary row per website. The example compares two open-source docs sites with a news publisher that blocks AI training crawlers; use your own domains.

Try for free

SEO Audit & Site Audit: Broken Links & AI Bot Checklindenwerk/seo-audit-crawler

Row

URL

HTTP

Score

+4 fields

Input

Websites(required):https://svelte.dev+2

Mode:crawl

Max pages per site:1

Add site summary row:true

Output fields

Row

URL

HTTP

Score

Issues

Title

Indexable

How it works

Sign up on Apify or log in

The SEO Audit & Site Audit: Broken Links & AI Bot Check Actor is ready to run with the inputs already set for "Check llms.txt and AI crawler rules (GPTBot) in robots.txt". Sign up or log in and you'll land in Apify Console with a copy of this exact task.

Start the "Check llms.txt and AI crawler rules (GPTBot) in robots.txt" task

The SEO Audit & Site Audit: Broken Links & AI Bot Check Actor will start running based on the input automatically. You can also change all the input fields, like websites, mode, and more, to match your specific use case.

Receive the "Check llms.txt and AI crawler rules (GPTBot) in robots.txt" output

Monitor the progress in real time. You will be notified as soon as your dataset is complete and ready for review. Each result includes data such as row, URL, HTTP, and more.