AI Crawler Readiness Audit & llms.txt Draft

Inspect a public website before preparing it for AI crawlers. One domain produces one structured readiness report, an action list and an optional llms.txt draft built from observed site information.

What you get

• robots.txt directives for eight named crawler tokens: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot, Google-Extended and CCBot.
• Homepage response headers, llms.txt and llms-full.txt observations, JSON-LD schema types and discovered identity links.
• Local readiness scores, supporting HTTP evidence and prioritized changes to review.
• An optional llms.txt draft. Review it before publishing; the Actor never changes your website.

This is a crawler-readiness audit. It does not query paid AI models, measure citations in ChatGPT/Claude/Perplexity answers, prove accessibility from real crawler IPs, or guarantee search rankings. Autocomplete suggestions can be irrelevant to ambiguous brand names and are not AI citations. Grades are local heuristics, not certification.

Start with one site

{"domain":"linear.app","competitor_domains":[],"category_keywords":[],"generate_ready_llms_txt":true}

Optional competitors are limited to three and category phrases to ten. Start without them for a focused audit. The default example has been tested on Apify; the saved sample is an owner test, not a customer endorsement.

Output and files

The default dataset contains one report for the primary domain when its homepage returns HTTP 200. Read http_evidence, limitations, score_type and ai_answer_citations_verified before interpreting scores. OUTPUT contains the structured report; EXECUTIVE_SUMMARY contains the summary and RECOMMENDED_LLMS_TXT the optional draft.

If the homepage is blocked or unavailable, diagnostics are saved in OUTPUT with audit_status=homepage_unavailable, and no dataset result event is emitted. A run can finish successfully with diagnostic output and an empty dataset. Other individual checks may remain unknown even when the homepage is available.

Pilot pricing

$0.02 per returned primary-domain report, charged once via Apify's default-dataset-item event. No Actor start fee and no additional custom charge. Optional comparisons are included in the same report. See the live Pricing tab and your spending cap before running; Apify's post-run storage access rules still apply. No paid external API is used.

Evidence and support

The October 22 owner smoke returned HTTP 200 for Linear's homepage and robots.txt. Automated tests cover parsing, public-network protections and delivery billing guards. This pilot is maintained by HirePulse. Report issues or request help at edtool79@gmail.com. Do not send credentials in the domain field.