NLWeb Compliance Checker

Built by John Rippy | johnrippy.link

🏆 2025 Zapier Automation Hero of the Year — Project Phoenix: A 95-step AI sales pipeline cutting development time by 50%. Read more →

Check if websites implement Microsoft's NLWeb (Natural Language Web) protocol for AI assistant compatibility. Get a comprehensive AI-readiness score and actionable recommendations.

What is NLWeb?

NLWeb (Natural Language Web) is Microsoft's emerging protocol that helps AI assistants understand and interact with websites. By implementing NLWeb, websites can:

• Enable AI conversations - Let AI assistants query your site naturally
• Provide structured data - Expose schemas AI can understand
• Define capabilities - Specify what interactions are supported
• Improve discoverability - Help AI systems find and use your content

Features

• NLWeb Detection: Check for /.well-known/nlweb.json presence and validity
• Schema Validation: Verify referenced JSON-LD schemas are valid
• llms.txt Check: Detect presence of the llms.txt AI information file
• robots.txt Analysis: Find AI bot directives (GPTBot, Claude, Bard, etc.)
• AI Readiness Score: 0-100 score based on multiple factors
• Actionable Recommendations: Specific steps to improve AI compatibility
• Batch Processing: Check multiple sites at once
• Webhook Integration: Get notified when checks complete

Use Cases

Website Audit

Check if your website is ready for the AI-first web. Identify gaps and get specific recommendations.

Competitive Analysis

Compare your AI readiness against competitors. See who's ahead in AI accessibility.

Agency Reporting

Generate AI readiness reports for clients. Show the value of implementing modern web standards.

Developer Onboarding

Quickly understand a site's AI capabilities before integration work.

Input Parameters

Parameter	Type	Description
urls	array	List of website URLs to check
checkSchemas	boolean	Validate schemas in nlweb.json (default: true)
checkLLMsTxt	boolean	Check for llms.txt file (default: true)
checkRobotsTxt	boolean	Analyze robots.txt for AI directives (default: true)
webhookUrl	string	Webhook URL for completion notification
demoMode	boolean	Run with sample data (default: true)

Output

{
  "totalSites": 3,
  "sitesWithNLWeb": 1,
  "sitesAIReady": 1,
  "averageScore": 65,
  "results": [
    {
      "url": "https://example.com",
      "hasNLWeb": true,
      "nlwebUrl": "https://example.com/.well-known/nlweb.json",
      "nlwebConfig": {
        "version": "1.0",
        "description": "Example website with NLWeb support",
        "capabilities": ["query", "search"],
        "schemas": ["https://example.com/schema/main.json"]
      },
      "isValid": true,
      "validationErrors": [],
      "schemas": [
        {"url": "...", "isValid": true, "type": "WebSite"}
      ],
      "hasLLMsTxt": true,
      "robotsTxtAIDirectives": [
        {"userAgent": "GPTBot", "rule": "allow"}
      ],
      "score": 95,
      "aiReady": true,
      "recommendations": []
    }
  ]
}

Scoring System

Factor	Max Points	Description
NLWeb Presence	20	Has /.well-known/nlweb.json
Valid Config	20	nlweb.json passes validation
Schema Validity	20	Referenced schemas are valid
llms.txt	15	Has /llms.txt file
robots.txt	25	Allows AI bots (GPTBot, Claude, etc.)

AI Ready: Score ≥ 70

AI Bots Checked

The actor checks robots.txt for these AI user agents:

• GPTBot (OpenAI)
• ChatGPT-User (OpenAI)
• Google-Extended (Gemini)
• Claude-Web (Anthropic)
• anthropic-ai (Anthropic)
• CCBot (Common Crawl)
• Applebot-Extended (Apple)
• PerplexityBot (Perplexity)
• Bytespider (ByteDance)
• Amazonbot (Amazon)

Example Usage

Basic Check

{
  "urls": ["https://microsoft.com", "https://google.com"],
  "demoMode": false
}

With Webhook

{
  "urls": ["https://yourdomain.com"],
  "webhookUrl": "https://hooks.zapier.com/your-webhook",
  "demoMode": false
}

Schema Validation Only

{
  "urls": ["https://yourdomain.com"],
  "checkSchemas": true,
  "checkLLMsTxt": false,
  "checkRobotsTxt": false,
  "demoMode": false
}

Pricing

This actor uses pay-per-event pricing:

Event	Price
apify-actor-start	$0.05 per run
site_checked	$0.01 per site

Example costs:

• 10 sites: $0.05 + (10 × $0.01) = $0.15
• 50 sites: $0.05 + (50 × $0.01) = $0.55
• 100 sites: $0.05 + (100 × $0.01) = $1.05

NLWeb Implementation Guide

To make your site AI-ready:

1. Create nlweb.json

// /.well-known/nlweb.json
{
  "version": "1.0",
  "description": "Your site description for AI assistants",
  "contact": "webmaster@yourdomain.com",
  "capabilities": ["query", "search", "navigate"],
  "schemas": [
    "https://yourdomain.com/schema/organization.json"
  ],
  "endpoints": {
    "query": "https://api.yourdomain.com/nlweb/query",
    "search": "https://api.yourdomain.com/nlweb/search"
  }
}

2. Add llms.txt

# YourDomain.com

## About
Brief description of your website for AI systems.

## Pages
- /about - Company information
- /products - Product catalog
- /contact - Contact details

## API
- /api/v1 - REST API documentation

3. Update robots.txt

User-agent: GPTBot
Allow: /

User-agent: Claude-Web
Allow: /

User-agent: Google-Extended
Allow: /

Related Actors

• IndexNow Submitter - Submit URLs for instant indexing
• robots.txt Checker - Comprehensive robots.txt analysis
• Technical SEO Auditor - Full site SEO audit

Support

For issues or feature requests, contact support.

---

Keywords

nlweb, microsoft nlweb, ai ready websites, natural language web, llms.txt, ai seo, chatgpt compatibility, ai accessibility, structured data, json-ld, ai assistant integration