NLWeb Compliance Checker

Check if websites implement Microsoft's NLWeb (Natural Language Web) protocol for AI assistant compatibility. Validates /.well-known/nlweb.json and associated schemas.

Features

• NLWeb Detection: Check for /.well-known/nlweb.json presence and validity
• Schema Validation: Verify referenced JSON-LD schemas are valid
• llms.txt Check: Detect presence of the llms.txt AI information file
• robots.txt Analysis: Find AI bot directives (GPTBot, Claude, Bard, etc.)
• AI Readiness Score: 0-100 score based on multiple factors
• Actionable Recommendations: Specific steps to improve AI compatibility
• Batch Processing: Check multiple sites at once
• Webhook Integration: Get notified when checks complete

Quick Start

{
  "input": "your input here"
}

Demo Mode

Set demoMode: true to test with sample data (no charges). When you're ready for real results, set demoMode: false or omit it.

{
  "demoMode": true,
  ...
}

Input Parameters

Parameter	Type	Description
urls	array	List of website URLs to check
checkSchemas	boolean	Validate schemas in nlweb.json (default: true)
checkLLMsTxt	boolean	Check for llms.txt file (default: true)
checkRobotsTxt	boolean	Analyze robots.txt for AI directives (default: true)
webhookUrl	string	Webhook URL for completion notification
demoMode	boolean	Run with sample data (default: true)

Output Format

{
  "totalSites": 3,
  "sitesWithNLWeb": 1,
  "sitesAIReady": 1,
  "averageScore": 65,
  "results": [
    {
      "url": "https://example.com",
      "hasNLWeb": true,
      "nlwebUrl": "https://example.com/.well-known/nlweb.json",
      "nlwebConfig": {
        "version": "1.0",
        "description": "Example website with NLWeb support",
        "capabilities": ["query", "search"],
        "schemas": ["https://example.com/schema/main.json"]
      },
      "isValid": true,
      "validationErrors": [],
      "schemas": [
        {"url": "...", "isValid": true, "type": "WebSite"}
      ],
      "hasLLMsTxt": true,
      "robotsTxtAIDirectives": [
        {"userAgent": "GPTBot", "rule": "allow"}
      ],
      "score": 95,
      "aiReady": true,
      "recommendations": []
    }
  ]
}

Pricing

This actor uses pay-per-event billing: This actor uses pay-per-event pricing:

Event	Price
apify-actor-start	$0.05 per run
site_checked	$0.01 per site

Example costs:

• 10 sites: $0.05 + (10 × $0.01) = $0.15
• 50 sites: $0.05 + (50 × $0.01) = $0.55
• 100 sites: $0.05 + (100 × $0.01) = $1.05

Use Cases

Website Audit

Check if your website is ready for the AI-first web. Identify gaps and get specific recommendations.

Competitive Analysis

Compare your AI readiness against competitors. See who's ahead in AI accessibility.

Agency Reporting

Generate AI readiness reports for clients. Show the value of implementing modern web standards.

Developer Onboarding

Quickly understand a site's AI capabilities before integration work.

---

Common Problems & Solutions

"Invalid API key" error

Cause: Your API key is wrong, expired, or doesn't have the right permissions. Fix: Double-check your API key. Make sure you copied it exactly without extra spaces.

"Rate limit exceeded" error

Cause: You've hit the API's rate limits. Fix: Wait a few minutes, then try again. Consider reducing the number of concurrent requests.

Empty or incomplete results

Cause: The target may have anti-scraping protection or the data doesn't exist. Fix:

• Check if the URL/search query is correct
• Try with different parameters
• Some sites may block automated access

Demo data showing instead of real results

Cause: demoMode is still set to true. Fix: Set demoMode: false and provide your API key(s).

---

Built by John Rippy | Actor Arsenal