Agent Web Access Plan

Sitemap Extractor - Hardened URL Inventory

8tp/sitemap-url-extractor-coverage-api

Extract URL inventories from public HTTPS sitemap XML and gzip files with robots.txt discovery, bounded index traversal, and machine-readable coverage receipts. No page crawling, login, browser, proxy, or private-network access.

Structured Data Extractor for AI Agents

bmiller1009/structured-data-extractor

Extract JSON-LD, schema.org types, OpenGraph, and microdata from a public page for machine-readable consumption.

Agent Accessibility Auditor llms txt Robots AI Schema Check

mambalabs/agent-accessibility-auditor

Can an AI agent read this site. Checks llms.txt, robots AI crawler rules and the Content Signal policy, structured data, render mode, and machine readable endpoints like sitemap, OpenAPI and well known files. Every field is a fact read off a fetch, never a model opinion.

SEO Audit Tool: Title, Meta, Schema, Robots, Sitemap

eliai/website-seo-tech-auditor

SEO audit for any URL or whole site: crawl mode finds redirect chains, broken links, duplicate titles; page mode checks title/meta, headings, schema, robots.txt, sitemap, canonical. Machine-readable JSON with a pre-sorted fix list. $0.10 per page audited, failures free.

Extract bounded plan, price, billing-period, feature, and evidence signals from public SaaS pricing pages with useful-result pricing.

Robots Meta X-Robots Auditor

junipr/robots-meta-xrobots-auditor

Audit page-level robots directives from meta tags and X-Robots-Tag headers and detect conflicting indexability signals.

Audit robots.txt policies for major AI crawlers and inspect llms.txt discovery files for public websites with structured machine-readable results.

Convert any web page into clean, readable Markdown or plain text — ads, navigation and boilerplate stripped out. Built for feeding AI agents, LLMs and RAG pipelines. Pay only per page.