Agent Interface Discovery

Agent-facing, structured discovery of public machine-readable site interfaces. Given up to 20 public HTTPS domains, inspect robots.txt, llms.txt, llms-full.txt, /.well-known/openapi.json, and /openapi.json without following redirects. Returns one JSON dataset record per domain with HTTP statuses, policy summary for a selected user-agent token, llms.txt links, OpenAPI metadata, and timestamps. It does not crawl pages, bypass robots rules, authenticate, or claim that an endpoint is usable without testing.

Input

{"domains":["https://example.com"],"userAgent":"GPTBot"}

Buyers / uses

AI-agent developers, retrieval pipeline builders, integration catalog maintainers and documentation auditors can discover whether a site advertises agent-readable docs and API specifications. Outputs are deterministic and useful as an agent tool response. No LLM API cost.

Security and limitations

Only public HTTPS hostnames, no IP literals, private DNS results rejected, redirects not followed, 12-second timeout per resource, 250 KB maximum per response. Public DNS may change between resolution and fetch; deploy behind egress restrictions for strong SSRF protection. robots.txt is a path-specific directive, not a blanket authorization; returned root policy is only a summary. Missing files are reported as HTTP statuses, not treated as errors. No automatic paid runs.

Availability

This repository contains Actor source code. It is separate from an Apify Store listing; verify current Store availability before trying to run it.

Run-to-run change detection

The first run for a domain and user-agent establishes a private baseline in the Actor's key-value store. Later runs report changes to robots rules, llms.txt headings and links, OpenAPI metadata and endpoint paths, resource availability, and llms-full.txt fingerprints. OpenAPI paths are capped at 100 and robots rules at 200 in each result to keep output bounded. Failed all-resource checks do not replace a prior baseline.