Well-Known Extractor

Probes a domain's /.well-known/ endpoints and returns each file's raw body plus a parsed representation. Runs as a batch over many domains, or as a Standby HTTP API for one domain per request.

Input

Field	Type	Default	Description
domains	array	— (required)	Domains to probe, e.g. google.com. Each is normalized to its https origin.
profiles	array	["core"]	Endpoint groups to probe. Use ["all"] for all 35.
endpoints	array	—	Individual endpoint ids, added on top of profiles.
includeNotFound	boolean	false	Also write dataset rows for endpoints that were absent.

profiles and endpoints are combined and deduplicated by path. If both are empty, core is used. At most 50 endpoints are probed per domain; extras are skipped with a warning.

Endpoint profiles

Profile	Endpoints
core (11)	security.txt, gpc.json, openid-configuration, oauth-authorization-server, oauth-protected-resource, assetlinks.json, apple-app-site-association, nodeinfo, host-meta, agent-card.json, llms.txt
security (4)	security.txt, gpc.json, dnt-policy.txt, openpgpkey-policy
identity (7)	openid-configuration, oauth-authorization-server, oauth-protected-resource, openid-federation, jwks.json, did.json, microsoft-identity-association.json
mobile (4)	assetlinks.json, apple-app-site-association, apple-developer-merchantid-domain-association, microsoft-identity-association.json
email (2)	mta-sts.txt, openpgpkey-policy
social (9)	did.json, nodeinfo, host-meta, host-meta.json, matrix-server, matrix-client, matrix-support, atproto-did, discord
ai (6)	oauth-protected-resource, agent-card.json, agent.json, mcp.json, ai.txt, llms.txt
adtech (3)	ads.txt, app-ads.txt, sellers.json
web (5)	traffic-advice, related-website-set.json, privacy-sandbox-attestations.json, web-app-origin-association, discord
all (35)	All of the above.

Profiles overlap; an endpoint in two selected profiles is probed once. 34 of the 35 endpoints have a parser; openpgpkey-policy returns raw only. llms.txt, ads.txt, app-ads.txt, and sellers.json live at the site root, not under /.well-known/.

Output

A batch run writes one dataset row per endpoint found:

{
    "domain": "www.google.com",
    "origin": "https://www.google.com",
    "scannedAt": "2026-08-04T12:00:00.000Z",
    "id": "security.txt",
    "path": "/.well-known/security.txt",
    "title": "Security contact policy",
    "spec": "https://www.rfc-editor.org/rfc/rfc9116",
    "url": "https://www.google.com/.well-known/security.txt",
    "finalUrl": "https://www.google.com/.well-known/security.txt",
    "found": true,
    "httpStatus": 200,
    "contentType": "text/plain",
    "raw": "Contact: https://g.co/vulnz\nExpires: 2027-01-01T00:00:00Z\n",
    "parsed": { "contact": ["https://g.co/vulnz"], "expires": "2027-01-01T00:00:00Z" },
    "error": null
}

Field	Description
domain / origin	Scanned host and its normalized origin.
id / path / title / spec	Which endpoint this row is, and the spec that defines it.
url / finalUrl	Requested URL and the URL after redirects.
found	Whether the response counted as a real hit.
httpStatus / contentType	Response status and content type. null status means a transport error.
raw	Response body, capped at 256 KB. null when not found.
parsed	Structured form of raw, or null if no parser applies or parsing failed.
error	Transport error message, otherwise null.

found is false for a 4xx, and also for a 200 that returns HTML or an unexpected body shape, since sites commonly answer unknown paths with a catch-all page.

You can download the dataset as JSON, CSV, Excel, or HTML.

Standby API

GET  /?domain=google.com
GET  /?domain=google.com&profiles=ai,security
GET  /?domain=google.com&endpoints=security.txt,openid-configuration
POST /            { "domain": "google.com", "profiles": ["core"] }

One domain per request. The response wraps the same probes in wellKnown[] and adds a summary:

{
    "domain": "www.google.com",
    "url": "https://www.google.com",
    "scannedAt": "2026-08-04T12:00:00.000Z",
    "wellKnown": [{ "id": "security.txt", "found": true, "httpStatus": 200, "parsed": {} }],
    "summary": {
        "probed": 11,
        "found": ["security.txt"],
        "notFound": ["assetlinks.json"],
        "errored": [],
        "security": { "contacts": ["https://g.co/vulnz"], "expires": "2027-01-01T00:00:00Z" },
        "openidIssuer": "https://accounts.google.com"
    }
}

Pricing

One domain-scanned event per domain, charged whether or not anything is found. Cost is the number of domains times the per-domain price.