URL to Structured Fact — Source Verification for AI Agents

Verifies a URL and returns a structured fact sheet about the source itself (not its content): institution type (government / education / NGO / commercial), domain registration age, freshness signals (last-modified), and canonical metadata (title, language, server). 100% deterministic, side-effect-light — the kind of fast, read-only verification step a RAG or citation agent calls between finding a source and citing it.

When should an AI agent use this?

• "I found a source I want to cite — is it an official/government source or a random blog, and how old is the domain?"
• "My RAG pipeline retrieved a URL — give me a quick trust signal (domain age, institution type) before I include it in the answer."
• "I need to verify that https://www.rijksoverheid.nl/... is actually a government source and find its canonical metadata (title, language, last-modified)."
• "Batch-verify a list of URLs: classify each by institution type and flag any with very young domains (possible spam/disinformation signal)."
• "Give me the canonical URL, page title, and detected language for these URLs so I can format proper citations."
• "Check whether this source was recently updated (last-modified header or sitemap) before I rely on its data."

What this Actor does

For each URL it returns one flat, MCP-friendly record:

• Institution classification — detects government (*.overheid.nl, *.gov, *.gouv.*, *.europa.eu, rijksoverheid.nl, admin.ch, gc.ca, bund.de, belgium.be), education (*.ac.*, *.edu, uni-*.de), NGO (*.org, *.ngo, *.ong), and commercial (*.com, *.net, *.io, *.co) domains. Returns the detected type plus the matching signals.
• Domain age — looks up the domain registration date via RDAP (the modern, standardized WHOIS replacement) using the IANA bootstrap, and computes domainAgeDays. A domain registered two weeks ago is a very different trust signal than one from 1999.
• Freshness — extracts Last-Modified from the HTTP response headers. Optionally falls back to the site's sitemap.xml (includeSitemap: true) when the header is absent. Reports lastModifiedSource ("http-header" or "sitemap") so you know where the signal came from.
• Canonical metadata — the final URL (after redirects), the HTML <title>, the Content-Language, and the Server header, so you can build a proper citation without re-fetching the page.

Input

Field	Type	Default	Description
urls	array of strings	required	URL(s) to verify (max 1000 per run). Returns one fact sheet per URL.
includeSitemap	boolean	false	Also fetch the site's sitemap.xml as a fallback for last-modified. Adds one extra HTTP request per host.
timeoutSec	integer	10	Per-request timeout in seconds for HTTP and RDAP fetches. Range 3–60.

Output

One flat JSON record per verified URL:

{
  "url": "https://www.rijksoverheid.nl/onderwerpen/klimaatbeleid",
  "canonicalUrl": "https://www.rijksoverheid.nl/onderwerpen/klimaatbeleid",
  "domain": "www.rijksoverheid.nl",
  "title": "Klimaatbeleid | Rijksoverheid.nl",
  "language": "nl",
  "lastModified": "2025-08-15T10:30:00Z",
  "lastModifiedSource": "http-header",
  "server": "nginx",
  "domainRegistrationDate": "1999-07-06T10:37:04Z",
  "domainAgeDays": 9919,
  "institutionType": "government",
  "institutionSignals": ["rijksoverheid.nl"],
  "isGovernment": true,
  "fetchStatus": "ok",
  "errorCode": null,
  "error": null

Field	Description
url	The URL as supplied in the input
canonicalUrl	Final URL after following redirects
domain	Lowercased hostname
title	HTML <title> (streaming extraction, decodes entities)
language	Content-Language header, if present
lastModified	Last-modified timestamp, if available
lastModifiedSource	"http-header", "sitemap", or null
server	Server header, if present
domainRegistrationDate	ISO timestamp from RDAP, if available
domainAgeDays	Whole days since registration (computed)
institutionType	"government", "education", "ngo", "commercial", or "unknown"
institutionSignals	Which domain patterns matched
isGovernment	true when institution type is "government"
fetchStatus	"ok", "partial" (one of HTTP/RDAP failed), or "error"
errorCode	Machine-readable error class when not "ok"
error	Human-readable error detail, if any

Use cases

• RAG/citation guard — call this between retrieving a URL and citing it in an answer. A government source with a 25-year-old domain and a recent last-modified is a strong trust signal; a 3-week-old .com with no freshness data is not.
• Source triage for research agents — batch-verify a list of candidate URLs and rank them by domain age and institution type before doing expensive content extraction.
• Citation formatting — get the canonical URL, title, and language for a clean citation without a separate scraping pass.
• Spam/disinformation signal — a very young domain (domainAgeDays in the single/double digits) is a lightweight red flag worth surfacing to a downstream verifier.
• Compliance-by-default crawling — pair with a permission check: verify the source is authoritative before committing to a full scrape.

Pricing

This Actor uses Apify's Pay-Per-Event (PPE) pricing model.

• Actor Start: $0.00005 (Apify default)
• url-verified: $0.002 per URL verified — charged when the verification pipeline completed for that URL (fetchStatus ok or partial). Not charged when the URL could not be fetched at all (fetchStatus: "error", e.g. unreachable host, invalid URL).

See STOREINFO.md for the full pricing table and rationale.

Legal

This Actor reads only publicly available technical metadata: HTTP response headers, the HTML <title>, RDAP domain registration data (the standardized, machine-readable successor to WHOIS, explicitly intended for automated querying), and optionally sitemap.xml. It does not scrape page content, does not collect personal data, and does not access any paywalled or authenticated resource. Requests to private/loopback/link-local IP ranges are blocked (including via redirects), and response bodies are read with a strict size cap. The institution-type classification is a heuristic based on domain-name patterns — it is not an official registry check and comes with no guarantee of correctness (documented in the README so users treat it as a signal, not a legal determination). You remain responsible for how you use the verification results downstream.

FAQ

Does this scrape the page content? No. It fetches the HTTP headers and streams just enough of the HTML to extract the <title> tag (typically the first ~16 KB), then stops. No content is stored or returned.

Why is lastModified often null? Many sites (including major ones like rijksoverheid.nl) don't send a Last-Modified header. Set includeSitemap: true to try the site's sitemap.xml as a fallback — but note that many sitemaps are index files without per-URL lastmod dates, so null is a common and honest answer.

Why is domainRegistrationDate sometimes null? RDAP coverage is broad but not universal. The actor uses the IANA RDAP bootstrap to find the right server per TLD. Some TLDs (notably .eu) are not yet in the bootstrap; for those, domainRegistrationDate is null and fetchStatus is "partial". This is expected and documented, not a bug.

Can I verify many URLs in one run? Yes — urls is an array. Each URL is processed independently, and one failure does not stop the rest. PPE is charged per URL.

Is the institution classification guaranteed correct? No. It is a heuristic based on well-known domain patterns (.gov, .ac.uk, .org, etc.). It covers the most common cases but will return "unknown" for anything that doesn't match. Treat it as a trust signal, not a legal determination.

Can I verify internal/intranet URLs? No. Requests to private, loopback, link-local (e.g. 169.254.169.254), and other non-public IP ranges are blocked, including when reached through redirects. The actor only verifies publicly reachable sources — that is its purpose and a deliberate safety boundary.

Keywords

url verification, source verification, domain age check, institution type detector, government source check, rdap lookup, whois alternative, citation verification, rag guard, source trust signal, domain registration date, last-modified check, canonical url extractor, ai agent tool, mcp tool, fact checking source, spam domain signal

Related Actors

• llm-output-validator — validate LLM/tool-call JSON against a schema with repair hints, another developer-tool/agent-hulpmiddel from the same portfolio
• legacy-code-license-scanner — audit a repo or dependency manifest for license compatibility

Changelog

0.1.0

• Initial release: institution classification (government/education/NGO/commercial) via domain-pattern heuristics, domain-age lookup via RDAP (IANA bootstrap), HTTP Last-Modified + optional sitemap fallback, canonical metadata (title, language, server), per-URL error isolation.
• Hardening: private/loopback/link-local address blocking with per-hop redirect validation, size-capped response reads, max 1000 URLs per run, PPE charged only when the verification pipeline completed (not on total failures), per-domain RDAP memoization within a run, and concurrent HTTP+RDAP probing per URL.