# URL to Structured Fact — Source Verification for AI Agents (`codeclouds/url-to-structured-fact`) Actor

Verify a URL and get a structured fact sheet: institution type (government/education/NGO/commercial), domain age via RDAP, freshness signals, and canonical metadata. A deterministic, read-only verification step for RAG and citation agents.

- **URL**: https://apify.com/codeclouds/url-to-structured-fact.md
- **Developed by:** [Dennis](https://apify.com/codeclouds) (community)
- **Categories:** Developer tools, Other
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $2.00 / 1,000 url verifieds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## URL to Structured Fact — Source Verification for AI Agents

Verifies a URL and returns a **structured fact sheet about the source itself** (not its content): institution type (government / education / NGO / commercial), domain registration age, freshness signals (last-modified), and canonical metadata (title, language, server). 100% deterministic, side-effect-light — the kind of fast, read-only verification step a RAG or citation agent calls between finding a source and citing it.

### When should an AI agent use this?

- "I found a source I want to cite — is it an official/government source or a random blog, and how old is the domain?"
- "My RAG pipeline retrieved a URL — give me a quick trust signal (domain age, institution type) before I include it in the answer."
- "I need to verify that `https://www.rijksoverheid.nl/...` is actually a government source and find its canonical metadata (title, language, last-modified)."
- "Batch-verify a list of URLs: classify each by institution type and flag any with very young domains (possible spam/disinformation signal)."
- "Give me the canonical URL, page title, and detected language for these URLs so I can format proper citations."
- "Check whether this source was recently updated (last-modified header or sitemap) before I rely on its data."

### What this Actor does

For each URL it returns one flat, MCP-friendly record:

- **Institution classification** — detects government (`*.overheid.nl`, `*.gov`, `*.gouv.*`, `*.europa.eu`, `rijksoverheid.nl`, `admin.ch`, `gc.ca`, `bund.de`, `belgium.be`), education (`*.ac.*`, `*.edu`, `uni-*.de`), NGO (`*.org`, `*.ngo`, `*.ong`), and commercial (`*.com`, `*.net`, `*.io`, `*.co`) domains. Returns the detected type plus the matching signals.
- **Domain age** — looks up the domain registration date via RDAP (the modern, standardized WHOIS replacement) using the IANA bootstrap, and computes `domainAgeDays`. A domain registered two weeks ago is a very different trust signal than one from 1999.
- **Freshness** — extracts `Last-Modified` from the HTTP response headers. Optionally falls back to the site's `sitemap.xml` (`includeSitemap: true`) when the header is absent. Reports `lastModifiedSource` (`"http-header"` or `"sitemap"`) so you know where the signal came from.
- **Canonical metadata** — the final URL (after redirects), the HTML `<title>`, the `Content-Language`, and the `Server` header, so you can build a proper citation without re-fetching the page.

### Input

| Field | Type | Default | Description |
|---|---|---|---|
| `urls` | array of strings | *required* | URL(s) to verify (max 1000 per run). Returns one fact sheet per URL. |
| `includeSitemap` | boolean | `false` | Also fetch the site's `sitemap.xml` as a fallback for `last-modified`. Adds one extra HTTP request per host. |
| `timeoutSec` | integer | `10` | Per-request timeout in seconds for HTTP and RDAP fetches. Range 3–60. |

### Output

One flat JSON record per verified URL:

```json
{
  "url": "https://www.rijksoverheid.nl/onderwerpen/klimaatbeleid",
  "canonicalUrl": "https://www.rijksoverheid.nl/onderwerpen/klimaatbeleid",
  "domain": "www.rijksoverheid.nl",
  "title": "Klimaatbeleid | Rijksoverheid.nl",
  "language": "nl",
  "lastModified": "2025-08-15T10:30:00Z",
  "lastModifiedSource": "http-header",
  "server": "nginx",
  "domainRegistrationDate": "1999-07-06T10:37:04Z",
  "domainAgeDays": 9919,
  "institutionType": "government",
  "institutionSignals": ["rijksoverheid.nl"],
  "isGovernment": true,
  "fetchStatus": "ok",
  "errorCode": null,
  "error": null
```

| Field | Description |
|---|---|
| `url` | The URL as supplied in the input |
| `canonicalUrl` | Final URL after following redirects |
| `domain` | Lowercased hostname |
| `title` | HTML `<title>` (streaming extraction, decodes entities) |
| `language` | `Content-Language` header, if present |
| `lastModified` | Last-modified timestamp, if available |
| `lastModifiedSource` | `"http-header"`, `"sitemap"`, or `null` |
| `server` | `Server` header, if present |
| `domainRegistrationDate` | ISO timestamp from RDAP, if available |
| `domainAgeDays` | Whole days since registration (computed) |
| `institutionType` | `"government"`, `"education"`, `"ngo"`, `"commercial"`, or `"unknown"` |
| `institutionSignals` | Which domain patterns matched |
| `isGovernment` | `true` when institution type is `"government"` |
| `fetchStatus` | `"ok"`, `"partial"` (one of HTTP/RDAP failed), or `"error"` |
| `errorCode` | Machine-readable error class when not `"ok"` |
| `error` | Human-readable error detail, if any |

### Use cases

- **RAG/citation guard** — call this between retrieving a URL and citing it in an answer. A government source with a 25-year-old domain and a recent `last-modified` is a strong trust signal; a 3-week-old `.com` with no freshness data is not.
- **Source triage for research agents** — batch-verify a list of candidate URLs and rank them by domain age and institution type before doing expensive content extraction.
- **Citation formatting** — get the canonical URL, title, and language for a clean citation without a separate scraping pass.
- **Spam/disinformation signal** — a very young domain (`domainAgeDays` in the single/double digits) is a lightweight red flag worth surfacing to a downstream verifier.
- **Compliance-by-default crawling** — pair with a permission check: verify the source is authoritative *before* committing to a full scrape.

### Pricing

This Actor uses Apify's Pay-Per-Event (PPE) pricing model.

- **Actor Start:** $0.00005 (Apify default)
- **`url-verified`:** $0.002 per URL verified — charged when the verification pipeline completed for that URL (`fetchStatus` `ok` or `partial`). **Not charged** when the URL could not be fetched at all (`fetchStatus: "error"`, e.g. unreachable host, invalid URL).

See [STOREINFO.md](STOREINFO.md) for the full pricing table and rationale.

### Legal

This Actor reads only publicly available technical metadata: HTTP response headers, the HTML `<title>`, RDAP domain registration data (the standardized, machine-readable successor to WHOIS, explicitly intended for automated querying), and optionally `sitemap.xml`. It does **not** scrape page content, does not collect personal data, and does not access any paywalled or authenticated resource. Requests to private/loopback/link-local IP ranges are blocked (including via redirects), and response bodies are read with a strict size cap. The institution-type classification is a heuristic based on domain-name patterns — it is **not** an official registry check and comes with no guarantee of correctness (documented in the README so users treat it as a signal, not a legal determination). You remain responsible for how you use the verification results downstream.

### FAQ

**Does this scrape the page content?**
No. It fetches the HTTP headers and streams just enough of the HTML to extract the `<title>` tag (typically the first ~16 KB), then stops. No content is stored or returned.

**Why is `lastModified` often `null`?**
Many sites (including major ones like `rijksoverheid.nl`) don't send a `Last-Modified` header. Set `includeSitemap: true` to try the site's `sitemap.xml` as a fallback — but note that many sitemaps are index files without per-URL lastmod dates, so `null` is a common and honest answer.

**Why is `domainRegistrationDate` sometimes `null`?**
RDAP coverage is broad but not universal. The actor uses the IANA RDAP bootstrap to find the right server per TLD. Some TLDs (notably `.eu`) are not yet in the bootstrap; for those, `domainRegistrationDate` is `null` and `fetchStatus` is `"partial"`. This is expected and documented, not a bug.

**Can I verify many URLs in one run?**
Yes — `urls` is an array. Each URL is processed independently, and one failure does not stop the rest. PPE is charged per URL.

**Is the institution classification guaranteed correct?**
No. It is a heuristic based on well-known domain patterns (`.gov`, `.ac.uk`, `.org`, etc.). It covers the most common cases but will return `"unknown"` for anything that doesn't match. Treat it as a trust *signal*, not a legal determination.

**Can I verify internal/intranet URLs?**
No. Requests to private, loopback, link-local (e.g. `169.254.169.254`), and other non-public IP ranges are blocked, including when reached through redirects. The actor only verifies publicly reachable sources — that is its purpose and a deliberate safety boundary.

### Keywords

url verification, source verification, domain age check, institution type detector, government source check, rdap lookup, whois alternative, citation verification, rag guard, source trust signal, domain registration date, last-modified check, canonical url extractor, ai agent tool, mcp tool, fact checking source, spam domain signal

### Related Actors

- [llm-output-validator](https://apify.com/codeclouds/llm-output-validator) — validate LLM/tool-call JSON against a schema with repair hints, another developer-tool/agent-hulpmiddel from the same portfolio
- [legacy-code-license-scanner](https://apify.com/codeclouds/legacy-code-license-scanner) — audit a repo or dependency manifest for license compatibility

### Changelog

#### 0.1.0

- Initial release: institution classification (government/education/NGO/commercial) via domain-pattern heuristics, domain-age lookup via RDAP (IANA bootstrap), HTTP `Last-Modified` + optional sitemap fallback, canonical metadata (title, language, server), per-URL error isolation.
- Hardening: private/loopback/link-local address blocking with per-hop redirect validation, size-capped response reads, max 1000 URLs per run, PPE charged only when the verification pipeline completed (not on total failures), per-domain RDAP memoization within a run, and concurrent HTTP+RDAP probing per URL.

# Actor input Schema

## `urls` (type: `array`):

URL(s) to verify. Returns a structured fact sheet per URL: institution type, domain age, freshness signals, and canonical metadata.

## `includeSitemap` (type: `boolean`):

Also fetch the sites sitemap.xml as a fallback for last-modified. Adds one extra HTTP request per host.

## `timeoutSec` (type: `integer`):

Per-request timeout in seconds for HTTP and RDAP fetches.

## Actor input object example

```json
{
  "urls": [
    "https://www.wikipedia.org"
  ],
  "includeSitemap": false,
  "timeoutSec": 10
}
```

# Actor output Schema

## `results` (type: `string`):

Results stored in the default dataset.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "urls": [
        "https://www.wikipedia.org"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("codeclouds/url-to-structured-fact").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "urls": ["https://www.wikipedia.org"] }

# Run the Actor and wait for it to finish
run = client.actor("codeclouds/url-to-structured-fact").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "urls": [
    "https://www.wikipedia.org"
  ]
}' |
apify call codeclouds/url-to-structured-fact --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,codeclouds/url-to-structured-fact"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/FLWDp8181UfxEKYle/builds/tYz3f5Aeb6ztWVhBT/openapi.json