URL Intelligence — Web Data for AI Agents

Turn any public URL into compact, predictable JSON for AI agents, RAG pipelines, monitoring, enrichment, SEO checks, and automation.

Pricing: $1 per 1,000 processed URLs ($0.001 per result).

Why use this Actor

Most agents do not need a full browser session just to understand a page. This Actor gives them the useful structure directly:

• final URL and HTTP status
• title, meta description, canonical, and robots directives
• H1/H2/H3 heading structure
• JSON-LD structured data
• normalized links with anchor text
• cleaned body text
• SHA-256 content fingerprint for change detection
• response headers including ETag and Last-Modified

One URL in → one machine-readable result out.

Good for

• AI agent web research
• RAG ingestion and preprocessing
• website monitoring and change detection
• SEO and metadata audits
• lead and company-page enrichment
• content pipelines
• link discovery
• page fingerprinting
• automation workflows in n8n, Make, Zapier, Python, JavaScript, or MCP-connected agents

Input

{
  "urls": [
    "https://example.com"
  ],
  "includeLinks": true,
  "includeJsonLd": true,
  "maxLinks": 100
}

You can send multiple public HTTP(S) URLs in one run.

Output example

{
  "inputUrl": "https://example.com",
  "ok": true,
  "status": 200,
  "finalUrl": "https://example.com/",
  "title": "Example Domain",
  "description": null,
  "canonical": null,
  "robots": null,
  "h1": ["Example Domain"],
  "headings": [
    {"level": "h1", "text": "Example Domain"}
  ],
  "jsonLd": [],
  "links": [],
  "content": {
    "text": "Example Domain ...",
    "characters": 142,
    "sha256": "..."
  },
  "headers": {
    "cacheControl": null,
    "etag": null,
    "lastModified": null,
    "server": null
  }
}

Machine-to-machine use

The Actor is intentionally cheap and deterministic so autonomous systems can call it repeatedly without committing to a large per-request purchase.

Each successfully processed URL produces one dataset item and one billable URL processed event.

Limits and behavior

• public HTTP(S) URLs only
• up to 100 URLs per run
• 20-second fetch timeout per URL
• up to 500 links returned per page
• cleaned body text capped at 50,000 characters
• invalid or unreachable URLs return an error item instead of crashing the whole run

Reliability

The Actor follows redirects, reports the final URL and HTTP status, and returns partial structured data when a page does not provide every metadata field.

Built and maintained by Revenue Swarm.