Url Metadata Crawler

Fetch public HTTP/HTTPS pages and extract semantic metadata without downloading or publishing the page body. The actor follows redirects when requested, applies bounded retries, detects common block pages, filters sensitive metadata, and protects against private-network and metadata-service URLs.

What it does

• Processes a list of public URLs.
• Processes up to five independent public URLs concurrently while preserving input order.
• Extracts titles, descriptions, language, canonical links, favicons, Open Graph/Twitter cards, selected headers, and sanitized JSON-LD.
• Preserves diagnostics for blocked, non-HTML, and failed responses.
• Writes one dataset record per URL and a count summary to OUTPUT.

Input

{
  "urls": [
    "https://example.com",
    "https://github.com"
  ],
  "followRedirects": true,
  "timeoutSecs": 30,
  "maxItems": 100
}

urls must contain HTTP or HTTPS URLs without credentials. The actor deduplicates normalized URLs and caps processing at 1,000. timeoutSecs accepts 5–120 seconds. Proxy configuration is optional and invalid requested settings fail closed.

Output

Successful records contain page metadata and safe response diagnostics. Non-success records still identify the requested/final URL, status, timestamp, and normalized fetchStatus (blocked, error, or non-html).

Example:

{
  "inputUrl": "https://example.com/",
  "finalUrl": "https://example.com/",
  "httpStatus": 200,
  "fetchStatus": "success",
  "title": "Example Domain",
  "description": "Example metadata",
  "canonical": "https://example.com/",
  "fetchedAt": "2026-01-01T00:00:00.000Z"
}

OUTPUT contains requested, successful, blocked, error, non-HTML, and completion counts. Sensitive metadata such as tokens, cookies, signatures, and authorization values is excluded.

Cost and limits

Runtime depends on URL count, response time, redirects, retries, and proxy use. Responses above 5 MiB are rejected for safety, and each URL is attempted at most three times. Apify compute and proxy charges are separate.

Tips and FAQ

• Start with a small URL list to confirm the target pages expose HTML metadata.
• non-html means the response was not HTML/XHTML; binary content is not parsed.
• blocked means a challenge/block response or status was detected, not that the page is necessarily unavailable to a normal browser.
• JSON-LD is sanitized and may contain different shapes across sites.

Disclaimer

Use this actor only with permission and in accordance with target-site terms, applicable law, and access policies. It does not bypass authentication, CAPTCHAs, or network access controls.