URL Metadata & OpenGraph Extractor

Give it a list of page URLs and get back the metadata each page publishes for previews and search: OpenGraph (og:*), Twitter card (twitter:*), <title>, meta description, meta robots, keywords, author, theme color, published and modified times, canonical link, declared favicon, page language, hreflang alternates, feed links, the first h1, and schema.org structured data (JSON-LD and microdata types). Flat rows, built for link previews and for feeding RAG pipelines with consistent per-link metadata.

It reads schema.org JSON-LD and microdata types, hreflang alternates and feed links from the page itself. JSON-LD scripts are parsed as written, and a script that does not parse is counted in jsonLdErrors instead of being guessed at.

Input

• URLs: one per line.
• Respect robots.txt: when on (default), the host's robots.txt is checked and disallowed URLs are skipped.
• Max delivered URLs: cap on billed rows (0 = no cap).

Output

One row per URL: url, finalUrl, httpStatus, title, description, canonical, the og* fields, the twitter* fields, favicon, lang, metaRobots, keywords, author, themeColor, publishedTime, modifiedTime, hreflang, feeds, h1, h1Count, jsonLd, jsonLdErrors, jsonLdTypes, microdataTypes, plus provenance (sourceUrl, retrievedAt, confidence, dataSource).

How it works

Sites publish these tags specifically so other tools can render previews and read structured data. The actor fetches each page politely with a declared User-Agent and copies the tags verbatim. The preview fields come from the head only; JSON-LD, microdata types, and the h1 are also read from the body. Relative og:image, canonical, favicon, hreflang, and feed URLs are resolved to absolute against the page URL; nothing else is transformed, dates are not reformatted, and a tag the page does not declare is null (or an empty list), never invented. JSON-LD is capped at 20 objects or 50000 characters per page. A URL that robots disallows, or that fails to fetch, is written to the free rejected dataset and is not billed. A site owner can ask us to skip their domain at https://ponodata.com/opt-out ; opted-out hosts are skipped and never charged.

Billing

Pay per URL successfully read. A page counts when it has head metadata or at least one JSON-LD type; an h1 alone does not. Robots-disallowed and failed URLs cost nothing.

Sample output

A real run reading each page's own public head tags (one row per URL):

URL	title	description	OG type
https://www.cloudflare.com	Cloudflare: Build for the…	Welcome to Cloudflare - Powering …	website
https://stripe.com	Stripe / Financial Infras…	Stripe is a financial services pl…	website
https://www.python.org	Welcome to Python.org	The official home of the Python P…	website
https://kubernetes.io	Kubernetes	Kubernetes, also known as K8s, is…	website

Every row carries a sourceUrl (the page read), for example https://www.cloudflare.com. Pages that return no metadata route to the free reject dataset.

Why this one

• Reads only the page's own public tags and structured data and copies them verbatim, with a sourceUrl to the page. Nothing is guessed.
• Follows redirects safely: it records the final URL and screens every redirect hop, so a public URL cannot be turned into a request to an internal host.
• You pay only for a URL that returns metadata. Robots-disallowed and failed URLs are free.
• Typed, flat rows with a stable schema, so an agent or a pipeline can consume them by field name.

Use cases

• Render link previews at scale: title, description, and og:image for a list of URLs, the same tags a chat app or social card reads.
• Feed a RAG pipeline consistent per-link metadata: one typed row per URL, so a retriever indexes a clean title and description instead of raw HTML.
• Audit a site's social and SEO tags: find pages missing a description, a canonical link, an OpenGraph image, or an h1, or pages with more than one h1 or a noindex robots tag.
• Check schema markup: list the JSON-LD and microdata types each page declares and find ld+json scripts that fail to parse.
• Collect hreflang alternates and RSS, Atom, or JSON Feed links across a list of sites.
• Normalize a mixed link list: each row carries the finalUrl after redirects and the page language.

FAQ

• Does it read the whole page? It reads the head for the preview and meta fields, and the full document for JSON-LD, microdata types, and the h1. Values are copied verbatim; a tag the page does not declare is null, never invented.
• Does it respect robots.txt? Yes by default; a disallowed URL is skipped to the free rejected dataset, unbilled.
• Is it safe against redirect tricks? Yes. It screens every redirect hop, so a public URL that points at an internal address is refused before it connects.
• How am I billed? Per URL that returns metadata or JSON-LD types. Robots-disallowed and failed URLs cost nothing.

See also

More pay-only-for-results data tools from Pono Data:

• Sitemap Extractor - every URL from any sitemap
• Bulk DNS Lookup - DNS records plus SPF, DMARC, and CAA
• Domain WHOIS via RDAP - registration data, structured from RDAP

Full catalog: https://apify.com/thoob